TCP stack of Solaris - Solaris

This is a discussion on TCP stack of Solaris - Solaris ; Sun says that they improved their TCP stack. But I think their is still a lot of work to do. See those 2 graph : http://www.box.net/public/tdi16rjlsy http://www.box.net/public/a4pa9cdpd4 I used NetPipe to generate them ( http://www.scl.ameslab.gov/netpipe/ ). They have been generated ...

+ Reply to Thread
Results 1 to 4 of 4

Thread: TCP stack of Solaris

  1. TCP stack of Solaris

    Sun says that they improved their TCP stack. But I think their is
    still a lot of work to do.

    See those 2 graph :
    http://www.box.net/public/tdi16rjlsy
    http://www.box.net/public/a4pa9cdpd4

    I used NetPipe to generate them (http://www.scl.ameslab.gov/netpipe/).

    They have been generated using two DL385, linked by a Cisco 3750
    switch.

    Can someone explains to me why the Solaris graphs are so ugly ?

  2. Re: TCP stack of Solaris

    fabrice-pas-despame.bacchella@worldonline.fr wrote:
    > Sun says that they improved their TCP stack. But I think their is
    > still a lot of work to do.


    > See those 2 graph :
    > http://www.box.net/public/tdi16rjlsy
    > http://www.box.net/public/a4pa9cdpd4


    > I used NetPipe to generate them (http://www.scl.ameslab.gov/netpipe/).


    Might be nice to label each axis and provide a bit more detail about
    things like socket buffer sizes, send sizes etc etc.

    It would also be good to include "service demand" figures. Service
    demand is what netperf calculates based on the throughput and the CPU
    utilization - it says how much CPU was used to perform a given unit of
    work. I've not looked closely at netpipe, but would expect it to have
    some sort of similar thing available.

    > They have been generated using two DL385, linked by a Cisco 3750
    > switch.


    Using the core interface(s) on the DL385? I am not from the ProLiant
    side of the house, but I believe there are several variations on
    "DL385" so more details would be goodness.

    > Can someone explains to me why the Solaris graphs are so ugly ?


    For the latency, that rather looks like results I've gotten in other
    situations with netperf TCP_RR tests when the NIC (typically gigabit -
    is that what you were using?) has been hard set to a certain interrupt
    limit. Latency tests will not often hit the "packets to trigger an
    interrupt" component which then often leaves waiting for the timer on
    the NIC to expire. That then does arguably nasty things to the
    minimum latency reported by single-instance tests. Often as not the
    tradeoff was for (potentially) a higher aggregate packet-per-second
    limits. Basically, tuning for bandwidth rather than latency.

    Of course, I've no idea that is exactly what is going-on here, but it
    would be an avenue to explore. On the Linux side I suspect you would
    manipulate the "coalescing" settings via ethtool or perhaps
    modules.config (?). For Solaris I suspect it would be ndd.

    rick jones
    --
    The glass is neither half-empty nor half-full. The glass has a leak.
    The real question is "Can it be patched?"
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  3. Re: TCP stack of Solaris

    On Mon, 26 Jun 2006 21:32:15 GMT, Rick Jones
    wrote:

    >fabrice-pas-despame.bacchella@worldonline.fr wrote:
    >> Sun says that they improved their TCP stack. But I think their is
    >> still a lot of work to do.

    >
    >> See those 2 graph :
    >> http://www.box.net/public/tdi16rjlsy
    >> http://www.box.net/public/a4pa9cdpd4

    >
    >> I used NetPipe to generate them (http://www.scl.ameslab.gov/netpipe/).

    >
    >Might be nice to label each axis and provide a bit more detail about
    >things like socket buffer sizes, send sizes etc etc.


    I've just updated them with more informations. The new URLS are :
    http://www.box.net/public/8i95tcavfr
    http://www.box.net/public/urxqioy5hs

    Solaris 11's build is 41, not 44, don't dream about a secret release,
    my fault.

    socket buffer size where default one :
    16384 and 87380 for send and receive buffers on Linux
    49152 for both on Solaris 10 & 11
    The program says : A bug in Linux doubles the requested buffer sizes.

    Send size is what NetPipe change

    >
    >It would also be good to include "service demand" figures. Service
    >demand is what netperf calculates based on the throughput and the CPU
    >utilization - it says how much CPU was used to perform a given unit of
    >work. I've not looked closely at netpipe, but would expect it to have
    >some sort of similar thing available.

    Nop, but I will have a look using netperf.

    >
    >> They have been generated using two DL385, linked by a Cisco 3750
    >> switch.

    >
    >Using the core interface(s) on the DL385? I am not from the ProLiant
    >side of the house, but I believe there are several variations on
    >"DL385" so more details would be goodness.


    The NIC :
    03:06.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
    Gigabit Ethernet (rev 10)

    The processors : 2 x Opteron 265 ( 2 x 1.8 Ghz)

    The RAM : 16 GBytes

    By the way, could please tell the Proliant guys to finish porting all
    the tools to Solaris ? We're still missing hpacucli and things like
    that here.

    >Of course, I've no idea that is exactly what is going-on here, but it
    >would be an avenue to explore. On the Linux side I suspect you would
    >manipulate the "coalescing" settings via ethtool or perhaps
    >modules.config (?). For Solaris I suspect it would be ndd.

    I don't think there is this kind of tuning on Solaris this ndd, but
    playing with /etc/system might be the way.

    >
    >rick jones


  4. Re: TCP stack of Solaris

    fabrice-pas-despame.bacchella@worldonline.fr wrote:
    > socket buffer size where default one :
    > 16384 and 87380 for send and receive buffers on Linux
    > 49152 for both on Solaris 10 & 11
    > The program says : A bug in Linux doubles the requested buffer sizes.


    While I agree with the netpipe people that Linux's behaviour WRT
    SO_*BUF in setsockopt/getsockopt is annoying, my understanding is that
    it isn't so much doubling the requested size as it is setting the
    "overhead" (size of the skb's (eg mbufs or mblks depending on your
    history)) limit to 2X what one asks to be the data limit in a
    setsockopt() call, and then getsockopt() returns the overhead limit
    not necessarily the data limit.

    I could though be wrong there.

    > Send size is what NetPipe change


    >>
    >>It would also be good to include "service demand" figures. Service
    >>demand is what netperf calculates based on the throughput and the CPU
    >>utilization - it says how much CPU was used to perform a given unit of
    >>work. I've not looked closely at netpipe, but would expect it to have
    >>some sort of similar thing available.

    > Nop, but I will have a look using netperf.


    BTW, I should warn you that this may be the first time the "kstat10"
    CPU method in netperf has been used for Solaris x86. It may only have
    been used on SPARC to this point. It would be good for you to take a
    close look at the caveats in the comments in src/netcpu_kstat10.c


    > The NIC :
    > 03:06.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
    > Gigabit Ethernet (rev 10)


    > The processors : 2 x Opteron 265 ( 2 x 1.8 Ghz)


    > The RAM : 16 GBytes


    All good things to include in a report.

    > By the way, could please tell the Proliant guys to finish porting all
    > the tools to Solaris ? We're still missing hpacucli and things like
    > that here.


    What, and take all the fun away from Sun?-) If I happen across some of
    those folks I will try to let them know.

    >>Of course, I've no idea that is exactly what is going-on here, but it
    >>would be an avenue to explore. On the Linux side I suspect you would
    >>manipulate the "coalescing" settings via ethtool or perhaps
    >>modules.config (?). For Solaris I suspect it would be ndd.

    > I don't think there is this kind of tuning on Solaris this ndd, but
    > playing with /etc/system might be the way.


    try:

    ndd -get /dev/ ?

    where is whatever Solaris decided to call the interfaces
    with the PPA stripped - eg ipge if you have ipge0 and ipge1

    rick jones
    --
    The glass is neither half-empty nor half-full. The glass has a leak.
    The real question is "Can it be patched?"
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

+ Reply to Thread