fabrice-pas-despame.bacchella@worldonline.fr wrote:
> Sun says that they improved their TCP stack. But I think their is
> still a lot of work to do.

> See those 2 graph :
> http://www.box.net/public/tdi16rjlsy
> http://www.box.net/public/a4pa9cdpd4

> I used NetPipe to generate them (http://www.scl.ameslab.gov/netpipe/).

Might be nice to label each axis and provide a bit more detail about
things like socket buffer sizes, send sizes etc etc.

It would also be good to include "service demand" figures. Service
demand is what netperf calculates based on the throughput and the CPU
utilization - it says how much CPU was used to perform a given unit of
work. I've not looked closely at netpipe, but would expect it to have
some sort of similar thing available.

> They have been generated using two DL385, linked by a Cisco 3750
> switch.

Using the core interface(s) on the DL385? I am not from the ProLiant
side of the house, but I believe there are several variations on
"DL385" so more details would be goodness.

> Can someone explains to me why the Solaris graphs are so ugly ?

For the latency, that rather looks like results I've gotten in other
situations with netperf TCP_RR tests when the NIC (typically gigabit -
is that what you were using?) has been hard set to a certain interrupt
limit. Latency tests will not often hit the "packets to trigger an
interrupt" component which then often leaves waiting for the timer on
the NIC to expire. That then does arguably nasty things to the
minimum latency reported by single-instance tests. Often as not the
tradeoff was for (potentially) a higher aggregate packet-per-second
limits. Basically, tuning for bandwidth rather than latency.

Of course, I've no idea that is exactly what is going-on here, but it
would be an avenue to explore. On the Linux side I suspect you would
manipulate the "coalescing" settings via ethtool or perhaps
modules.config (?). For Solaris I suspect it would be ndd.

rick jones
--
The glass is neither half-empty nor half-full. The glass has a leak.
The real question is "Can it be patched?"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

On Mon, 26 Jun 2006 21:32:15 GMT, Rick Jones
wrote:

>fabrice-pas-despame.bacchella@worldonline.fr wrote:
>> Sun says that they improved their TCP stack. But I think their is
>> still a lot of work to do.
>
>> See those 2 graph :
>> http://www.box.net/public/tdi16rjlsy
>> http://www.box.net/public/a4pa9cdpd4
>
>> I used NetPipe to generate them (http://www.scl.ameslab.gov/netpipe/).
>
>Might be nice to label each axis and provide a bit more detail about
>things like socket buffer sizes, send sizes etc etc.

I've just updated them with more informations. The new URLS are :
http://www.box.net/public/8i95tcavfr
http://www.box.net/public/urxqioy5hs

Solaris 11's build is 41, not 44, don't dream about a secret release,
my fault.

socket buffer size where default one :
16384 and 87380 for send and receive buffers on Linux
49152 for both on Solaris 10 & 11
The program says : A bug in Linux doubles the requested buffer sizes.

Send size is what NetPipe change

>
>It would also be good to include "service demand" figures. Service
>demand is what netperf calculates based on the throughput and the CPU
>utilization - it says how much CPU was used to perform a given unit of
>work. I've not looked closely at netpipe, but would expect it to have
>some sort of similar thing available.
Nop, but I will have a look using netperf.

>
>> They have been generated using two DL385, linked by a Cisco 3750
>> switch.
>
>Using the core interface(s) on the DL385? I am not from the ProLiant
>side of the house, but I believe there are several variations on
>"DL385" so more details would be goodness.

The NIC :
03:06.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
Gigabit Ethernet (rev 10)

The processors : 2 x Opteron 265 ( 2 x 1.8 Ghz)

The RAM : 16 GBytes

By the way, could please tell the Proliant guys to finish porting all
the tools to Solaris ? We're still missing hpacucli and things like
that here.

>Of course, I've no idea that is exactly what is going-on here, but it
>would be an avenue to explore. On the Linux side I suspect you would
>manipulate the "coalescing" settings via ethtool or perhaps
>modules.config (?). For Solaris I suspect it would be ndd.
I don't think there is this kind of tuning on Solaris this ndd, but
playing with /etc/system might be the way.

>
>rick jones

fabrice-pas-despame.bacchella@worldonline.fr wrote:
> socket buffer size where default one :
> 16384 and 87380 for send and receive buffers on Linux
> 49152 for both on Solaris 10 & 11
> The program says : A bug in Linux doubles the requested buffer sizes.

While I agree with the netpipe people that Linux's behaviour WRT
SO_*BUF in setsockopt/getsockopt is annoying, my understanding is that
it isn't so much doubling the requested size as it is setting the
"overhead" (size of the skb's (eg mbufs or mblks depending on your
history)) limit to 2X what one asks to be the data limit in a
setsockopt() call, and then getsockopt() returns the overhead limit
not necessarily the data limit.

I could though be wrong there.

> Send size is what NetPipe change

>>
>>It would also be good to include "service demand" figures. Service
>>demand is what netperf calculates based on the throughput and the CPU
>>utilization - it says how much CPU was used to perform a given unit of
>>work. I've not looked closely at netpipe, but would expect it to have
>>some sort of similar thing available.
> Nop, but I will have a look using netperf.

BTW, I should warn you that this may be the first time the "kstat10"
CPU method in netperf has been used for Solaris x86. It may only have
been used on SPARC to this point. It would be good for you to take a
close look at the caveats in the comments in src/netcpu_kstat10.c


> The NIC :
> 03:06.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
> Gigabit Ethernet (rev 10)

> The processors : 2 x Opteron 265 ( 2 x 1.8 Ghz)

> The RAM : 16 GBytes

All good things to include in a report.

> By the way, could please tell the Proliant guys to finish porting all
> the tools to Solaris ? We're still missing hpacucli and things like
> that here.

What, and take all the fun away from Sun?-) If I happen across some of
those folks I will try to let them know.

>>Of course, I've no idea that is exactly what is going-on here, but it
>>would be an avenue to explore. On the Linux side I suspect you would
>>manipulate the "coalescing" settings via ethtool or perhaps
>>modules.config (?). For Solaris I suspect it would be ndd.
> I don't think there is this kind of tuning on Solaris this ndd, but
> playing with /etc/system might be the way.

try:

ndd -get /dev/ ?

where is whatever Solaris decided to call the interfaces
with the PPA stripped - eg ipge if you have ipge0 and ipge1

rick jones
--
The glass is neither half-empty nor half-full. The glass has a leak.
The real question is "Can it be patched?"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...