NFS and MTU

A customer asked me recently if increasing the maximum transmission unit on
TCP/IP packets would increase network throughput. Good question, and I'd
like to solicit opinions here.

I'm the administrator of a small, local ATM network of 8 systems. My client
systems are connected at OC-3 and access separate file and data servers
connected at OC-12. The major data access functions retrieve, edit and
write data in NFS mounted directories from the data server. The volume of
data is on the order of tens to hundreds of gigbytes and thousands to tens
of thousands of files. The MTU is currently set to default: 1500 bytes. My
question is: Would increasing the MTU result in an increase in throughput?
I'm guessing there would be some noticeable difference from the reduced
packet count, but I don't know what that would be: I have no experience in
this matter.

Your opinions are greatly appreciated.

Dale

> if increasing the maximum transmission unit on[color=blue]
> TCP/IP packets would increase network throughput.[/color]

The answer, as usual, is "it depends". Within an ATM network, "MTU" is
a fairly useless notion because ATM payloads are always 48 bytes.

MTU does force fragmentation of IP datagrams and (usually) limits the
size of TCP segments. As you clearly know, fewer, larger packets means
less per-packet overhead. However, those overheads are paid only by
devices at the edges of the ATM network. RFC 1626 specifies a default
MTU of 9180 bytes over ATM AAL5.

However, increasing TCP segment and IP datagram sizes (by any means)
_could_ be counter-productive. If there is any measurable loss in the
link layer (i.e., ATM payloads), the larger sizes mean bigger impacts
due to loss, as TCP ultimately detects failures and retransmits in
terms of segments.

Because this is an ATM network rather than Ethernet, your network adapters
are going to be breaking the "MTU" into a stream of 48-byte ATM payloads
anyway. As long as the adapters can keep the links busy, increasing the
MTU will have only very marginal effects, limited to lowering processing
overheads within the hosts. It will have no further effect on the ATM
network utilization or overheads.

However, NFS and TCP may be rate-limited in ways that starve your adapters
somewhat. First, if you haven't already done so, increase your NFS read
and write sizes to the maximum both clients and servers can support,
through
client mount options. Second, enable the TCP Window Scale and Selective
Acknowledgement options at both clients and servers. It's possible your
TCP implementations already have these options and quietly negotiate using
them when setting up a connection.

Increasing read and write sizes obviously reduces the number of NFS
operations. This has only a modest effect on the amount of network
traffic, but reduces total round trip latency and improves client and
server host overheads somewhat.

If your clients were connected via OC-12, the Window Scale option would
become imperative to get maximum TCP performance. Limited to OC-3, it's
still useful, but probably not critical.

The Selective Acknowledgment option, if supported, reduces the performance
hit TCP takes from a lost packet by enabling it to retransmit just a single
TCP segment in a single IP datagram. Without it, TCP must retransmit all
data from the beginning of the lost TCP segment.

By the way, are you running Classical IP over ATM (RFC 1577), LAN
emulation,
MPOA, vendor-specific multilevel switching, other???

Stu Friedberg

"Dale" <dschmitz4@cox.net> writes:
[color=blue]
>A customer asked me recently if increasing the maximum transmission unit on
>TCP/IP packets would increase network throughput. Good question, and I'd
>like to solicit opinions here.[/color]
[color=blue]
>I'm the administrator of a small, local ATM network of 8 systems. My client
>systems are connected at OC-3 and access separate file and data servers
>connected at OC-12. The major data access functions retrieve, edit and
>write data in NFS mounted directories from the data server. The volume of
>data is on the order of tens to hundreds of gigbytes and thousands to tens
>of thousands of files. The MTU is currently set to default: 1500 bytes. My
>question is: Would increasing the MTU result in an increase in throughput?
>I'm guessing there would be some noticeable difference from the reduced
>packet count, but I don't know what that would be: I have no experience in
>this matter.[/color]

A larger MTU may help, but the first things to look at would be:

- Are you using NFS over TCP?
- What TCP window sizes are you using?
- What NFS read/write sizes are you using?


I'd increase all of the above before using a bigger MTU.

Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

> - Are you using NFS over TCP?[color=blue]
> - What TCP window sizes are you using?
> - What NFS read/write sizes are you using?[/color]

Add to those:

What is your current TCP retransmission rate?

rick jones
--
No need to believe in either side, or any side. There is no cause.
There's only yourself. The belief is in your own precision. - Jobert
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Thanks Stu, I will look into some of these things (those I understand), and
try to get some education on the rest. What I'm getting out of all this is
that despite what I do at the extreme ends of the connection, while that may
do some good, I'm still limited by ATM performance dependencies.

Dale

"Stuart Friedberg" <stuartf@polyserve.com> wrote in message
news:op.sroe5gepgowx4s@m-stuartf...[color=blue][color=green]
>> if increasing the maximum transmission unit on
>> TCP/IP packets would increase network throughput.[/color]
>
> The answer, as usual, is "it depends". Within an ATM network, "MTU" is
> a fairly useless notion because ATM payloads are always 48 bytes.
>
> MTU does force fragmentation of IP datagrams and (usually) limits the
> size of TCP segments. As you clearly know, fewer, larger packets means
> less per-packet overhead. However, those overheads are paid only by
> devices at the edges of the ATM network. RFC 1626 specifies a default
> MTU of 9180 bytes over ATM AAL5.
>
> However, increasing TCP segment and IP datagram sizes (by any means)
> _could_ be counter-productive. If there is any measurable loss in the
> link layer (i.e., ATM payloads), the larger sizes mean bigger impacts
> due to loss, as TCP ultimately detects failures and retransmits in
> terms of segments.
>
> Because this is an ATM network rather than Ethernet, your network adapters
> are going to be breaking the "MTU" into a stream of 48-byte ATM payloads
> anyway. As long as the adapters can keep the links busy, increasing the
> MTU will have only very marginal effects, limited to lowering processing
> overheads within the hosts. It will have no further effect on the ATM
> network utilization or overheads.
>
> However, NFS and TCP may be rate-limited in ways that starve your adapters
> somewhat. First, if you haven't already done so, increase your NFS read
> and write sizes to the maximum both clients and servers can support,
> through
> client mount options. Second, enable the TCP Window Scale and Selective
> Acknowledgement options at both clients and servers. It's possible your
> TCP implementations already have these options and quietly negotiate using
> them when setting up a connection.
>
> Increasing read and write sizes obviously reduces the number of NFS
> operations. This has only a modest effect on the amount of network
> traffic, but reduces total round trip latency and improves client and
> server host overheads somewhat.
>
> If your clients were connected via OC-12, the Window Scale option would
> become imperative to get maximum TCP performance. Limited to OC-3, it's
> still useful, but probably not critical.
>
> The Selective Acknowledgment option, if supported, reduces the performance
> hit TCP takes from a lost packet by enabling it to retransmit just a
> single
> TCP segment in a single IP datagram. Without it, TCP must retransmit all
> data from the beginning of the lost TCP segment.
>
> By the way, are you running Classical IP over ATM (RFC 1577), LAN
> emulation,
> MPOA, vendor-specific multilevel switching, other???
>
> Stu Friedberg[/color]

I wish I could answer your questions right off the bat, Casper, but I have
to find out what it is you're asking for to begin with. Stu Friedberg's
post mentioned those same attributes--window size and NFS read/write
sizes--and I'll have to find out what they're set to. They do seem the most
likely place to start.

Thanks,
Dale

"Casper H.S. Dik" <Casper.Dik@Sun.COM> wrote in message
news:429d8216$0$23350$e4fe514c@news.xs4all.nl...[color=blue]
> "Dale" <dschmitz4@cox.net> writes:
>[color=green]
>>A customer asked me recently if increasing the maximum transmission unit
>>on
>>TCP/IP packets would increase network throughput. Good question, and I'd
>>like to solicit opinions here.[/color]
>[color=green]
>>I'm the administrator of a small, local ATM network of 8 systems. My
>>client
>>systems are connected at OC-3 and access separate file and data servers
>>connected at OC-12. The major data access functions retrieve, edit and
>>write data in NFS mounted directories from the data server. The volume of
>>data is on the order of tens to hundreds of gigbytes and thousands to tens
>>of thousands of files. The MTU is currently set to default: 1500 bytes.
>>My
>>question is: Would increasing the MTU result in an increase in throughput?
>>I'm guessing there would be some noticeable difference from the reduced
>>packet count, but I don't know what that would be: I have no experience in
>>this matter.[/color]
>
> A larger MTU may help, but the first things to look at would be:
>
> - Are you using NFS over TCP?
> - What TCP window sizes are you using?
> - What NFS read/write sizes are you using?
>
>
> I'd increase all of the above before using a bigger MTU.
>
> Casper
> --
> Expressed in this posting are my opinions. They are in no way related
> to opinions held by my employer, Sun Microsystems.
> Statements on Sun products included here are not gospel and may
> be fiction rather than truth.[/color]

Thanks, I'll look at that. I believe that's a parameter capture by
oracallator so it should be graphed. I presume the concern here is that if
it's excessive, it represents overhead. What would excessive be?

Dale

"Rick Jones" <rick.jones2@hp.com> wrote in message
news:oDone.6329$Rs2.5189@news.cpqcorp.net...[color=blue][color=green]
>> - Are you using NFS over TCP?
>> - What TCP window sizes are you using?
>> - What NFS read/write sizes are you using?[/color]
>
> Add to those:
>
> What is your current TCP retransmission rate?
>
> rick jones
> --
> No need to believe in either side, or any side. There is no cause.
> There's only yourself. The belief is in your own precision. - Jobert
> these opinions are mine, all mine; HP might not want them anyway... :)
> feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...[/color]

Dale <dschmitz4@cox.net> wrote:[color=blue]
> Thanks, I'll look at that. I believe that's a parameter capture by
> oracallator so it should be graphed. I presume the concern here is
> that if it's excessive, it represents overhead. What would
> excessive be?[/color]

NFS is (in broad handwaving terms) a latency sensitive
application/protocol/whatever. ATM networks imply WAN which implies
high latencies. They also imply bandwidth constraints.

When latency is high, the only way to maintain throughput is with
increased parallelism. This means having more NFS requests
outstanding at any one time, and also leads into Casper's mention of
TCP windows (since the TCP window limits how much data can be
outstanding at one time on a connection, which then may or may not be
less than what NFS is trying to have on that connection at one time).

However, if there is much in the way of packet loss, TCP cannot
maintain a large window and throughput can suffer.

rick jones
--
a wide gulf separates "what if" from "if only"
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Uhhh...I'll have to decipher that one for a bit. Thanks for responding.

Dale

"Rick Jones" <rick.jones2@hp.com> wrote in message
news:Bxqne.6343$VB2.3340@news.cpqcorp.net...[color=blue]
> Dale <dschmitz4@cox.net> wrote:[color=green]
>> Thanks, I'll look at that. I believe that's a parameter capture by
>> oracallator so it should be graphed. I presume the concern here is
>> that if it's excessive, it represents overhead. What would
>> excessive be?[/color]
>
> NFS is (in broad handwaving terms) a latency sensitive
> application/protocol/whatever. ATM networks imply WAN which implies
> high latencies. They also imply bandwidth constraints.
>
> When latency is high, the only way to maintain throughput is with
> increased parallelism. This means having more NFS requests
> outstanding at any one time, and also leads into Casper's mention of
> TCP windows (since the TCP window limits how much data can be
> outstanding at one time on a connection, which then may or may not be
> less than what NFS is trying to have on that connection at one time).
>
> However, if there is much in the way of packet loss, TCP cannot
> maintain a large window and throughput can suffer.
>
> rick jones
> --
> a wide gulf separates "what if" from "if only"
> these opinions are mine, all mine; HP might not want them anyway... :)
> feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...[/color]

Here is another approach that might shed some light on the
issue.


Get ttcp. This application is I understand optimised to generate
network traffic with minimum CPU load. OOPS, or netperf:-) however
I have no experience of the latter mostly since IIRC there
were difficulties in finding Windows binaries.
[url]ftp://ftp.cup.hp.com/dist/networking/benchmarks/netperf/binaries/2.1pl1/win32/x86/[/url]
Not now!

Test between the machines. If when using write sizes of the same order
as
NFS is using you nearly saturate the network and the CPU is low
on the machines then changing the MTU will not likely make much
difference.

If on the other hand the network can not be saturated then
there is likely some network related bottleneck.

If the CPU is low then said bottleneck is likely in the
network hardware somewhere and you are probably stuck with it.
There is I guess a small chance that the NIC is unable
to (oh, I forget the correct term) efficienty convert the
packets into cells in the case of small packets.

If the CPU is high then it is more likely that changing the
MTU will help.

If you use UDP packets you will be able to check for
lost packets directly. In UDP mode the packets are just blasted out
as fast as possible and the other end counts them in.

I used Microsoft's ttcp.

There are some performance numbers on Rick's web site.
[url]http://www.netperf.org/netperf/numbers/NetperfBrowse.html[/url]
Set network type to ATM.


C:\Program Files\Support Tools>ttcp -r -u
ttcp-r: me 0.0.0.0
ttcp-r: buflen=8192, nbuf=2048, align=16384/+0, port=5001 udp
ttcp-r: recvfrom 127.0.0.1
ttcp-r: 16777216 bytes in 224 real milliseconds = 73142 KB/sec
ttcp-r: 2050 I/O calls, msec/call = 0, calls/sec = 9151, bytes/call =
8184

C:\Program Files\Support Tools>

C:\Program Files\Support Tools>ttcp -t -u 127.0.0.1
ttcp-t: me 0.0.0.0 -> him 127.0.0.1
ttcp-t: buflen=8192, nbuf=2048, align=16384/+0, port=5001 udp ->
127.0.0.1
ttcp-t: done sending, nbuf = -1
ttcp-t: 16777216 bytes in 224 real milliseconds = 73142 KB/sec
ttcp-t: 2054 I/O calls, msec/call = 0, calls/sec = 9169, bytes/call =
8168


Or just try it out. Get a good idea of the CPU and network traffic
before you start.

[email]anybody43@hotmail.com[/email] wrote:[color=blue]
> Here is another approach that might shed some light on the
> issue.[/color]

[color=blue]
> Get ttcp. This application is I understand optimised to generate
> network traffic with minimum CPU load. OOPS, or netperf:-) however
> I have no experience of the latter mostly since IIRC there
> were difficulties in finding Windows binaries.[/color]
[color=blue]
> [url]ftp://ftp.cup.hp.com/dist/networking/benchmarks/netperf/binaries/2.1pl1/win32/x86/[/url]
> Not now![/color]

Alas, those are a bit old, but still servicable - netperf is up to
2.4.0 now.
[color=blue]
> Test between the machines. If when using write sizes of the same
> order as NFS is using you nearly saturate the network and the CPU is
> low on the machines then changing the MTU will not likely make much
> difference.[/color]

That is likely true.

I should point-out though there is a very key difference between a
ttcp test and NFS. Ttcp (ditto netperf *_STREAM ) will be a
unidirectional stream of data. Under the hood, NFS is a
request/response protocol.

At this point, if I were trying to simulate NFS read/write with
netperf, I would use either a TCP_RR or UDP_RR test and enable the
DO_FIRST_BURST stuff, setting the first burst value to the maximum
number of outstanding NFS requests that the client would support.
DO_FIRST_BURST (or --enable-burst - IIRC - on 2.4.0 ./configure) is
still a bit experimental, so observe it carefully )

If one is going to have separate mounts/connections, then run
concurrent, single-request *RR tests.

In each case, match the request,response sizes (-r) with what your NFS
will be doing.
[color=blue]
> There are some performance numbers on Rick's web site.
> [url]http://www.netperf.org/netperf/numbers/NetperfBrowse.html[/url]
> Set network type to ATM.[/color]

I have _got_ to get that updated one of these days.

rick jones
--
Wisdom Teeth are impacted, people are affected by the effects of events.
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

> > ... not likely make much[color=blue][color=green]
> > difference.[/color]
> That is likely true.[/color]

Are we "Likely Lads"?
[color=blue]
> At this point, if I were trying to simulate NFS read/write with
> netperf, I would use either a TCP_RR or UDP_RR test and enable the
> DO_FIRST_BURST stuff[/color]

The question that I was trying to answer with the Network Test tools
was:
Is the MTU size limiting the performance of the network?
This is I think a simpler problem than "is the MTU size
limiting the performance of NFS?". Clearly though if the network
performance is limited then NFS will be similarly limited.
(Well I think that it is clear anyway;)

In order to do this I think that it helps to eliminate
other behaviour and to just blast out traffic. I will refer to ttcp
since I am slightly familiar with it however I would
be astonished if netperf did not offer the facilities that
I describe here. If UDP is used then the traffic just gets
sent out as fast as possible "ttcp -t -u". It doesn't
care if the receiver exists or not. If there is a receiver
"ttcp -r -u" the receiver counts the packets in and produces
some stats.

With TCP on the other hand the traffic flow rate will be affected by
lost packets, tcp rx windows, ...... more I am sure. This is
therefore a more polluted (or less pure) test of the
systems' capacity to sent network traffic.

Anyway, enough I think for some progress on the issue.

The point is to eliminate (factor out) some of the NFS (like)
behaviour.

As much as possible in fact.

Clearly simulating NFS would at some times be useful
and I thank you for your suggestions.

[email]anybody43@hotmail.com[/email] wrote:[color=blue][color=green][color=darkred]
>> > ... not likely make much difference.[/color]
>> That is likely true.[/color][/color]
[color=blue]
> Are we "Likely Lads"?[/color]

No idea as to the reference. Sorry.
[color=blue][color=green]
>> At this point, if I were trying to simulate NFS read/write with
>> netperf, I would use either a TCP_RR or UDP_RR test and enable the
>> DO_FIRST_BURST stuff[/color][/color]
[color=blue]
> The question that I was trying to answer with the Network Test tools
> was:
> Is the MTU size limiting the performance of the network? This is I
> think a simpler problem than "is the MTU size limiting the
> performance of NFS?". Clearly though if the network performance is
> limited then NFS will be similarly limited. (Well I think that it
> is clear anyway;)[/color]

True, if the system cannot get decent performance on ttcp/netperf
unidirectional send it is unlikely to get decent performance on NFS.
Depends on where one wants to start I suppose - after ttcp/netperf
runs ok, the "more like NFS stuff" has to happen any way. 6 of one,
1/2 dozen of the other I suppose.

Now, if we really want to pick nits, isn't it that the system would be
CPU limited and that increasing the MTU mitigates the CPU
limitation?-)
[color=blue]
> In order to do this I think that it helps to eliminate
> other behaviour and to just blast out traffic. I will refer to ttcp
> since I am slightly familiar with it however I would
> be astonished if netperf did not offer the facilities that
> I describe here. If UDP is used then the traffic just gets
> sent out as fast as possible "ttcp -t -u". It doesn't
> care if the receiver exists or not. If there is a receiver
> "ttcp -r -u" the receiver counts the packets in and produces
> some stats.[/color]

Yes, netperf [UDP|TCP]_STREAM will behave much like ttcp. As for UDP
being send as fast as possible, there is at least one stack that
appears to have intra-stack flow control when using UDP, so things
like socket buffer settings (the intra-stack "window" as it were) can
still matter. Not quite just fire and forget.

rick jones
--
a wide gulf separates "what if" from "if only"
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Dale wrote:
[color=blue]
> I wish I could answer your questions right off the bat, Casper, but I have
> to find out what it is you're asking for to begin with. Stu Friedberg's
> post mentioned those same attributes--window size and NFS read/write
> sizes--and I'll have to find out what they're set to. They do seem the most
> likely place to start.[/color]

Before that, be sure you are running over TCP.

Traditionally NFS ran over UDP, and the answers to those questions
are very different in that case.

The read size and write size were important, then, as they determined
the size of the UDP packet before IP fragmentation. Fragmented UDP
worked pretty well over a LAN, but much less well over long distance
Internet links.

Most newer systems will run TCP, it may or may not be the default.
As others say, TCP is probably your best choice.

-- glen