NFS and MTU - NFS

This is a discussion on NFS and MTU - NFS ; A customer asked me recently if increasing the maximum transmission unit on TCP/IP packets would increase network throughput. Good question, and I'd like to solicit opinions here. I'm the administrator of a small, local ATM network of 8 systems. My ...

+ Reply to Thread
Results 1 to 14 of 14

Thread: NFS and MTU

  1. NFS and MTU

    A customer asked me recently if increasing the maximum transmission unit on
    TCP/IP packets would increase network throughput. Good question, and I'd
    like to solicit opinions here.

    I'm the administrator of a small, local ATM network of 8 systems. My client
    systems are connected at OC-3 and access separate file and data servers
    connected at OC-12. The major data access functions retrieve, edit and
    write data in NFS mounted directories from the data server. The volume of
    data is on the order of tens to hundreds of gigbytes and thousands to tens
    of thousands of files. The MTU is currently set to default: 1500 bytes. My
    question is: Would increasing the MTU result in an increase in throughput?
    I'm guessing there would be some noticeable difference from the reduced
    packet count, but I don't know what that would be: I have no experience in
    this matter.

    Your opinions are greatly appreciated.

    Dale



  2. Re: NFS and MTU

    > if increasing the maximum transmission unit on
    > TCP/IP packets would increase network throughput.


    The answer, as usual, is "it depends". Within an ATM network, "MTU" is
    a fairly useless notion because ATM payloads are always 48 bytes.

    MTU does force fragmentation of IP datagrams and (usually) limits the
    size of TCP segments. As you clearly know, fewer, larger packets means
    less per-packet overhead. However, those overheads are paid only by
    devices at the edges of the ATM network. RFC 1626 specifies a default
    MTU of 9180 bytes over ATM AAL5.

    However, increasing TCP segment and IP datagram sizes (by any means)
    _could_ be counter-productive. If there is any measurable loss in the
    link layer (i.e., ATM payloads), the larger sizes mean bigger impacts
    due to loss, as TCP ultimately detects failures and retransmits in
    terms of segments.

    Because this is an ATM network rather than Ethernet, your network adapters
    are going to be breaking the "MTU" into a stream of 48-byte ATM payloads
    anyway. As long as the adapters can keep the links busy, increasing the
    MTU will have only very marginal effects, limited to lowering processing
    overheads within the hosts. It will have no further effect on the ATM
    network utilization or overheads.

    However, NFS and TCP may be rate-limited in ways that starve your adapters
    somewhat. First, if you haven't already done so, increase your NFS read
    and write sizes to the maximum both clients and servers can support,
    through
    client mount options. Second, enable the TCP Window Scale and Selective
    Acknowledgement options at both clients and servers. It's possible your
    TCP implementations already have these options and quietly negotiate using
    them when setting up a connection.

    Increasing read and write sizes obviously reduces the number of NFS
    operations. This has only a modest effect on the amount of network
    traffic, but reduces total round trip latency and improves client and
    server host overheads somewhat.

    If your clients were connected via OC-12, the Window Scale option would
    become imperative to get maximum TCP performance. Limited to OC-3, it's
    still useful, but probably not critical.

    The Selective Acknowledgment option, if supported, reduces the performance
    hit TCP takes from a lost packet by enabling it to retransmit just a single
    TCP segment in a single IP datagram. Without it, TCP must retransmit all
    data from the beginning of the lost TCP segment.

    By the way, are you running Classical IP over ATM (RFC 1577), LAN
    emulation,
    MPOA, vendor-specific multilevel switching, other???

    Stu Friedberg

  3. Re: NFS and MTU

    "Dale" writes:

    >A customer asked me recently if increasing the maximum transmission unit on
    >TCP/IP packets would increase network throughput. Good question, and I'd
    >like to solicit opinions here.


    >I'm the administrator of a small, local ATM network of 8 systems. My client
    >systems are connected at OC-3 and access separate file and data servers
    >connected at OC-12. The major data access functions retrieve, edit and
    >write data in NFS mounted directories from the data server. The volume of
    >data is on the order of tens to hundreds of gigbytes and thousands to tens
    >of thousands of files. The MTU is currently set to default: 1500 bytes. My
    >question is: Would increasing the MTU result in an increase in throughput?
    >I'm guessing there would be some noticeable difference from the reduced
    >packet count, but I don't know what that would be: I have no experience in
    >this matter.


    A larger MTU may help, but the first things to look at would be:

    - Are you using NFS over TCP?
    - What TCP window sizes are you using?
    - What NFS read/write sizes are you using?


    I'd increase all of the above before using a bigger MTU.

    Casper
    --
    Expressed in this posting are my opinions. They are in no way related
    to opinions held by my employer, Sun Microsystems.
    Statements on Sun products included here are not gospel and may
    be fiction rather than truth.

  4. Re: NFS and MTU

    > - Are you using NFS over TCP?
    > - What TCP window sizes are you using?
    > - What NFS read/write sizes are you using?


    Add to those:

    What is your current TCP retransmission rate?

    rick jones
    --
    No need to believe in either side, or any side. There is no cause.
    There's only yourself. The belief is in your own precision. - Jobert
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  5. Re: NFS and MTU

    Thanks Stu, I will look into some of these things (those I understand), and
    try to get some education on the rest. What I'm getting out of all this is
    that despite what I do at the extreme ends of the connection, while that may
    do some good, I'm still limited by ATM performance dependencies.

    Dale

    "Stuart Friedberg" wrote in message
    newsp.sroe5gepgowx4s@m-stuartf...
    >> if increasing the maximum transmission unit on
    >> TCP/IP packets would increase network throughput.

    >
    > The answer, as usual, is "it depends". Within an ATM network, "MTU" is
    > a fairly useless notion because ATM payloads are always 48 bytes.
    >
    > MTU does force fragmentation of IP datagrams and (usually) limits the
    > size of TCP segments. As you clearly know, fewer, larger packets means
    > less per-packet overhead. However, those overheads are paid only by
    > devices at the edges of the ATM network. RFC 1626 specifies a default
    > MTU of 9180 bytes over ATM AAL5.
    >
    > However, increasing TCP segment and IP datagram sizes (by any means)
    > _could_ be counter-productive. If there is any measurable loss in the
    > link layer (i.e., ATM payloads), the larger sizes mean bigger impacts
    > due to loss, as TCP ultimately detects failures and retransmits in
    > terms of segments.
    >
    > Because this is an ATM network rather than Ethernet, your network adapters
    > are going to be breaking the "MTU" into a stream of 48-byte ATM payloads
    > anyway. As long as the adapters can keep the links busy, increasing the
    > MTU will have only very marginal effects, limited to lowering processing
    > overheads within the hosts. It will have no further effect on the ATM
    > network utilization or overheads.
    >
    > However, NFS and TCP may be rate-limited in ways that starve your adapters
    > somewhat. First, if you haven't already done so, increase your NFS read
    > and write sizes to the maximum both clients and servers can support,
    > through
    > client mount options. Second, enable the TCP Window Scale and Selective
    > Acknowledgement options at both clients and servers. It's possible your
    > TCP implementations already have these options and quietly negotiate using
    > them when setting up a connection.
    >
    > Increasing read and write sizes obviously reduces the number of NFS
    > operations. This has only a modest effect on the amount of network
    > traffic, but reduces total round trip latency and improves client and
    > server host overheads somewhat.
    >
    > If your clients were connected via OC-12, the Window Scale option would
    > become imperative to get maximum TCP performance. Limited to OC-3, it's
    > still useful, but probably not critical.
    >
    > The Selective Acknowledgment option, if supported, reduces the performance
    > hit TCP takes from a lost packet by enabling it to retransmit just a
    > single
    > TCP segment in a single IP datagram. Without it, TCP must retransmit all
    > data from the beginning of the lost TCP segment.
    >
    > By the way, are you running Classical IP over ATM (RFC 1577), LAN
    > emulation,
    > MPOA, vendor-specific multilevel switching, other???
    >
    > Stu Friedberg




  6. Re: NFS and MTU

    I wish I could answer your questions right off the bat, Casper, but I have
    to find out what it is you're asking for to begin with. Stu Friedberg's
    post mentioned those same attributes--window size and NFS read/write
    sizes--and I'll have to find out what they're set to. They do seem the most
    likely place to start.

    Thanks,
    Dale

    "Casper H.S. Dik" wrote in message
    news:429d8216$0$23350$e4fe514c@news.xs4all.nl...
    > "Dale" writes:
    >
    >>A customer asked me recently if increasing the maximum transmission unit
    >>on
    >>TCP/IP packets would increase network throughput. Good question, and I'd
    >>like to solicit opinions here.

    >
    >>I'm the administrator of a small, local ATM network of 8 systems. My
    >>client
    >>systems are connected at OC-3 and access separate file and data servers
    >>connected at OC-12. The major data access functions retrieve, edit and
    >>write data in NFS mounted directories from the data server. The volume of
    >>data is on the order of tens to hundreds of gigbytes and thousands to tens
    >>of thousands of files. The MTU is currently set to default: 1500 bytes.
    >>My
    >>question is: Would increasing the MTU result in an increase in throughput?
    >>I'm guessing there would be some noticeable difference from the reduced
    >>packet count, but I don't know what that would be: I have no experience in
    >>this matter.

    >
    > A larger MTU may help, but the first things to look at would be:
    >
    > - Are you using NFS over TCP?
    > - What TCP window sizes are you using?
    > - What NFS read/write sizes are you using?
    >
    >
    > I'd increase all of the above before using a bigger MTU.
    >
    > Casper
    > --
    > Expressed in this posting are my opinions. They are in no way related
    > to opinions held by my employer, Sun Microsystems.
    > Statements on Sun products included here are not gospel and may
    > be fiction rather than truth.




  7. Re: NFS and MTU

    Thanks, I'll look at that. I believe that's a parameter capture by
    oracallator so it should be graphed. I presume the concern here is that if
    it's excessive, it represents overhead. What would excessive be?

    Dale

    "Rick Jones" wrote in message
    newsDone.6329$Rs2.5189@news.cpqcorp.net...
    >> - Are you using NFS over TCP?
    >> - What TCP window sizes are you using?
    >> - What NFS read/write sizes are you using?

    >
    > Add to those:
    >
    > What is your current TCP retransmission rate?
    >
    > rick jones
    > --
    > No need to believe in either side, or any side. There is no cause.
    > There's only yourself. The belief is in your own precision. - Jobert
    > these opinions are mine, all mine; HP might not want them anyway...
    > feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...




  8. Re: NFS and MTU

    Dale wrote:
    > Thanks, I'll look at that. I believe that's a parameter capture by
    > oracallator so it should be graphed. I presume the concern here is
    > that if it's excessive, it represents overhead. What would
    > excessive be?


    NFS is (in broad handwaving terms) a latency sensitive
    application/protocol/whatever. ATM networks imply WAN which implies
    high latencies. They also imply bandwidth constraints.

    When latency is high, the only way to maintain throughput is with
    increased parallelism. This means having more NFS requests
    outstanding at any one time, and also leads into Casper's mention of
    TCP windows (since the TCP window limits how much data can be
    outstanding at one time on a connection, which then may or may not be
    less than what NFS is trying to have on that connection at one time).

    However, if there is much in the way of packet loss, TCP cannot
    maintain a large window and throughput can suffer.

    rick jones
    --
    a wide gulf separates "what if" from "if only"
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  9. Re: NFS and MTU

    Uhhh...I'll have to decipher that one for a bit. Thanks for responding.

    Dale

    "Rick Jones" wrote in message
    news:Bxqne.6343$VB2.3340@news.cpqcorp.net...
    > Dale wrote:
    >> Thanks, I'll look at that. I believe that's a parameter capture by
    >> oracallator so it should be graphed. I presume the concern here is
    >> that if it's excessive, it represents overhead. What would
    >> excessive be?

    >
    > NFS is (in broad handwaving terms) a latency sensitive
    > application/protocol/whatever. ATM networks imply WAN which implies
    > high latencies. They also imply bandwidth constraints.
    >
    > When latency is high, the only way to maintain throughput is with
    > increased parallelism. This means having more NFS requests
    > outstanding at any one time, and also leads into Casper's mention of
    > TCP windows (since the TCP window limits how much data can be
    > outstanding at one time on a connection, which then may or may not be
    > less than what NFS is trying to have on that connection at one time).
    >
    > However, if there is much in the way of packet loss, TCP cannot
    > maintain a large window and throughput can suffer.
    >
    > rick jones
    > --
    > a wide gulf separates "what if" from "if only"
    > these opinions are mine, all mine; HP might not want them anyway...
    > feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...




  10. Re: NFS and MTU

    Here is another approach that might shed some light on the
    issue.


    Get ttcp. This application is I understand optimised to generate
    network traffic with minimum CPU load. OOPS, or netperf:-) however
    I have no experience of the latter mostly since IIRC there
    were difficulties in finding Windows binaries.
    ftp://ftp.cup.hp.com/dist/networking...pl1/win32/x86/
    Not now!

    Test between the machines. If when using write sizes of the same order
    as
    NFS is using you nearly saturate the network and the CPU is low
    on the machines then changing the MTU will not likely make much
    difference.

    If on the other hand the network can not be saturated then
    there is likely some network related bottleneck.

    If the CPU is low then said bottleneck is likely in the
    network hardware somewhere and you are probably stuck with it.
    There is I guess a small chance that the NIC is unable
    to (oh, I forget the correct term) efficienty convert the
    packets into cells in the case of small packets.

    If the CPU is high then it is more likely that changing the
    MTU will help.

    If you use UDP packets you will be able to check for
    lost packets directly. In UDP mode the packets are just blasted out
    as fast as possible and the other end counts them in.

    I used Microsoft's ttcp.

    There are some performance numbers on Rick's web site.
    http://www.netperf.org/netperf/numbe...erfBrowse.html
    Set network type to ATM.


    C:\Program Files\Support Tools>ttcp -r -u
    ttcp-r: me 0.0.0.0
    ttcp-r: buflen=8192, nbuf=2048, align=16384/+0, port=5001 udp
    ttcp-r: recvfrom 127.0.0.1
    ttcp-r: 16777216 bytes in 224 real milliseconds = 73142 KB/sec
    ttcp-r: 2050 I/O calls, msec/call = 0, calls/sec = 9151, bytes/call =
    8184

    C:\Program Files\Support Tools>

    C:\Program Files\Support Tools>ttcp -t -u 127.0.0.1
    ttcp-t: me 0.0.0.0 -> him 127.0.0.1
    ttcp-t: buflen=8192, nbuf=2048, align=16384/+0, port=5001 udp ->
    127.0.0.1
    ttcp-t: done sending, nbuf = -1
    ttcp-t: 16777216 bytes in 224 real milliseconds = 73142 KB/sec
    ttcp-t: 2054 I/O calls, msec/call = 0, calls/sec = 9169, bytes/call =
    8168


    Or just try it out. Get a good idea of the CPU and network traffic
    before you start.


  11. Re: NFS and MTU

    anybody43@hotmail.com wrote:
    > Here is another approach that might shed some light on the
    > issue.



    > Get ttcp. This application is I understand optimised to generate
    > network traffic with minimum CPU load. OOPS, or netperf:-) however
    > I have no experience of the latter mostly since IIRC there
    > were difficulties in finding Windows binaries.


    > ftp://ftp.cup.hp.com/dist/networking...pl1/win32/x86/
    > Not now!


    Alas, those are a bit old, but still servicable - netperf is up to
    2.4.0 now.

    > Test between the machines. If when using write sizes of the same
    > order as NFS is using you nearly saturate the network and the CPU is
    > low on the machines then changing the MTU will not likely make much
    > difference.


    That is likely true.

    I should point-out though there is a very key difference between a
    ttcp test and NFS. Ttcp (ditto netperf *_STREAM ) will be a
    unidirectional stream of data. Under the hood, NFS is a
    request/response protocol.

    At this point, if I were trying to simulate NFS read/write with
    netperf, I would use either a TCP_RR or UDP_RR test and enable the
    DO_FIRST_BURST stuff, setting the first burst value to the maximum
    number of outstanding NFS requests that the client would support.
    DO_FIRST_BURST (or --enable-burst - IIRC - on 2.4.0 ./configure) is
    still a bit experimental, so observe it carefully )

    If one is going to have separate mounts/connections, then run
    concurrent, single-request *RR tests.

    In each case, match the request,response sizes (-r) with what your NFS
    will be doing.

    > There are some performance numbers on Rick's web site.
    > http://www.netperf.org/netperf/numbe...erfBrowse.html
    > Set network type to ATM.


    I have _got_ to get that updated one of these days.

    rick jones
    --
    Wisdom Teeth are impacted, people are affected by the effects of events.
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  12. Re: NFS and MTU

    > > ... not likely make much
    > > difference.

    > That is likely true.


    Are we "Likely Lads"?

    > At this point, if I were trying to simulate NFS read/write with
    > netperf, I would use either a TCP_RR or UDP_RR test and enable the
    > DO_FIRST_BURST stuff


    The question that I was trying to answer with the Network Test tools
    was:
    Is the MTU size limiting the performance of the network?
    This is I think a simpler problem than "is the MTU size
    limiting the performance of NFS?". Clearly though if the network
    performance is limited then NFS will be similarly limited.
    (Well I think that it is clear anyway

    In order to do this I think that it helps to eliminate
    other behaviour and to just blast out traffic. I will refer to ttcp
    since I am slightly familiar with it however I would
    be astonished if netperf did not offer the facilities that
    I describe here. If UDP is used then the traffic just gets
    sent out as fast as possible "ttcp -t -u". It doesn't
    care if the receiver exists or not. If there is a receiver
    "ttcp -r -u" the receiver counts the packets in and produces
    some stats.

    With TCP on the other hand the traffic flow rate will be affected by
    lost packets, tcp rx windows, ...... more I am sure. This is
    therefore a more polluted (or less pure) test of the
    systems' capacity to sent network traffic.

    Anyway, enough I think for some progress on the issue.

    The point is to eliminate (factor out) some of the NFS (like)
    behaviour.

    As much as possible in fact.

    Clearly simulating NFS would at some times be useful
    and I thank you for your suggestions.


  13. Re: NFS and MTU

    anybody43@hotmail.com wrote:
    >> > ... not likely make much difference.

    >> That is likely true.


    > Are we "Likely Lads"?


    No idea as to the reference. Sorry.

    >> At this point, if I were trying to simulate NFS read/write with
    >> netperf, I would use either a TCP_RR or UDP_RR test and enable the
    >> DO_FIRST_BURST stuff


    > The question that I was trying to answer with the Network Test tools
    > was:
    > Is the MTU size limiting the performance of the network? This is I
    > think a simpler problem than "is the MTU size limiting the
    > performance of NFS?". Clearly though if the network performance is
    > limited then NFS will be similarly limited. (Well I think that it
    > is clear anyway


    True, if the system cannot get decent performance on ttcp/netperf
    unidirectional send it is unlikely to get decent performance on NFS.
    Depends on where one wants to start I suppose - after ttcp/netperf
    runs ok, the "more like NFS stuff" has to happen any way. 6 of one,
    1/2 dozen of the other I suppose.

    Now, if we really want to pick nits, isn't it that the system would be
    CPU limited and that increasing the MTU mitigates the CPU
    limitation?-)

    > In order to do this I think that it helps to eliminate
    > other behaviour and to just blast out traffic. I will refer to ttcp
    > since I am slightly familiar with it however I would
    > be astonished if netperf did not offer the facilities that
    > I describe here. If UDP is used then the traffic just gets
    > sent out as fast as possible "ttcp -t -u". It doesn't
    > care if the receiver exists or not. If there is a receiver
    > "ttcp -r -u" the receiver counts the packets in and produces
    > some stats.


    Yes, netperf [UDP|TCP]_STREAM will behave much like ttcp. As for UDP
    being send as fast as possible, there is at least one stack that
    appears to have intra-stack flow control when using UDP, so things
    like socket buffer settings (the intra-stack "window" as it were) can
    still matter. Not quite just fire and forget.

    rick jones
    --
    a wide gulf separates "what if" from "if only"
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  14. Re: NFS and MTU

    Dale wrote:

    > I wish I could answer your questions right off the bat, Casper, but I have
    > to find out what it is you're asking for to begin with. Stu Friedberg's
    > post mentioned those same attributes--window size and NFS read/write
    > sizes--and I'll have to find out what they're set to. They do seem the most
    > likely place to start.


    Before that, be sure you are running over TCP.

    Traditionally NFS ran over UDP, and the answers to those questions
    are very different in that case.

    The read size and write size were important, then, as they determined
    the size of the UDP packet before IP fragmentation. Fragmented UDP
    worked pretty well over a LAN, but much less well over long distance
    Internet links.

    Most newer systems will run TCP, it may or may not be the default.
    As others say, TCP is probably your best choice.

    -- glen


+ Reply to Thread