Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3? - Mandrake

This is a discussion on Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3? - Mandrake ; Maybe somebody can help me with this problem: I export several directories from a Novell server and mount them on a Linux machine to have some remote backup and storage facilities. A few years back I have installed a Mandrake ...

+ Reply to Thread
Results 1 to 13 of 13

Thread: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

  1. Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    Maybe somebody can help me with this problem:
    I export several directories from a Novell server and mount them on a
    Linux machine to have some remote backup and storage facilities.
    A few years back I have installed a Mandrake 10.0 box with kernel
    2.6.3 and after some tinkering the NFS performance was about 3.5
    MByte/s on a 100 Mbps full duplex interface card. The FTP performance
    to and from that same Mandrake box was about 7 MByte/s.

    This year I upgraded my hardware and I installed Mandriva 2006 using
    kernel 2.12.12 on the Linux machine. Using exactly the same mount
    options I get a throughput of about 0.050 MByte/s read to the Linux
    and about 0.400 MBytes/s on write from the Linux machine. The FTP
    performance to and from the Mandriva box is around 8 MBytes/s.

    To test if the problem is the kernel or something else I installed
    Mandriva 2007 with kernel 2.6.17 on a little test box and this one
    does exactly the same low NFS performance while giving around 6
    MBytes/s on FTP up- and download.

    I have done some tweaking on the NFS settings and reached the maximum
    of 0.600 MBytes/s for a read and 1 MBytes/s for a write operation. I
    have tweaked rsize, wsize, timeo, udp/tcp, v3, /proc settings
    (ip_frag_threshold etc.) but I just can't get back to the decent
    performance.

    As a last test I picked up my old hardware again, booted it up,
    changed the IP address and checked what the throughput was on this
    one, it was again the expected 3.5 MBytes/s.

    Does anybody have any clue as to what is going on?

    I did notice that on the newer boxed there are other files stated at
    the modprobe and rpcinfo commands: sunrpc.ko.gz and auth_gss which do
    not appear on the 2.6.3 kernel box. And after deleting a sunrpc.ko.gz
    I am now unable to mount any nfs export on my test machine so I reckon
    they do have an impact.
    A waste is a terrible thing to mind

  2. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    On Sat, 14 Oct 2006 12:10:09 +0200, VanderBilt wrote:

    > As a last test I picked up my old hardware again, booted it up, changed
    > the IP address and checked what the throughput was on this one, it was
    > again the expected 3.5 MBytes/s.


    > Does anybody have any clue as to what is going on?


    I would expect hardware from that right there. I've seen no change in NFS
    performance across many kernels. To my knowledge it has not been broken
    in the last few years at any time.

  3. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    On Sat, 14 Oct 2006 18:29:16 GMT, Ron Gibson
    wrote:

    >On Sat, 14 Oct 2006 12:10:09 +0200, VanderBilt wrote:
    >
    >> As a last test I picked up my old hardware again, booted it up, changed
    >> the IP address and checked what the throughput was on this one, it was
    >> again the expected 3.5 MBytes/s.

    >
    >> Does anybody have any clue as to what is going on?

    >
    >I would expect hardware from that right there. I've seen no change in NFS
    >performance across many kernels. To my knowledge it has not been broken
    >in the last few years at any time.


    Yeah, I had that thought as well, but then that doesn't explain the
    good performance on the FTP up- and downloads. Unless there is a
    difference between UDP and TCP based services? Hardware would only
    consider the ethernet layer, or at best IP, and the computers are on
    the same 100 Mbps switch.

    Can it have something to do with a "stated" process that is now
    appearing in my rpcinfo output? That is not present on the old box and
    I have no idea how it got there on my new machine. I tried shutting it
    down on my little test-box but that didn't improve the throughput. I'm
    just wondering now what it is that this stated process does.
    A waste is a terrible thing to mind

  4. Re: Performance of NFS much lower on Linux kernel 2.6.17 then onLinux kernel 2.6.3?

    VanderBilt writes:

    > On Sat, 14 Oct 2006 18:29:16 GMT, Ron Gibson
    > wrote:
    >
    >>On Sat, 14 Oct 2006 12:10:09 +0200, VanderBilt wrote:
    >>
    >>> As a last test I picked up my old hardware again, booted it up, changed
    >>> the IP address and checked what the throughput was on this one, it was
    >>> again the expected 3.5 MBytes/s.

    >>
    >>> Does anybody have any clue as to what is going on?

    >>
    >>I would expect hardware from that right there. I've seen no change in NFS
    >>performance across many kernels. To my knowledge it has not been broken
    >>in the last few years at any time.

    >
    > Yeah, I had that thought as well, but then that doesn't explain the
    > good performance on the FTP up- and downloads. Unless there is a
    > difference between UDP and TCP based services? Hardware would only
    > consider the ethernet layer, or at best IP, and the computers are on
    > the same 100 Mbps switch.
    >
    > Can it have something to do with a "stated" process that is now
    > appearing in my rpcinfo output? That is not present on the old box and
    > I have no idea how it got there on my new machine. I tried shutting it
    > down on my little test-box but that didn't improve the throughput. I'm
    > just wondering now what it is that this stated process does.
    > A waste is a terrible thing to mind


    Just a guess, do:

    ifconfig

    look for collisions, errors.

  5. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    On Sat, 14 Oct 2006 23:18:04 +0200, VanderBilt wrote:

    >>> Does anybody have any clue as to what is going on?


    >>I would expect hardware from that right there. I've seen no change in NFS
    >>performance across many kernels. To my knowledge it has not been broken
    >>in the last few years at any time.


    > Yeah, I had that thought as well, but then that doesn't explain the good
    > performance on the FTP up- and downloads. Unless there is a difference
    > between UDP and TCP based services? Hardware would only consider the
    > ethernet layer, or at best IP, and the computers are on the same 100 Mbps
    > switch.


    Ah! I have it narrowed down! It's either hardware or software :-)

    You know you I have used Webmin, wizards, control center, etc and the
    best way for me is to edit the files by hand. Here is what you need (one
    text and the other HTML)...

    http://www.tldp.org/HOWTO/text/NFS-HOWTO

    http://tldp.org/HOWTO/NFS-HOWTO/index.html

    > Can it have something to do with a "stated" process that is now appearing
    > in my rpcinfo output?


    The above tells you you need exactly and will cover the various daemons.

  6. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    On Sunday 15 October 2006 02:12, Ron Gibson stood up and addressed the
    masses in /alt.os.linux.mandrake/ as follows...:

    > On Sat, 14 Oct 2006 23:18:04 +0200, VanderBilt wrote:
    >
    >>>> Does anybody have any clue as to what is going on?

    >
    >>>I would expect hardware from that right there. I've seen no change in NFS
    >>>performance across many kernels. To my knowledge it has not been broken
    >>>in the last few years at any time.

    >
    >> Yeah, I had that thought as well, but then that doesn't explain the good
    >> performance on the FTP up- and downloads. Unless there is a difference
    >> between UDP and TCP based services? Hardware would only consider the
    >> ethernet layer, or at best IP, and the computers are on the same 100 Mbps
    >> switch.

    >
    > http://tldp.org/HOWTO/NFS-HOWTO/index.html


    It might also be useful to the OP to mention that later 2.6 kernels also
    support NFS over TCP/IP, and that this /could/ indeed yield a performance
    hit versus NFS over UDP, especially if there are certain network-related
    bugs in the kernel which exist in the TCP/IP layer but do not affect UDP.
    I don't know what NFS options the Mandriva 2.6.12 kernel was compiled with.

    Another thing to mention is that the 2.6.12 kernel introduced a number of
    bugs and broke a lot of things. Perhaps trying a later kernel version - I
    am alluding to installing Mandrake 2007 - would yield better performance,
    and then the culprit would be known right away. ;-)

    --
    With kind regards,

    *Aragorn*
    (registered GNU/Linux user #223157)

  7. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    VanderBilt wrote:
    :
    > This year I upgraded my hardware and I installed Mandriva 2006 using
    > kernel 2.12.12 on the Linux machine. Using exactly the same mount
    > options I get a throughput of about 0.050 MByte/s read to the Linux
    > and about 0.400 MBytes/s on write from the Linux machine. The FTP
    > performance to and from the Mandriva box is around 8 MBytes/s.


    I have some machines running nfs
    NFS Server: Mandriva 2006 (kernel 2.6.12)
    NFS Client: Mandriva 2007 (kernel 2.6.17)
    As an example, while copying 100MB files I get between
    4.5 - 5.3 MBytes/sec in both directions using nfs version 3
    with default setting on protocol and rsize,wsize.
    % mount -t nfs -o nfsvers=3 :/home/export /mnt/nfs

    Have you tried using default mount options ?

    /Karl D


  8. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    On Sun, 15 Oct 2006 05:29:11 GMT, Aragorn
    wrote:

    >On Sunday 15 October 2006 02:12, Ron Gibson stood up and addressed the
    >masses in /alt.os.linux.mandrake/ as follows...:
    >
    >> On Sat, 14 Oct 2006 23:18:04 +0200, VanderBilt wrote:
    >>
    >>>>> Does anybody have any clue as to what is going on?

    >>
    >>>>I would expect hardware from that right there. I've seen no change in NFS
    >>>>performance across many kernels. To my knowledge it has not been broken
    >>>>in the last few years at any time.

    >>
    >>> Yeah, I had that thought as well, but then that doesn't explain the good
    >>> performance on the FTP up- and downloads. Unless there is a difference
    >>> between UDP and TCP based services? Hardware would only consider the
    >>> ethernet layer, or at best IP, and the computers are on the same 100 Mbps
    >>> switch.

    >>
    >> http://tldp.org/HOWTO/NFS-HOWTO/index.html

    >
    >It might also be useful to the OP to mention that later 2.6 kernels also
    >support NFS over TCP/IP, and that this /could/ indeed yield a performance
    >hit versus NFS over UDP, especially if there are certain network-related
    >bugs in the kernel which exist in the TCP/IP layer but do not affect UDP.
    >I don't know what NFS options the Mandriva 2.6.12 kernel was compiled with.
    >
    >Another thing to mention is that the 2.6.12 kernel introduced a number of
    >bugs and broke a lot of things. Perhaps trying a later kernel version - I
    >am alluding to installing Mandrake 2007 - would yield better performance,
    >and then the culprit would be known right away. ;-)


    I have tried installing another box with Mandriva 2007, using kernel
    2.6.17, but this gives me exactly the same lousy throughput. And I
    have also tried TCP, but with that I did not even get into the 0.100
    MBytes/s range.

    I do still have this question: Why does this "stated" suddenly appear
    on the rpcinfo list? Can somebody explain that to me? Is this because
    the nfs protocol now supports v4? And if so, how can I downgrade to an
    nfs client like the one shipped with Mandrake 10.0, so that I do not
    get all this new stuff but just performance?
    A waste is a terrible thing to mind

  9. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    On 15 Oct 2006 00:17:43 -0700, "KarlD" wrote:

    >VanderBilt wrote:
    >:
    >> This year I upgraded my hardware and I installed Mandriva 2006 using
    >> kernel 2.12.12 on the Linux machine. Using exactly the same mount
    >> options I get a throughput of about 0.050 MByte/s read to the Linux
    >> and about 0.400 MBytes/s on write from the Linux machine. The FTP
    >> performance to and from the Mandriva box is around 8 MBytes/s.

    >
    >I have some machines running nfs
    >NFS Server: Mandriva 2006 (kernel 2.6.12)
    >NFS Client: Mandriva 2007 (kernel 2.6.17)
    >As an example, while copying 100MB files I get between
    >4.5 - 5.3 MBytes/sec in both directions using nfs version 3
    >with default setting on protocol and rsize,wsize.
    >% mount -t nfs -o nfsvers=3 :/home/export /mnt/nfs
    >
    >Have you tried using default mount options ?
    >
    >/Karl D


    Oh yes, I have tried many options. The best result I get with this:
    rw,nosuid,timeo=1,rsize=32768,wsize=2048,udp,intr.

    It seems that one of the most effective option changes was to set the
    timeo value to 1. Before that it was almost like the server received
    all the data but the client was still waiting for something.

    A waste is a terrible thing to mind

  10. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    On Sat, 14 Oct 2006 18:06:05 -0400, Dan Espen
    wrote:

    >VanderBilt writes:
    >
    >> On Sat, 14 Oct 2006 18:29:16 GMT, Ron Gibson
    >> wrote:
    >>
    >>>On Sat, 14 Oct 2006 12:10:09 +0200, VanderBilt wrote:
    >>>
    >>>> As a last test I picked up my old hardware again, booted it up, changed
    >>>> the IP address and checked what the throughput was on this one, it was
    >>>> again the expected 3.5 MBytes/s.
    >>>
    >>>> Does anybody have any clue as to what is going on?
    >>>
    >>>I would expect hardware from that right there. I've seen no change in NFS
    >>>performance across many kernels. To my knowledge it has not been broken
    >>>in the last few years at any time.

    >>
    >> Yeah, I had that thought as well, but then that doesn't explain the
    >> good performance on the FTP up- and downloads. Unless there is a
    >> difference between UDP and TCP based services? Hardware would only
    >> consider the ethernet layer, or at best IP, and the computers are on
    >> the same 100 Mbps switch.
    >>
    >> Can it have something to do with a "stated" process that is now
    >> appearing in my rpcinfo output? That is not present on the old box and
    >> I have no idea how it got there on my new machine. I tried shutting it
    >> down on my little test-box but that didn't improve the throughput. I'm
    >> just wondering now what it is that this stated process does.
    >> A waste is a terrible thing to mind

    >
    >Just a guess, do:
    >
    >ifconfig
    >
    >look for collisions, errors.


    It's not the hardware, or else the FTP would be impacted as well:
    RX packets:441270460 errors:0 dropped:0 overruns:0 frame:0
    TX packets:441375129 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:18323115 (17.4 MiB) TX bytes:2628631720 (2.4 GiB)

    I suspect it has something to do with the new NFS capabilities, like
    stated and v4. But how can I downgrade to a former NFS version, like
    the one that was used in Mandrake 10.0? It seems that it is impossible
    to find a single package involved in this application.
    A waste is a terrible thing to mind

  11. Re: Performance of NFS much lower on Linux kernel 2.6.17 then onLinux kernel 2.6.3?



    VanderBilt wrote:

    >On 15 Oct 2006 00:17:43 -0700, "KarlD" wrote:
    >
    >
    >
    >>VanderBilt wrote:
    >>:
    >>
    >>
    >>>This year I upgraded my hardware and I installed Mandriva 2006 using
    >>>kernel 2.12.12 on the Linux machine. Using exactly the same mount
    >>>options I get a throughput of about 0.050 MByte/s read to the Linux
    >>>and about 0.400 MBytes/s on write from the Linux machine. The FTP
    >>>performance to and from the Mandriva box is around 8 MBytes/s.
    >>>
    >>>

    >>I have some machines running nfs
    >>NFS Server: Mandriva 2006 (kernel 2.6.12)
    >>NFS Client: Mandriva 2007 (kernel 2.6.17)
    >>As an example, while copying 100MB files I get between
    >>4.5 - 5.3 MBytes/sec in both directions using nfs version 3
    >>with default setting on protocol and rsize,wsize.
    >>% mount -t nfs -o nfsvers=3 :/home/export /mnt/nfs
    >>
    >>Have you tried using default mount options ?
    >>
    >>/Karl D
    >>
    >>

    >
    >Oh yes, I have tried many options. The best result I get with this:
    >rw,nosuid,timeo=1,rsize=32768,wsize=2048,udp,intr.
    >
    >


    Try adding async in the mix. I have good results with
    rsize=8192,wsize=8192,noatime,nfsvers=3,async,intr

    The change could be due to the change in the default for sync/async. It
    used to be async and then changed to sync.

    Steve


  12. Re: Performance of NFS much lower on Linux kernel 2.6.17 then onLinux kernel 2.6.3?

    Steve Cousins wrote:
    >
    >
    > VanderBilt wrote:
    >
    >> On 15 Oct 2006 00:17:43 -0700, "KarlD" wrote:
    >>
    >>
    >>
    >>> VanderBilt wrote:
    >>> :
    >>>
    >>>> This year I upgraded my hardware and I installed Mandriva 2006 using
    >>>> kernel 2.12.12 on the Linux machine. Using exactly the same mount
    >>>> options I get a throughput of about 0.050 MByte/s read to the Linux
    >>>> and about 0.400 MBytes/s on write from the Linux machine. The FTP
    >>>> performance to and from the Mandriva box is around 8 MBytes/s.
    >>>>
    >>> I have some machines running nfs
    >>> NFS Server: Mandriva 2006 (kernel 2.6.12)
    >>> NFS Client: Mandriva 2007 (kernel 2.6.17)
    >>> As an example, while copying 100MB files I get between
    >>> 4.5 - 5.3 MBytes/sec in both directions using nfs version 3
    >>> with default setting on protocol and rsize,wsize.
    >>> % mount -t nfs -o nfsvers=3 :/home/export /mnt/nfs
    >>>
    >>> Have you tried using default mount options ?
    >>>
    >>> /Karl D
    >>>

    >>
    >> Oh yes, I have tried many options. The best result I get with this:
    >> rw,nosuid,timeo=1,rsize=32768,wsize=2048,udp,intr.
    >>

    >
    > Try adding async in the mix. I have good results with
    > rsize=8192,wsize=8192,noatime,nfsvers=3,async,intr
    >
    > The change could be due to the change in the default for sync/async. It
    > used to be async and then changed to sync.
    > Steve
    >


    I have two suggestions and you may have already tried these....

    1) Disable firewall software on both machines. If TCP traffic is workin
    g ok then it's likely to be with the protocol itself. Certainly wont
    hurt to rule it out.

    2) Install ethereal and monitor the network traffic between the two
    boxes. Will hopefully tell you which machine is slowing things down.

    I'm currently getting burst rates of 130MB/s on mandriva 2007 via nfs
    V4, admittedly I have a gigabit switch inbetween, but I certainly didn't
    do any tweaking to NFs on Mandriva 2007.

    trb



  13. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    Sorry Tony, there is no security of any kind running on either
    machine. The Netware server or the Mandriva client are connected to
    each other on the same 100 Mbps LAN switch.

    I know which machine is slowing things down, it's the Mandriva client.
    If I re-attach the old Mandrake 10.0 machine to the Netware exports I
    get a decent 3.5 MB/s performance.

    I have tried using different kernels (2.6.16, 2.6.18.1) each
    hand-compiled with different options for the NFS (without Kerberos,
    not as a module, without NFSv4 support etc.) but it always stays at
    this low 400 kB/s read and 900 kB/s writes. I am starting to suspect
    the portmapper or rpcio, but how can I downgrade those to the point
    where they are the same as the old Mandrake ones? They don't seem to
    be a RPM of their own, but I'll figure that one out.

    On Thu, 19 Oct 2006 19:18:47 +0200, Tony Barnard
    wrote:

    >>>> Have you tried using default mount options ?
    >>>> /Karl D
    >>>
    >>> Oh yes, I have tried many options. The best result I get with this:
    >>> rw,nosuid,timeo=1,rsize=32768,wsize=2048,udp,intr.

    >>
    >> Try adding async in the mix. I have good results with
    >> rsize=8192,wsize=8192,noatime,nfsvers=3,async,intr
    >>
    >> The change could be due to the change in the default for sync/async. It
    >> used to be async and then changed to sync.

    >
    >I have two suggestions and you may have already tried these....
    >
    >1) Disable firewall software on both machines. If TCP traffic is workin
    >g ok then it's likely to be with the protocol itself. Certainly wont
    >hurt to rule it out.
    >
    >2) Install ethereal and monitor the network traffic between the two
    >boxes. Will hopefully tell you which machine is slowing things down.
    >
    >I'm currently getting burst rates of 130MB/s on mandriva 2007 via nfs
    >V4, admittedly I have a gigabit switch inbetween, but I certainly didn't
    >do any tweaking to NFs on Mandriva 2007.
    >
    >trb
    >



+ Reply to Thread