Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3? - NFS

This is a discussion on Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3? - NFS ; Maybe somebody can help me with this problem: I export several directories from a Novell server and mount them on a Linux machine to have some remote backup and storage facilities. A few years back I have installed a Mandrake ...

+ Reply to Thread
Results 1 to 10 of 10

Thread: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

  1. Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    Maybe somebody can help me with this problem:
    I export several directories from a Novell server and mount them on a
    Linux machine to have some remote backup and storage facilities.
    A few years back I have installed a Mandrake 10.0 box with kernel
    2.6.3 and after some tinkering the NFS performance was about 3.5
    MByte/s on a 100 Mbps full duplex interface card. The FTP performance
    to and from that same Mandrake box was about 7 MByte/s.

    This year I upgraded my hardware and I installed Mandriva 2006 using
    kernel 2.12.12 on the Linux machine. Using exactly the same mount
    options I get a throughput of about 0.050 MByte/s read to the Linux
    and about 0.400 MBytes/s on write from the Linux machine. The FTP
    performance to and from the Mandriva box is around 8 MBytes/s.

    To test if the problem is the kernel or something else I installed
    Mandriva 2007 with kernel 2.6.17 on a little test box and this one
    does exactly the same low NFS performance while giving around 6
    MBytes/s on FTP up- and download.

    I have done some tweaking on the NFS settings and reached the maximum
    of 0.600 MBytes/s for a read and 1 MBytes/s for a write operation. I
    have tweaked rsize, wsize, timeo, udp/tcp, v3, /proc settings
    (ip_frag_threshold etc.) but I just can't get back to the decent
    performance.

    As a last test I picked up my old hardware again, booted it up,
    changed the IP address and checked what the throughput was on this
    one, it was again the expected 3.5 MBytes/s.

    Does anybody have any clue as to what is going on?

    I did notice that on the newer boxed there are other files stated at
    the modprobe and rpcinfo commands: sunrpc.ko.gz and auth_gss which do
    not appear on the 2.6.3 kernel box. And after deleting a sunrpc.ko.gz
    I am now unable to mount any nfs export on my test machine so I reckon
    they do have an impact.
    A waste is a terrible thing to mind

  2. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    VanderBilt wrote:
    :
    > This year I upgraded my hardware and I installed Mandriva 2006 using
    > kernel 2.12.12 on the Linux machine. Using exactly the same mount
    > options I get a throughput of about 0.050 MByte/s read to the Linux
    > and about 0.400 MBytes/s on write from the Linux machine. The FTP
    > performance to and from the Mandriva box is around 8 MBytes/s.


    I have some machines running nfs
    NFS Server: Mandriva 2006 (kernel 2.6.12)
    NFS Client: Mandriva 2007 (kernel 2.6.17)
    As an example, while copying 100MB files I get between
    4.5 - 5.3 MBytes/sec in both directions using nfs version 3
    with default setting on protocol and rsize,wsize.
    % mount -t nfs -o nfsvers=3 :/home/export /mnt/nfs

    Have you tried using default mount options ?

    /Karl D


  3. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    On 15 Oct 2006 00:17:43 -0700, "KarlD" wrote:

    >VanderBilt wrote:
    >:
    >> This year I upgraded my hardware and I installed Mandriva 2006 using
    >> kernel 2.12.12 on the Linux machine. Using exactly the same mount
    >> options I get a throughput of about 0.050 MByte/s read to the Linux
    >> and about 0.400 MBytes/s on write from the Linux machine. The FTP
    >> performance to and from the Mandriva box is around 8 MBytes/s.

    >
    >I have some machines running nfs
    >NFS Server: Mandriva 2006 (kernel 2.6.12)
    >NFS Client: Mandriva 2007 (kernel 2.6.17)
    >As an example, while copying 100MB files I get between
    >4.5 - 5.3 MBytes/sec in both directions using nfs version 3
    >with default setting on protocol and rsize,wsize.
    >% mount -t nfs -o nfsvers=3 :/home/export /mnt/nfs
    >
    >Have you tried using default mount options ?
    >
    >/Karl D


    Oh yes, I have tried many options. The best result I get with this:
    rw,nosuid,timeo=1,rsize=32768,wsize=2048,udp,intr.

    It seems that one of the most effective option changes was to set the
    timeo value to 1. Before that it was almost like the server received
    all the data but the client was still waiting for something.

    A waste is a terrible thing to mind

  4. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    On Sat, 14 Oct 2006 12:10:09 +0200, VanderBilt
    wrote:

    >Maybe somebody can help me with this problem:
    >I export several directories from a Novell server and mount them on a
    >Linux machine to have some remote backup and storage facilities.
    >A few years back I have installed a Mandrake 10.0 box with kernel
    >2.6.3 and after some tinkering the NFS performance was about 3.5
    >MByte/s on a 100 Mbps full duplex interface card. The FTP performance
    >to and from that same Mandrake box was about 7 MByte/s.
    >
    >This year I upgraded my hardware and I installed Mandriva 2006 using
    >kernel 2.12.12 on the Linux machine. Using exactly the same mount
    >options I get a throughput of about 0.050 MByte/s read to the Linux
    >and about 0.400 MBytes/s on write from the Linux machine. The FTP
    >performance to and from the Mandriva box is around 8 MBytes/s.
    >
    >To test if the problem is the kernel or something else I installed
    >Mandriva 2007 with kernel 2.6.17 on a little test box and this one
    >does exactly the same low NFS performance while giving around 6
    >MBytes/s on FTP up- and download.
    >
    >I have done some tweaking on the NFS settings and reached the maximum
    >of 0.600 MBytes/s for a read and 1 MBytes/s for a write operation. I
    >have tweaked rsize, wsize, timeo, udp/tcp, v3, /proc settings
    >(ip_frag_threshold etc.) but I just can't get back to the decent
    >performance.
    >
    >As a last test I picked up my old hardware again, booted it up,
    >changed the IP address and checked what the throughput was on this
    >one, it was again the expected 3.5 MBytes/s.
    >
    >Does anybody have any clue as to what is going on?
    >
    >I did notice that on the newer boxed there are other files stated at
    >the modprobe and rpcinfo commands: sunrpc.ko.gz and auth_gss which do
    >not appear on the 2.6.3 kernel box. And after deleting a sunrpc.ko.gz
    >I am now unable to mount any nfs export on my test machine so I reckon
    >they do have an impact.
    >A waste is a terrible thing to mind


    I have no idea what is going on, only that I've seen the same type of
    issue and found that upgrading the kernel fixed it.

    I was running a SLES9 kernel that gave me random read performance of
    under 1MB/sec consistently. I checked server, network ports, etc.
    Nothing out of the ordinary.

    I then upgraded to a SLES 9 sp3 kernel and the performance went back
    up to 70+MB/sec (gig attached). 2.6.16 was the "good" kernel.

    ~F

  5. Re: Performance of NFS much lower on Linux kernel 2.6.17 then onLinux kernel 2.6.3?



    VanderBilt wrote:

    >On 15 Oct 2006 00:17:43 -0700, "KarlD" wrote:
    >
    >
    >
    >>VanderBilt wrote:
    >>:
    >>
    >>
    >>>This year I upgraded my hardware and I installed Mandriva 2006 using
    >>>kernel 2.12.12 on the Linux machine. Using exactly the same mount
    >>>options I get a throughput of about 0.050 MByte/s read to the Linux
    >>>and about 0.400 MBytes/s on write from the Linux machine. The FTP
    >>>performance to and from the Mandriva box is around 8 MBytes/s.
    >>>
    >>>

    >>I have some machines running nfs
    >>NFS Server: Mandriva 2006 (kernel 2.6.12)
    >>NFS Client: Mandriva 2007 (kernel 2.6.17)
    >>As an example, while copying 100MB files I get between
    >>4.5 - 5.3 MBytes/sec in both directions using nfs version 3
    >>with default setting on protocol and rsize,wsize.
    >>% mount -t nfs -o nfsvers=3 :/home/export /mnt/nfs
    >>
    >>Have you tried using default mount options ?
    >>
    >>/Karl D
    >>
    >>

    >
    >Oh yes, I have tried many options. The best result I get with this:
    >rw,nosuid,timeo=1,rsize=32768,wsize=2048,udp,intr.
    >
    >


    Try adding async in the mix. I have good results with
    rsize=8192,wsize=8192,noatime,nfsvers=3,async,intr

    The change could be due to the change in the default for sync/async. It
    used to be async and then changed to sync.

    Steve


  6. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    Thanks, I'll test that out.

    Now of course the question is: How do I get a 2.6.16 kernel? I guess
    I'll be compiling it myself then, over the weekend :-)

    On Mon, 16 Oct 2006 20:01:42 GMT, Faeandar
    wrote:

    >On Sat, 14 Oct 2006 12:10:09 +0200, VanderBilt
    > wrote:
    >
    >
    >I have no idea what is going on, only that I've seen the same type of
    >issue and found that upgrading the kernel fixed it.
    >
    >I was running a SLES9 kernel that gave me random read performance of
    >under 1MB/sec consistently. I checked server, network ports, etc.
    >Nothing out of the ordinary.
    >
    >I then upgraded to a SLES 9 sp3 kernel and the performance went back
    >up to 70+MB/sec (gig attached). 2.6.16 was the "good" kernel.
    >
    >~F



  7. Re: Performance of NFS much lower on Linux kernel 2.6.17 then onLinux kernel 2.6.3?



    VanderBilt wrote:

    >Thanks, I'll test that out.
    >
    >Now of course the question is: How do I get a 2.6.16 kernel? I guess
    >I'll be compiling it myself then, over the weekend :-)
    >


    I have a number of machines on 2.6.17 without this issue. Possibly your
    distribution has a kernel update that would make it easy to get a new
    kernel (urpmi ?) but I doubt if this is a kernel issue. A simpler test
    is to try the async parameter to see if that speeds things up.

    How are other network protocols working? Is s/ftp fast but nfs is
    slow? Maybe it is a duplex/speed negotiation issue with the network
    card/switch (it isn't a Cisco switch is it?)



    >On Mon, 16 Oct 2006 20:01:42 GMT, Faeandar
    >wrote:
    >
    >
    >
    >>On Sat, 14 Oct 2006 12:10:09 +0200, VanderBilt
    >> wrote:
    >>
    >>
    >>I have no idea what is going on, only that I've seen the same type of
    >>issue and found that upgrading the kernel fixed it.
    >>
    >>I was running a SLES9 kernel that gave me random read performance of
    >>under 1MB/sec consistently. I checked server, network ports, etc.
    >>Nothing out of the ordinary.
    >>
    >>I then upgraded to a SLES 9 sp3 kernel and the performance went back
    >>up to 70+MB/sec (gig attached). 2.6.16 was the "good" kernel.
    >>
    >>~F
    >>
    >>

    >
    >
    >



  8. Re: Performance of NFS much lower on Linux kernel 2.6.17 then onLinux kernel 2.6.3?

    Steve Cousins wrote:
    >
    >
    > VanderBilt wrote:
    >
    >> On 15 Oct 2006 00:17:43 -0700, "KarlD" wrote:
    >>
    >>
    >>
    >>> VanderBilt wrote:
    >>> :
    >>>
    >>>> This year I upgraded my hardware and I installed Mandriva 2006 using
    >>>> kernel 2.12.12 on the Linux machine. Using exactly the same mount
    >>>> options I get a throughput of about 0.050 MByte/s read to the Linux
    >>>> and about 0.400 MBytes/s on write from the Linux machine. The FTP
    >>>> performance to and from the Mandriva box is around 8 MBytes/s.
    >>>>
    >>> I have some machines running nfs
    >>> NFS Server: Mandriva 2006 (kernel 2.6.12)
    >>> NFS Client: Mandriva 2007 (kernel 2.6.17)
    >>> As an example, while copying 100MB files I get between
    >>> 4.5 - 5.3 MBytes/sec in both directions using nfs version 3
    >>> with default setting on protocol and rsize,wsize.
    >>> % mount -t nfs -o nfsvers=3 :/home/export /mnt/nfs
    >>>
    >>> Have you tried using default mount options ?
    >>>
    >>> /Karl D
    >>>

    >>
    >> Oh yes, I have tried many options. The best result I get with this:
    >> rw,nosuid,timeo=1,rsize=32768,wsize=2048,udp,intr.
    >>

    >
    > Try adding async in the mix. I have good results with
    > rsize=8192,wsize=8192,noatime,nfsvers=3,async,intr
    >
    > The change could be due to the change in the default for sync/async. It
    > used to be async and then changed to sync.
    > Steve
    >


    I have two suggestions and you may have already tried these....

    1) Disable firewall software on both machines. If TCP traffic is workin
    g ok then it's likely to be with the protocol itself. Certainly wont
    hurt to rule it out.

    2) Install ethereal and monitor the network traffic between the two
    boxes. Will hopefully tell you which machine is slowing things down.

    I'm currently getting burst rates of 130MB/s on mandriva 2007 via nfs
    V4, admittedly I have a gigabit switch inbetween, but I certainly didn't
    do any tweaking to NFs on Mandriva 2007.

    trb



  9. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    I have tried different kernels over the weekend, compiling them with
    NFS as a module, with or without Kerberos parts, with or without
    NFSv4, nothing helped, the performance always stuck at around 400 kbps
    read and 900 kbps write speed.

    I am suspecting the rpcio parts by now, or perhaps the portmapper. But
    how do I downgrade those to the ones I had in Mandrake 10.0? They
    don't seem to be a part of any RPM.

    Ah well, maybe re-installing the whole machine with Mandrake 10.0
    would solve my problem


    On Tue, 17 Oct 2006 17:57:30 -0400, Steve Cousins
    wrote:

    >
    >
    >VanderBilt wrote:
    >
    >>Thanks, I'll test that out.
    >>
    >>Now of course the question is: How do I get a 2.6.16 kernel? I guess
    >>I'll be compiling it myself then, over the weekend :-)
    >>

    >
    >I have a number of machines on 2.6.17 without this issue. Possibly your
    >distribution has a kernel update that would make it easy to get a new
    >kernel (urpmi ?) but I doubt if this is a kernel issue. A simpler test
    >is to try the async parameter to see if that speeds things up.
    >
    >How are other network protocols working? Is s/ftp fast but nfs is
    >slow? Maybe it is a duplex/speed negotiation issue with the network
    >card/switch (it isn't a Cisco switch is it?)
    >
    >
    >
    >>On Mon, 16 Oct 2006 20:01:42 GMT, Faeandar
    >>wrote:
    >>
    >>
    >>
    >>>On Sat, 14 Oct 2006 12:10:09 +0200, VanderBilt
    >>> wrote:
    >>>
    >>>
    >>>I have no idea what is going on, only that I've seen the same type of
    >>>issue and found that upgrading the kernel fixed it.
    >>>
    >>>I was running a SLES9 kernel that gave me random read performance of
    >>>under 1MB/sec consistently. I checked server, network ports, etc.
    >>>Nothing out of the ordinary.
    >>>
    >>>I then upgraded to a SLES 9 sp3 kernel and the performance went back
    >>>up to 70+MB/sec (gig attached). 2.6.16 was the "good" kernel.
    >>>
    >>>~F
    >>>
    >>>

    >>
    >>
    >>



  10. Re: Performance of NFS much lower on Linux kernel 2.6.17 then on Linux kernel 2.6.3?

    Sorry Tony, there is no security of any kind running on either
    machine. The Netware server or the Mandriva client are connected to
    each other on the same 100 Mbps LAN switch.

    I know which machine is slowing things down, it's the Mandriva client.
    If I re-attach the old Mandrake 10.0 machine to the Netware exports I
    get a decent 3.5 MB/s performance.

    I have tried using different kernels (2.6.16, 2.6.18.1) each
    hand-compiled with different options for the NFS (without Kerberos,
    not as a module, without NFSv4 support etc.) but it always stays at
    this low 400 kB/s read and 900 kB/s writes. I am starting to suspect
    the portmapper or rpcio, but how can I downgrade those to the point
    where they are the same as the old Mandrake ones? They don't seem to
    be a RPM of their own, but I'll figure that one out.

    On Thu, 19 Oct 2006 19:18:47 +0200, Tony Barnard
    wrote:

    >>>> Have you tried using default mount options ?
    >>>> /Karl D
    >>>
    >>> Oh yes, I have tried many options. The best result I get with this:
    >>> rw,nosuid,timeo=1,rsize=32768,wsize=2048,udp,intr.

    >>
    >> Try adding async in the mix. I have good results with
    >> rsize=8192,wsize=8192,noatime,nfsvers=3,async,intr
    >>
    >> The change could be due to the change in the default for sync/async. It
    >> used to be async and then changed to sync.

    >
    >I have two suggestions and you may have already tried these....
    >
    >1) Disable firewall software on both machines. If TCP traffic is workin
    >g ok then it's likely to be with the protocol itself. Certainly wont
    >hurt to rule it out.
    >
    >2) Install ethereal and monitor the network traffic between the two
    >boxes. Will hopefully tell you which machine is slowing things down.
    >
    >I'm currently getting burst rates of 130MB/s on mandriva 2007 via nfs
    >V4, admittedly I have a gigabit switch inbetween, but I certainly didn't
    >do any tweaking to NFs on Mandriva 2007.
    >
    >trb
    >



+ Reply to Thread