Poor NFS server performance under local load - IRIX 6.5 - SGI

This is a discussion on Poor NFS server performance under local load - IRIX 6.5 - SGI ; We've got a 4 processor Onyx2 running 6.5 (.17m I think) that we use both as a NFS server and to do local computing. I'm seeing abysmal NFS throughput (94KB/s) when the server is running 4 CPU bound processes, even ...

+ Reply to Thread
Results 1 to 9 of 9

Thread: Poor NFS server performance under local load - IRIX 6.5

  1. Poor NFS server performance under local load - IRIX 6.5

    We've got a 4 processor Onyx2 running 6.5 (.17m I think) that we use
    both as a NFS server and to do local computing. I'm seeing abysmal NFS
    throughput (94KB/s) when the server is running 4 CPU bound processes,
    even though they are running maximally niced. If I stop the local
    processes, the nfsd processes start to take up some cpu time and NFS
    performance improves.

    Is there anyway to boost the NFS performance when under local load?

    TIA,

    Orion


  2. Re: Poor NFS server performance under local load - IRIX 6.5

    Jerry Jorgenson wrote:

    >Hi Orion,
    >
    >Try these:
    >
    >Use cpusets to limit the other processes to certain CPUs.
    >
    >
    >

    Not sure I want to dedicate a cpu for NFS - it doesn't use that much cpu
    time (10-20% when very busy).

    >Use snoop -P to be sure that the packet size matches the other box(es).
    >
    >
    >

    Not quite sure about this, but here is a sample:

    kolea -> lewis NFS C WRITE3 FH=5077 at 2674688 for 4096 (ASYNC)
    lewis -> kolea NFS R WRITE3 OK 4096 (ASYNC)
    lewis -> kolea NFS R WRITE3 OK 4096 (ASYNC)
    kolea -> lewis NFS C COMMIT3 FH=5077 at 1753088 for 0
    lewis -> kolea NFS R COMMIT3 OK
    kolea -> lewis UDP continuation ID=48286
    kolea -> lewis UDP continuation ID=48286
    kolea -> lewis NFS C WRITE3 FH=5077 at 2801664 for 4096
    (ASYNC)
    kolea -> lewis UDP continuation ID=48287
    kolea -> lewis UDP continuation ID=48287
    kolea -> lewis NFS C WRITE3 FH=5077 at 2805760 for 4096
    (ASYNC)
    lewis -> kolea NFS R WRITE3 OK 4096 (ASYNC)
    lewis -> kolea NFS R WRITE3 OK 4096 (ASYNC)
    kolea -> lewis NFS C COMMIT3 FH=5077 at 2801664 for 0
    lewis -> kolea NFS R COMMIT3 OK

    >Ensure that you're using NFS3
    >
    >
    >

    Yup.

    >Check the systune parameters and nfs parameters to insure there is enough
    >memory.
    >
    >
    >

    Not sure about this. But remember, I get good performance when the
    machine is idle. It just doesn't seem to give cpu time to the nfsd
    processes when it is busy, even though the nfsd processes have high
    priority and the local processes low priority.

    >Use par to see what the nfs processes are doing.
    >
    >


    Seem to be doing the normal io stuff, just slower:

    4322mS[ 2] nfsd( 242): I/O queued; flags 0x1000000008 dev
    0,477 count 8192 blkno 237552
    4322mS[ 2] nfsd( 242): I/O started; flags 0x1000000008
    dev 0,477 count 8192 blkno 237552

    7621mS[ 3] nfsd( 244): I/O queued; flags 0x1000004108 dev
    0,477 count 114688 blkno 237344
    7621mS[ 3] nfsd( 244): I/O started; flags 0x1000004108
    dev 0,477 count 114688 blkno 237344

    >Use sar to make sure that you aren't swapping (sar -w).
    >
    >
    >

    Plenty of memory.

    Watch a dump of the NFS packets on the client side, I just start to see
    the delay between the client call and the server reply increase from
    about .0001-.0003 sec to .016 sec with spikes to a couple tenths. Still
    seems like a scheduling issue on the server.

    Thanks for the response.

    --
    Orion Poplawski
    System Administrator 303-415-9701 x222
    Colorado Research Associates/NWRA FAX: 303-415-9702
    3380 Mitchell Lane, Boulder CO 80301 http://www.co-ra.com


  3. Re: Poor NFS server performance under local load - IRIX 6.5

    Orion Poplawski wrote:
    > We've got a 4 processor Onyx2 running 6.5 (.17m I think) that we use
    > both as a NFS server and to do local computing. I'm seeing abysmal NFS
    > throughput (94KB/s) when the server is running 4 CPU bound processes,
    > even though they are running maximally niced. If I stop the local
    > processes, the nfsd processes start to take up some cpu time and NFS
    > performance improves.
    >


    NFS is on the top of my list for waste of an Onyx2...

    build yourself a cheapass peecee(tm) NFS server for pennies on the dollar
    and use that lovely Onyx2 for the real work.



  4. Re: Poor NFS server performance under local load - IRIX 6.5

    Jerry Jorgenson wrote:

    >On Wed, 31 Dec 2003 10:38:50 -0700, Orion Poplawski wrote:
    >
    >
    >
    >>Jerry Jorgenson wrote:
    >>
    >>
    >>
    >>>Hi Orion,
    >>>
    >>>Try these:
    >>>
    >>>Use cpusets to limit the other processes to certain CPUs.
    >>>
    >>>
    >>>

    >>Not sure I want to dedicate a cpu for NFS - it doesn't use that much cpu
    >>time (10-20% when very busy).
    >>
    >>

    >
    >I was thinking more of dedicating CPUS to other tasks and letting the NFS
    >float.
    >
    >
    >

    Isn't the effect the same if I prevent other jobs from using a
    particular CPU?

    >4096 looks like a really small size. 16K or 32K is more like it. This is
    >done in /etc/fstab on the client side with and entry like this:
    >
    >
    >kolea:/vol /mnt nfs nosuid,rw,rsize=16384,wsize=16384,vers=3,bg 0 0
    >
    >
    >
    >

    Thanks, just started poking around with bumping up the block sizes.

    >>>Check the systune parameters and nfs parameters to insure there is enough
    >>>memory.
    >>>
    >>>
    >>>

    >>Not sure about this. But remember, I get good performance when the
    >>machine is idle. It just doesn't seem to give cpu time to the nfsd
    >>processes when it is busy, even though the nfsd processes have high
    >>priority and the local processes low priority.
    >>
    >>

    >
    >Right, which means that it's possible that swapping or memory reallocation
    >is going on. It's something to check anyway.
    >
    >
    >
    >

    No swapping, LOTS of memory. Not sure about NFS/socket queues etc, but
    don't know how to check
    on IRIX.

    >Which is why if you restrict the heavy hitter processes to a cpuset, then
    >the other processes might work better.
    >
    >
    >

    But that is what I find puzzling - these aren't "heavy hitter
    processes". They are CPU bound, but they have been niced to the lowest
    possible priority. I also reniced the nfsd process to a high priority.

    I can start a NFS transfer with the server idle and then see the nfsd
    processes get starved out for CPU time after starting the other
    processes, which surprises me because of the great differences in
    scheduling priority.

    Is this a time-slice issue?

    Thanks again.

    PS - Jerry - your email server seems to think I'm a spammer or doesn't
    like your address. Hard to tell which.

    --
    Orion Poplawski
    System Administrator 303-415-9701 x222
    Colorado Research Associates/NWRA FAX: 303-415-9702
    3380 Mitchell Lane, Boulder CO 80301 http://www.co-ra.com


  5. Re: Poor NFS server performance under local load - IRIX 6.5



    Orion Poplawski wrote:
    > They are CPU bound, but they have been niced to the lowest
    > possible priority.


    What nice value? Can you show us what you see when you run "top"?

    Steve
    __________________________________________________ ___________
    Steve Cousins Email: cousins@umit.maine.edu
    Research Associate Phone: (207) 581-4302
    Ocean Modeling Group
    School of Marine Sciences 208 Libby Hall,
    University of Maine Orono ME 04469

  6. Re: Poor NFS server performance under local load - IRIX 6.5

    Benjamin Goldsteen wrote:
    >
    > Last, I find that cheapass peecee(tm) running Linux do not make for
    > especially good NFS servers. Ignoring the hetrogenous networking
    > issues and greater administration issues if there aren't already Linux
    > machines at the site, I have not found Linux's NFS implementation very
    > solid. There are patches out there that improve the state of things
    > but I am still not very impressed. I recall someone not too long ago
    > bragging about 3MB/sec transfer rates...which doesn't compare well
    > with 9MB/sec easily obtainable from any OCTANE-class or better SGI.
    > The Linux world evolves constantly, but I wouldn't add another machine
    > to the network just to solve this problem.


    who said they should use Linux? certainly not the post you were
    responding to. Linux is not fit for anything let alone an NFS server.

    from the original servers post it did not appear they were doing anything
    beyond sharing files via NFS. poorly utilizing the onyx2 and something
    a BSD or Solaris peecee could easily accomplish.

    high performance NFS as you stated is another story.

    cheers.

  7. Re: Poor NFS server performance under local load - IRIX 6.5

    Hi,

    I would definitely encourage you to upgrade to a new IRIX release like
    6.5.22/6.5.23.
    There has been boatloads of NFS performance improvements. The best part of
    the latest NFS stack on IRIX is the fact that direct i/o is supported for
    both NFS clients and servers.
    So upgrading would be an option to your problem.
    Also from first hand experience,i can assert that NFS implementation on
    Linux is simply not in the same league in the 2.4 kernels.
    Linux 2.6 has done some cleanups wrt performance,but still not enterprise
    class performance especially for large NFS installations.

    Hope that helps.

    Cheers,
    Arun



  8. Re: Poor NFS server performance under local load - IRIX 6.5

    >>>>> "arunr" == Arun Ramakrishnan writes:

    arunr> The best part of the latest NFS stack on IRIX is the fact that
    arunr> direct i/o is supported for both NFS clients and servers.
    Alas, not yet.

    And even if it was, Direct IO is not a magic pudding to help starving
    nfsds.

    max




  9. Re: Poor NFS server performance under local load - IRIX 6.5

    IRIX Central wrote in message news:...
    > Benjamin Goldsteen wrote:
    > >
    > > Last, I find that cheapass peecee(tm) running Linux do not make for
    > > especially good NFS servers.

    >
    > who said they should use Linux? certainly not the post you were
    > responding to. Linux is not fit for anything let alone an NFS server.
    >



    Yes. The IBM NFS implementation on OS/2 is far better :-)

+ Reply to Thread