nfs server Calls almost equals badcalls - Aix

This is a discussion on nfs server Calls almost equals badcalls - Aix ; Hi, AIX 5.2.5.4 NFS v3 Machine has been a NIM server for a while, plenty of NIM tasks have been executed. NFS is integral to NIM, so we assume NFS has been fine for a while. Then we have the ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: nfs server Calls almost equals badcalls

  1. nfs server Calls almost equals badcalls

    Hi,

    AIX 5.2.5.4

    NFS v3

    Machine has been a NIM server for a while, plenty of NIM tasks have
    been executed. NFS is integral to NIM, so we assume NFS has been fine
    for a while.

    Then we have the period of time where 'nothing changes'

    Then NIM stops working, NFS is running extremely slowly (3 days to
    copy over a few GB).

    stopsrc -g nfs

    intention to restart, but nfsd hangs trying to come down.

    reboot machine.

    Machine back up, NFS operations (mount from remote client) are taking
    an age and timing out. Can;t get it to work for love nor money.
    rpcinfo, ps -ef etc.. all show correct damons running.

    nfso shows one oddity, nfs_max_threads is 1.

    minimum value is 5!, look
    > nfso -onfs_max_threads=1

    nfso: 1485-111 Invalid value 1 for tunable nfs_max_threads
    Minimal value for tunable nfs_max_threads is 5

    setting nfs_max_threads to 3891, any hanging client mounts suddenly
    finish ok... and then any further nfs operation experience same
    problems. Massive timeouts.

    so, we tried making a permanent change to nfs_max_threads
    nfso -ponfs_max_threads=3891

    .... but upon reboot., nfs_max_threads is 1 again!
    /etc/tunables/lastboot.log has no record of changing nfs_max_threads

    Also,

    nfsstat -s

    shows calls almost equal to badcalls, IE

    Server nfs:
    calls badcalls public_v2 public_v3
    122586 122580 0 0

    So the RPC layer is rejecting calls like nobodys business, but _WHY?!

    So, summary

    why does nfs_max_threads always become 1 after reboot?
    How comes it's 1 when the minimum is 5?
    Is that causing calls to almost equal badcalls?
    ....is any of that the casue of ultra-slow/failing nfs operations for
    clients?
    .....How could this situation have occurred to begin with?


    Any help muchly appreciated,

    Matthew Warren.


  2. Re: nfs server Calls almost equals badcalls

    Sorry , that should be AIX level 5300-05


  3. Re: nfs server Calls almost equals badcalls

    Also, I've looked everywhere but havent yet discovered exactly what
    can lead to a bad call. Most places either say because of an
    'authentication error' or just that they are calls rejected by the RPC
    layer..


  4. Re: nfs server Calls almost equals badcalls

    AlienBaby wrote:

    > why does nfs_max_threads always become 1 after reboot?


    check the sys0-Attribute pre520tune: lsattr -El sys0
    if it is "enable", settings from /etc/tunables/nextboot are not applied
    at boot time.
    On a freshly installed 5.2, pre520tune is disabled, but a Migration
    installation enables it by default.

    Joachim

  5. Re: nfs server Calls almost equals badcalls

    On Jun 3, 11:19 am, Joachim Gann wrote:
    > AlienBaby wrote:
    > > why does nfs_max_threads always become 1 after reboot?

    >
    > check the sys0-Attribute pre520tune: lsattr -El sys0
    > if it is "enable", settings from /etc/tunables/nextboot are not applied
    > at boot time.
    > On a freshly installed 5.2, pre520tune is disabled, but a Migration
    > installation enables it by default.
    >
    > Joachim


    Hi, thanks for the reply.

    Unfortunatley it is currently 'disable' so I'm still not certain what
    is happening with nfs_max_threads


  6. Re: nfs server Calls almost equals badcalls

    I'm still chipping away at this, trying to resolve it.

    One more thing I have noticed

    ps -ef | grep nfsd

    root 1528002 1396822 0 13:11:28 - 0:00 /usr/sbin/nfsd 1


    So, nfsd was started with a max threads value of 1


    nfso -onfs_max_threads=3891
    nfso -a | grep nfs_max_threads
    nfs_max_threads = 3891

    So, right now, nfs_max_threads is 3891

    stopsrc -s nfsd
    startsrc -s nfsd

    ps -ef | grep nfsd
    root 2342437 1396822 0 13:22:14 - 0:00 /usr/sbin/nfsd 1


    It has still only started with one thread.
    Testing this on a separate system showed startsrc had read 3891 from
    nfs_max_threads and issued a /usr/sbin/nfsd 3891 to start the daemon



    Anyone know why I cant get nfs_max_threads to change permanently from
    1, or why once I set it manually after reboot then startsrc still uses
    the value of 1?



    On Jun 5, 11:25 am, AlienBaby wrote:
    > On Jun 3, 11:19 am, Joachim Gann wrote:
    >
    > > AlienBaby wrote:
    > > > why does nfs_max_threads always become 1 after reboot?

    >
    > > check the sys0-Attribute pre520tune: lsattr -El sys0
    > > if it is "enable", settings from /etc/tunables/nextboot are not applied
    > > at boot time.
    > > On a freshly installed 5.2, pre520tune is disabled, but a Migration
    > > installation enables it by default.

    >
    > > Joachim

    >
    > Hi, thanks for the reply.
    >
    > Unfortunatley it is currently 'disable' so I'm still not certain what
    > is happening with nfs_max_threads




  7. Re: nfs server Calls almost equals badcalls

    On Jun 5, 2:25 pm, AlienBaby wrote:
    > I'm still chipping away at this, trying to resolve it.

    .....


    It depends how you start the nfsd
    Using /etc/rc.nfs use chnfs
    Using startsrc use chssys

    Example:
    $ chnfs -n 3891
    # chssys -s nfsd -a 3891

    To verify the settings:
    $ lssrc -S -s nfsd
    $ cat /etc/rc.nfs | grep nfsd

    hth
    Hajo


+ Reply to Thread