'ssh node "ls -lR"': consumes all memory on the client! - NFS

This is a discussion on 'ssh node "ls -lR"': consumes all memory on the client! - NFS ; This issue is with a volume exported from the server, when accessed indirectly via SSH from the server, i.e. not logging into the node: ssh node "ls -lR /files/export > /dev/null" The command takes up all available memory (near 4Gb) ...

+ Reply to Thread
Results 1 to 6 of 6

Thread: 'ssh node "ls -lR"': consumes all memory on the client!

  1. 'ssh node "ls -lR"': consumes all memory on the client!

    This issue is with a volume exported from the server, when accessed
    indirectly via SSH from the server, i.e. not logging into the node:

    ssh node "ls -lR /files/export > /dev/null"

    The command takes up all available memory (near 4Gb) on the node,
    while generating hundreds of thousand of slabs (nfs_inode_cache and
    dentry_cache) as seen with slabtop. The system is rather plain Ubuntu:
    2.6.15-26-amd64-k8 kernel.

    /etc/exports on the server:
    /files/export
    10.68.0.0/255.255.0.0(sync,no_wdelay,rw,no_root_squash)

    /etc/fstab entry on the client node:
    home:/files/export /files/export nfs
    rw,nosuid,rsize=32768,wsize=32768,tcp 0 0

    I'd very much appreciate any pointers on how to treat this problem.

    Zoli


  2. Re: 'ssh node "ls -lR"': consumes all memory on the client!

    On Sep 13, 7:04 pm, Bill Marcum wrote:
    >[...] Is /dev/null correct?
    > crw-rw-rw- 1 root root 1, 3 2006-05-22 10:25 /dev/null

    Yes, it is.

    > Maybe the sorting causes ls to take up memory. Try ls -lRU

    I just have, with very interesting results: when looking for the line-
    count with 'ls -lRU /files/export/ |wc >export.ls-lRU_wc' while logged
    into the client, it provoked the same memory consumption I had only
    seen via SSH call previously! The total count came out as 5179046
    lines, and slabtop says:

    OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE
    NAME
    2306142 2306141 99% 1.02K 768714 3 3074856K
    nfs_inode_cache
    1520616 1519124 99% 0.22K 89448 17 357792K dentry_cache

    And 'free' gives:

    total used free shared buffers
    cached
    Mem: 4053584 3796608 256976 0 68220
    240680
    -/+ buffers/cache: 3487708 565876
    Swap: 4000144 92 4000052

    After umount/mount the volume on the client, memory comes back to
    normal:
    total used free shared buffers
    cached
    Mem: 4053584 131988 3921596 0 68248
    8908
    -/+ buffers/cache: 54832 3998752
    Swap: 4000144 92 4000052

    Now trying a test with the output discarded, 'ls -lRU /files/export/
    >/dev/null' launched from the client: results in essentially the same

    memory lost!
    Then I repeated with 'ls -lR /files/export/ >/dev/null' (which had
    not give the problem before, when logged into the client), this time
    giving the same problem... So it seems at least we can take the SSH
    call out of the picture, but I'm as baffled as before! (The same is
    happening after a clean reboot, too.)

    > Or maybe it's the recursion. Do you have directories containing
    > thousands of files or thousands of subdirectories?

    Recursion certainly has something to do with it (plain 'ls -l' is not
    a problem, at least at the top-level where I tried) - but why? And
    yes, its a big RAID disk with lots of stuff, including some huge
    directories...

    > Or maybe you have circular hard links. fsck should detect that.

    Although I'm fairly certain there are no circular links (actually I
    think there are only soft links on that volume), I'll give fsck a try
    when have a chance.

    Thanks again - Zoli


  3. Re: 'ssh node "ls -lR"': consumes all memory on the client!

    On Sep 13, 8:37 am, Zoli wrote:

    > This issue is with a volume exported from the server, when accessed
    > indirectly via SSH from the server, i.e. not logging into the node:
    >
    > ssh node "ls -lR /files/export > /dev/null"
    >
    > The command takes up all available memory (near 4Gb) on the node,
    > while generating hundreds of thousand of slabs (nfs_inode_cache and
    > dentry_cache) as seen with slabtop. The system is rather plain Ubuntu:
    > 2.6.15-26-amd64-k8 kernel.


    > I'd very much appreciate any pointers on how to treat this problem.


    What's the problem? Is the memory not freed when it's needed for
    something else?

    The most efficient thing the system can do is keep the data it read
    from the disk in memory just in case it's needed later. There is no
    reason to throw the data away by freeing the memory just in case the
    memory is needed later -- if it's needed later, it can be freed later.

    If you don't want your memory used, put it on the shelf.

    DS


  4. Re: 'ssh node "ls -lR"': consumes all memory on the client!

    On Sep 14, 5:15 pm, David Schwartz wrote:
    > > ssh node "ls -lR /files/export > /dev/null"
    > > The command takes up all available memory (near 4Gb) on the node,


    > What's the problem?
    >Is the memory not freed when it's needed for something else?


    Exactly.

    Alas, in the meantime I found that it's probably not related to NFS
    as such (so I reset followup-to): the same occurs on the server with
    the host volume...

    -- Zoli


  5. Re: 'ssh node "ls -lR"': consumes all memory on the client!

    On 13 Sep, 16:37, Zoli wrote:
    > This issue is with a volume exported from the server, when accessed
    > indirectly via SSH from the server, i.e. not logging into the node:
    >
    > ssh node "ls -lR /files/export > /dev/null"
    >
    > The command takes up all available memory (near 4Gb) on the node,
    > while generating hundreds of thousand of slabs (nfs_inode_cache and
    > dentry_cache) as seen with slabtop. The system is rather plain Ubuntu:
    > 2.6.15-26-amd64-k8 kernel.
    >
    > /etc/exports on the server:
    > /files/export
    > 10.68.0.0/255.255.0.0(sync,no_wdelay,rw,no_root_squash)
    >
    > /etc/fstab entry on the client node:
    > home:/files/export /files/export nfs
    > rw,nosuid,rsize=32768,wsize=32768,tcp 0 0
    >
    > I'd very much appreciate any pointers on how to treat this problem.
    >
    > Zoli


    What is *in* /files/exports?


  6. Re: 'ssh node "ls -lR"': consumes all memory on the client!

    On Sep 15, 2:20 am, Nico wrote:
    > What is *in* /files/exports?


    It's a 450Gb RAID volume: as I've noted in my detailed report
    <1189756269.837684.164380@o80g2000hse.googlegroups. com>, the full file
    listing is some 5M lines.

    Also in the meantime I found that the trouble is probably not related
    to NFS
    as such (so I reset followup-to): the same occurs on the server with
    the host volume, too...

    -- Zoli


+ Reply to Thread