multi-second hang in getdents (Linux 2.4, x86, ext3fs) - Unix

This is a discussion on multi-second hang in getdents (Linux 2.4, x86, ext3fs) - Unix ; I'm seeing a strange performance problem on one of our systems. When running a find on one filesystem, there are long hangs (up to 30 seconds plus) during the find. Strace reveals that some of the getdents calls are taking ...

+ Reply to Thread
Results 1 to 3 of 3

Thread: multi-second hang in getdents (Linux 2.4, x86, ext3fs)

  1. multi-second hang in getdents (Linux 2.4, x86, ext3fs)

    I'm seeing a strange performance problem on one of our systems.
    When running a find on one filesystem, there are long hangs (up to 30
    seconds plus) during the find. Strace reveals that some of the getdents
    calls are taking that much time!

    From 'strace -T':
    getdents64(0x4, 0x805ed28, 0x1000, 0x805ed28) = 48 <35.798045>

    Oddly enough, not all of them are taking so long:
    getdents64(0x4, 0x805ed28, 0x1000, 0x805ed28) = 48 <0.000016>
    getdents64(0x4, 0x8059cd0, 0x1000, 0x8059cd0) = 240 <0.000017>

    The system is otherwise almost completely idle during the find.
    Nothing else is putting much of a demand on the filesystem. As you can
    see, the time the getdents takes does not seem to be a function of how
    much it returns. None of the directories the find traverses have very
    many entries - none have more than 100, even. During the hang, vmstat
    shows a blocked process, but no time spent in iowait. Any attempt to
    access the same filesystem during a hang also hangs, so it's not just
    the one process. The hangs are not repeatable at the same directory -
    if I try just running a find at the point where a find hung before, it's
    more likely than not to proceed just fine.

    The filesystem is ext3, on a mirrored pair of Maxtor 180 GB SATA
    drives attached to a 3Ware 8500-4. The 3Ware software does not show any
    problems with the mirror. These are the only drives in the system, so I
    can't directly compare them against anything else. Nothing out of the
    ordinary appears in dmesg or /var/log/messages. I'm pretty out of
    ideas...the length of time feels like an ATA timeout to me, but I don't
    have any real evidence for that. Any suggestions would be much
    appreciated.

    JDW


  2. Re: multi-second hang in getdents (Linux 2.4, x86, ext3fs)

    > I'm seeing a strange performance problem on one of our systems.
    > When running a find on one filesystem, there are long hangs (up to 30
    > seconds plus) during the find. Strace reveals that some of the getdents
    > calls are taking that much time!


    It is exceedingly probable that the delay is caused by writeback
    to the disk of the updated access time for all of the directories
    that have been visited by the 'find'. This saturates the disk
    queues for a while.

    The workaround is easy: add "nodiratime" to the mount options in
    /etc/fstab. For example:
    /dev/hda3 / ext3 defaults,nodiratime 1 1
    Of course, this means that the recorded time of last access on
    directories won't be "correct", but almost nobody uses it anyway,
    because it is _not_ updated just because a file within it gets
    changed. Thus access time on a directory cannot be used as a hint
    of whether a backup program needs to descend into the directory
    in order to see all files that were modified after a given date.

    --

  3. Re: multi-second hang in getdents (Linux 2.4, x86, ext3fs)

    John Reiser wrote:
    > The workaround is easy: add "nodiratime" to the mount options in
    > /etc/fstab.


    This looks like it does the trick. Thanks very much!

    JDW


+ Reply to Thread