du indicates directory usage when there are no files - Unix

This is a discussion on du indicates directory usage when there are no files - Unix ; hi, when i do a du command on a directory i get a response indicating it has 14MB worth of data. if i cd to this directory and to a cat/strings on "." it returns a huge list of files. ...

+ Reply to Thread
Results 1 to 10 of 10

Thread: du indicates directory usage when there are no files

  1. du indicates directory usage when there are no files

    hi,
    when i do a du command on a directory i get a response indicating it
    has 14MB worth of data. if i cd to this directory and to a cat/strings
    on "." it returns a huge list of files. from my unix knowledge i
    understand that the "." file contains inode information which ties in
    with what i saw. because there have been millions of files processed by
    this directory there are millions of entries in this file. what i would
    like assistance from people out there is:
    - am i correct in my assumptions ?
    - does the system (as i expect) perfom less well when the "." file is
    large?
    - what processes (if any) regularly cleans up the contents of this
    file?
    thanks for any assistance. i appreciate that i may not be explaining
    myself that well. will gladly elaborate if you need more information to
    respond.
    thanks,
    g


  2. Re: du indicates directory usage when there are no files

    hi,
    thanks for the response. the directory contained millions of files in
    the past but we have made sure that this will not occur in the future.
    i want to understand if there is a mechanism for removing old entries
    in the "." file to improve directory listing using ls. is there any
    native feature of unix which does this apart from the system backup and
    restore suggested? may be hard to achieve this on production box.
    g


  3. Re: du indicates directory usage when there are no files

    strepxe@yahoo.co.uk wrote:
    > hi,
    > thanks for the response. the directory contained millions of files in
    > the past but we have made sure that this will not occur in the future.
    > i want to understand if there is a mechanism for removing old entries
    > in the "." file to improve directory listing using ls. is there any
    > native feature of unix which does this apart from the system backup and
    > restore suggested? may be hard to achieve this on production box.
    > g
    >


    You don't have to go as far as a backup/restore. You just need to
    replace the directory with something similar to...
    mv mydir mydir.bak
    mkdir mydir
    mv mydir.bak/* mydir
    rmdir mydir.bak
    The third step will require use of find/xargs if you have a lot of
    files in the directory.

    -- ced

    --
    Chuck Dillon
    Senior Software Engineer
    NimbleGen Systems Inc.

  4. Re: du indicates directory usage when there are no files

    strepxe@yahoo.co.uk wrote:
    > when i do a du command on a directory i get a response indicating it
    > has 14MB worth of data. if i cd to this directory and to a cat/strings
    > on "." it returns a huge list of files.


    Just out of curiosity, what flavor of Unix and what kind of
    filesystem are you using? Some Unixes, or some tools, don't like to let
    you cat or strings a directory.


    > from my unix knowledge i understand that the "." file


    You do know that '.' is not a "file" in the usual sense of the term,
    yes? It is a reference to "this directory".


    > - am i correct in my assumptions ?


    More or less, except that most people wouldn't call '.' a file.


    > - does the system (as i expect) perfom less well when the "." file is
    > large?


    I suppose that a getdents() or a readdir() would be slower on such a
    directory, although I have no idea how much slower, or whether the
    slowdown would be linear or exponential or what. I suspect it depends
    very strongly on the filesystem type.


    > - what processes (if any) regularly cleans up the contents of this
    > file?


    Nothing. As others have pointed out, if you back up and restore the
    directory, you'll reduce its size. Now a question for you: is there any
    particular reason you feel it's important to reduce the size of the
    directory? Are you seeing performance problems, and have you tracked
    down the source of the problem to the size of the directory?

    --
    Oh to have a lodge in some vast wilderness. Where rumors of oppression
    and deceit, of unsuccessful and successful wars may never reach me
    anymore.
    -- William Cowper

  5. Re: du indicates directory usage when there are no files

    hi,

    thanks for getting back to me. the unix o/s is hp-ux 11iv1 and the
    filesystem is vxfs. was unsure about how to refer to "." and appreciate
    you putting me right on the terminology.

    am i correct in understanding that this reference to the directory will
    never be cleared up and that it will always contain a complete list of
    all files created in that directory ever?

    yes i am seeing a performance problem. the application does an lstat()
    on the directory to get a list of files to process:
    - when i do so on the filesystem "as is" (i.e. with huge list of files
    in dot file) it takes a number of seconds to list the contents of
    directory (good indication for application processing time)
    - when i do so on the filesystem after backup and recovery the time to
    list the contents of a directory is almost instantaneous


  6. Re: du indicates directory usage when there are no files

    the system contained millions of files in the past but will not do so
    in the future. the application in question performs "less well" with
    millions of small files and not so with a few large files. so i am keen
    to understand if this reference to the directory is ever emptied or how
    maintenance is done? i notice a perciptible difference in the issuing
    of ls commands on the current and restored directories which will
    impact the performance of the application (uses lstat() to get list of
    files to process).


  7. Re: du indicates directory usage when there are no files

    hi chuck,
    thanks for responding. am i correct in understanding that this
    reference to the directory containing files created therein will never
    be cleared out? or does it depend on the o/s? i am seeing different
    behaviour on solaris and hp-ux. still feeling my way around on this in
    an effort to understand the process for file management in unix.
    g


  8. Re: du indicates directory usage when there are no files

    strepxe@yahoo.co.uk wrote:
    > hi chuck,
    > thanks for responding. am i correct in understanding that this
    > reference to the directory containing files created therein will never
    > be cleared out? or does it depend on the o/s? i am seeing different
    > behaviour on solaris and hp-ux. still feeling my way around on this in
    > an effort to understand the process for file management in unix.
    > g
    >


    Generally speaking, as I understand it, it's a function of the
    filesystem not the OS per se. This link:
    http://www.unix.org.ua/orelly/unix/upt/ch24_16.htm
    discusses the issues. It's part of chapter 24 of "UNIX Power Tools".
    Google led me to it.

    If you need to follow-up I suggest you ask in comp.unix.programmer
    where there are folks who have knowledge of implementation internals.
    You might also ask in groups specific to the OSes you are using.

    -- ced

    --
    Chuck Dillon
    Senior Software Engineer
    NimbleGen Systems Inc.

  9. Re: du indicates directory usage when there are no files

    strepxe@yahoo.co.uk wrote:
    > am i correct in understanding that this reference to the directory will
    > never be cleared up and that it will always contain a complete list of
    > all files created in that directory ever?


    I think that depends on the details of the filesystem
    implementation. I've used VxFS but I'm no expert in it, sorry. An
    expert might be able to tell you.


    > yes i am seeing a performance problem. the application does an lstat()
    > on the directory to get a list of files to process:
    > - when i do so on the filesystem "as is" (i.e. with huge list of files
    > in dot file) it takes a number of seconds to list the contents of
    > directory (good indication for application processing time)


    OK, it does sound like you have a real issue. I'm a little
    surprised, as I thought VxFS was supposed to be better than that, but
    that's neither here nor there. Do you know if it is in fact the lstat()
    call that's taking so long? You should be able to tell with trace or
    perhaps tusc (sorry, my experience is mostly in the Solaris and Linux
    worlds). There may be some tuning you can do to the filesystem or some
    parameter you can set. I would seek out a VxFS expert or put in a
    support call to Veritas.

    JDW

    --
    Oh to have a lodge in some vast wilderness. Where rumors of oppression
    and deceit, of unsuccessful and successful wars may never reach me
    anymore.
    -- William Cowper

  10. Re: du indicates directory usage when there are no files

    hi,
    i am a bit surprised about this behaviour of vxfs but i am no real
    expert on any filesystems other than being able to tell basic
    differences and uses of each. i have used tusc on an ls of the
    directory and found lstat was the offending call. this function is also
    used by the application to get contents of directory hence my problem.
    i'm going to see if i can raise a call about this and i'll post my
    findings here. thanks for the assistance. much appreciated!
    g


+ Reply to Thread