filesystem 100% full, but files do not occupy the space - Setup

This is a discussion on filesystem 100% full, but files do not occupy the space - Setup ; Hi, I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as a LVM. The problem now is that (after some monts of operation) the root filesystem is 100% full (see "df -h" output below), but ...

+ Reply to Thread
Results 1 to 13 of 13

Thread: filesystem 100% full, but files do not occupy the space

  1. filesystem 100% full, but files do not occupy the space

    Hi,

    I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
    a LVM. The problem now is that (after some monts of operation) the
    root filesystem is 100% full (see "df -h" output below), but I can not
    locate how the space is occupied (see du output further below). All
    files take up maybe 4GB - where is the rest? I rebooted with forced
    fsck, but this didn't change anything, same with vgck. Any hints?

    thanks, Joachim

    [root@maxperf tmp]# uname -r
    2.6.9-42.0.2.EL
    [root@maxperf ~]# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/VolGroup00-LogVol00
    27G 27G 0 100% /
    /dev/hda1 99M 13M 82M 13% /boot
    none 125M 0 125M 0% /dev/shm
    /dev/sda5 147G 136G 3.5G 98% /media/usbdisk2
    /dev/sdb1 230G 213G 5.6G 98% /media/usbdisk
    [root@maxperf ~]# du -sh /*
    0 /backup
    5.8M /bin
    6.7M /boot
    168K /dev
    60M /etc
    0 /export
    36M /home
    8.0K /initrd
    85M /lib
    16K /lost+found
    8.0K /misc
    8.0K /mnt
    8.0K /opt
    258M /proc
    3.4M /root
    16M /sbin
    8.0K /selinux
    8.0K /srv
    0 /sys
    76K /tmp
    2.5G /usr
    226M /var
    [root@maxperf tmp]# vgdisplay
    --- Volume group ---
    VG Name VolGroup00
    System ID
    Format lvm2
    Metadata Areas 1
    Metadata Sequence No 5
    VG Access read/write
    VG Status resizable
    MAX LV 0
    Cur LV 2
    Open LV 2
    Max PV 0
    Cur PV 1
    Act PV 1
    VG Size 27.81 GB
    PE Size 32.00 MB
    Total PE 890
    Alloc PE / Size 890 / 27.81 GB
    Free PE / Size 0 / 0
    VG UUID t9Vx9Q-uU2l-lG3w-Vk9z-bmck-L5h6-7Lwx1h


  2. Re: filesystem 100% full, but files do not occupy the space

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Joachim Worringen wrote:
    > Hi,
    >
    > I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
    > a LVM. The problem now is that (after some monts of operation) the
    > root filesystem is 100% full (see "df -h" output below), but I can not
    > locate how the space is occupied (see du output further below). All
    > files take up maybe 4GB - where is the rest? I rebooted with forced
    > fsck, but this didn't change anything, same with vgck. Any hints?
    >
    > thanks, Joachim
    >
    > [root@maxperf tmp]# uname -r
    > 2.6.9-42.0.2.EL
    > [root@maxperf ~]# df -h
    > Filesystem Size Used Avail Use% Mounted on
    > /dev/mapper/VolGroup00-LogVol00
    > 27G 27G 0 100% /

    [snip]
    > [root@maxperf ~]# du -sh /*

    [snip]

    The du(1) utility looks through directories, summing the space used by the
    files named therein.

    The df(1) utility looks at the filesystem allocation and freespace tables,
    displaying the values recorded there.

    When du(1) tells you that there is less used space (and thus more free space)
    than du(f) tells you, then you can be assured that the difference is due to a
    file that has been deleted from the directory tree, but has /not/ been
    returned to the filesystem's freespace.

    This situation happens when you delete all hardlinks to a file that is still
    opened by some process. Because all the directory entries that link to the
    file's inode have been deleted, the space is not counted in the du(1) count.
    However, because the file is still open, the system will not return the space
    allocated to that file to the filesystem freespace list.

    To fix this, make the process that has the file open close the file. To find
    the file, use lsof(8).

    My guess is that you deleted/reallocated one of the log files (say
    /var/log/messages) and didn't restart syslog. syslog keeps the old files open
    (rather than go through the work of opening the file, writing a line, and
    closing it each time a log record must be written), and so the space allocated
    to the old /var/log/messages file didn't get returned to the freespace list.

    HTH
    - --
    Lew Pitcher

    Master Codewright & JOAT-in-training | Registered Linux User #112576
    http://pitcher.digitalfreehold.ca/ | GPG public key available by request
    - ---------- Slackware - Because I know what I'm doing. ------


    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.7 (GNU/Linux)
    Comment: Armoured with GnuPG

    iD8DBQFGaIlDagVFX4UWr64RAsX6AJ92IHd3Eog7Z0zo49odfS pebI4NdwCglFW+
    VPLji4tZb0THDGFUpXJnxpc=
    =xOKE
    -----END PGP SIGNATURE-----

  3. Re: filesystem 100% full, but files do not occupy the space

    "Joachim Worringen" wrote
    > I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
    > a LVM. The problem now is that (after some months of operation) the
    > root filesystem is 100% full (see "df -h" output below), but I can not
    > locate how the space is occupied (see du output further below). All
    > files take up maybe 4GB - where is the rest?
    >
    > [root@maxperf ~]# df -h
    > Filesystem Size Used Avail Use% Mounted on
    > /dev/mapper/VolGroup00-LogVol00
    > 27G 27G 0 100% /
    > /dev/hda1 99M 13M 82M 13% /boot




    > [root@maxperf ~]# du -sh /*
    > 0 /backup
    > 5.8M /bin
    > 6.7M /boot
    > 168K /dev
    > 60M /etc




    You've done well so far -- no need to do an elaborate size-based
    find
    command when the situation is so pronounced that a mere
    du
    will highlight the problem area,
    but you forget that your du command wildcard will not show
    files/directories that start with a '.' character.

    So try
    cd / ; du -sh .* * | more
    and look for those missing umpteen gigs there.

    Of course, ignore the result for file '.',
    which is 'the whole current directory.'
    Don't try this command in a subdir, as for example
    cd /tmp ; du -sh .*
    would include .. which is the parent directory -- how tedious.

    Good luck hunting! And do this as root, as tightly-secured subdirs
    cannot be surveyed by unpriviledged users.




  4. Re: filesystem 100% full, but files do not occupy the space

    Lew Pitcher wrote:
    > -----BEGIN PGP SIGNED MESSAGE-----
    > Hash: SHA1
    >
    > Joachim Worringen wrote:
    >> Hi,
    >>
    >> I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
    >> a LVM. The problem now is that (after some monts of operation) the
    >> root filesystem is 100% full (see "df -h" output below), but I can not
    >> locate how the space is occupied (see du output further below). All
    >> files take up maybe 4GB - where is the rest? I rebooted with forced
    >> fsck, but this didn't change anything, same with vgck. Any hints?
    >>
    >> thanks, Joachim
    >>
    >> [root@maxperf tmp]# uname -r
    >> 2.6.9-42.0.2.EL
    >> [root@maxperf ~]# df -h
    >> Filesystem Size Used Avail Use% Mounted on
    >> /dev/mapper/VolGroup00-LogVol00
    >> 27G 27G 0 100% /

    > [snip]
    >> [root@maxperf ~]# du -sh /*

    > [snip]
    >
    > The du(1) utility looks through directories, summing the space used by the
    > files named therein.
    >
    > The df(1) utility looks at the filesystem allocation and freespace tables,
    > displaying the values recorded there.
    >
    > When du(1) tells you that there is less used space (and thus more free space)
    > than du(f) tells you, then you can be assured that the difference is due to a
    > file that has been deleted from the directory tree, but has /not/ been
    > returned to the filesystem's freespace.
    >
    > This situation happens when you delete all hardlinks to a file that is still
    > opened by some process. Because all the directory entries that link to the
    > file's inode have been deleted, the space is not counted in the du(1) count.
    > However, because the file is still open, the system will not return the space
    > allocated to that file to the filesystem freespace list.
    >
    > To fix this, make the process that has the file open close the file. To find
    > the file, use lsof(8).
    >
    > My guess is that you deleted/reallocated one of the log files (say
    > /var/log/messages) and didn't restart syslog. syslog keeps the old files open
    > (rather than go through the work of opening the file, writing a line, and
    > closing it each time a log record must be written), and so the space allocated
    > to the old /var/log/messages file didn't get returned to the freespace list.


    Would not going to single user and fsck'ing everything sort this? or
    simply a reboot?

    Your issues with log files ring a bell..

    ISTR we used to rename the current file to log.1 or something, 'touch' a
    new log file to create it., and then HUP the syslogd..so it reopened all
    its files anew..all this under cron..

  5. Re: filesystem 100% full, but files do not occupy the space

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    The Natural Philosopher wrote:
    > Lew Pitcher wrote:
    >> -----BEGIN PGP SIGNED MESSAGE-----
    >> Hash: SHA1
    >>
    >> Joachim Worringen wrote:
    >>> Hi,
    >>>
    >>> I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
    >>> a LVM. The problem now is that (after some monts of operation) the
    >>> root filesystem is 100% full (see "df -h" output below), but I can not
    >>> locate how the space is occupied (see du output further below). All
    >>> files take up maybe 4GB - where is the rest? I rebooted with forced
    >>> fsck, but this didn't change anything, same with vgck. Any hints?
    >>>
    >>> thanks, Joachim
    >>>
    >>> [root@maxperf tmp]# uname -r
    >>> 2.6.9-42.0.2.EL
    >>> [root@maxperf ~]# df -h
    >>> Filesystem Size Used Avail Use% Mounted on
    >>> /dev/mapper/VolGroup00-LogVol00
    >>> 27G 27G 0 100% /

    >> [snip]
    >>> [root@maxperf ~]# du -sh /*

    >> [snip]
    >>
    >> The du(1) utility looks through directories, summing the space used by
    >> the
    >> files named therein.
    >>
    >> The df(1) utility looks at the filesystem allocation and freespace
    >> tables,
    >> displaying the values recorded there.
    >>
    >> When du(1) tells you that there is less used space (and thus more free
    >> space)
    >> than du(f) tells you, then you can be assured that the difference is
    >> due to a
    >> file that has been deleted from the directory tree, but has /not/ been
    >> returned to the filesystem's freespace.

    >[snip]
    >
    > Would not going to single user and fsck'ing everything sort this? or
    > simply a reboot?


    Sure. The simplest solution is just to reboot.
    Going to single user might not free up everything (some processes may continue
    to run even in single user mode), and there's no real need to fsck unless the
    filesystem is really corrupt.

    > Your issues with log files ring a bell..
    >
    > ISTR we used to rename the current file to log.1 or something, 'touch' a
    > new log file to create it., and then HUP the syslogd..so it reopened all
    > its files anew..all this under cron..


    Yah. It's one of those things that catches most just-past-newbies (you know,
    no longer a newbie, but not a guru :-) )


    - --
    Lew Pitcher

    Master Codewright & JOAT-in-training | Registered Linux User #112576
    http://pitcher.digitalfreehold.ca/ | GPG public key available by request
    - ---------- Slackware - Because I know what I'm doing. ------


    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.7 (GNU/Linux)
    Comment: Armoured with GnuPG

    iD8DBQFGaJkVagVFX4UWr64RAvQjAJ9CJKX2y24bl7PLr5zR/ryoo1OVOwCg8vsU
    Jaf4tTWLHMuh8+8L4FcDCaQ=
    =DtfL
    -----END PGP SIGNATURE-----

  6. Re: filesystem 100% full, but files do not occupy the space

    On Jun 8, 2:04 am, "Frank Winans" wrote:
    > "Joachim Worringen" wrote
    >
    > > I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
    > > a LVM. The problem now is that (after some months of operation) the
    > > root filesystem is 100% full (see "df -h" output below), but I can not
    > > locate how the space is occupied (see du output further below). All
    > > files take up maybe 4GB - where is the rest?

    [...]
    > but you forget that your du command wildcard will not show
    > files/directories that start with a '.' character.
    >
    > So try
    > cd / ; du -sh .* * | more
    > and look for those missing umpteen gigs there.


    Thanks all for responding. My feedback:
    - There are no .-dirs in / (checked with ls -la), and .-dirs from
    subdirectories will be counted anyway, won't they?
    - I already rebooted with fsck, which did not change anything
    - log files (/var/log) are a few MBs only, and I did not rotate them.

    Any more suggestions?

    thanks, Joachim


  7. Re: filesystem 100% full, but files do not occupy the space

    Joachim Worringen wrote:
    > On Jun 8, 2:04 am, "Frank Winans" wrote:
    >> "Joachim Worringen" wrote
    >>
    >>> I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
    >>> a LVM. The problem now is that (after some months of operation) the
    >>> root filesystem is 100% full (see "df -h" output below), but I can not
    >>> locate how the space is occupied (see du output further below). All
    >>> files take up maybe 4GB - where is the rest?

    > [...]
    >> but you forget that your du command wildcard will not show
    >> files/directories that start with a '.' character.
    >>
    >> So try
    >> cd / ; du -sh .* * | more
    >> and look for those missing umpteen gigs there.

    >
    > Thanks all for responding. My feedback:
    > - There are no .-dirs in / (checked with ls -la), and .-dirs from
    > subdirectories will be counted anyway, won't they?
    > - I already rebooted with fsck, which did not change anything
    > - log files (/var/log) are a few MBs only, and I did not rotate them.
    >
    > Any more suggestions?
    >
    > thanks, Joachim
    >


    IIRC you can use 'find' to search directory structures for odd named files.

    Tis a puzzle to be sure.


  8. Re: filesystem 100% full, but files do not occupy the space

    In comp.os.linux.setup Lew Pitcher :
    >> Lew Pitcher wrote:
    >>> Joachim Worringen wrote:


    >>>> [..] I rebooted with forced fsck, but this didn't change
    >>>> anything, same with vgck. Any hints?

    [..]

    >> Would not going to single user and fsck'ing everything sort this? or
    >> simply a reboot?


    > Sure. The simplest solution is just to reboot.


    As well as reading the OP...

    In addition this example exactly shows why it is a good idea to
    have several FS and not slam anything in /.

    --
    Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
    mail: echo zvpunry@urvzvat.qr | perl -pe 'y/a-z/n-za-m/'
    #bofh excuse 359: YOU HAVE AN I/O ERROR -> Incompetent
    Operator error

  9. Re: filesystem 100% full, but files do not occupy the space

    Michael Heiming wrote:
    > In comp.os.linux.setup Lew Pitcher :
    >>> Lew Pitcher wrote:
    >>>> Joachim Worringen wrote:

    >
    >>>>> [..] I rebooted with forced fsck, but this didn't change
    >>>>> anything, same with vgck. Any hints?

    > [..]
    >
    >>> Would not going to single user and fsck'ing everything sort this? or
    >>> simply a reboot?

    >
    >> Sure. The simplest solution is just to reboot.

    >
    > As well as reading the OP...
    >
    > In addition this example exactly shows why it is a good idea to
    > have several FS and not slam anything in /.
    >

    I just had a thought..infeasibly large inode block sizes would show a
    difference between du and df wouldn't they?

    df shows the space used, whereas du summarises file sizes..loads of one
    byte files would use a lot of space, but not show up as a large number
    of bytes of actual file usage..wonder if the OP has loads of teeny files..

  10. Re: filesystem 100% full, but files do not occupy the space

    On Jun 8, 2:51 pm, The Natural Philosopher wrote:
    > Michael Heiming wrote:
    > > In comp.os.linux.setup Lew Pitcher :
    > >>> Lew Pitcher wrote:
    > >>>> Joachim Worringen wrote:

    >
    > >>>>> [..] I rebooted with forced fsck, but this didn't change
    > >>>>> anything, same with vgck. Any hints?

    > > [..]

    >
    > >>> Would not going to single user and fsck'ing everything sort this? or
    > >>> simply a reboot?

    >
    > >> Sure. The simplest solution is just to reboot.

    >
    > > As well as reading the OP...

    >
    > > In addition this example exactly shows why it is a good idea to
    > > have several FS and not slam anything in /.


    Yes, but I more often than this came into situations (esp. on Solaris)
    where /var is full, but /home has GB's of free space. Ok, volume
    managers and things like ZFS will solve this kind of problem, but this
    is my home server and not a data center... ;-)

    > I just had a thought..infeasibly large inode block sizes would show a
    > difference between du and df wouldn't they?
    >
    > df shows the space used, whereas du summarises file sizes..loads of one
    > byte files would use a lot of space, but not show up as a large number
    > of bytes of actual file usage..wonder if the OP has loads of teeny files..


    I checked with 'find / -mount | wc -l' which gave a little bit more
    than 138k files. Not insane, I'd say?

    Joachim


  11. Re: filesystem 100% full, but files do not occupy the space

    The Natural Philosopher wrote:
    > Michael Heiming wrote:
    > > Lew Pitcher :
    > >>> Lew Pitcher wrote:
    > >>>> Joachim Worringen wrote:

    >
    > >>>>> [..] I rebooted with forced fsck, but this didn't change
    > >>>>> anything, same with vgck. Any hints?

    >
    > >>> Would not going to single user and fsck'ing everything sort this? or
    > >>> simply a reboot?

    >
    > >> Sure. The simplest solution is just to reboot.


    Sure. If your pet dog chews up the newspaper it works to
    shoot the dog, go to the pound and get another dog. It
    also works to train your dog to chew its own toys instead.

    It can be tempting to apply the Microsoft approach at times
    but it never does actually solve the problem. It just moves
    the next repeat forward in time.

    > > As well as reading the OP...


    Mention of reboot and fsck, check.

    > > In addition this example exactly shows why it is a good idea to
    > > have several FS and not slam anything in /.


    When / fills, very-bad things happen. When /var fills
    much-less-bad things happen.

    > I just had a thought..infeasibly large inode block sizes would show a
    > difference between du and df wouldn't they?


    I thought fragment size not block size, but I probably
    got this from write-ups of obsolete filesystem formats.

    One thing that du would miss but df would find: Manage
    to remove a directory that is not empty. The problem in
    these case - fsck will restore the lost files in /lost+found.

    > df shows the space used, whereas du summarises file sizes..loads of one
    > byte files would use a lot of space, but not show up as a large number
    > of bytes of actual file usage..wonder if the OP has loads of teeny files..


    df -i would report that.


  12. Re: filesystem 100% full, but files do not occupy the space

    On Jun 7, 2:03 pm, Joachim Worringen wrote:
    > Hi,
    >
    > I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
    > a LVM ...

    ....
    > [root@maxperf ~]# df -h
    > Filesystem Size Used Avail Use% Mounted on
    > /dev/mapper/VolGroup00-LogVol00

    ....
    > [root@maxperf ~]# du -sh /*
    > 0 /backup


    Are you running snapshots at the FS or LVM layer? I wonder if there
    are backup copies being generated on a regular basis but not being
    removed on a regular basis. I think /backup would not look empty
    if that is happening ...


  13. Re: filesystem 100% full, but files do not occupy the space

    "Joachim Worringen" wrote
    > "Frank Winans" wrote:
    > > "Joachim Worringen" wrote
    > >
    > > > I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
    > > > a LVM. The problem now is that (after some months of operation) the
    > > > root filesystem is 100% full (see "df -h" output below), but I can not
    > > > locate how the space is occupied (see du output further below). All
    > > > files take up maybe 4GB - where is the rest?

    > [...]
    > > but you forget that your du command wildcard will not show
    > > files/directories that start with a '.' character.
    > >
    > > So try
    > > cd / ; du -sh .* * | more
    > > and look for those missing umpteen gigs there.

    >
    > Thanks all for responding. My feedback:
    > - There are no .-dirs in / (checked with ls -la), and .-dirs from
    > subdirectories will be counted anyway, won't they?
    > - I already rebooted with fsck, which did not change anything
    > - log files (/var/log) are a few MBs only, and I did not rotate them.
    >
    > Any more suggestions?
    >
    > thanks, Joachim
    >

    Ooh! Good call about subdir .-dirs being counted anyway...

    "more suggestions" ? Only of the 'clutching at straws' variety,

    like
    "do you use chattr to set files as {able to be undeled later}?
    but that is the 'u' feature>

    or like
    "have you used exotic mke2fs settings to retain 95 percent
    of the filesystem space reserved for root userid admin purposes,
    instead of the default 5 percent of total capacity?"
    <>

    or like
    "if you mount some other computer's disk share on this box,
    and copy your various major directories over there,
    does this phantom disk usage get copied over too?



+ Reply to Thread