filesystem 100% full, but files do not occupy the space - Setup

Hi,

I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
a LVM. The problem now is that (after some monts of operation) the
root filesystem is 100% full (see "df -h" output below), but I can not
locate how the space is occupied (see du output further below). All
files take up maybe 4GB - where is the rest? I rebooted with forced
fsck, but this didn't change anything, same with vgck. Any hints?

thanks, Joachim

[root@maxperf tmp]# uname -r
2.6.9-42.0.2.EL
[root@maxperf ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
27G 27G 0 100% /
/dev/hda1 99M 13M 82M 13% /boot
none 125M 0 125M 0% /dev/shm
/dev/sda5 147G 136G 3.5G 98% /media/usbdisk2
/dev/sdb1 230G 213G 5.6G 98% /media/usbdisk
[root@maxperf ~]# du -sh /*
0 /backup
5.8M /bin
6.7M /boot
168K /dev
60M /etc
0 /export
36M /home
8.0K /initrd
85M /lib
16K /lost+found
8.0K /misc
8.0K /mnt
8.0K /opt
258M /proc
3.4M /root
16M /sbin
8.0K /selinux
8.0K /srv
0 /sys
76K /tmp
2.5G /usr
226M /var
[root@maxperf tmp]# vgdisplay
--- Volume group ---
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 27.81 GB
PE Size 32.00 MB
Total PE 890
Alloc PE / Size 890 / 27.81 GB
Free PE / Size 0 / 0
VG UUID t9Vx9Q-uU2l-lG3w-Vk9z-bmck-L5h6-7Lwx1h

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Joachim Worringen wrote:
> Hi,
>
> I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
> a LVM. The problem now is that (after some monts of operation) the
> root filesystem is 100% full (see "df -h" output below), but I can not
> locate how the space is occupied (see du output further below). All
> files take up maybe 4GB - where is the rest? I rebooted with forced
> fsck, but this didn't change anything, same with vgck. Any hints?
>
> thanks, Joachim
>
> [root@maxperf tmp]# uname -r
> 2.6.9-42.0.2.EL
> [root@maxperf ~]# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/VolGroup00-LogVol00
> 27G 27G 0 100% /
[snip]
> [root@maxperf ~]# du -sh /*
[snip]

The du(1) utility looks through directories, summing the space used by the
files named therein.

The df(1) utility looks at the filesystem allocation and freespace tables,
displaying the values recorded there.

When du(1) tells you that there is less used space (and thus more free space)
than du(f) tells you, then you can be assured that the difference is due to a
file that has been deleted from the directory tree, but has /not/ been
returned to the filesystem's freespace.

This situation happens when you delete all hardlinks to a file that is still
opened by some process. Because all the directory entries that link to the
file's inode have been deleted, the space is not counted in the du(1) count.
However, because the file is still open, the system will not return the space
allocated to that file to the filesystem freespace list.

To fix this, make the process that has the file open close the file. To find
the file, use lsof(8).

My guess is that you deleted/reallocated one of the log files (say
/var/log/messages) and didn't restart syslog. syslog keeps the old files open
(rather than go through the work of opening the file, writing a line, and
closing it each time a log record must be written), and so the space allocated
to the old /var/log/messages file didn't get returned to the freespace list.

HTH
- --
Lew Pitcher

Master Codewright & JOAT-in-training | Registered Linux User #112576
http://pitcher.digitalfreehold.ca/ | GPG public key available by request
- ---------- Slackware - Because I know what I'm doing. ------


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Armoured with GnuPG

iD8DBQFGaIlDagVFX4UWr64RAsX6AJ92IHd3Eog7Z0zo49odfS pebI4NdwCglFW+
VPLji4tZb0THDGFUpXJnxpc=
=xOKE
-----END PGP SIGNATURE-----

"Joachim Worringen" wrote
> I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
> a LVM. The problem now is that (after some months of operation) the
> root filesystem is 100% full (see "df -h" output below), but I can not
> locate how the space is occupied (see du output further below). All
> files take up maybe 4GB - where is the rest?
>
> [root@maxperf ~]# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/VolGroup00-LogVol00
> 27G 27G 0 100% /
> /dev/hda1 99M 13M 82M 13% /boot



> [root@maxperf ~]# du -sh /*
> 0 /backup
> 5.8M /bin
> 6.7M /boot
> 168K /dev
> 60M /etc



You've done well so far -- no need to do an elaborate size-based
find
command when the situation is so pronounced that a mere
du
will highlight the problem area,
but you forget that your du command wildcard will not show
files/directories that start with a '.' character.

So try
cd / ; du -sh .* * | more
and look for those missing umpteen gigs there.

Of course, ignore the result for file '.',
which is 'the whole current directory.'
Don't try this command in a subdir, as for example
cd /tmp ; du -sh .*
would include .. which is the parent directory -- how tedious.

Good luck hunting! And do this as root, as tightly-secured subdirs
cannot be surveyed by unpriviledged users.

Lew Pitcher wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Joachim Worringen wrote:
>> Hi,
>>
>> I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
>> a LVM. The problem now is that (after some monts of operation) the
>> root filesystem is 100% full (see "df -h" output below), but I can not
>> locate how the space is occupied (see du output further below). All
>> files take up maybe 4GB - where is the rest? I rebooted with forced
>> fsck, but this didn't change anything, same with vgck. Any hints?
>>
>> thanks, Joachim
>>
>> [root@maxperf tmp]# uname -r
>> 2.6.9-42.0.2.EL
>> [root@maxperf ~]# df -h
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/mapper/VolGroup00-LogVol00
>> 27G 27G 0 100% /
> [snip]
>> [root@maxperf ~]# du -sh /*
> [snip]
>
> The du(1) utility looks through directories, summing the space used by the
> files named therein.
>
> The df(1) utility looks at the filesystem allocation and freespace tables,
> displaying the values recorded there.
>
> When du(1) tells you that there is less used space (and thus more free space)
> than du(f) tells you, then you can be assured that the difference is due to a
> file that has been deleted from the directory tree, but has /not/ been
> returned to the filesystem's freespace.
>
> This situation happens when you delete all hardlinks to a file that is still
> opened by some process. Because all the directory entries that link to the
> file's inode have been deleted, the space is not counted in the du(1) count.
> However, because the file is still open, the system will not return the space
> allocated to that file to the filesystem freespace list.
>
> To fix this, make the process that has the file open close the file. To find
> the file, use lsof(8).
>
> My guess is that you deleted/reallocated one of the log files (say
> /var/log/messages) and didn't restart syslog. syslog keeps the old files open
> (rather than go through the work of opening the file, writing a line, and
> closing it each time a log record must be written), and so the space allocated
> to the old /var/log/messages file didn't get returned to the freespace list.

Would not going to single user and fsck'ing everything sort this? or
simply a reboot?

Your issues with log files ring a bell..

ISTR we used to rename the current file to log.1 or something, 'touch' a
new log file to create it., and then HUP the syslogd..so it reopened all
its files anew..all this under cron..

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The Natural Philosopher wrote:
> Lew Pitcher wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Joachim Worringen wrote:
>>> Hi,
>>>
>>> I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
>>> a LVM. The problem now is that (after some monts of operation) the
>>> root filesystem is 100% full (see "df -h" output below), but I can not
>>> locate how the space is occupied (see du output further below). All
>>> files take up maybe 4GB - where is the rest? I rebooted with forced
>>> fsck, but this didn't change anything, same with vgck. Any hints?
>>>
>>> thanks, Joachim
>>>
>>> [root@maxperf tmp]# uname -r
>>> 2.6.9-42.0.2.EL
>>> [root@maxperf ~]# df -h
>>> Filesystem Size Used Avail Use% Mounted on
>>> /dev/mapper/VolGroup00-LogVol00
>>> 27G 27G 0 100% /
>> [snip]
>>> [root@maxperf ~]# du -sh /*
>> [snip]
>>
>> The du(1) utility looks through directories, summing the space used by
>> the
>> files named therein.
>>
>> The df(1) utility looks at the filesystem allocation and freespace
>> tables,
>> displaying the values recorded there.
>>
>> When du(1) tells you that there is less used space (and thus more free
>> space)
>> than du(f) tells you, then you can be assured that the difference is
>> due to a
>> file that has been deleted from the directory tree, but has /not/ been
>> returned to the filesystem's freespace.
>[snip]
>
> Would not going to single user and fsck'ing everything sort this? or
> simply a reboot?

Sure. The simplest solution is just to reboot.
Going to single user might not free up everything (some processes may continue
to run even in single user mode), and there's no real need to fsck unless the
filesystem is really corrupt.

> Your issues with log files ring a bell..
>
> ISTR we used to rename the current file to log.1 or something, 'touch' a
> new log file to create it., and then HUP the syslogd..so it reopened all
> its files anew..all this under cron..

Yah. It's one of those things that catches most just-past-newbies (you know,
no longer a newbie, but not a guru :-) )


- --
Lew Pitcher

Master Codewright & JOAT-in-training | Registered Linux User #112576
http://pitcher.digitalfreehold.ca/ | GPG public key available by request
- ---------- Slackware - Because I know what I'm doing. ------


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Armoured with GnuPG

iD8DBQFGaJkVagVFX4UWr64RAvQjAJ9CJKX2y24bl7PLr5zR/ryoo1OVOwCg8vsU
Jaf4tTWLHMuh8+8L4FcDCaQ=
=DtfL
-----END PGP SIGNATURE-----

On Jun 8, 2:04 am, "Frank Winans" wrote:
> "Joachim Worringen" wrote
>
> > I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
> > a LVM. The problem now is that (after some months of operation) the
> > root filesystem is 100% full (see "df -h" output below), but I can not
> > locate how the space is occupied (see du output further below). All
> > files take up maybe 4GB - where is the rest?
[...]
> but you forget that your du command wildcard will not show
> files/directories that start with a '.' character.
>
> So try
> cd / ; du -sh .* * | more
> and look for those missing umpteen gigs there.

Thanks all for responding. My feedback:
- There are no .-dirs in / (checked with ls -la), and .-dirs from
subdirectories will be counted anyway, won't they?
- I already rebooted with fsck, which did not change anything
- log files (/var/log) are a few MBs only, and I did not rotate them.

Any more suggestions?

thanks, Joachim

Joachim Worringen wrote:
> On Jun 8, 2:04 am, "Frank Winans" wrote:
>> "Joachim Worringen" wrote
>>
>>> I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
>>> a LVM. The problem now is that (after some months of operation) the
>>> root filesystem is 100% full (see "df -h" output below), but I can not
>>> locate how the space is occupied (see du output further below). All
>>> files take up maybe 4GB - where is the rest?
> [...]
>> but you forget that your du command wildcard will not show
>> files/directories that start with a '.' character.
>>
>> So try
>> cd / ; du -sh .* * | more
>> and look for those missing umpteen gigs there.
>
> Thanks all for responding. My feedback:
> - There are no .-dirs in / (checked with ls -la), and .-dirs from
> subdirectories will be counted anyway, won't they?
> - I already rebooted with fsck, which did not change anything
> - log files (/var/log) are a few MBs only, and I did not rotate them.
>
> Any more suggestions?
>
> thanks, Joachim
>

IIRC you can use 'find' to search directory structures for odd named files.

Tis a puzzle to be sure.

In comp.os.linux.setup Lew Pitcher :
>> Lew Pitcher wrote:
>>> Joachim Worringen wrote:

>>>> [..] I rebooted with forced fsck, but this didn't change
>>>> anything, same with vgck. Any hints?
[..]

>> Would not going to single user and fsck'ing everything sort this? or
>> simply a reboot?

> Sure. The simplest solution is just to reboot.

As well as reading the OP...

In addition this example exactly shows why it is a good idea to
have several FS and not slam anything in /.

--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo zvpunry@urvzvat.qr | perl -pe 'y/a-z/n-za-m/'
#bofh excuse 359: YOU HAVE AN I/O ERROR -> Incompetent
Operator error

Michael Heiming wrote:
> In comp.os.linux.setup Lew Pitcher :
>>> Lew Pitcher wrote:
>>>> Joachim Worringen wrote:
>
>>>>> [..] I rebooted with forced fsck, but this didn't change
>>>>> anything, same with vgck. Any hints?
> [..]
>
>>> Would not going to single user and fsck'ing everything sort this? or
>>> simply a reboot?
>
>> Sure. The simplest solution is just to reboot.
>
> As well as reading the OP...
>
> In addition this example exactly shows why it is a good idea to
> have several FS and not slam anything in /.
>
I just had a thought..infeasibly large inode block sizes would show a
difference between du and df wouldn't they?

df shows the space used, whereas du summarises file sizes..loads of one
byte files would use a lot of space, but not show up as a large number
of bytes of actual file usage..wonder if the OP has loads of teeny files..

On Jun 8, 2:51 pm, The Natural Philosopher wrote:
> Michael Heiming wrote:
> > In comp.os.linux.setup Lew Pitcher :
> >>> Lew Pitcher wrote:
> >>>> Joachim Worringen wrote:
>
> >>>>> [..] I rebooted with forced fsck, but this didn't change
> >>>>> anything, same with vgck. Any hints?
> > [..]
>
> >>> Would not going to single user and fsck'ing everything sort this? or
> >>> simply a reboot?
>
> >> Sure. The simplest solution is just to reboot.
>
> > As well as reading the OP...
>
> > In addition this example exactly shows why it is a good idea to
> > have several FS and not slam anything in /.

Yes, but I more often than this came into situations (esp. on Solaris)
where /var is full, but /home has GB's of free space. Ok, volume
managers and things like ZFS will solve this kind of problem, but this
is my home server and not a data center... ;-)

> I just had a thought..infeasibly large inode block sizes would show a
> difference between du and df wouldn't they?
>
> df shows the space used, whereas du summarises file sizes..loads of one
> byte files would use a lot of space, but not show up as a large number
> of bytes of actual file usage..wonder if the OP has loads of teeny files..

I checked with 'find / -mount | wc -l' which gave a little bit more
than 138k files. Not insane, I'd say?

Joachim

The Natural Philosopher wrote:
> Michael Heiming wrote:
> > Lew Pitcher :
> >>> Lew Pitcher wrote:
> >>>> Joachim Worringen wrote:
>
> >>>>> [..] I rebooted with forced fsck, but this didn't change
> >>>>> anything, same with vgck. Any hints?
>
> >>> Would not going to single user and fsck'ing everything sort this? or
> >>> simply a reboot?
>
> >> Sure. The simplest solution is just to reboot.

Sure. If your pet dog chews up the newspaper it works to
shoot the dog, go to the pound and get another dog. It
also works to train your dog to chew its own toys instead.

It can be tempting to apply the Microsoft approach at times
but it never does actually solve the problem. It just moves
the next repeat forward in time.

> > As well as reading the OP...

Mention of reboot and fsck, check.

> > In addition this example exactly shows why it is a good idea to
> > have several FS and not slam anything in /.

When / fills, very-bad things happen. When /var fills
much-less-bad things happen.

> I just had a thought..infeasibly large inode block sizes would show a
> difference between du and df wouldn't they?

I thought fragment size not block size, but I probably
got this from write-ups of obsolete filesystem formats.

One thing that du would miss but df would find: Manage
to remove a directory that is not empty. The problem in
these case - fsck will restore the lost files in /lost+found.

> df shows the space used, whereas du summarises file sizes..loads of one
> byte files would use a lot of space, but not show up as a large number
> of bytes of actual file usage..wonder if the OP has loads of teeny files..

df -i would report that.

On Jun 7, 2:03 pm, Joachim Worringen wrote:
> Hi,
>
> I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
> a LVM ...
....
> [root@maxperf ~]# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/VolGroup00-LogVol00
....
> [root@maxperf ~]# du -sh /*
> 0 /backup

Are you running snapshots at the FS or LVM layer? I wonder if there
are backup copies being generated on a regular basis but not being
removed on a regular basis. I think /backup would not look empty
if that is happening ...

"Joachim Worringen" wrote
> "Frank Winans" wrote:
> > "Joachim Worringen" wrote
> >
> > > I have a CentOS 4.4 machine with a 30GB hard disk. / is configured as
> > > a LVM. The problem now is that (after some months of operation) the
> > > root filesystem is 100% full (see "df -h" output below), but I can not
> > > locate how the space is occupied (see du output further below). All
> > > files take up maybe 4GB - where is the rest?
> [...]
> > but you forget that your du command wildcard will not show
> > files/directories that start with a '.' character.
> >
> > So try
> > cd / ; du -sh .* * | more
> > and look for those missing umpteen gigs there.
>
> Thanks all for responding. My feedback:
> - There are no .-dirs in / (checked with ls -la), and .-dirs from
> subdirectories will be counted anyway, won't they?
> - I already rebooted with fsck, which did not change anything
> - log files (/var/log) are a few MBs only, and I did not rotate them.
>
> Any more suggestions?
>
> thanks, Joachim
>
Ooh! Good call about subdir .-dirs being counted anyway...

"more suggestions" ? Only of the 'clutching at straws' variety,

like
"do you use chattr to set files as {able to be undeled later}?
but that is the 'u' feature>

or like
"have you used exotic mke2fs settings to retain 95 percent
of the filesystem space reserved for root userid admin purposes,
instead of the default 5 percent of total capacity?"
<>

or like
"if you mount some other computer's disk share on this box,
and copy your various major directories over there,
does this phantom disk usage get copied over too?