Printable View
I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
/tmp file system got up to 95% but their were no large files in /tmp.
Usually when our /tmp gets large it's when one of my users have started a
report pulling from one of our large databases and forgot to put range
selections in. Any ideas on where or how this got so high and how I can
tell in the future where the problem might be? In the 8 years of running an
AIX box I don't remember this ever happening.
Thanks....Mark
Mark McWilliams wrote:
[color=blue]
> I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
> /tmp file system got up to 95% but their were no large files in /tmp.
> Usually when our /tmp gets large it's when one of my users have started a
> report pulling from one of our large databases and forgot to put range
> selections in. Any ideas on where or how this got so high and how I can
> tell in the future where the problem might be? In the 8 years of running
> an AIX box I don't remember this ever happening.
>
> Thanks....Mark[/color]
Mark;
You may just have to make /tmp a little larger (chfs -size=+1 /tmp) will
increase it to the next logical level (not just one block). You can use
"df" to see what all your file systems look like capacity-wise and adjust
accordingly assuming you have some free space available. You can also use
"lsvg -a" to see if there is any unallocated space in the volume group(s).
If so you can allocate them for more space.
Good luck.
bobmct
Mark McWilliams wrote:[color=blue]
> I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
> /tmp file system got up to 95% but their were no large files in /tmp.
> Usually when our /tmp gets large it's when one of my users have started a
> report pulling from one of our large databases and forgot to put range
> selections in. Any ideas on where or how this got so high and how I can
> tell in the future where the problem might be? In the 8 years of running an
> AIX box I don't remember this ever happening.[/color]
Well, a *lot* of small files will eventually fill your /tmp too ;-)
Have a look at /tmp and/or its subdirs with:
du -sk /tmp/* | sort -n
or
find /tmp -type d | xargs du -sk
if you are using a lot of subdirs in /tmp. Clean out old files that
didn't get removed properly (find /tmp -mtime ...).
If the 'du -sk /tmp' command reports less usage than 'df' does, you/
someone has most likely deleted a file from the filesystem, but
there is still a process accessing it. The file will be around until
the last file handle to it is closed, i.e. not process is accessing
it anymore. To find if that is the case use the 'fuser -d ...'
command.
Regards,
Frank
"Frank Fegert" <fra.nospam.nk@gmx.de> wrote in message
news:eo2294$gla$02$1@news.t-online.com...[color=blue]
> Mark McWilliams wrote:[color=green]
>> I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
>> /tmp file system got up to 95% but their were no large files in /tmp.
>> Usually when our /tmp gets large it's when one of my users have started a
>> report pulling from one of our large databases and forgot to put range
>> selections in. Any ideas on where or how this got so high and how I can
>> tell in the future where the problem might be? In the 8 years of running
>> an
>> AIX box I don't remember this ever happening.[/color]
>
> Well, a *lot* of small files will eventually fill your /tmp too ;-)
> Have a look at /tmp and/or its subdirs with:
> du -sk /tmp/* | sort -n
> or
> find /tmp -type d | xargs du -sk
> if you are using a lot of subdirs in /tmp. Clean out old files that
> didn't get removed properly (find /tmp -mtime ...).
> If the 'du -sk /tmp' command reports less usage than 'df' does, you/
> someone has most likely deleted a file from the filesystem, but
> there is still a process accessing it. The file will be around until
> the last file handle to it is closed, i.e. not process is accessing
> it anymore. To find if that is the case use the 'fuser -d ...'
> command.
>
> Regards,
>
> Frank[/color]
Frank,
Thanks for the response. If I remember right, on the day that this
happened their weren't a lot of small files either. I was trying to cd into
some of the subdirectories and it wouldn't allow me even though I was logged
in as root. Can you not look within subdirectories on /tmp? We do a backup
every night and the system reboots afterwards and this should flush out
/tmp. I'm hoping this doesn't happen again but if it does the commands you
gave should help out a lot in narrowing down the problem.
Mark
Mark McWilliams wrote:[color=blue]
> "Frank Fegert" <fra.nospam.nk@gmx.de> wrote in message
> news:eo2294$gla$02$1@news.t-online.com...[color=green]
>> Mark McWilliams wrote:[color=darkred]
>>> I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
>>> /tmp file system got up to 95% but their were no large files in /tmp.
>>> Usually when our /tmp gets large it's when one of my users have started a
>>> report pulling from one of our large databases and forgot to put range
>>> selections in. Any ideas on where or how this got so high and how I can
>>> tell in the future where the problem might be? In the 8 years of running
>>> an
>>> AIX box I don't remember this ever happening.[/color]
>> Well, a *lot* of small files will eventually fill your /tmp too ;-)
>> Have a look at /tmp and/or its subdirs with:
>> du -sk /tmp/* | sort -n
>> or
>> find /tmp -type d | xargs du -sk
>> if you are using a lot of subdirs in /tmp. Clean out old files that
>> didn't get removed properly (find /tmp -mtime ...).
>> If the 'du -sk /tmp' command reports less usage than 'df' does, you/
>> someone has most likely deleted a file from the filesystem, but
>> there is still a process accessing it. The file will be around until
>> the last file handle to it is closed, i.e. not process is accessing
>> it anymore. To find if that is the case use the 'fuser -d ...'
>> command.
>>
>> Regards,
>>
>> Frank[/color]
>
> Frank,
>
> Thanks for the response. If I remember right, on the day that this
> happened their weren't a lot of small files either. I was trying to cd into
> some of the subdirectories and it wouldn't allow me even though I was logged
> in as root. Can you not look within subdirectories on /tmp? We do a backup
> every night and the system reboots afterwards and this should flush out
> /tmp. I'm hoping this doesn't happen again but if it does the commands you
> gave should help out a lot in narrowing down the problem.
>
> Mark
>
>[/color]
Oh contraire........
AIX does not do that. Solaris does, AIX does not. If you don't clean
out /tmp, everything will hang around forever.
0xDEADABE wrote:[color=blue]
> Mark McWilliams wrote:[color=green]
>> Thanks for the response. If I remember right, on the day that
>> this happened their weren't a lot of small files either. I was trying
>> to cd into some of the subdirectories and it wouldn't allow me even
>> though I was logged in as root. Can you not look within
>> subdirectories on /tmp? We do a backup every night and the system
>> reboots afterwards and this should flush out /tmp. I'm hoping this
>> doesn't happen again but if it does the commands you gave should help
>> out a lot in narrowing down the problem.[/color]
>
> Oh contraire........
>
> AIX does not do that. Solaris does, AIX does not. If you don't clean
> out /tmp, everything will hang around forever.[/color]
As well as certain Linux distributions, which have rc-scripts to
clean out /tmp.
To make my DWTF ([url]http://thedailywtf.com[/url]) i'm really curious what
setup you're having that needs a reboot after backup/nightly re-
boots? Why don't you just let the system do what you paid for i.e.
running?
Regards,
Frank
There is probably an unlinked, but open file in the file system. If
you have lsof installed, run it and look for a file with 0 links.
Mark McWilliams wrote:[color=blue]
> I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
> /tmp file system got up to 95% but their were no large files in /tmp.
> Usually when our /tmp gets large it's when one of my users have started a
> report pulling from one of our large databases and forgot to put range
> selections in. Any ideas on where or how this got so high and how I can
> tell in the future where the problem might be? In the 8 years of running an
> AIX box I don't remember this ever happening.
>
> Thanks....Mark[/color]
Mark McWilliams wrote:
[color=blue]
> I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
> /tmp file system got up to 95% but their were no large files in /tmp.
> Usually when our /tmp gets large it's when one of my users have started a
> report pulling from one of our large databases and forgot to put range
> selections in. Any ideas on where or how this got so high and how I can
> tell in the future where the problem might be? In the 8 years of running an
> AIX box I don't remember this ever happening.[/color]
Could be a file that has been removed but is still open - check with
"fuser -dV /tmp".
0xDEADABE wrote:[color=blue]
> Mark McWilliams wrote:[color=green]
> > "Frank Fegert" <fra.nospam.nk@gmx.de> wrote in message
> > news:eo2294$gla$02$1@news.t-online.com...[color=darkred]
> >> Mark McWilliams wrote:
> >>> I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
> >>> /tmp file system got up to 95% but their were no large files in /tmp.
> >>> Usually when our /tmp gets large it's when one of my users have started a
> >>> report pulling from one of our large databases and forgot to put range
> >>> selections in. Any ideas on where or how this got so high and how I can
> >>> tell in the future where the problem might be? In the 8 years of running
> >>> an
> >>> AIX box I don't remember this ever happening.
> >> Well, a *lot* of small files will eventually fill your /tmp too ;-)
> >> Have a look at /tmp and/or its subdirs with:
> >> du -sk /tmp/* | sort -n
> >> or
> >> find /tmp -type d | xargs du -sk
> >> if you are using a lot of subdirs in /tmp. Clean out old files that
> >> didn't get removed properly (find /tmp -mtime ...).
> >> If the 'du -sk /tmp' command reports less usage than 'df' does, you/
> >> someone has most likely deleted a file from the filesystem, but
> >> there is still a process accessing it. The file will be around until
> >> the last file handle to it is closed, i.e. not process is accessing
> >> it anymore. To find if that is the case use the 'fuser -d ...'
> >> command.
> >>
> >> Regards,
> >>
> >> Frank[/color]
> >
> > Frank,
> >
> > Thanks for the response. If I remember right, on the day that this
> > happened their weren't a lot of small files either. I was trying to cd into
> > some of the subdirectories and it wouldn't allow me even though I was logged
> > in as root. Can you not look within subdirectories on /tmp? We do a backup
> > every night and the system reboots afterwards and this should flush out
> > /tmp. I'm hoping this doesn't happen again but if it does the commands you
> > gave should help out a lot in narrowing down the problem.
> >
> > Mark
> >
> >[/color]
>
> Oh contraire........
>
> AIX does not do that. Solaris does, AIX does not. If you don't clean
> out /tmp, everything will hang around forever.[/color]
Once the OP has identified and rectified the problem that has filled
his /tmp, he should probably also look into configuring and activating
the skulker cron job. One of skulker's jobs is to clean old files out
of /tmp, so that the directory (and fs underneath it) does not fill up.
HTH
--
Lew
lsof is the best tool to investigate such cases - it will give you a
size of the file even if it is deleted but still opened by the running
proces (ths is the most likely scenario) and it will give you a
process pid also. You can find a version that is compiled for AIX
4.3.3 e.g in bullfreeware archives. fuser command was not very
reliable on AIX 4.3.3. Good luck.
Leszek