/tmp filled up---No Large Files?? - Aix

This is a discussion on /tmp filled up---No Large Files?? - Aix ; I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the /tmp file system got up to 95% but their were no large files in /tmp. Usually when our /tmp gets large it's when one of ...

+ Reply to Thread
Results 1 to 10 of 10

Thread: /tmp filled up---No Large Files??

  1. /tmp filled up---No Large Files??

    I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
    /tmp file system got up to 95% but their were no large files in /tmp.
    Usually when our /tmp gets large it's when one of my users have started a
    report pulling from one of our large databases and forgot to put range
    selections in. Any ideas on where or how this got so high and how I can
    tell in the future where the problem might be? In the 8 years of running an
    AIX box I don't remember this ever happening.

    Thanks....Mark



  2. Re: /tmp filled up---No Large Files??

    Mark McWilliams wrote:

    > I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
    > /tmp file system got up to 95% but their were no large files in /tmp.
    > Usually when our /tmp gets large it's when one of my users have started a
    > report pulling from one of our large databases and forgot to put range
    > selections in. Any ideas on where or how this got so high and how I can
    > tell in the future where the problem might be? In the 8 years of running
    > an AIX box I don't remember this ever happening.
    >
    > Thanks....Mark

    Mark;

    You may just have to make /tmp a little larger (chfs -size=+1 /tmp) will
    increase it to the next logical level (not just one block). You can use
    "df" to see what all your file systems look like capacity-wise and adjust
    accordingly assuming you have some free space available. You can also use
    "lsvg -a" to see if there is any unallocated space in the volume group(s).
    If so you can allocate them for more space.

    Good luck.

    bobmct

  3. Re: /tmp filled up---No Large Files??

    Mark McWilliams wrote:
    > I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
    > /tmp file system got up to 95% but their were no large files in /tmp.
    > Usually when our /tmp gets large it's when one of my users have started a
    > report pulling from one of our large databases and forgot to put range
    > selections in. Any ideas on where or how this got so high and how I can
    > tell in the future where the problem might be? In the 8 years of running an
    > AIX box I don't remember this ever happening.


    Well, a *lot* of small files will eventually fill your /tmp too ;-)
    Have a look at /tmp and/or its subdirs with:
    du -sk /tmp/* | sort -n
    or
    find /tmp -type d | xargs du -sk
    if you are using a lot of subdirs in /tmp. Clean out old files that
    didn't get removed properly (find /tmp -mtime ...).
    If the 'du -sk /tmp' command reports less usage than 'df' does, you/
    someone has most likely deleted a file from the filesystem, but
    there is still a process accessing it. The file will be around until
    the last file handle to it is closed, i.e. not process is accessing
    it anymore. To find if that is the case use the 'fuser -d ...'
    command.

    Regards,

    Frank

  4. Re: /tmp filled up---No Large Files??


    "Frank Fegert" wrote in message
    news:eo2294$gla$02$1@news.t-online.com...
    > Mark McWilliams wrote:
    >> I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
    >> /tmp file system got up to 95% but their were no large files in /tmp.
    >> Usually when our /tmp gets large it's when one of my users have started a
    >> report pulling from one of our large databases and forgot to put range
    >> selections in. Any ideas on where or how this got so high and how I can
    >> tell in the future where the problem might be? In the 8 years of running
    >> an
    >> AIX box I don't remember this ever happening.

    >
    > Well, a *lot* of small files will eventually fill your /tmp too ;-)
    > Have a look at /tmp and/or its subdirs with:
    > du -sk /tmp/* | sort -n
    > or
    > find /tmp -type d | xargs du -sk
    > if you are using a lot of subdirs in /tmp. Clean out old files that
    > didn't get removed properly (find /tmp -mtime ...).
    > If the 'du -sk /tmp' command reports less usage than 'df' does, you/
    > someone has most likely deleted a file from the filesystem, but
    > there is still a process accessing it. The file will be around until
    > the last file handle to it is closed, i.e. not process is accessing
    > it anymore. To find if that is the case use the 'fuser -d ...'
    > command.
    >
    > Regards,
    >
    > Frank


    Frank,

    Thanks for the response. If I remember right, on the day that this
    happened their weren't a lot of small files either. I was trying to cd into
    some of the subdirectories and it wouldn't allow me even though I was logged
    in as root. Can you not look within subdirectories on /tmp? We do a backup
    every night and the system reboots afterwards and this should flush out
    /tmp. I'm hoping this doesn't happen again but if it does the commands you
    gave should help out a lot in narrowing down the problem.

    Mark



  5. Re: /tmp filled up---No Large Files??

    Mark McWilliams wrote:
    > "Frank Fegert" wrote in message
    > news:eo2294$gla$02$1@news.t-online.com...
    >> Mark McWilliams wrote:
    >>> I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
    >>> /tmp file system got up to 95% but their were no large files in /tmp.
    >>> Usually when our /tmp gets large it's when one of my users have started a
    >>> report pulling from one of our large databases and forgot to put range
    >>> selections in. Any ideas on where or how this got so high and how I can
    >>> tell in the future where the problem might be? In the 8 years of running
    >>> an
    >>> AIX box I don't remember this ever happening.

    >> Well, a *lot* of small files will eventually fill your /tmp too ;-)
    >> Have a look at /tmp and/or its subdirs with:
    >> du -sk /tmp/* | sort -n
    >> or
    >> find /tmp -type d | xargs du -sk
    >> if you are using a lot of subdirs in /tmp. Clean out old files that
    >> didn't get removed properly (find /tmp -mtime ...).
    >> If the 'du -sk /tmp' command reports less usage than 'df' does, you/
    >> someone has most likely deleted a file from the filesystem, but
    >> there is still a process accessing it. The file will be around until
    >> the last file handle to it is closed, i.e. not process is accessing
    >> it anymore. To find if that is the case use the 'fuser -d ...'
    >> command.
    >>
    >> Regards,
    >>
    >> Frank

    >
    > Frank,
    >
    > Thanks for the response. If I remember right, on the day that this
    > happened their weren't a lot of small files either. I was trying to cd into
    > some of the subdirectories and it wouldn't allow me even though I was logged
    > in as root. Can you not look within subdirectories on /tmp? We do a backup
    > every night and the system reboots afterwards and this should flush out
    > /tmp. I'm hoping this doesn't happen again but if it does the commands you
    > gave should help out a lot in narrowing down the problem.
    >
    > Mark
    >
    >


    Oh contraire........

    AIX does not do that. Solaris does, AIX does not. If you don't clean
    out /tmp, everything will hang around forever.

  6. Re: /tmp filled up---No Large Files??

    0xDEADABE wrote:
    > Mark McWilliams wrote:
    >> Thanks for the response. If I remember right, on the day that
    >> this happened their weren't a lot of small files either. I was trying
    >> to cd into some of the subdirectories and it wouldn't allow me even
    >> though I was logged in as root. Can you not look within
    >> subdirectories on /tmp? We do a backup every night and the system
    >> reboots afterwards and this should flush out /tmp. I'm hoping this
    >> doesn't happen again but if it does the commands you gave should help
    >> out a lot in narrowing down the problem.

    >
    > Oh contraire........
    >
    > AIX does not do that. Solaris does, AIX does not. If you don't clean
    > out /tmp, everything will hang around forever.


    As well as certain Linux distributions, which have rc-scripts to
    clean out /tmp.
    To make my DWTF (http://thedailywtf.com) i'm really curious what
    setup you're having that needs a reboot after backup/nightly re-
    boots? Why don't you just let the system do what you paid for i.e.
    running?

    Regards,

    Frank

  7. Re: /tmp filled up---No Large Files??

    There is probably an unlinked, but open file in the file system. If
    you have lsof installed, run it and look for a file with 0 links.

    Mark McWilliams wrote:
    > I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
    > /tmp file system got up to 95% but their were no large files in /tmp.
    > Usually when our /tmp gets large it's when one of my users have started a
    > report pulling from one of our large databases and forgot to put range
    > selections in. Any ideas on where or how this got so high and how I can
    > tell in the future where the problem might be? In the 8 years of running an
    > AIX box I don't remember this ever happening.
    >
    > Thanks....Mark



  8. Re: /tmp filled up---No Large Files??


    Mark McWilliams wrote:

    > I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
    > /tmp file system got up to 95% but their were no large files in /tmp.
    > Usually when our /tmp gets large it's when one of my users have started a
    > report pulling from one of our large databases and forgot to put range
    > selections in. Any ideas on where or how this got so high and how I can
    > tell in the future where the problem might be? In the 8 years of running an
    > AIX box I don't remember this ever happening.


    Could be a file that has been removed but is still open - check with
    "fuser -dV /tmp".


  9. Re: /tmp filled up---No Large Files??


    0xDEADABE wrote:
    > Mark McWilliams wrote:
    > > "Frank Fegert" wrote in message
    > > news:eo2294$gla$02$1@news.t-online.com...
    > >> Mark McWilliams wrote:
    > >>> I had a strange occurrence on our IBM RISC 6F1 running AIX 4.3 where the
    > >>> /tmp file system got up to 95% but their were no large files in /tmp.
    > >>> Usually when our /tmp gets large it's when one of my users have started a
    > >>> report pulling from one of our large databases and forgot to put range
    > >>> selections in. Any ideas on where or how this got so high and how I can
    > >>> tell in the future where the problem might be? In the 8 years of running
    > >>> an
    > >>> AIX box I don't remember this ever happening.
    > >> Well, a *lot* of small files will eventually fill your /tmp too ;-)
    > >> Have a look at /tmp and/or its subdirs with:
    > >> du -sk /tmp/* | sort -n
    > >> or
    > >> find /tmp -type d | xargs du -sk
    > >> if you are using a lot of subdirs in /tmp. Clean out old files that
    > >> didn't get removed properly (find /tmp -mtime ...).
    > >> If the 'du -sk /tmp' command reports less usage than 'df' does, you/
    > >> someone has most likely deleted a file from the filesystem, but
    > >> there is still a process accessing it. The file will be around until
    > >> the last file handle to it is closed, i.e. not process is accessing
    > >> it anymore. To find if that is the case use the 'fuser -d ...'
    > >> command.
    > >>
    > >> Regards,
    > >>
    > >> Frank

    > >
    > > Frank,
    > >
    > > Thanks for the response. If I remember right, on the day that this
    > > happened their weren't a lot of small files either. I was trying to cd into
    > > some of the subdirectories and it wouldn't allow me even though I was logged
    > > in as root. Can you not look within subdirectories on /tmp? We do a backup
    > > every night and the system reboots afterwards and this should flush out
    > > /tmp. I'm hoping this doesn't happen again but if it does the commands you
    > > gave should help out a lot in narrowing down the problem.
    > >
    > > Mark
    > >
    > >

    >
    > Oh contraire........
    >
    > AIX does not do that. Solaris does, AIX does not. If you don't clean
    > out /tmp, everything will hang around forever.


    Once the OP has identified and rectified the problem that has filled
    his /tmp, he should probably also look into configuring and activating
    the skulker cron job. One of skulker's jobs is to clean old files out
    of /tmp, so that the directory (and fs underneath it) does not fill up.

    HTH
    --
    Lew


  10. Re: /tmp filled up---No Large Files??

    lsof is the best tool to investigate such cases - it will give you a
    size of the file even if it is deleted but still opened by the running
    proces (ths is the most likely scenario) and it will give you a
    process pid also. You can find a version that is compiled for AIX
    4.3.3 e.g in bullfreeware archives. fuser command was not very
    reliable on AIX 4.3.3. Good luck.
    Leszek


+ Reply to Thread