How to get an "alert" when a process dies - Security

This is a discussion on How to get an "alert" when a process dies - Security ; Hi all, Even though I use Linux a lot, I have not been doing a lot of hard core Linux sysadmin stuff for some time, hence the question here :-) Is there a way to trigger an event (like running ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 22

Thread: How to get an "alert" when a process dies

  1. How to get an "alert" when a process dies

    Hi all,

    Even though I use Linux a lot, I have not been doing a lot of hard
    core Linux sysadmin stuff for some time, hence the question here :-)

    Is there a way to trigger an event (like running a script) when a
    process dies? Perhaps even a script to restart a process if it detects
    that it's dead?

    I have been using cron for monitoring processes up to now, but the
    monitoring interval is 1 minute apart. I have a project (streaming of
    nature) that should not go down, but the problem is that sometimes it
    dies and it takes up to a minute before the traditional monitoring
    script will restart it.

    Any ideas?

    Thanks - Nico.

  2. Re: How to get an "alert" when a process dies

    nicc777 wrote:
    > Hi all,
    >
    > Even though I use Linux a lot, I have not been doing a lot of hard
    > core Linux sysadmin stuff for some time, hence the question here :-)
    >
    > Is there a way to trigger an event (like running a script) when a
    > process dies? Perhaps even a script to restart a process if it detects
    > that it's dead?
    >
    > I have been using cron for monitoring processes up to now, but the
    > monitoring interval is 1 minute apart. I have a project (streaming of
    > nature) that should not go down, but the problem is that sometimes it
    > dies and it takes up to a minute before the traditional monitoring
    > script will restart it.
    >
    > Any ideas?
    >
    > Thanks - Nico.


    Nagios.

    Nico, too

  3. Re: How to get an "alert" when a process dies

    Nico Kadel-Garcia :

    > nicc777 wrote:
    >> Hi all,
    >>
    >> Even though I use Linux a lot, I have not been doing a lot of hard
    >> core Linux sysadmin stuff for some time, hence the question here :-)
    >>
    >> Is there a way to trigger an event (like running a script) when a
    >> process dies? Perhaps even a script to restart a process if it detects
    >> that it's dead?
    >>
    >> I have been using cron for monitoring processes up to now, but the
    >> monitoring interval is 1 minute apart. I have a project (streaming of
    >> nature) that should not go down, but the problem is that sometimes it
    >> dies and it takes up to a minute before the traditional monitoring
    >> script will restart it.
    >>
    >> Any ideas?
    >>
    >> Thanks - Nico.

    >
    > Nagios.


    Suffers from the very problem, the OP was trying to avoid: A quite great
    delay between the death of the specific process and the monitoring tool
    running the check and thus noticing the dead process. Iirc, nagios uses
    even higher intervals than one minute between checks.

    I guess, the OP wants something like inotify for processes: Passive
    checking, yielding an _immediate_ notification about state changes, without
    the unavoidable latency of active checks as nagios performs them.

    --
    Freedom is always the freedom of dissenters.
    (Rosa Luxemburg)

  4. Re: How to get an "alert" when a process dies

    nicc777 wrote:

    > Hi all,
    >
    > Even though I use Linux a lot, I have not been doing a lot of hard
    > core Linux sysadmin stuff for some time, hence the question here :-)
    >
    > Is there a way to trigger an event (like running a script) when a
    > process dies? Perhaps even a script to restart a process if it detects
    > that it's dead?
    >
    > I have been using cron for monitoring processes up to now, but the
    > monitoring interval is 1 minute apart. I have a project (streaming of
    > nature) that should not go down, but the problem is that sometimes it
    > dies and it takes up to a minute before the traditional monitoring
    > script will restart it.


    Totally untested, but it should get your creative juices flowing:

    #!/bin/bash
    PROC=someprocess # The process to check
    WAIT=10 # Seconds to wait between checks

    while [ 1 ]
    do
    ps -ef|grep $PROC &>/dev/null
    if [ $? -ne 0 ]; then
    # grep didn't find "$PROC"...dead?
    /etc/init.d/$PROC restart &>/dev/null

    # Send some notification saying what happened.
    # Maybe re-run the process check??
    fi

    # Now sleep and re-check the process after a specified delay
    sleep $WAIT
    done


    Now all you need to do, is start that, fork it to the background and voila.
    Add whatever notification method you want in the "if...fi" block. This
    script will run until you (or something else) kills it by virtue of
    the "while" loop.

    If you really wanted to get fancy and check a few processes, change "PROC"
    into an array and iterate through it, or use a "for PROC in proc1 proc2
    etc" type construct. There are many ways to extend this, but whatever you
    need to do can be based on this to provide sub-minute scheduling.

    Cheers,

    James
    --
    It's lucky you're going so slowly, because you're going in the wrong
    direction.


  5. Re: How to get an "alert" when a process dies

    On Jun 30, 1:25*am, "Sebastian \"lunar\" Wiesner"
    wrote:
    > Nico Kadel-Garcia :
    >
    >
    >
    > > nicc777 wrote:
    > >> Hi all,

    >
    > >> Even though I use Linux a lot, I have not been doing a lot of hard
    > >> core Linux sysadmin stuff for some time, hence the question here :-)

    >
    > >> Is there a way to trigger an event (like running a script) when a
    > >> process dies? Perhaps even a script to restart a process if it detects
    > >> that it's dead?

    >
    > >> I have been using cron for monitoring processes up to now, but the
    > >> monitoring interval is 1 minute apart. I have a project (streaming of
    > >> nature) that should not go down, but the problem is that sometimes it
    > >> dies and it takes up to a minute before the traditional monitoring
    > >> script will restart it.

    >
    > >> Any ideas?

    >
    > >> Thanks - Nico.

    >
    > > Nagios.

    >
    > Suffers from the very problem, the OP was trying to avoid: *A quite great
    > delay between the death of the specific process and the monitoring tool
    > running the check and thus noticing the dead process. *Iirc, nagios uses
    > even higher intervals than one minute between checks.
    >
    > I guess, the OP wants something like inotify for processes: *Passive
    > checking, yielding an _immediate_ notification about state changes, without
    > the unavoidable latency of active checks as nagios performs them.
    >
    > --
    > Freedom is always the freedom of dissenters.
    > * * * * * * * * * * * * * * * * * * * (Rosa Luxemburg)


    Exactly - Time delays is what I try to get away from. Even seconds can
    mean potential customers leaving your service because there's no
    service :-(

    Thanks for explaining my problem more clearly !

  6. Re: How to get an "alert" when a process dies

    On Jun 30, 5:03*am, Centurion wrote:
    > nicc777 wrote:
    > > Hi all,

    >
    > > Even though I use Linux a lot, I have not been doing a lot of hard
    > > core Linux sysadmin stuff for some time, hence the question here :-)

    >
    > > Is there a way to trigger an event (like running a script) when a
    > > process dies? Perhaps even a script to restart a process if it detects
    > > that it's dead?

    >
    > > I have been using cron for monitoring processes up to now, but the
    > > monitoring interval is 1 minute apart. I have a project (streaming of
    > > nature) that should not go down, but the problem is that sometimes it
    > > dies and it takes up to a minute before the traditional monitoring
    > > script will restart it.

    >
    > Totally untested, but it should get your creative juices flowing:
    >
    > #!/bin/bash
    > PROC=someprocess *# The process to check
    > WAIT=10 * * * * * # Seconds to wait between checks
    >
    > while [ 1 ]
    > do
    > * * * * ps -ef|grep $PROC &>/dev/null
    > * * * * if [ $? -ne 0 ]; then
    > * * * * * * * * # grep didn't find "$PROC"...dead?
    > * * * * * * * * /etc/init.d/$PROC restart &>/dev/null
    >
    > * * * * * * * * # Send some notification saying what happened.
    > * * * * * * * * # Maybe re-run the process check??
    > * * * * fi
    >
    > * * * * # Now sleep and re-check the process after a specified delay
    > * * * * sleep $WAIT
    > done
    >
    > Now all you need to do, is start that, fork it to the background and voila.
    > Add whatever notification method you want in the "if...fi" block. *This
    > script will run until you (or something else) kills it by virtue of
    > the "while" loop.
    >
    > If you really wanted to get fancy and check a few processes, change "PROC"
    > into an array and iterate through it, or use a "for PROC in proc1 proc2
    > etc" type construct. *There are many ways to extend this, but whatever you
    > need to do can be based on this to provide sub-minute scheduling.
    >
    > Cheers,
    >
    > James
    > --
    > It's lucky you're going so slowly, because you're going in the wrong
    > direction.


    Thanks a lot - I came to the same conclusion using another scripting
    language, but I still sit with a 1 second delay - which I assume is
    sufficient for my current scenario.

    Still - I would like to know if there is a real time notification
    mechanism.

    Does any body know if it's possible to hook something into the Kernel
    (custom module) maybe? My thinking is maybe something that can "catch"
    signals on the kernel level. Thanks again :-)

  7. Re: How to get an "alert" when a process dies

    nicc777 writes:

    > Is there a way to trigger an event (like running a script) when a
    > process dies? Perhaps even a script to restart a process if it detects
    > that it's dead?
    >
    > I have been using cron for monitoring processes up to now, but the
    > monitoring interval is 1 minute apart. I have a project (streaming of
    > nature) that should not go down, but the problem is that sometimes it
    > dies and it takes up to a minute before the traditional monitoring
    > script will restart it.


    See http://upstart.ubuntu.com

    In the end you are looking for some sort of process supervisor.

  8. Re: How to get an "alert" when a process dies

    * nicc777
    | Is there a way to trigger an event (like running a script) when a
    | process dies? Perhaps even a script to restart a process if it detects
    | that it's dead?

    #!/bin/sh
    while true ; do
    run_your_process
    status=$?
    case $status in
    0) : probably ok? ; break ;;
    *) echo "process has exited with status $status, re-run"
    esac
    donex

    ?

    R'

  9. Re: How to get an "alert" when a process dies

    nicc777 wrote:
    > On Jun 30, 1:25 am, "Sebastian \"lunar\" Wiesner"
    > wrote:
    >> Nico Kadel-Garcia :
    >>
    >>
    >>
    >>> nicc777 wrote:
    >>>> Hi all,
    >>>> Even though I use Linux a lot, I have not been doing a lot of hard
    >>>> core Linux sysadmin stuff for some time, hence the question here :-)
    >>>> Is there a way to trigger an event (like running a script) when a
    >>>> process dies? Perhaps even a script to restart a process if it detects
    >>>> that it's dead?
    >>>> I have been using cron for monitoring processes up to now, but the
    >>>> monitoring interval is 1 minute apart. I have a project (streaming of
    >>>> nature) that should not go down, but the problem is that sometimes it
    >>>> dies and it takes up to a minute before the traditional monitoring
    >>>> script will restart it.
    >>>> Any ideas?
    >>>> Thanks - Nico.
    >>> Nagios.

    >> Suffers from the very problem, the OP was trying to avoid: A quite great
    >> delay between the death of the specific process and the monitoring tool
    >> running the check and thus noticing the dead process. Iirc, nagios uses
    >> even higher intervals than one minute between checks.
    >>
    >> I guess, the OP wants something like inotify for processes: Passive
    >> checking, yielding an _immediate_ notification about state changes, without
    >> the unavoidable latency of active checks as nagios performs them.
    >>
    >> --
    >> Freedom is always the freedom of dissenters.
    >> (Rosa Luxemburg)

    >
    > Exactly - Time delays is what I try to get away from. Even seconds can
    > mean potential customers leaving your service because there's no
    > service :-(
    >
    > Thanks for explaining my problem more clearly !


    OK. You need a monitor task that checks much more frequently. A cron script
    could assure that the *MONITOR* is up and running, but the monitor could be a
    simple shell script with a built-in 'restart if not working' setup.

  10. Re: How to get an "alert" when a process dies

    In article ,
    nicc777 wrote:
    :Hi all,
    :
    :Even though I use Linux a lot, I have not been doing a lot of hard
    :core Linux sysadmin stuff for some time, hence the question here :-)
    :
    :Is there a way to trigger an event (like running a script) when a
    rocess dies? Perhaps even a script to restart a process if it detects
    :that it's dead?
    :
    :I have been using cron for monitoring processes up to now, but the
    :monitoring interval is 1 minute apart. I have a project (streaming of
    :nature) that should not go down, but the problem is that sometimes it
    :dies and it takes up to a minute before the traditional monitoring
    :script will restart it.
    :
    :Any ideas?

    Can't you just run your process from a simple script that loops,
    restarting your process whenever it terminates:

    while :
    do
    myprocess [args ... ]
    done

    Alternatively, if this process is one that you want to start when the
    system boots then you could run it directly from /etc/inittab, using
    "respawn" in the action field. If you do that, you'll probably want
    to invoke your process via 'su' or else it will be running with root
    privileges.

    --
    Bob Nichols AT comcast.net I am "RNichols42"

  11. Re: How to get an "alert" when a process dies

    Robert Nichols wrote:
    > In article ,
    > nicc777 wrote:
    > :Hi all,
    > :
    > :Even though I use Linux a lot, I have not been doing a lot of hard
    > :core Linux sysadmin stuff for some time, hence the question here :-)
    > :
    > :Is there a way to trigger an event (like running a script) when a
    > rocess dies? Perhaps even a script to restart a process if it detects
    > :that it's dead?
    > :
    > :I have been using cron for monitoring processes up to now, but the
    > :monitoring interval is 1 minute apart. I have a project (streaming of
    > :nature) that should not go down, but the problem is that sometimes it
    > :dies and it takes up to a minute before the traditional monitoring
    > :script will restart it.
    > :
    > :Any ideas?
    >
    > Can't you just run your process from a simple script that loops,
    > restarting your process whenever it terminates:
    >
    > while :
    > do
    > myprocess [args ... ]
    > done
    >
    > Alternatively, if this process is one that you want to start when the
    > system boots then you could run it directly from /etc/inittab, using
    > "respawn" in the action field. If you do that, you'll probably want
    > to invoke your process via 'su' or else it will be running with root
    > privileges.
    >


    Put a pause in there, of say a second, your script is going to hoover up
    whatever CPU is available for itself.

  12. Re: How to get an "alert" when a process dies

    On Jun 30, 12:36*pm, Ralf Fassel wrote:
    > * nicc777
    > | Is there a way to trigger an event (like running a script) when a
    > | process dies? Perhaps even a script to restart a process if it detects
    > | that it's dead?
    >
    > * #!/bin/sh
    > * while true ; do
    > * * *run_your_process
    > * * *status=$?
    > * * *case $status in
    > * * * *0) : probably ok? ; break ;;
    > * * * **) echo "process has exited with status $status, re-run"
    > * * *esac
    > * donex
    >
    > ?
    >
    > R'


    Cool - not what I expected, but it does work perfectly :-)

    I think I was making the problem to complicated.

    Thanks a mil

  13. Re: How to get an "alert" when a process dies

    Thanks for all the replies - really helpful!

  14. Re: How to get an "alert" when a process dies

    In article <486902DF.3040302@gmail.com>,
    Nico Kadel-Garcia wrote:
    :Robert Nichols wrote:
    :> In article ,
    :> nicc777 wrote:
    :> :Hi all,
    :> :
    :> :Even though I use Linux a lot, I have not been doing a lot of hard
    :> :core Linux sysadmin stuff for some time, hence the question here :-)
    :> :
    :> :Is there a way to trigger an event (like running a script) when a
    :> rocess dies? Perhaps even a script to restart a process if it detects
    :> :that it's dead?
    :> :
    :> :I have been using cron for monitoring processes up to now, but the
    :> :monitoring interval is 1 minute apart. I have a project (streaming of
    :> :nature) that should not go down, but the problem is that sometimes it
    :> :dies and it takes up to a minute before the traditional monitoring
    :> :script will restart it.
    :> :
    :> :Any ideas?
    :>
    :> Can't you just run your process from a simple script that loops,
    :> restarting your process whenever it terminates:
    :>
    :> while :
    :> do
    :> myprocess [args ... ]
    :> done
    :>
    :> Alternatively, if this process is one that you want to start when the
    :> system boots then you could run it directly from /etc/inittab, using
    :> "respawn" in the action field. If you do that, you'll probably want
    :> to invoke your process via 'su' or else it will be running with root
    :> privileges.
    :>
    :
    :Put a pause in there, of say a second, your script is going to hoover up
    :whatever CPU is available for itself.

    Curious. I've got a couple of scripts that basically look like that,
    and the shell obediently waits, consuming no CPU whatever. It will use
    CPU time only if 'myprocess' keeps terminating immediately. And, if
    that process is as vital as the OP suggests, having a shell sitting in a
    loop trying to restart it isn't a big problem, though flooding some log
    file with error messages could indeed be nasty.

    Anyway, my suggestion wasn't really intended as a complete solution.
    For bulletproofing, you'd want to check whether the program you are
    trying to invoke exists and is executable, and limit the restart
    rate if it keeps dying immediately. Something like:

    #!/bin/bash
    PROG=/bin/true
    N=0
    while :
    do
    if [ ! -x $PROG ]; then
    echo "$PROG is not executable" >&2
    exit 1
    fi
    if [ $N -lt $SECONDS ]; then
    N=$SECONDS
    elif [ $(($N - 5)) -ge $SECONDS ]; then
    echo "$PROG restarting too frequently -- sleeping" >&2
    sleep 60
    fi
    N=$(($N+1))
    $PROG [args ... ]
    done

    --
    Bob Nichols AT comcast.net I am "RNichols42"

  15. Re: How to get an "alert" when a process dies

    On Jul 1, 2:16*am, Robert Nichols
    wrote:
    > In article <486902DF.3040...@gmail.com>,
    > Nico Kadel-Garcia * wrote::Robert Nichols wrote:
    >
    > :> In article ,
    > :> nicc777 * wrote:
    > :> :Hi all,
    > :> :
    > :> :Even though I use Linux a lot, I have not been doing a lot of hard
    > :> :core Linux sysadmin stuff for some time, hence the question here :-)
    > :> :
    > :> :Is there a way to trigger an event (like running a script) when a
    > :> rocess dies? Perhaps even a script to restart a process if it detects
    > :> :that it's dead?
    > :> :
    > :> :I have been using cron for monitoring processes up to now, but the
    > :> :monitoring interval is 1 minute apart. I have a project (streaming of
    > :> :nature) that should not go down, but the problem is that sometimes it
    > :> :dies and it takes up to a minute before the traditional monitoring
    > :> :script will restart it.
    > :> :
    > :> :Any ideas?
    > :>
    > :> Can't you just run your process from a simple script that loops,
    > :> restarting your process whenever it terminates:
    > :>
    > :> * * * *while :
    > :> * * * *do
    > :> * * * * * *myprocess [args ... ]
    > :> * * * *done
    > :>
    > :> Alternatively, if this process is one that you want to start when the
    > :> system boots then you could run it directly from /etc/inittab, using
    > :> "respawn" in the action field. *If you do that, you'll probably want
    > :> to invoke your process via 'su' or else it will be running with root
    > :> privileges.
    > :>
    > :
    > :Put a pause in there, of say a second, your script is going to hoover up
    > :whatever CPU is available for itself.
    >
    > Curious. *I've got a couple of scripts that basically look like that,
    > and the shell obediently waits, consuming no CPU whatever. *It will use
    > CPU time only if 'myprocess' keeps terminating immediately. *And, if
    > that process is as vital as the OP suggests, having a shell sitting in a
    > loop trying to restart it isn't a big problem, though flooding some log
    > file with error messages could indeed be nasty.
    >
    > Anyway, my suggestion wasn't really intended as a complete solution.
    > For bulletproofing, you'd want to check whether the program you are
    > trying to invoke exists and is executable, and limit the restart
    > rate if it keeps dying immediately. *Something like:
    >
    > * * #!/bin/bash
    > * * PROG=/bin/true
    > * * N=0
    > * * while :
    > * * do
    > * * * * if [ ! -x $PROG ]; then
    > * * * * * * echo "$PROG is not executable" >&2
    > * * * * * * exit 1
    > * * * * fi
    > * * * * if [ $N -lt $SECONDS ]; then
    > * * * * * * N=$SECONDS
    > * * * * elif [ $(($N - 5)) -ge $SECONDS ]; then
    > * * * * * * echo "$PROG restarting too frequently -- sleeping">&2
    > * * * * * * sleep 60
    > * * * * fi
    > * * * * N=$(($N+1))
    > * * * * $PROG [args ... ]
    > * * done
    >
    > --
    > Bob Nichols * * * * AT comcast.net I am "RNichols42"


    Ahh - this is excellent as well. Thanks a mil!

  16. Re: How to get an "alert" when a process dies

    On Jun 29, 10:35 am, nicc777 wrote:
    > Is there a way to trigger an event (like running a script) when a
    > process dies? Perhaps even a script to restart a process if it detects
    > that it's dead?


    you want any kind of exit code? You may try logger tool...

    ex:

    updatedb || logger -t updatedb -- Process Terminated &

    You will get a message at your log's (usualy /var/log/*) or you may
    specify a socket to send packets to (see man logger). And you may also
    change the logging priority (see man again).

  17. Re: How to get an "alert" when a process dies

    On Jul 1, 12:38*pm, pedroarthur.j...@gmail.com wrote:
    > On Jun 29, 10:35 am, nicc777 wrote:
    >
    > > Is there a way to trigger an event (like running a script) when a
    > > process dies? Perhaps even a script to restart a process if it detects
    > > that it's dead?

    >
    > you want any kind of exit code? You may try logger tool...
    >
    > ex:
    >
    > updatedb || logger -t updatedb -- Process Terminated &
    >
    > You will get a message at your log's (usualy /var/log/*) or you may
    > specify a socket to send packets to (see man logger). And you may also
    > change the logging priority (see man again).




    Thanks - it's already in my script :-)


  18. Re: How to get an "alert" when a process dies

    On 06/30/2008 04:45 PM, nicc777 wrote:
    > On Jun 30, 12:36 pm, Ralf Fassel wrote:
    >> * nicc777
    >> | Is there a way to trigger an event (like running a script) when a
    >> | process dies? Perhaps even a script to restart a process if it detects
    >> | that it's dead?
    >>
    >> #!/bin/sh
    >> while true ; do
    >> run_your_process
    >> status=$?
    >> case $status in
    >> 0) : probably ok? ; break ;;
    >> *) echo "process has exited with status $status, re-run"
    >> esac
    >> donex
    >>
    >> ?
    >>
    >> R'

    >
    > Cool - not what I expected, but it does work perfectly :-)
    >
    > I think I was making the problem to complicated.
    >
    > Thanks a mil


    In /etc/inittab:
    yp:2345:respawn:/your/bin/script

    Don't re-invent the wheel when you don't have to.

    Doug

  19. Re: How to get an "alert" when a process dies

    nicc777 wrote:
    > Hi all,
    >
    > Even though I use Linux a lot, I have not been doing a lot of hard
    > core Linux sysadmin stuff for some time, hence the question here :-)
    >
    > Is there a way to trigger an event (like running a script) when a
    > process dies? Perhaps even a script to restart a process if it detects
    > that it's dead?


    For your purposes, start the program from inittab. The init process will
    relaunch if it dies.

    >
    > I have been using cron for monitoring processes up to now, but the
    > monitoring interval is 1 minute apart. I have a project (streaming of
    > nature) that should not go down, but the problem is that sometimes it
    > dies and it takes up to a minute before the traditional monitoring
    > script will restart it.
    >
    > Any ideas?
    >
    > Thanks - Nico.


  20. Re: How to get an "alert" when a process dies

    In article ,
    nicc777 wrote:
    >Hi all,
    >
    >Even though I use Linux a lot, I have not been doing a lot of hard
    >core Linux sysadmin stuff for some time, hence the question here :-)
    >
    >Is there a way to trigger an event (like running a script) when a
    >process dies? Perhaps even a script to restart a process if it detects
    >that it's dead?
    >
    >I have been using cron for monitoring processes up to now, but the
    >monitoring interval is 1 minute apart. I have a project (streaming of
    >nature) that should not go down, but the problem is that sometimes it
    >dies and it takes up to a minute before the traditional monitoring
    >script will restart it.
    >
    >Any ideas?
    >
    >Thanks - Nico.



    If you know the pid of your process, won't the "wait" shell
    command do what you want, without any sleep/polling?

    --Ken

    --
    Ken R. Dye an optimist is a guy |
    Chicago, Illinois that has never had |
    www.geocities.com/MotorCity/Track/8746 much experience |
    dye1146 at sbcglobal dot net archy |

+ Reply to Thread
Page 1 of 2 1 2 LastLast