Hi all,

Even though I use Linux a lot, I have not been doing a lot of hard
core Linux sysadmin stuff for some time, hence the question here :-)

Is there a way to trigger an event (like running a script) when a
process dies? Perhaps even a script to restart a process if it detects
that it's dead?

I have been using cron for monitoring processes up to now, but the
monitoring interval is 1 minute apart. I have a project (streaming of
nature) that should not go down, but the problem is that sometimes it
dies and it takes up to a minute before the traditional monitoring
script will restart it.

Any ideas?

Thanks - Nico.

nicc777 wrote:
> Hi all,
>
> Even though I use Linux a lot, I have not been doing a lot of hard
> core Linux sysadmin stuff for some time, hence the question here :-)
>
> Is there a way to trigger an event (like running a script) when a
> process dies? Perhaps even a script to restart a process if it detects
> that it's dead?
>
> I have been using cron for monitoring processes up to now, but the
> monitoring interval is 1 minute apart. I have a project (streaming of
> nature) that should not go down, but the problem is that sometimes it
> dies and it takes up to a minute before the traditional monitoring
> script will restart it.
>
> Any ideas?
>
> Thanks - Nico.

Nagios.

Nico, too

Nico Kadel-Garcia :

> nicc777 wrote:
>> Hi all,
>>
>> Even though I use Linux a lot, I have not been doing a lot of hard
>> core Linux sysadmin stuff for some time, hence the question here :-)
>>
>> Is there a way to trigger an event (like running a script) when a
>> process dies? Perhaps even a script to restart a process if it detects
>> that it's dead?
>>
>> I have been using cron for monitoring processes up to now, but the
>> monitoring interval is 1 minute apart. I have a project (streaming of
>> nature) that should not go down, but the problem is that sometimes it
>> dies and it takes up to a minute before the traditional monitoring
>> script will restart it.
>>
>> Any ideas?
>>
>> Thanks - Nico.
>
> Nagios.

Suffers from the very problem, the OP was trying to avoid: A quite great
delay between the death of the specific process and the monitoring tool
running the check and thus noticing the dead process. Iirc, nagios uses
even higher intervals than one minute between checks.

I guess, the OP wants something like inotify for processes: Passive
checking, yielding an _immediate_ notification about state changes, without
the unavoidable latency of active checks as nagios performs them.

--
Freedom is always the freedom of dissenters.
(Rosa Luxemburg)

nicc777 wrote:

> Hi all,
>
> Even though I use Linux a lot, I have not been doing a lot of hard
> core Linux sysadmin stuff for some time, hence the question here :-)
>
> Is there a way to trigger an event (like running a script) when a
> process dies? Perhaps even a script to restart a process if it detects
> that it's dead?
>
> I have been using cron for monitoring processes up to now, but the
> monitoring interval is 1 minute apart. I have a project (streaming of
> nature) that should not go down, but the problem is that sometimes it
> dies and it takes up to a minute before the traditional monitoring
> script will restart it.

Totally untested, but it should get your creative juices flowing:

#!/bin/bash
PROC=someprocess # The process to check
WAIT=10 # Seconds to wait between checks

while [ 1 ]
do
ps -ef|grep $PROC &>/dev/null
if [ $? -ne 0 ]; then
# grep didn't find "$PROC"...dead?
/etc/init.d/$PROC restart &>/dev/null

# Send some notification saying what happened.
# Maybe re-run the process check??
fi

# Now sleep and re-check the process after a specified delay
sleep $WAIT
done


Now all you need to do, is start that, fork it to the background and voila.
Add whatever notification method you want in the "if...fi" block. This
script will run until you (or something else) kills it by virtue of
the "while" loop.

If you really wanted to get fancy and check a few processes, change "PROC"
into an array and iterate through it, or use a "for PROC in proc1 proc2
etc" type construct. There are many ways to extend this, but whatever you
need to do can be based on this to provide sub-minute scheduling.

Cheers,

James
--
It's lucky you're going so slowly, because you're going in the wrong
direction.

On Jun 30, 1:25*am, "Sebastian \"lunar\" Wiesner"
wrote:
> Nico Kadel-Garcia :
>
>
>
> > nicc777 wrote:
> >> Hi all,
>
> >> Even though I use Linux a lot, I have not been doing a lot of hard
> >> core Linux sysadmin stuff for some time, hence the question here :-)
>
> >> Is there a way to trigger an event (like running a script) when a
> >> process dies? Perhaps even a script to restart a process if it detects
> >> that it's dead?
>
> >> I have been using cron for monitoring processes up to now, but the
> >> monitoring interval is 1 minute apart. I have a project (streaming of
> >> nature) that should not go down, but the problem is that sometimes it
> >> dies and it takes up to a minute before the traditional monitoring
> >> script will restart it.
>
> >> Any ideas?
>
> >> Thanks - Nico.
>
> > Nagios.
>
> Suffers from the very problem, the OP was trying to avoid: *A quite great
> delay between the death of the specific process and the monitoring tool
> running the check and thus noticing the dead process. *Iirc, nagios uses
> even higher intervals than one minute between checks.
>
> I guess, the OP wants something like inotify for processes: *Passive
> checking, yielding an _immediate_ notification about state changes, without
> the unavoidable latency of active checks as nagios performs them.
>
> --
> Freedom is always the freedom of dissenters.
> * * * * * * * * * * * * * * * * * * * (Rosa Luxemburg)

Exactly - Time delays is what I try to get away from. Even seconds can
mean potential customers leaving your service because there's no
service :-(

Thanks for explaining my problem more clearly !

On Jun 30, 5:03*am, Centurion wrote:
> nicc777 wrote:
> > Hi all,
>
> > Even though I use Linux a lot, I have not been doing a lot of hard
> > core Linux sysadmin stuff for some time, hence the question here :-)
>
> > Is there a way to trigger an event (like running a script) when a
> > process dies? Perhaps even a script to restart a process if it detects
> > that it's dead?
>
> > I have been using cron for monitoring processes up to now, but the
> > monitoring interval is 1 minute apart. I have a project (streaming of
> > nature) that should not go down, but the problem is that sometimes it
> > dies and it takes up to a minute before the traditional monitoring
> > script will restart it.
>
> Totally untested, but it should get your creative juices flowing:
>
> #!/bin/bash
> PROC=someprocess *# The process to check
> WAIT=10 * * * * * # Seconds to wait between checks
>
> while [ 1 ]
> do
> * * * * ps -ef|grep $PROC &>/dev/null
> * * * * if [ $? -ne 0 ]; then
> * * * * * * * * # grep didn't find "$PROC"...dead?
> * * * * * * * * /etc/init.d/$PROC restart &>/dev/null
>
> * * * * * * * * # Send some notification saying what happened.
> * * * * * * * * # Maybe re-run the process check??
> * * * * fi
>
> * * * * # Now sleep and re-check the process after a specified delay
> * * * * sleep $WAIT
> done
>
> Now all you need to do, is start that, fork it to the background and voila.
> Add whatever notification method you want in the "if...fi" block. *This
> script will run until you (or something else) kills it by virtue of
> the "while" loop.
>
> If you really wanted to get fancy and check a few processes, change "PROC"
> into an array and iterate through it, or use a "for PROC in proc1 proc2
> etc" type construct. *There are many ways to extend this, but whatever you
> need to do can be based on this to provide sub-minute scheduling.
>
> Cheers,
>
> James
> --
> It's lucky you're going so slowly, because you're going in the wrong
> direction.

Thanks a lot - I came to the same conclusion using another scripting
language, but I still sit with a 1 second delay - which I assume is
sufficient for my current scenario.

Still - I would like to know if there is a real time notification
mechanism.

Does any body know if it's possible to hook something into the Kernel
(custom module) maybe? My thinking is maybe something that can "catch"
signals on the kernel level. Thanks again :-)

nicc777 writes:

> Is there a way to trigger an event (like running a script) when a
> process dies? Perhaps even a script to restart a process if it detects
> that it's dead?
>
> I have been using cron for monitoring processes up to now, but the
> monitoring interval is 1 minute apart. I have a project (streaming of
> nature) that should not go down, but the problem is that sometimes it
> dies and it takes up to a minute before the traditional monitoring
> script will restart it.

See http://upstart.ubuntu.com

In the end you are looking for some sort of process supervisor.

* nicc777
| Is there a way to trigger an event (like running a script) when a
| process dies? Perhaps even a script to restart a process if it detects
| that it's dead?

#!/bin/sh
while true ; do
run_your_process
status=$?
case $status in
0) : probably ok? ; break ;;
*) echo "process has exited with status $status, re-run"
esac
donex

?

R'

nicc777 wrote:
> On Jun 30, 1:25 am, "Sebastian \"lunar\" Wiesner"
> wrote:
>> Nico Kadel-Garcia :
>>
>>
>>
>>> nicc777 wrote:
>>>> Hi all,
>>>> Even though I use Linux a lot, I have not been doing a lot of hard
>>>> core Linux sysadmin stuff for some time, hence the question here :-)
>>>> Is there a way to trigger an event (like running a script) when a
>>>> process dies? Perhaps even a script to restart a process if it detects
>>>> that it's dead?
>>>> I have been using cron for monitoring processes up to now, but the
>>>> monitoring interval is 1 minute apart. I have a project (streaming of
>>>> nature) that should not go down, but the problem is that sometimes it
>>>> dies and it takes up to a minute before the traditional monitoring
>>>> script will restart it.
>>>> Any ideas?
>>>> Thanks - Nico.
>>> Nagios.
>> Suffers from the very problem, the OP was trying to avoid: A quite great
>> delay between the death of the specific process and the monitoring tool
>> running the check and thus noticing the dead process. Iirc, nagios uses
>> even higher intervals than one minute between checks.
>>
>> I guess, the OP wants something like inotify for processes: Passive
>> checking, yielding an _immediate_ notification about state changes, without
>> the unavoidable latency of active checks as nagios performs them.
>>
>> --
>> Freedom is always the freedom of dissenters.
>> (Rosa Luxemburg)
>
> Exactly - Time delays is what I try to get away from. Even seconds can
> mean potential customers leaving your service because there's no
> service :-(
>
> Thanks for explaining my problem more clearly !

OK. You need a monitor task that checks much more frequently. A cron script
could assure that the *MONITOR* is up and running, but the monitor could be a
simple shell script with a built-in 'restart if not working' setup.

In article ,
nicc777 wrote:
:Hi all,
:
:Even though I use Linux a lot, I have not been doing a lot of hard
:core Linux sysadmin stuff for some time, hence the question here :-)
:
:Is there a way to trigger an event (like running a script) when a
rocess dies? Perhaps even a script to restart a process if it detects
:that it's dead?
:
:I have been using cron for monitoring processes up to now, but the
:monitoring interval is 1 minute apart. I have a project (streaming of
:nature) that should not go down, but the problem is that sometimes it
:dies and it takes up to a minute before the traditional monitoring
:script will restart it.
:
:Any ideas?

Can't you just run your process from a simple script that loops,
restarting your process whenever it terminates:

while :
do
myprocess [args ... ]
done

Alternatively, if this process is one that you want to start when the
system boots then you could run it directly from /etc/inittab, using
"respawn" in the action field. If you do that, you'll probably want
to invoke your process via 'su' or else it will be running with root
privileges.

--
Bob Nichols AT comcast.net I am "RNichols42"

Robert Nichols wrote:
> In article ,
> nicc777 wrote:
> :Hi all,
> :
> :Even though I use Linux a lot, I have not been doing a lot of hard
> :core Linux sysadmin stuff for some time, hence the question here :-)
> :
> :Is there a way to trigger an event (like running a script) when a
> rocess dies? Perhaps even a script to restart a process if it detects
> :that it's dead?
> :
> :I have been using cron for monitoring processes up to now, but the
> :monitoring interval is 1 minute apart. I have a project (streaming of
> :nature) that should not go down, but the problem is that sometimes it
> :dies and it takes up to a minute before the traditional monitoring
> :script will restart it.
> :
> :Any ideas?
>
> Can't you just run your process from a simple script that loops,
> restarting your process whenever it terminates:
>
> while :
> do
> myprocess [args ... ]
> done
>
> Alternatively, if this process is one that you want to start when the
> system boots then you could run it directly from /etc/inittab, using
> "respawn" in the action field. If you do that, you'll probably want
> to invoke your process via 'su' or else it will be running with root
> privileges.
>

Put a pause in there, of say a second, your script is going to hoover up
whatever CPU is available for itself.

On Jun 30, 12:36*pm, Ralf Fassel wrote:
> * nicc777
> | Is there a way to trigger an event (like running a script) when a
> | process dies? Perhaps even a script to restart a process if it detects
> | that it's dead?
>
> * #!/bin/sh
> * while true ; do
> * * *run_your_process
> * * *status=$?
> * * *case $status in
> * * * *0) : probably ok? ; break ;;
> * * * **) echo "process has exited with status $status, re-run"
> * * *esac
> * donex
>
> ?
>
> R'

Cool - not what I expected, but it does work perfectly :-)

I think I was making the problem to complicated.

Thanks a mil

Thanks for all the replies - really helpful!

In article <486902DF.3040302@gmail.com>,
Nico Kadel-Garcia wrote:
:Robert Nichols wrote:
:> In article ,
:> nicc777 wrote:
:> :Hi all,
:> :
:> :Even though I use Linux a lot, I have not been doing a lot of hard
:> :core Linux sysadmin stuff for some time, hence the question here :-)
:> :
:> :Is there a way to trigger an event (like running a script) when a
:> rocess dies? Perhaps even a script to restart a process if it detects
:> :that it's dead?
:> :
:> :I have been using cron for monitoring processes up to now, but the
:> :monitoring interval is 1 minute apart. I have a project (streaming of
:> :nature) that should not go down, but the problem is that sometimes it
:> :dies and it takes up to a minute before the traditional monitoring
:> :script will restart it.
:> :
:> :Any ideas?
:>
:> Can't you just run your process from a simple script that loops,
:> restarting your process whenever it terminates:
:>
:> while :
:> do
:> myprocess [args ... ]
:> done
:>
:> Alternatively, if this process is one that you want to start when the
:> system boots then you could run it directly from /etc/inittab, using
:> "respawn" in the action field. If you do that, you'll probably want
:> to invoke your process via 'su' or else it will be running with root
:> privileges.
:>
:
:Put a pause in there, of say a second, your script is going to hoover up
:whatever CPU is available for itself.

Curious. I've got a couple of scripts that basically look like that,
and the shell obediently waits, consuming no CPU whatever. It will use
CPU time only if 'myprocess' keeps terminating immediately. And, if
that process is as vital as the OP suggests, having a shell sitting in a
loop trying to restart it isn't a big problem, though flooding some log
file with error messages could indeed be nasty.

Anyway, my suggestion wasn't really intended as a complete solution.
For bulletproofing, you'd want to check whether the program you are
trying to invoke exists and is executable, and limit the restart
rate if it keeps dying immediately. Something like:

#!/bin/bash
PROG=/bin/true
N=0
while :
do
if [ ! -x $PROG ]; then
echo "$PROG is not executable" >&2
exit 1
fi
if [ $N -lt $SECONDS ]; then
N=$SECONDS
elif [ $(($N - 5)) -ge $SECONDS ]; then
echo "$PROG restarting too frequently -- sleeping" >&2
sleep 60
fi
N=$(($N+1))
$PROG [args ... ]
done

--
Bob Nichols AT comcast.net I am "RNichols42"

On Jul 1, 2:16*am, Robert Nichols
wrote:
> In article <486902DF.3040...@gmail.com>,
> Nico Kadel-Garcia * wrote::Robert Nichols wrote:
>
> :> In article ,
> :> nicc777 * wrote:
> :> :Hi all,
> :> :
> :> :Even though I use Linux a lot, I have not been doing a lot of hard
> :> :core Linux sysadmin stuff for some time, hence the question here :-)
> :> :
> :> :Is there a way to trigger an event (like running a script) when a
> :> rocess dies? Perhaps even a script to restart a process if it detects
> :> :that it's dead?
> :> :
> :> :I have been using cron for monitoring processes up to now, but the
> :> :monitoring interval is 1 minute apart. I have a project (streaming of
> :> :nature) that should not go down, but the problem is that sometimes it
> :> :dies and it takes up to a minute before the traditional monitoring
> :> :script will restart it.
> :> :
> :> :Any ideas?
> :>
> :> Can't you just run your process from a simple script that loops,
> :> restarting your process whenever it terminates:
> :>
> :> * * * *while :
> :> * * * *do
> :> * * * * * *myprocess [args ... ]
> :> * * * *done
> :>
> :> Alternatively, if this process is one that you want to start when the
> :> system boots then you could run it directly from /etc/inittab, using
> :> "respawn" in the action field. *If you do that, you'll probably want
> :> to invoke your process via 'su' or else it will be running with root
> :> privileges.
> :>
> :
> :Put a pause in there, of say a second, your script is going to hoover up
> :whatever CPU is available for itself.
>
> Curious. *I've got a couple of scripts that basically look like that,
> and the shell obediently waits, consuming no CPU whatever. *It will use
> CPU time only if 'myprocess' keeps terminating immediately. *And, if
> that process is as vital as the OP suggests, having a shell sitting in a
> loop trying to restart it isn't a big problem, though flooding some log
> file with error messages could indeed be nasty.
>
> Anyway, my suggestion wasn't really intended as a complete solution.
> For bulletproofing, you'd want to check whether the program you are
> trying to invoke exists and is executable, and limit the restart
> rate if it keeps dying immediately. *Something like:
>
> * * #!/bin/bash
> * * PROG=/bin/true
> * * N=0
> * * while :
> * * do
> * * * * if [ ! -x $PROG ]; then
> * * * * * * echo "$PROG is not executable" >&2
> * * * * * * exit 1
> * * * * fi
> * * * * if [ $N -lt $SECONDS ]; then
> * * * * * * N=$SECONDS
> * * * * elif [ $(($N - 5)) -ge $SECONDS ]; then
> * * * * * * echo "$PROG restarting too frequently -- sleeping">&2
> * * * * * * sleep 60
> * * * * fi
> * * * * N=$(($N+1))
> * * * * $PROG [args ... ]
> * * done
>
> --
> Bob Nichols * * * * AT comcast.net I am "RNichols42"

Ahh - this is excellent as well. Thanks a mil!

On Jun 29, 10:35 am, nicc777 wrote:
> Is there a way to trigger an event (like running a script) when a
> process dies? Perhaps even a script to restart a process if it detects
> that it's dead?

you want any kind of exit code? You may try logger tool...

ex:

updatedb || logger -t updatedb -- Process Terminated &

You will get a message at your log's (usualy /var/log/*) or you may
specify a socket to send packets to (see man logger). And you may also
change the logging priority (see man again).

On Jul 1, 12:38*pm, pedroarthur.j...@gmail.com wrote:
> On Jun 29, 10:35 am, nicc777 wrote:
>
> > Is there a way to trigger an event (like running a script) when a
> > process dies? Perhaps even a script to restart a process if it detects
> > that it's dead?
>
> you want any kind of exit code? You may try logger tool...
>
> ex:
>
> updatedb || logger -t updatedb -- Process Terminated &
>
> You will get a message at your log's (usualy /var/log/*) or you may
> specify a socket to send packets to (see man logger). And you may also
> change the logging priority (see man again).



Thanks - it's already in my script :-)

On 06/30/2008 04:45 PM, nicc777 wrote:
> On Jun 30, 12:36 pm, Ralf Fassel wrote:
>> * nicc777
>> | Is there a way to trigger an event (like running a script) when a
>> | process dies? Perhaps even a script to restart a process if it detects
>> | that it's dead?
>>
>> #!/bin/sh
>> while true ; do
>> run_your_process
>> status=$?
>> case $status in
>> 0) : probably ok? ; break ;;
>> *) echo "process has exited with status $status, re-run"
>> esac
>> donex
>>
>> ?
>>
>> R'
>
> Cool - not what I expected, but it does work perfectly :-)
>
> I think I was making the problem to complicated.
>
> Thanks a mil

In /etc/inittab:
yp:2345:respawn:/your/bin/script

Don't re-invent the wheel when you don't have to.

Doug

nicc777 wrote:
> Hi all,
>
> Even though I use Linux a lot, I have not been doing a lot of hard
> core Linux sysadmin stuff for some time, hence the question here :-)
>
> Is there a way to trigger an event (like running a script) when a
> process dies? Perhaps even a script to restart a process if it detects
> that it's dead?

For your purposes, start the program from inittab. The init process will
relaunch if it dies.

>
> I have been using cron for monitoring processes up to now, but the
> monitoring interval is 1 minute apart. I have a project (streaming of
> nature) that should not go down, but the problem is that sometimes it
> dies and it takes up to a minute before the traditional monitoring
> script will restart it.
>
> Any ideas?
>
> Thanks - Nico.

In article ,
nicc777 wrote:
>Hi all,
>
>Even though I use Linux a lot, I have not been doing a lot of hard
>core Linux sysadmin stuff for some time, hence the question here :-)
>
>Is there a way to trigger an event (like running a script) when a
>process dies? Perhaps even a script to restart a process if it detects
>that it's dead?
>
>I have been using cron for monitoring processes up to now, but the
>monitoring interval is 1 minute apart. I have a project (streaming of
>nature) that should not go down, but the problem is that sometimes it
>dies and it takes up to a minute before the traditional monitoring
>script will restart it.
>
>Any ideas?
>
>Thanks - Nico.


If you know the pid of your process, won't the "wait" shell
command do what you want, without any sleep/polling?

--Ken

--
Ken R. Dye an optimist is a guy |
Chicago, Illinois that has never had |
www.geocities.com/MotorCity/Track/8746 much experience |
dye1146 at sbcglobal dot net archy |