pb with remote command cleanup - SSH

This is a discussion on pb with remote command cleanup - SSH ; Hi, I have a cleanup problem when I remote execute a command using ssh. If I kill ssh client during remote execution (with ctrl-C or by closing the terminal) then remote command still exists. Here is what's happening : [mat@myfirsthost ...

+ Reply to Thread
Results 1 to 12 of 12

Thread: pb with remote command cleanup

  1. pb with remote command cleanup

    Hi,

    I have a cleanup problem when I remote execute a command using ssh. If I
    kill ssh client during remote execution (with ctrl-C or by closing the
    terminal) then remote command still exists.

    Here is what's happening :

    [mat@myfirsthost ~]$ ssh mysecondhost sleep 3600
    Killed by signal 2.
    [mat@myfirsthost ~]$ ssh mysecondhost ps -fu mat | grep sleep
    mat 3570 1 0 22:04 ? 00:00:00 sleep 3600
    mat 3685 3492 0 22:09 pts/1 00:00:00 grep sleep


    Is there a way for sshd to kill all childs when connection is
    closed/lost ? Maybe there is an option in sshd_config that can do it but
    I didn't find any.

    To cope with this probleme, I can use "-t" to force remote tty, and then
    child are killed but I think it's not a good solution. Furthermore, I
    can't assure that all users do remote execution using this parameter. As
    a result, there is a lot of useless resident program that run on hosts
    and use cpu cycles for nothing.


    Thank you in advance

    Mat

  2. Re: pb with remote command cleanup

    Mat wrote:
    > I have a cleanup problem when I remote execute a command using ssh. If I
    > kill ssh client during remote execution (with ctrl-C or by closing the
    > terminal) then remote command still exists.


    > Here is what's happening :


    > [mat@myfirsthost ~]$ ssh mysecondhost sleep 3600
    > Killed by signal 2.
    > [mat@myfirsthost ~]$ ssh mysecondhost ps -fu mat | grep sleep
    > mat 3570 1 0 22:04 ? 00:00:00 sleep 3600
    > mat 3685 3492 0 22:09 pts/1 00:00:00 grep sleep



    > Is there a way for sshd to kill all childs when connection is
    > closed/lost ? Maybe there is an option in sshd_config that can do it but
    > I didn't find any.


    You can set keepalives (like ClientAliveInterval/ClientAliveCountMax).
    If the client isn't heard from, then it will poll and eventually kill
    the connection if the client doesn't respond.

    Don't set them too low or it will kill the connection even during
    temporary network outages.

    Most idle sshd processes don't take up much resources (certainly not
    much CPU), so I'd normally set the time limit several hours or greater.
    Your environment may demand otherwise.

    --
    Darren Dunham ddunham@taos.com
    Senior Technical Consultant TAOS http://www.taos.com/
    Got some Dr Pepper? San Francisco, CA bay area
    < This line left intentionally blank to confuse you. >

  3. Re: pb with remote command cleanup

    Thank you for your help but my problem is not a connection problem. I'm
    worrying about the remaining command that is still running on the remote
    side after connection lost.

    Keepalives enable me to kill connection sooner or later but doesn't
    guarantee any remote command destruction and that is the real problem :
    if a user launch, using ssh, a remote process that is CPU intensive and
    then kill his terminal (or hit ctrl-C), the CPU intensive task is still
    running on the remote side despite the lost of the connection.

    I'm looking for a way to handle remote command destruction after
    connection lost ? Maybe an sshd_config parameter ? Maybe a patch ?

    Thanks

    Darren Dunham a écrit :
    > Mat wrote:
    >> I have a cleanup problem when I remote execute a command using ssh. If I
    >> kill ssh client during remote execution (with ctrl-C or by closing the
    >> terminal) then remote command still exists.

    >
    >> Here is what's happening :

    >
    >> [mat@myfirsthost ~]$ ssh mysecondhost sleep 3600
    >> Killed by signal 2.
    >> [mat@myfirsthost ~]$ ssh mysecondhost ps -fu mat | grep sleep
    >> mat 3570 1 0 22:04 ? 00:00:00 sleep 3600
    >> mat 3685 3492 0 22:09 pts/1 00:00:00 grep sleep

    >
    >
    >> Is there a way for sshd to kill all childs when connection is
    >> closed/lost ? Maybe there is an option in sshd_config that can do it but
    >> I didn't find any.

    >
    > You can set keepalives (like ClientAliveInterval/ClientAliveCountMax).
    > If the client isn't heard from, then it will poll and eventually kill
    > the connection if the client doesn't respond.
    >
    > Don't set them too low or it will kill the connection even during
    > temporary network outages.
    >
    > Most idle sshd processes don't take up much resources (certainly not
    > much CPU), so I'd normally set the time limit several hours or greater.
    > Your environment may demand otherwise.
    >


  4. Re: pb with remote command cleanup

    Mat wrote:
    > Thank you for your help but my problem is not a connection problem. I'm
    > worrying about the remaining command that is still running on the remote
    > side after connection lost.


    > Keepalives enable me to kill connection sooner or later but doesn't
    > guarantee any remote command destruction and that is the real problem :
    > if a user launch, using ssh, a remote process that is CPU intensive and
    > then kill his terminal (or hit ctrl-C), the CPU intensive task is still
    > running on the remote side despite the lost of the connection.


    Then that is a problem with the command at some level. When the
    connection dies, it is traditional for it to send a HUP (as in when we
    were using modems, a notification that the phone line "hung up") to the
    child process group. However children can choose to ignore signals (see
    'nohup').

    > I'm looking for a way to handle remote command destruction after
    > connection lost ? Maybe an sshd_config parameter ? Maybe a patch ?


    The behavior is to send a signal. Whether the jobs run by the server
    dissociate themselves so the signal is not delivered or choose to ignore
    it is beyond the ability of ssh or other connection mechanisms to deal
    with.

    When the user logs in, presumably they are put into a shell. Does the
    shell exit when the connection dies?

    --
    Darren Dunham ddunham@taos.com
    Senior Technical Consultant TAOS http://www.taos.com/
    Got some Dr Pepper? San Francisco, CA bay area
    < This line left intentionally blank to confuse you. >

  5. Re: pb with remote command cleanup

    In article Darren
    Dunham writes:
    >Mat wrote:
    >> Thank you for your help but my problem is not a connection problem. I'm
    >> worrying about the remaining command that is still running on the remote
    >> side after connection lost.

    >
    >> Keepalives enable me to kill connection sooner or later but doesn't
    >> guarantee any remote command destruction and that is the real problem :
    >> if a user launch, using ssh, a remote process that is CPU intensive and
    >> then kill his terminal (or hit ctrl-C), the CPU intensive task is still
    >> running on the remote side despite the lost of the connection.

    >
    >Then that is a problem with the command at some level. When the
    >connection dies, it is traditional for it to send a HUP (as in when we
    >were using modems, a notification that the phone line "hung up") to the
    >child process group.


    Actually it's the tty driver that sends this signal - the OP already
    said that it worked as he wanted when using 'ssh -t', but he wasn't
    happy with requiring it. I guess it could be reasonable, at least as an
    option, for sshd to take over this part of the tty driver's job when
    running tty-less, but AFAIK it doesn't currently and (AFAIK) there's no
    way to make it do it (other than modifying the source).

    >When the user logs in, presumably they are put into a shell.


    A shell is run, but I don't think it's a good description to say that
    the user is "put into it" when running 'ssh host '. In any case
    that shell will not specifically notice that the connection goes away,
    it has no business even looking at the socket file descriptor (which is
    the stdin/out/err of ).

    --Per Hedeland
    per@hedeland.org

  6. Re: pb with remote command cleanup

    Per Hedeland a écrit :
    > In article Darren
    > Dunham writes:
    >> Mat wrote:
    >>> Thank you for your help but my problem is not a connection problem. I'm
    >>> worrying about the remaining command that is still running on the remote
    >>> side after connection lost.
    >>> Keepalives enable me to kill connection sooner or later but doesn't
    >>> guarantee any remote command destruction and that is the real problem :
    >>> if a user launch, using ssh, a remote process that is CPU intensive and
    >>> then kill his terminal (or hit ctrl-C), the CPU intensive task is still
    >>> running on the remote side despite the lost of the connection.

    >> Then that is a problem with the command at some level. When the
    >> connection dies, it is traditional for it to send a HUP (as in when we
    >> were using modems, a notification that the phone line "hung up") to the
    >> child process group.

    >
    > Actually it's the tty driver that sends this signal - the OP already
    > said that it worked as he wanted when using 'ssh -t', but he wasn't
    > happy with requiring it. I guess it could be reasonable, at least as an
    > option, for sshd to take over this part of the tty driver's job when
    > running tty-less, but AFAIK it doesn't currently and (AFAIK) there's no
    > way to make it do it (other than modifying the source).
    >,


    Thank you for your response, do you know the way to ask for this kind of
    improvement in openSSH ? Maybe I could make it myself but I think I'm
    not the only one to be interested in using such an option and that it
    will be better done by the official developpers.

    Best regards,
    Mat

    >> When the user logs in, presumably they are put into a shell.

    >
    > A shell is run, but I don't think it's a good description to say that
    > the user is "put into it" when running 'ssh host '. In any case
    > that shell will not specifically notice that the connection goes away,
    > it has no business even looking at the socket file descriptor (which is
    > the stdin/out/err of ).
    >
    > --Per Hedeland
    > per@hedeland.org


  7. Re: pb with remote command cleanup

    In article <468014f2$0$31597$426a74cc@news.free.fr> Mat
    writes:
    >Per Hedeland a écrit :
    >> In article Darren
    >> Dunham writes:
    >>> Mat wrote:
    >>>> Thank you for your help but my problem is not a connection problem. I'm
    >>>> worrying about the remaining command that is still running on the remote
    >>>> side after connection lost.
    >>>> Keepalives enable me to kill connection sooner or later but doesn't
    >>>> guarantee any remote command destruction and that is the real problem :
    >>>> if a user launch, using ssh, a remote process that is CPU intensive and
    >>>> then kill his terminal (or hit ctrl-C), the CPU intensive task is still
    >>>> running on the remote side despite the lost of the connection.
    >>> Then that is a problem with the command at some level. When the
    >>> connection dies, it is traditional for it to send a HUP (as in when we
    >>> were using modems, a notification that the phone line "hung up") to the
    >>> child process group.

    >>
    >> Actually it's the tty driver that sends this signal - the OP already
    >> said that it worked as he wanted when using 'ssh -t', but he wasn't
    >> happy with requiring it. I guess it could be reasonable, at least as an
    >> option, for sshd to take over this part of the tty driver's job when
    >> running tty-less, but AFAIK it doesn't currently and (AFAIK) there's no
    >> way to make it do it (other than modifying the source).
    >>,

    >
    >Thank you for your response, do you know the way to ask for this kind of
    >improvement in openSSH ? Maybe I could make it myself but I think I'm
    >not the only one to be interested in using such an option and that it
    >will be better done by the official developpers.


    Well, maybe have a look at http://www.openssh.com/report.html (it
    mentions features too). In general (and mentioned there), I've found
    that suggestions for features/enhancements to open-source software
    stands a significantly greater chance of being accepted if accompanied
    by a well-implemented and tested patch, even if the developers don't use
    the patch exactly as-is. I've no specific experience with OpenSSH in
    that regard, though. Bringing up the subject on one of the mailing lists
    before attempting implementation is probably a good idea too - maybe
    there is some reason the change simply will not be considered.

    N.B. In this particular case, I would guess that a) the default would
    have to remain unchanged, and b) there's probably no way to have the
    client request this behaviour from the server (given that the protocol
    is standardized). I.e. it would presumably have to be a server-side
    config option to enable it - but maybe that fits your bill.

    --Per Hedeland
    per@hedeland.org

  8. Re: pb with remote command cleanup

    Per Hedeland a écrit :
    > In article <468014f2$0$31597$426a74cc@news.free.fr> Mat
    > writes:
    >> Per Hedeland a écrit :
    >>> In article Darren
    >>> Dunham writes:
    >>>> Mat wrote:
    >>>>> Thank you for your help but my problem is not a connection problem. I'm
    >>>>> worrying about the remaining command that is still running on the remote
    >>>>> side after connection lost.
    >>>>> Keepalives enable me to kill connection sooner or later but doesn't
    >>>>> guarantee any remote command destruction and that is the real problem :
    >>>>> if a user launch, using ssh, a remote process that is CPU intensive and
    >>>>> then kill his terminal (or hit ctrl-C), the CPU intensive task is still
    >>>>> running on the remote side despite the lost of the connection.
    >>>> Then that is a problem with the command at some level. When the
    >>>> connection dies, it is traditional for it to send a HUP (as in when we
    >>>> were using modems, a notification that the phone line "hung up") to the
    >>>> child process group.
    >>> Actually it's the tty driver that sends this signal - the OP already
    >>> said that it worked as he wanted when using 'ssh -t', but he wasn't
    >>> happy with requiring it. I guess it could be reasonable, at least as an
    >>> option, for sshd to take over this part of the tty driver's job when
    >>> running tty-less, but AFAIK it doesn't currently and (AFAIK) there's no
    >>> way to make it do it (other than modifying the source).
    >>> ,

    >> Thank you for your response, do you know the way to ask for this kind of
    >> improvement in openSSH ? Maybe I could make it myself but I think I'm
    >> not the only one to be interested in using such an option and that it
    >> will be better done by the official developpers.

    >
    > Well, maybe have a look at http://www.openssh.com/report.html (it
    > mentions features too). In general (and mentioned there), I've found
    > that suggestions for features/enhancements to open-source software
    > stands a significantly greater chance of being accepted if accompanied
    > by a well-implemented and tested patch, even if the developers don't use
    > the patch exactly as-is. I've no specific experience with OpenSSH in
    > that regard, though. Bringing up the subject on one of the mailing lists
    > before attempting implementation is probably a good idea too - maybe
    > there is some reason the change simply will not be considered.
    >
    > N.B. In this particular case, I would guess that a) the default would
    > have to remain unchanged, and b) there's probably no way to have the
    > client request this behaviour from the server (given that the protocol
    > is standardized). I.e. it would presumably have to be a server-side
    > config option to enable it - but maybe that fits your bill.
    >
    > --Per Hedeland
    > per@hedeland.org



    Thank you for your help, looking at openSSH 's bugzilla, I've found that
    this bug (id 396) is assigned since 2002. I'll try to reproduce it with
    the latest openSSH version ASAP and ask mailing list for information
    regarding this problem if it still exists.

    Best regards,
    Mat

  9. Re: pb with remote command cleanup

    On 2007-06-25, Mat wrote:
    [...]
    > Thank you for your help, looking at openSSH 's bugzilla, I've found that
    > this bug (id 396) is assigned since 2002. I'll try to reproduce it with
    > the latest openSSH version ASAP and ask mailing list for information
    > regarding this problem if it still exists.


    It probably does, although it's vaguely possible that the hang-on-exit
    changes resolved this by coincidence. If it still exists, it's not clear
    that sending pty signals to a program not running with a pty is the right
    thing to do.

    In theory, a running program should check to see if its input has closed
    and shut down in that case. It's only programs that don't that have this
    particular problem.

    --
    Darren Tucker (dtucker at zip.com.au)
    GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
    Good judgement comes with experience. Unfortunately, the experience
    usually comes from bad judgement.

  10. Re: pb with remote command cleanup

    In article Darren Tucker
    writes:
    >On 2007-06-25, Mat wrote:
    >[...]
    >> Thank you for your help, looking at openSSH 's bugzilla, I've found that
    >> this bug (id 396) is assigned since 2002. I'll try to reproduce it with
    >> the latest openSSH version ASAP and ask mailing list for information
    >> regarding this problem if it still exists.

    >
    >It probably does, although it's vaguely possible that the hang-on-exit
    >changes resolved this by coincidence. If it still exists, it's not clear
    >that sending pty signals to a program not running with a pty is the right
    >thing to do.


    I agree that it's not clear that it's the right thing - an argument can
    be made for either viewpoint. And in any case it is likely that there is
    usage that depends on the current behaviour of *not* sending the signal.
    Which is why I suggested that having it as an option (in sshd_config,
    default off) could be reasonable.

    Just for fun, I peeked into the current FreeBSD rshd source (that being
    the only program I can think of that is in the same situtation), and it
    doesn't send any signal if the socket is closed - in fact it can't even
    tell if the socket is closed since it has closed it itself after letting
    the command-runinng child inherit it. (Not an option for sshd of
    course.)

    >In theory, a running program should check to see if its input has closed
    >and shut down in that case. It's only programs that don't that have this
    >particular problem.


    I don't think such a theory exists though. You're suggesting that if I
    fire up a (e.g.) a long-running make with input redirected from
    /dev/null, it should exit immediately?:-) On the contrary, a program
    that doesn't expect input should specifically *not* check if its input
    has closed.

    --Per Hedeland
    per@hedeland.org

  11. Re: pb with remote command cleanup

    Per Hedeland a écrit :
    > In article Darren Tucker
    > writes:
    >> On 2007-06-25, Mat wrote:
    >> [...]
    >>> Thank you for your help, looking at openSSH 's bugzilla, I've found that
    >>> this bug (id 396) is assigned since 2002. I'll try to reproduce it with
    >>> the latest openSSH version ASAP and ask mailing list for information
    >>> regarding this problem if it still exists.

    >> It probably does, although it's vaguely possible that the hang-on-exit
    >> changes resolved this by coincidence. If it still exists, it's not clear
    >> that sending pty signals to a program not running with a pty is the right
    >> thing to do.

    >
    > I agree that it's not clear that it's the right thing - an argument can
    > be made for either viewpoint. And in any case it is likely that there is
    > usage that depends on the current behaviour of *not* sending the signal.
    > Which is why I suggested that having it as an option (in sshd_config,
    > default off) could be reasonable.
    >


    Hi, I have made a patch, based on the patch from bugzilla. I add an
    option to sshd_config, RemoteCommandCleanup. When this option is enabled
    sshd send a HUP signal to child process group when no tty was created.
    The reason why it send it to process group is that with some shell (csh)
    sshd wrappes remote command before execution (csh -c "..."). So if it
    doesn't send it to process group, remote command is still running after
    SIGHUP.

    I think I will send it on buhzilla and ask mailing list for news about
    the corresponding bug next week.

    To my mind, signal HUP is the right way to handle this problem because
    if you don't want that your remote command exits after ssh client
    termination, you can launch "ssh myhost nohup your_command" to inhibate
    the signal.


    > Just for fun, I peeked into the current FreeBSD rshd source (that being
    > the only program I can think of that is in the same situtation), and it
    > doesn't send any signal if the socket is closed - in fact it can't even
    > tell if the socket is closed since it has closed it itself after letting
    > the command-runinng child inherit it. (Not an option for sshd of
    > course.)
    >


    I don't use rsh but according to "man rsh", signals INT, QUIT and TERM
    are transmitted to remote command. So when you kill your client, remote
    side receives corresponding signal and relay it to command. With ssh,
    this behavior is active only when you use tty.
    Nethertheless, you must be right when you say that rshd doesn't send
    signal when connection is lost. The patch enables the cleanup in this
    particular case and I think that it's a better approach.

    Thank you
    Mat

    >> In theory, a running program should check to see if its input has closed
    >> and shut down in that case. It's only programs that don't that have this
    >> particular problem.

    >
    > I don't think such a theory exists though. You're suggesting that if I
    > fire up a (e.g.) a long-running make with input redirected from
    > /dev/null, it should exit immediately?:-) On the contrary, a program
    > that doesn't expect input should specifically *not* check if its input
    > has closed.
    >
    > --Per Hedeland
    > per@hedeland.org


  12. Re: pb with remote command cleanup

    In article <468e8dc6$0$10302$426a74cc@news.free.fr> Mat
    writes:
    >Per Hedeland a écrit :
    >> In article Darren Tucker
    >> writes:
    >>> On 2007-06-25, Mat wrote:
    >>> [...]
    >>>> Thank you for your help, looking at openSSH 's bugzilla, I've found that
    >>>> this bug (id 396) is assigned since 2002. I'll try to reproduce it with
    >>>> the latest openSSH version ASAP and ask mailing list for information
    >>>> regarding this problem if it still exists.
    >>> It probably does, although it's vaguely possible that the hang-on-exit
    >>> changes resolved this by coincidence. If it still exists, it's not clear
    >>> that sending pty signals to a program not running with a pty is the right
    >>> thing to do.

    >>
    >> I agree that it's not clear that it's the right thing - an argument can
    >> be made for either viewpoint. And in any case it is likely that there is
    >> usage that depends on the current behaviour of *not* sending the signal.
    >> Which is why I suggested that having it as an option (in sshd_config,
    >> default off) could be reasonable.
    >>

    >
    >Hi, I have made a patch, based on the patch from bugzilla. I add an
    >option to sshd_config, RemoteCommandCleanup. When this option is enabled
    >sshd send a HUP signal to child process group when no tty was created.


    Sounds good I think.

    >The reason why it send it to process group is that with some shell (csh)
    >sshd wrappes remote command before execution (csh -c "..."). So if it
    >doesn't send it to process group, remote command is still running after
    >SIGHUP.


    Hm, sshd will always "wrap" the command that way - i.e. it tells the
    user's login shell to run the command, which allows that shell to deal
    with I/O redirection, variables, pipelines, and other meta-information,
    using the often shell-specific way that such things are expressed. But
    anyway sending the signal to the process group is definitely right,
    that's what the tty driver does too.

    >To my mind, signal HUP is the right way to handle this problem because
    >if you don't want that your remote command exits after ssh client
    >termination, you can launch "ssh myhost nohup your_command" to inhibate
    >the signal.


    Or in a shell that supports job control (which most all do these days),
    it should be enough to run the command in the background with '&', since
    that causes it to be in a different process group. But HUP is right
    because that's what HUP is for.:-)

    >I don't use rsh


    Hey, neither do I!:-)

    > but according to "man rsh", signals INT, QUIT and TERM
    >are transmitted to remote command. So when you kill your client, remote
    >side receives corresponding signal and relay it to command.


    Well, yes and no - the client actually transmits the signal (as a byte
    value on the "extra" connection used for stderr with rsh) to the server,
    which sends it to the remote command - but AFAIK the signal doesn't kill
    the client, if it exits it is due to the remote command dying from the
    signal.

    > With ssh, this behavior is active only when you use tty.


    So this is again different - the ssh client doesn't transmit any signals
    (I would assume there is no way to do that within the protocol, as it's
    a pretty Unix-specific thing), but instead simply dies itself. And when
    a signal *is* sent in the tty case, it is due to the slave side of the
    pty driver "losing carrier", i.e. the master side controlled by sshd is
    closed.

    Alternatively, when you use remote tty and hit (e.g.) ^C on the client
    side, the client doesn't die but just sends the character across the
    normal data connection, and it gets turned into a signal by the remote
    tty driver if appropriate (otherwise it would be a pain to use emacs
    across a ssh connection:-).

    --Per Hedeland
    per@hedeland.org

+ Reply to Thread