rsync over stdin/stdout - Tools

This is a discussion on rsync over stdin/stdout - Tools ; So I've got this situation where I have two systems connected via a protocol that looks nothing at all like IP, and I'd like to be able to rsync between them. I can, though, open a perfectly normal looking socket ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: rsync over stdin/stdout

  1. rsync over stdin/stdout

    So I've got this situation where I have two systems connected via a
    protocol that looks nothing at all like IP, and I'd like to be able to
    rsync between them.

    I can, though, open a perfectly normal looking socket through various
    magic and trickery.

    I've managed to get rsync to work by using the "rsync
    --rsh=mymagictransportprogram" style invocation, but for a number of
    reasons this isn't really a viable long-term solution for me. What I'd
    like to be able to do is either:

    A: Set up stdin/stdout and exec rsync with some options that tell it
    that there's a remote rsync on the other end of those FDs, or

    B: Call rsync as a library, passing it a socket and directory(^Hies) to
    sync with. (librsync doesn't seem to be quite what I want)

    Basically I need to set up the socket beforehand and then hand it to
    rsync somehow. This seems to be possible on the 'remote' end, but I
    haven't managed to figure out how to do on the initiation side.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  2. Re: rsync over stdin/stdout

    On Mon, 2008-06-02 at 11:27 -0700, Tony Mantler wrote:
    > So I've got this situation where I have two systems connected via a
    > protocol that looks nothing at all like IP, and I'd like to be able to
    > rsync between them.
    >
    > I can, though, open a perfectly normal looking socket through various
    > magic and trickery.
    >
    > I've managed to get rsync to work by using the "rsync
    > --rsh=mymagictransportprogram" style invocation, but for a number of
    > reasons this isn't really a viable long-term solution for me. What I'd
    > like to be able to do is either:
    >
    > A: Set up stdin/stdout and exec rsync with some options that tell it
    > that there's a remote rsync on the other end of those FDs, or

    
    > Basically I need to set up the socket beforehand and then hand it to
    > rsync somehow. This seems to be possible on the 'remote' end, but I
    > haven't managed to figure out how to do on the initiation side.


    First, you should use an rsync daemon if you aren't already; with the
    remote shell mode, you have to get the server arguments across the
    socket yourself, whereas the daemon protocol takes care of that.

    There's an enhancement request for a "&3::module/src/" syntax to make
    the client access a daemon over a socket supplied on a fd:

    https://bugzilla.samba.org/show_bug.cgi?id=5220

    In the absence of this enhancement, you can accomplish the same thing
    using the RSYNC_CONNECT_PROG environment variable (or --rsh, but
    RSYNC_CONNECT_PROG is slightly more convenient). Have the
    RSYNC_CONNECT_PROG inherit the socket from rsync and bridge the socket
    to its own stdin/stdout for use by rsync. Here's an example, assuming
    the socket is already on fd 3 of the shell:

    RSYNC_CONNECT_PROG='cat >&3 & cat <&3; wait' rsync -a foo::module/src/ dest/

    The hostname, here "foo", doesn't matter, since RSYNC_CONNECT_PROG
    overrides it.

    > B: Call rsync as a library, passing it a socket and directory(^Hies) to
    > sync with. (librsync doesn't seem to be quite what I want)


    Rsync doesn't come as a library. Librsync is perhaps misnamed: it
    contains only rsync's delta-transfer algorithm for individual files.

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEABECAAYFAkhEW80ACgkQC+xSYN/RlftgXACfYBgpurmOcOJMGhCtN49vz2ts
    4YoAoLd13xshOC+Xq7umR76LjCFX3tPT
    =1d0m
    -----END PGP SIGNATURE-----


  3. Re: rsync over stdin/stdout

    On Mon, 2008-06-02 at 13:45 -0700, Matt McCutchen wrote:
    > On Mon, 2008-06-02 at 11:27 -0700, Tony Mantler wrote:
    > > Basically I need to set up the socket beforehand and then hand it to
    > > rsync somehow. This seems to be possible on the 'remote' end, but I
    > > haven't managed to figure out how to do on the initiation side.

    >
    > First, you should use an rsync daemon if you aren't already; with the
    > remote shell mode, you have to get the server arguments across the
    > socket yourself, whereas the daemon protocol takes care of that.


    The program I have handles that fairly well, actually. It's not too far
    removed from complete rsh functionality.


    > There's an enhancement request for a "&3::module/src/" syntax to make
    > the client access a daemon over a socket supplied on a fd:
    >
    > https://bugzilla.samba.org/show_bug.cgi?id=5220


    That's pretty much exactly what I'm looking for. It's a shame it's just
    sitting as an enhancement request, but I suppose the need for it is
    fairly rare.


    > In the absence of this enhancement, you can accomplish the same thing
    > using the RSYNC_CONNECT_PROG environment variable (or --rsh, but
    > RSYNC_CONNECT_PROG is slightly more convenient). Have the
    > RSYNC_CONNECT_PROG inherit the socket from rsync and bridge the socket
    > to its own stdin/stdout for use by rsync. Here's an example, assuming
    > the socket is already on fd 3 of the shell:
    >
    > RSYNC_CONNECT_PROG='cat >&3 & cat <&3; wait' rsync -a foo::module/src/ dest/
    >
    > The hostname, here "foo", doesn't matter, since RSYNC_CONNECT_PROG
    > overrides it.


    That probably won't work very well in my case.


    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  4. Re: rsync over stdin/stdout

    On Mon, 2008-06-02 at 14:29 -0700, Tony Mantler wrote:
    > On Mon, 2008-06-02 at 13:45 -0700, Matt McCutchen wrote:
    > > On Mon, 2008-06-02 at 11:27 -0700, Tony Mantler wrote:
    > > > Basically I need to set up the socket beforehand and then hand it to
    > > > rsync somehow. This seems to be possible on the 'remote' end, but I
    > > > haven't managed to figure out how to do on the initiation side.

    > >
    > > First, you should use an rsync daemon if you aren't already; with the
    > > remote shell mode, you have to get the server arguments across the
    > > socket yourself, whereas the daemon protocol takes care of that.

    >
    > The program I have handles that fairly well, actually. It's not too far
    > removed from complete rsh functionality.


    I don't understand. If your connection program is like rsh, it passes
    arguments and then gives you a socket to talk to the invoked program.
    But the server arguments are not determined until you start the rsync
    client, after the socket has been set up. So how could the server
    arguments be passed by the connection program? Or do you just mean that
    the connection program passes "rsync --server --daemon ." without you
    having to hardcode that on the remote side, in which case we're really
    talking about the same thing?

    > > In the absence of this enhancement, you can accomplish the same thing
    > > using the RSYNC_CONNECT_PROG environment variable [...]


    > > RSYNC_CONNECT_PROG='cat >&3 & cat <&3; wait' rsync -a foo::module/src/ dest/


    > That probably won't work very well in my case.


    Why not? It does exactly the same thing as the enhancement would, but
    with slightly higher overhead to pass the data back and forth.

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEABECAAYFAkhEas0ACgkQC+xSYN/RlfvDBACaAqczA/ZWvSp6Qet44I1qR21r
    L8wAoIKGMe7nNEBO39wcCT2EfWzxBp8h
    =qwHm
    -----END PGP SIGNATURE-----


  5. Re: rsync over stdin/stdout

    On Mon, 2008-06-02 at 14:49 -0700, Matt McCutchen wrote:
    > On Mon, 2008-06-02 at 14:29 -0700, Tony Mantler wrote:
    > > On Mon, 2008-06-02 at 13:45 -0700, Matt McCutchen wrote:
    > > > On Mon, 2008-06-02 at 11:27 -0700, Tony Mantler wrote:
    > > > > Basically I need to set up the socket beforehand and then hand it to
    > > > > rsync somehow. This seems to be possible on the 'remote' end, but I
    > > > > haven't managed to figure out how to do on the initiation side.
    > > >
    > > > First, you should use an rsync daemon if you aren't already; with the
    > > > remote shell mode, you have to get the server arguments across the
    > > > socket yourself, whereas the daemon protocol takes care of that.

    > >
    > > The program I have handles that fairly well, actually. It's not too far
    > > removed from complete rsh functionality.

    >
    > I don't understand. If your connection program is like rsh, it passes
    > arguments and then gives you a socket to talk to the invoked program.
    > But the server arguments are not determined until you start the rsync
    > client, after the socket has been set up. So how could the server
    > arguments be passed by the connection program? Or do you just mean that
    > the connection program passes "rsync --server --daemon ." without you
    > having to hardcode that on the remote side, in which case we're really
    > talking about the same thing?


    Well, when operating in the 'rsh'-like mode, rsync passes a bunch of
    arguments to my program which my program hands across to the other side
    to invoke the remote-side rsh.

    What I need to do is turn this whole thing inside out, so that my
    program launches rsync rather than rsync launching my program. The
    reason for this is that my program needs to operate as a plugin as well
    as standalone.


    > > > In the absence of this enhancement, you can accomplish the same thing
    > > > using the RSYNC_CONNECT_PROG environment variable [...]

    >
    > > > RSYNC_CONNECT_PROG='cat >&3 & cat <&3; wait' rsync -a foo::module/src/ dest/

    >
    > > That probably won't work very well in my case.

    >
    > Why not? It does exactly the same thing as the enhancement would, but
    > with slightly higher overhead to pass the data back and forth.


    Well, it won't work on windows for one thing, and that's apparently one
    of my target platforms. (not by choice, I assure you)

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  6. Re: rsync over stdin/stdout

    On Mon, 2008-06-02 at 14:59 -0700, Tony Mantler wrote:
    > > > On Mon, 2008-06-02 at 13:45 -0700, Matt McCutchen wrote:
    > > > > First, you should use an rsync daemon if you aren't already; with the
    > > > > remote shell mode, you have to get the server arguments across the
    > > > > socket yourself, whereas the daemon protocol takes care of that.


    > Well, when operating in the 'rsh'-like mode, rsync passes a bunch of
    > arguments to my program which my program hands across to the other side
    > to invoke the remote-side rsh.
    >
    > What I need to do is turn this whole thing inside out, so that my
    > program launches rsync rather than rsync launching my program. The
    > reason for this is that my program needs to operate as a plugin as well
    > as standalone.


    Right. I was suggesting an rsync daemon for the inside-out case where
    you can't easily pass the server arguments using your program.

    > > > > In the absence of this enhancement, you can accomplish the same thing
    > > > > using the RSYNC_CONNECT_PROG environment variable [...]

    > >
    > > > > RSYNC_CONNECT_PROG='cat >&3 & cat <&3; wait' rsync -a foo::module/src/ dest/

    > >
    > > > That probably won't work very well in my case.

    > >
    > > Why not? It does exactly the same thing as the enhancement would, but
    > > with slightly higher overhead to pass the data back and forth.

    >
    > Well, it won't work on windows for one thing, and that's apparently one
    > of my target platforms. (not by choice, I assure you)


    It might work under Cygwin, but you're right, that's iffy. I suggest
    you compile your own rsync with the proposed patch:

    https://bugzilla.samba.org/attachmen...16&action=view

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEABECAAYFAkhEb3MACgkQC+xSYN/RlfsilwCfWk8IN9k5hBruLXPboY4TeYWA
    QXkAnj3XxXMrP/ntmGsjXtMTRe+bPf5c
    =HVD5
    -----END PGP SIGNATURE-----


  7. Re: rsync over stdin/stdout

    On Mon, 2008-06-02 at 15:08 -0700, Matt McCutchen wrote:
    > On Mon, 2008-06-02 at 14:59 -0700, Tony Mantler wrote:
    > > What I need to do is turn this whole thing inside out, so that my
    > > program launches rsync rather than rsync launching my program. The
    > > reason for this is that my program needs to operate as a plugin as well
    > > as standalone.

    >
    > Right. I was suggesting an rsync daemon for the inside-out case where
    > you can't easily pass the server arguments using your program.


    Ah, ok. Sure.


    > > Well, it won't work on windows for one thing, and that's apparently one
    > > of my target platforms. (not by choice, I assure you)

    >
    > It might work under Cygwin, but you're right, that's iffy. I suggest
    > you compile your own rsync with the proposed patch:
    >
    > https://bugzilla.samba.org/attachmen...16&action=view


    I think that might be the way I end up going. Thanks.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


+ Reply to Thread