can I rsync on UDP? - Networking

This is a discussion on can I rsync on UDP? - Networking ; its a lame question, but let me ask you Gurus, I have task of downloading files sized little over 1GB using rsync. It is taking lot of time because of my lousy network speed. Just to test I want to ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 34

Thread: can I rsync on UDP?

  1. can I rsync on UDP?

    its a lame question, but let me ask you Gurus,

    I have task of downloading files sized little over 1GB using rsync. It
    is taking lot of time because of my lousy network speed. Just to test
    I want to try rsync over UDP if possible, I am sure someone should
    have done this trial, Can you share ideas about how to do it.

    Thanks

  2. Re: can I rsync on UDP?

    Jagadeesh schrieb:
    > I have task of downloading files sized little over 1GB using rsync. It
    > is taking lot of time because of my lousy network speed. Just to test
    > I want to try rsync over UDP if possible, I am sure someone should
    > have done this trial, Can you share ideas about how to do it.


    Most important question: why do you have lousy network speed? Or what do
    you consider to be "lousy speed"?

    There are protocols that you want to use over TCP because you can use
    jumbo frames (e.g. NFS). Albeit TCPs overhead for small packets has to
    be considered, overall it allows for much more throughput when you (are
    able to) use the right parameters. Simplified: file size over 1GB
    screams for TCP.

    I could be wrong. But I think you got hooked up at the wrong side of the
    problem.

    Regards,
    Benjamin

    P.S. do you really have to use rsync? Only access via a rsync server?
    There are cases were rsync really s*cks!

  3. Re: can I rsync on UDP?

    Jagadeesh writes:

    > its a lame question, but let me ask you Gurus,
    >
    > I have task of downloading files sized little over 1GB using rsync. It
    > is taking lot of time because of my lousy network speed. Just to test
    > I want to try rsync over UDP if possible, I am sure someone should
    > have done this trial, Can you share ideas about how to do it.


    Of course, 1GB files will take a long time with pretty darn good
    network speed (I for one would regard 5Mbit/sec as pretty good. 1GB
    at 5Mb/sec = 2000 seconds = half an hour). Why do you suspect network
    speed that would be improved by using UDP? Slow is slow, after
    all... FWIW, if there's a UDP-based protocol that rsync can use, I
    don't know of it.

  4. Re: can I rsync on UDP?

    Joe Pfeiffer writes:

    >Jagadeesh writes:


    >> its a lame question, but let me ask you Gurus,
    >>
    >> I have task of downloading files sized little over 1GB using rsync. It
    >> is taking lot of time because of my lousy network speed. Just to test
    >> I want to try rsync over UDP if possible, I am sure someone should
    >> have done this trial, Can you share ideas about how to do it.


    >Of course, 1GB files will take a long time with pretty darn good
    >network speed (I for one would regard 5Mbit/sec as pretty good. 1GB
    >at 5Mb/sec = 2000 seconds = half an hour). Why do you suspect network
    >speed that would be improved by using UDP? Slow is slow, after
    >all... FWIW, if there's a UDP-based protocol that rsync can use, I
    >don't know of it.


    Well, I would say that it depends on the connection. If he has Gb network
    between the machines, rsync over ssh should give about 30MB/s. with the
    encryption probably being the bottleneck.
    If it is over 100Mb link, 10MB/s is achievable.
    If it is a long distance transfer then an hour would not be out of place.

    Why he thinks UDP will change his "lousy network speed" I have no idea.


  5. Re: can I rsync on UDP?

    Unruh writes:

    > Joe Pfeiffer writes:
    >
    >>Jagadeesh writes:

    >
    >>> its a lame question, but let me ask you Gurus,
    >>>
    >>> I have task of downloading files sized little over 1GB using rsync. It
    >>> is taking lot of time because of my lousy network speed. Just to test
    >>> I want to try rsync over UDP if possible, I am sure someone should
    >>> have done this trial, Can you share ideas about how to do it.

    >
    >>Of course, 1GB files will take a long time with pretty darn good
    >>network speed (I for one would regard 5Mbit/sec as pretty good. 1GB
    >>at 5Mb/sec = 2000 seconds = half an hour). Why do you suspect network
    >>speed that would be improved by using UDP? Slow is slow, after
    >>all... FWIW, if there's a UDP-based protocol that rsync can use, I
    >>don't know of it.

    >
    > Well, I would say that it depends on the connection. If he has Gb network
    > between the machines, rsync over ssh should give about 30MB/s. with the
    > encryption probably being the bottleneck.
    > If it is over 100Mb link, 10MB/s is achievable.
    > If it is a long distance transfer then an hour would not be out of place.


    All good points -- I was starting from the assumption that when he
    said "lousy network speed" he was on something slower than LAN speeds,
    and pointing out that pretty good long distance transfer will still be
    taking quite a while.

    > Why he thinks UDP will change his "lousy network speed" I have no idea.


    I was wondering that too.

  6. Re: can I rsync on UDP?

    Unruh wrote:
    > Joe Pfeiffer writes:
    >
    >> Jagadeesh writes:

    >
    >>> its a lame question, but let me ask you Gurus,
    >>>
    >>> I have task of downloading files sized little over 1GB using rsync. It
    >>> is taking lot of time because of my lousy network speed. Just to test
    >>> I want to try rsync over UDP if possible, I am sure someone should
    >>> have done this trial, Can you share ideas about how to do it.

    >
    >> Of course, 1GB files will take a long time with pretty darn good
    >> network speed (I for one would regard 5Mbit/sec as pretty good. 1GB
    >> at 5Mb/sec = 2000 seconds = half an hour). Why do you suspect network
    >> speed that would be improved by using UDP? Slow is slow, after
    >> all... FWIW, if there's a UDP-based protocol that rsync can use, I
    >> don't know of it.

    >
    > Well, I would say that it depends on the connection. If he has Gb network
    > between the machines, rsync over ssh should give about 30MB/s. with the
    > encryption probably being the bottleneck.


    FWIW, I've found that rsync over ssh over a 1 Gbit link has a bottleneck
    at the platter, not the cpu. Of course, that's with
    electrically-connected drives (SAS, SATA).

    It would be interesting to do the same experiment with a good
    fiber-attached SAN (I'm thinking HP StorageWorks, e.g.), but I can only
    dream about what *that* answer might be, since I haven't got the
    hardware to actually do the experiment.

    > If it is over 100Mb link, 10MB/s is achievable.
    > If it is a long distance transfer then an hour would not be out of place.
    >
    > Why he thinks UDP will change his "lousy network speed" I have no idea.


  7. Re: can I rsync on UDP?

    Allen Kistler writes:

    >Unruh wrote:
    >> Joe Pfeiffer writes:
    >>
    >>> Jagadeesh writes:

    >>
    >>>> its a lame question, but let me ask you Gurus,
    >>>>
    >>>> I have task of downloading files sized little over 1GB using rsync. It
    >>>> is taking lot of time because of my lousy network speed. Just to test
    >>>> I want to try rsync over UDP if possible, I am sure someone should
    >>>> have done this trial, Can you share ideas about how to do it.

    >>
    >>> Of course, 1GB files will take a long time with pretty darn good
    >>> network speed (I for one would regard 5Mbit/sec as pretty good. 1GB
    >>> at 5Mb/sec = 2000 seconds = half an hour). Why do you suspect network
    >>> speed that would be improved by using UDP? Slow is slow, after
    >>> all... FWIW, if there's a UDP-based protocol that rsync can use, I
    >>> don't know of it.

    >>
    >> Well, I would say that it depends on the connection. If he has Gb network
    >> between the machines, rsync over ssh should give about 30MB/s. with the
    >> encryption probably being the bottleneck.


    >FWIW, I've found that rsync over ssh over a 1 Gbit link has a bottleneck
    >at the platter, not the cpu. Of course, that's with
    >electrically-connected drives (SAS, SATA).


    It depends on the computer. I ran tests with a SATA drive on a 2GH machine
    and Gb ethernet, and it was ssh that was the bottleneck. With a faster
    machine
    it is possible that disk access would be the bottleneck. ( If I ran rsync
    straight-- ie rsync->rsync daemon which does not go through ssh) I got
    about 50-80% higher throughput.


    >It would be interesting to do the same experiment with a good
    >fiber-attached SAN (I'm thinking HP StorageWorks, e.g.), but I can only
    >dream about what *that* answer might be, since I haven't got the
    >hardware to actually do the experiment.


    >> If it is over 100Mb link, 10MB/s is achievable.
    >> If it is a long distance transfer then an hour would not be out of place.
    >>
    >> Why he thinks UDP will change his "lousy network speed" I have no idea.


  8. Re: can I rsync on UDP?

    Jagadeesh writes:

    > I have task of downloading files sized little over 1GB using rsync. It
    > is taking lot of time because of my lousy network speed. Just to test
    > I want to try rsync over UDP if possible, I am sure someone should
    > have done this trial, Can you share ideas about how to do it.


    So you prefer a faster unreliable backup over a slower reliable one?

    Excuse me, but that sounds crazy.

  9. Re: can I rsync on UDP?

    On Wed, 22 Oct 2008 07:43:08 -0400, Maxwell Lol wrote:
    >Jagadeesh writes:


    >> I have task of downloading files sized little over 1GB using rsync. It
    >> is taking lot of time because of my lousy network speed. Just to test
    >> I want to try rsync over UDP if possible, I am sure someone should
    >> have done this trial, Can you share ideas about how to do it.


    >So you prefer a faster unreliable backup over a slower reliable one?

    rsync has error checking. It works perfectly over unreliable networks.

    >Excuse me, but that sounds crazy.


  10. Re: can I rsync on UDP?

    AZ Nomad writes:

    > On Wed, 22 Oct 2008 07:43:08 -0400, Maxwell Lol wrote:
    >>Jagadeesh writes:

    >
    >>> I have task of downloading files sized little over 1GB using rsync. It
    >>> is taking lot of time because of my lousy network speed. Just to test
    >>> I want to try rsync over UDP if possible, I am sure someone should
    >>> have done this trial, Can you share ideas about how to do it.

    >
    >>So you prefer a faster unreliable backup over a slower reliable one?

    > rsync has error checking. It works perfectly over unreliable networks.


    *ALL* networks are unreliable. That's why TCP was developed - to
    provide reliable transfer of data over a network.


    If rsync works over UDP, what is the command option? I don't see one.
    Rsync runs over rsh/ssh, which uses TCP.

    If there is a remote shell that is reliable and uses UDP as a
    transport, that could be specified. But I don't know of any.


    So please explain how rsync works over UDP......

  11. Re: can I rsync on UDP?

    Maxwell Lol wrote:
    > *ALL* networks are unreliable. That's why TCP was developed - to
    > provide reliable transfer of data over a network.


    Might it be more accurate to say that all IP networks are unreliable?

    rick jones
    --
    The glass is neither half-empty nor half-full. The glass has a leak.
    The real question is "Can it be patched?"
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  12. Re: can I rsync on UDP?

    On Wed, 22 Oct 2008 14:09:20 -0400, Maxwell Lol wrote:
    >AZ Nomad writes:


    >> On Wed, 22 Oct 2008 07:43:08 -0400, Maxwell Lol wrote:
    >>>Jagadeesh writes:

    >>
    >>>> I have task of downloading files sized little over 1GB using rsync. It
    >>>> is taking lot of time because of my lousy network speed. Just to test
    >>>> I want to try rsync over UDP if possible, I am sure someone should
    >>>> have done this trial, Can you share ideas about how to do it.

    >>
    >>>So you prefer a faster unreliable backup over a slower reliable one?

    >> rsync has error checking. It works perfectly over unreliable networks.


    >*ALL* networks are unreliable. That's why TCP was developed - to
    >provide reliable transfer of data over a network.


    rsync has it's own error checking protocol

  13. Re: can I rsync on UDP?

    Maxwell Lol writes:

    >AZ Nomad writes:


    >> On Wed, 22 Oct 2008 07:43:08 -0400, Maxwell Lol wrote:
    >>>Jagadeesh writes:

    >>
    >>>> I have task of downloading files sized little over 1GB using rsync. It
    >>>> is taking lot of time because of my lousy network speed. Just to test
    >>>> I want to try rsync over UDP if possible, I am sure someone should
    >>>> have done this trial, Can you share ideas about how to do it.

    >>
    >>>So you prefer a faster unreliable backup over a slower reliable one?

    >> rsync has error checking. It works perfectly over unreliable networks.


    >*ALL* networks are unreliable. That's why TCP was developed - to
    >provide reliable transfer of data over a network.



    >If rsync works over UDP, what is the command option? I don't see one.
    >Rsync runs over rsh/ssh, which uses TCP.


    >If there is a remote shell that is reliable and uses UDP as a
    >transport, that could be specified. But I don't know of any.



    >So please explain how rsync works over UDP......


    rsync has two modes-- one is over ssh (or rsh.) in that mode the local
    rsync makes an ssh connection to the remote machine and starts up a version
    of rsync on that remote machine to connect the two together.
    In the other mode, when the local rsync contacts the remote machines on
    port 873, the remote (x)inted starts up a version of rsync and intiates the
    transfer. That can be either tcp or udp.(Not sure how it decides which to
    use). That mode is NOT encrypted ( as rsync over ssh is) and can be about
    50% faster depending on your CPU , network,...

    It means that the remote machine MUST set up rsync to run as an (x)inetd
    daemon. In the first case all that is needed is that the user be able to
    log on with ssh, and run rsync. In the second the remote sysadmin must set
    up rsync to run as a daemon. This is obviously somewhat more risky since it
    potentially opens up the system to remote attacks. That can be mitigated
    with hosts.allow and /etc/rsync.conf








  14. Re: can I rsync on UDP?

    Rick Jones writes:

    > Maxwell Lol wrote:
    >> *ALL* networks are unreliable. That's why TCP was developed - to
    >> provide reliable transfer of data over a network.

    >
    > Might it be more accurate to say that all IP networks are unreliable?


    Sure. Also XNS, DECNET, SMB/NETBIOS and many others.

    I'm not sure if you are implying that other networks are reliable.

    But I suppose you can have a network of two machines in a bomb
    shelter, with a UPS that can be powered by a aux. generator. But that
    could be an IP network. So some IP networks could be 99.99999999%
    reliable.

  15. Re: can I rsync on UDP?

    AZ Nomad writes:

    >>*ALL* networks are unreliable. That's why TCP was developed - to
    >>provide reliable transfer of data over a network.

    >
    > rsync has it's own error checking protocol


    It will have a hard time syncing two directories if it can't transfer
    any files reliably. How many times will it try to transfer a file
    before it gives up?

    I suppose IF you had a UDP-based shell, and IF you let it run long
    enough, and IF rsync never stopped trying, eventually it would be able
    to transfer a few GB's reliably. Eventually.

    But I would bet that it would take a lot longer than a TCP-based shell.

    If someone has a RSH equiv that uses UDP, it's easy enough to try.

    I don't know of any, however....



  16. Re: can I rsync on UDP?

    Unruh writes:

    > rsync has two modes-- one is over ssh (or rsh.) in that mode the local
    > rsync makes an ssh connection to the remote machine and starts up a version
    > of rsync on that remote machine to connect the two together.
    > In the other mode, when the local rsync contacts the remote machines on
    > port 873, the remote (x)inted starts up a version of rsync and intiates the
    > transfer. That can be either tcp or udp.(Not sure how it decides which to
    > use). That mode is NOT encrypted ( as rsync over ssh is) and can be about
    > 50% faster depending on your CPU , network,...


    Thanks. This is new to me. But I didn't see anything about rsync
    using UDP in the man page. And does it use it to initiate a TCP
    connection? Or is EVERYTHING done with UDP?

    And are you use the speedup is due to UDP, or is the speedup duee to
    using RSH versus SSH (no encyprion vs. encryption)

    >
    > It means that the remote machine MUST set up rsync to run as an (x)inetd
    > daemon. In the first case all that is needed is that the user be able to
    > log on with ssh, and run rsync. In the second the remote sysadmin must set
    > up rsync to run as a daemon. This is obviously somewhat more risky since it
    > potentially opens up the system to remote attacks. That can be mitigated
    > with hosts.allow and /etc/rsync.conf


    I still don't see any config option that says to use UDP versus TCP.

    I'm not on a system where I can monitor a rsync session, but I still doubt
    file transfers can be done using UDP. They could be INITIATED using UDP.

    I suppose you can change/etc/initd.conf so it reads

    rsync stream udp nowait root /usr/bin/rsync rsyncd --daemon

    and try it out. Try it both ways and see if there is any difference.
    If UDP is not supported, it ought to generate an error.

    Anyone willling to try?

  17. Re: can I rsync on UDP?

    Maxwell Lol writes:

    >Unruh writes:


    >> rsync has two modes-- one is over ssh (or rsh.) in that mode the local
    >> rsync makes an ssh connection to the remote machine and starts up a version
    >> of rsync on that remote machine to connect the two together.
    >> In the other mode, when the local rsync contacts the remote machines on
    >> port 873, the remote (x)inted starts up a version of rsync and intiates the
    >> transfer. That can be either tcp or udp.(Not sure how it decides which to
    >> use). That mode is NOT encrypted ( as rsync over ssh is) and can be about
    >> 50% faster depending on your CPU , network,...


    >Thanks. This is new to me. But I didn't see anything about rsync
    >using UDP in the man page. And does it use it to initiate a TCP
    >connection? Or is EVERYTHING done with UDP?


    No idea. All I know is that port 873 is defined as rsync for both TCP and
    UDP. I have no idea if and how it uses it.


    >And are you use the speedup is due to UDP, or is the speedup duee to
    >using RSH versus SSH (no encyprion vs. encryption)


    It does not use rsh in the daemon mode. It connects directly to the daemon
    running on the other machine.



    >>
    >> It means that the remote machine MUST set up rsync to run as an (x)inetd
    >> daemon. In the first case all that is needed is that the user be able to
    >> log on with ssh, and run rsync. In the second the remote sysadmin must set
    >> up rsync to run as a daemon. This is obviously somewhat more risky since it
    >> potentially opens up the system to remote attacks. That can be mitigated
    >> with hosts.allow and /etc/rsync.conf


    >I still don't see any config option that says to use UDP versus TCP.


    >I'm not on a system where I can monitor a rsync session, but I still doubt
    >file transfers can be done using UDP. They could be INITIATED using UDP.


    >I suppose you can change/etc/initd.conf so it reads


    > rsync stream udp nowait root /usr/bin/rsync rsyncd --daemon


    Most modern systems use xinetd, and there is an rsync file in
    /etc/xinetd.d/ which you have to enable (disable=no)



    >and try it out. Try it both ways and see if there is any difference.
    >If UDP is not supported, it ought to generate an error.


    >Anyone willling to try?


  18. Re: can I rsync on UDP?

    Maxwell Lol wrote:
    > If rsync works over UDP, what is the command option? I don't see one.


    You could try something like this, I suppose:

    RSYNC_RSH='ssh -f %h nc -u -l 60873; nc -u %h 60873'
    rsync -avHP local/files remotehost:/remote/path

    Chris

  19. Re: can I rsync on UDP?

    Maxwell Lol wrote:
    > *ALL* networks are unreliable [...]


    Rick Jones wrote:
    > Might it be more accurate to say that all IP networks are unreliable?


    Consider two computers linked by {whatever}. I can't see a situation where
    you couldn't (theoretically) break the linkage described by {whatever}
    by some means.

    Chris

  20. Re: can I rsync on UDP?

    Unruh wrote:
    > No idea. All I know is that port 873 is defined as rsync for both TCP and
    > UDP. I have no idea if and how it uses it.


    That's an IANA issue. When someone registers a port it's assigned for
    both UDP and TCP. Many protocols use only one mechanism.

    Chris

+ Reply to Thread
Page 1 of 2 1 2 LastLast