can I rsync on UDP? - Networking

This is a discussion on can I rsync on UDP? - Networking ; Unruh writes: > Maxwell Lol writes: > >>Unruh writes: > >>> rsync has two modes-- one is over ssh (or rsh.) in that mode the local >>> rsync makes an ssh connection to the remote machine and starts up a ...

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2
Results 21 to 34 of 34

Thread: can I rsync on UDP?

  1. Re: can I rsync on UDP?

    Unruh writes:

    > Maxwell Lol writes:
    >
    >>Unruh writes:

    >
    >>> rsync has two modes-- one is over ssh (or rsh.) in that mode the local
    >>> rsync makes an ssh connection to the remote machine and starts up a version
    >>> of rsync on that remote machine to connect the two together.
    >>> In the other mode, when the local rsync contacts the remote machines on
    >>> port 873, the remote (x)inted starts up a version of rsync and intiates the
    >>> transfer. That can be either tcp or udp.(Not sure how it decides which to
    >>> use). That mode is NOT encrypted ( as rsync over ssh is) and can be about
    >>> 50% faster depending on your CPU , network,...

    >
    >>Thanks. This is new to me. But I didn't see anything about rsync
    >>using UDP in the man page. And does it use it to initiate a TCP
    >>connection? Or is EVERYTHING done with UDP?

    >
    > No idea. All I know is that port 873 is defined as rsync for both TCP and
    > UDP. I have no idea if and how it uses it.


    I looked into it a bit more.

    I only saw this reference to UDP/DGRAM in rsync 3.0.4:

    lib/addrinfo.h:#ifndef NI_DGRAM
    lib/addrinfo.h:#define NI_DGRAM 16
    lib/getaddrinfo.c: (flags & NI_DGRAM) ? "udp" : "tcp");


    This is just some utility code copied from another program.
    In other words, I see no way that UDP can be used.

    >>And are you use the speedup is due to UDP, or is the speedup duee to
    >>using RSH versus SSH (no encyprion vs. encryption)

    >
    > It does not use rsh in the daemon mode. It connects directly to the daemon
    > running on the other machine.


    I will have to set this up and test it.


  2. Re: can I rsync on UDP?

    On Oct 22, 12:23 am, Joe Pfeiffer wrote:
    > Unruh writes:
    > > Joe Pfeiffer writes:

    >
    > >>Jagadeesh writes:

    >
    > >>> its a lame question, but let me ask you Gurus,

    >
    > >>> I have task of downloading files sized little over 1GB using rsync. It
    > >>> is taking lot of time because of my lousy network speed. Just to test
    > >>> I want to try rsync over UDP if possible, I am sure someone should
    > >>> have done this trial, Can you share ideas about how to do it.

    >
    > >>Of course, 1GB files will take a long time with pretty darn good
    > >>network speed (I for one would regard 5Mbit/sec as pretty good. 1GB
    > >>at 5Mb/sec = 2000 seconds = half an hour). Why do you suspect network
    > >>speed that would be improved by using UDP? Slow is slow, after
    > >>all... FWIW, if there's a UDP-based protocol that rsync can use, I
    > >>don't know of it.

    >
    > > Well, I would say that it depends on the connection. If he has Gb network
    > > between the machines, rsync over ssh should give about 30MB/s. with the
    > > encryption probably being the bottleneck.
    > > If it is over 100Mb link, 10MB/s is achievable.
    > > If it is a long distance transfer then an hour would not be out of place.

    >
    > All good points -- I was starting from the assumption that when he
    > said "lousy network speed" he was on something slower than LAN speeds,
    > and pointing out that pretty good long distance transfer will still be
    > taking quite a while.
    >
    > > Why he thinks UDP will change his "lousy network speed" I have no idea.

    >
    > I was wondering that too.


    Well. I was talking about slow wan

    But good helpful replies.

    Thanks

  3. Re: can I rsync on UDP?

    Unruh writes:
    >
    > No idea. All I know is that port 873 is defined as rsync for both TCP and
    > UDP. I have no idea if and how it uses it.


    Near as I can tell from reading rsync's man page, it doesn't. It only
    actually uses tcp.

  4. Re: can I rsync on UDP?

    Jagadeesh writes:

    >On Oct 22, 12:23 am, Joe Pfeiffer wrote:
    >> Unruh writes:
    >> > Joe Pfeiffer writes:

    >>
    >> >>Jagadeesh writes:

    >>
    >> >>> its a lame question, but let me ask you Gurus,

    >>
    >> >>> I have task of downloading files sized little over 1GB using rsync. It
    >> >>> is taking lot of time because of my lousy network speed. Just to test
    >> >>> I want to try rsync over UDP if possible, I am sure someone should
    >> >>> have done this trial, Can you share ideas about how to do it.

    >>
    >> >>Of course, 1GB files will take a long time with pretty darn good
    >> >>network speed (I for one would regard 5Mbit/sec as pretty good. 1GB
    >> >>at 5Mb/sec = 2000 seconds = half an hour). Why do you suspect network
    >> >>speed that would be improved by using UDP? Slow is slow, after
    >> >>all... FWIW, if there's a UDP-based protocol that rsync can use, I
    >> >>don't know of it.

    >>
    >> > Well, I would say that it depends on the connection. If he has Gb network
    >> > between the machines, rsync over ssh should give about 30MB/s. with the
    >> > encryption probably being the bottleneck.
    >> > If it is over 100Mb link, 10MB/s is achievable.
    >> > If it is a long distance transfer then an hour would not be out of place.

    >>
    >> All good points -- I was starting from the assumption that when he
    >> said "lousy network speed" he was on something slower than LAN speeds,
    >> and pointing out that pretty good long distance transfer will still be
    >> taking quite a while.
    >>
    >> > Why he thinks UDP will change his "lousy network speed" I have no idea.

    >>
    >> I was wondering that too.


    >Well. I was talking about slow wan


    That of course tells me absolutely no more than did "lousy network speeds".
    And I still do not know why UDP would make a difference.



  5. Re: can I rsync on UDP?

    Maxwell Lol wrote:
    > Rick Jones writes:


    > > Maxwell Lol wrote:
    > >> *ALL* networks are unreliable. That's why TCP was developed - to
    > >> provide reliable transfer of data over a network.

    > >
    > > Might it be more accurate to say that all IP networks are unreliable?


    > Sure. Also XNS, DECNET, SMB/NETBIOS and many others.


    > I'm not sure if you are implying that other networks are reliable.


    I cannot recall the specifics, but IIRC X.25 has some degree of
    "reliability" along the lines of retransmission and whatnot.

    rick jones
    --
    oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  6. Re: can I rsync on UDP?

    Chris Davies wrote:
    > Maxwell Lol wrote:
    > > *ALL* networks are unreliable [...]


    > Rick Jones wrote:
    > > Might it be more accurate to say that all IP networks are
    > > unreliable?


    > Consider two computers linked by {whatever}. I can't see a situation
    > where you couldn't (theoretically) break the linkage described by
    > {whatever} by some means.


    Sure, the question is whether or not there is a network-layer (ish)
    protocol that provides reliability a la TCP or what not - ie one with
    retransmissions and flow control etc, such that a ULP (Upper Layer
    Protocol) sitting on top of it might presume reliability the way an
    application using TCP does.

    One can "break" a TCP connection but we still refer to TCP as a
    "reliable" protocol - even though the only "guarantee" that TCP makes
    is that it will tell you that it was _probably_ unable to get some of
    your data to the remote destination.

    rick jones
    --
    The glass is neither half-empty nor half-full. The glass has a leak.
    The real question is "Can it be patched?"
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  7. Re: can I rsync on UDP?

    Thank you very much all for you time and helping me.
    I can get high speed connection. That's my limitation.

    But all your replies saved my time that I would have spent on R&D
    without getting much success.
    Thank you again.

    Jagadeesh

  8. Re: can I rsync on UDP?

    Jagadeesh writes:

    >Thank you very much all for you time and helping me.
    >I can get high speed connection. That's my limitation.


    I think something went wrong witht he last two sentences.
    What doe you define as "high speed".

    >But all your replies saved my time that I would have spent on R&D
    >without getting much success.
    >Thank you again.


    >Jagadeesh


  9. Re: can I rsync on UDP?

    >>> rsync has two modes-- one is over ssh (or rsh.) in that mode the local
    >>> rsync makes an ssh connection to the remote machine and starts up a version
    >>> of rsync on that remote machine to connect the two together.
    >>> In the other mode, when the local rsync contacts the remote machines on
    >>> port 873, the remote (x)inted starts up a version of rsync and intiates the
    >>> transfer. That can be either tcp or udp.(Not sure how it decides which to
    >>> use).


    I have confirmed that UDP is NEVER used.
    As as extra validation, the manual page says

    >There are two different ways for rsync to contact a remote system:
    >using a remote-shell program as the transport (such as ssh or rsh) or
    >contacting an rsync daemon directly via TCP.


    >>>That mode is NOT encrypted ( as rsync over ssh is) and can be about
    >>> 50% faster depending on your CPU , network,...


    I suspect it's because encryption isn't used.

  10. Re: can I rsync on UDP?

    Chris Davies writes:

    > Maxwell Lol wrote:
    >> If rsync works over UDP, what is the command option? I don't see one.

    >
    > You could try something like this, I suppose:
    >
    > RSYNC_RSH='ssh -f %h nc -u -l 60873; nc -u %h 60873'
    > rsync -avHP local/files remotehost:/remote/path


    Since nc sends a one-way stream, I don't see how anything can be
    negotiated between the two systems. Which is the point of
    rsync......

    Nevertheless, I tried
    RSYNC_RSH='ssh -f 192.168.1.50 nc -u -l -p 60873; nc -u 192.168.1.50 -p 60873'
    export RSYNC_RSH

    and was not able to get it to work.







    >
    > Chris


  11. Re: can I rsync on UDP?

    Jagadeesh writes:

    > Thank you very much all for you time and helping me.
    > I can get high speed connection. That's my limitation.


    Assuming you mean you cannot get a high speed connection,
    there may be other problems.

    Encryption does slow things down. Do you want security or speed?

    Is rsync being done over a wide area network? There are ways to tune this.

    Do you have to use rsync? (instead of compressing a tar file).

  12. Re: can I rsync on UDP?

    Chris Davies writes:
    > You could try something like this, I suppose:
    > RSYNC_RSH='ssh -f %h nc -u -l 60873; nc -u %h 60873'


    Maxwell Lol wrote:
    > Since nc sends a one-way stream, I don't see how anything can be
    > negotiated between the two systems. Which is the point of
    > rsync......


    thishost$ nc -u otherhost 60873
    otherhost$ nc -u -l -p 60873

    The two terminal sessions in which I ran these command can now send
    packets between each other. Bidirectionally. (Granted, I missed a
    "-p". Or perhaps there are different versions of nc. I can't remember.)


    > RSYNC_RSH='ssh -f 192.168.1.50 nc -u -l -p 60873; nc -u 192.168.1.50 -p 60873'
    > and was not able to get it to work.


    It probably needs some more work, and since it was an illustration of
    futility (rsync over udp? bletch!) I hope you don't mind if I don't
    spend any more time on it.

    Chris

  13. Re: can I rsync on UDP?

    Maxwell Lol writes:

    >Jagadeesh writes:


    >> Thank you very much all for you time and helping me.
    >> I can get high speed connection. That's my limitation.


    >Assuming you mean you cannot get a high speed connection,
    >there may be other problems.


    >Encryption does slow things down. Do you want security or speed?


    >Is rsync being done over a wide area network? There are ways to tune this.


    >Do you have to use rsync? (instead of compressing a tar file).


    He refuses to tell us anything about his system. "Doctor I am sick". "What
    appears to be the problem" "Doctor I am sick, do you thing a lobotomy would
    help?"


  14. Re: can I rsync on UDP?

    On Tue, 21 Oct 2008 22:47:56 GMT, Unruh wrote:
    > Allen Kistler writes:
    >
    > >Unruh wrote:
    > >> Joe Pfeiffer writes:

    >
    > >FWIW, I've found that rsync over ssh over a 1 Gbit link has a bottleneck
    > >at the platter, not the cpu. Of course, that's with
    > >electrically-connected drives (SAS, SATA).

    >
    > It depends on the computer. I ran tests with a SATA drive on a 2GH machine
    > and Gb ethernet, and it was ssh that was the bottleneck. With a faster
    > machine
    > it is possible that disk access would be the bottleneck. ( If I ran rsync
    > straight-- ie rsync->rsync daemon which does not go through ssh) I got
    > about 50-80% higher throughput.


    As another data point, when I've compared sending a large file (e.g.
    knoppix iso image) via rsync/ssh and netcat, netcat usually took half
    as long. That was done on pata/sata attached drives in my office on
    either 100M or gig ethernet.


    Mike

    --
    Michael Zawrotny
    Institute of Molecular Biophysics
    Florida State University | email: zawrotny@sb.fsu.edu
    Tallahassee, FL 32306-4380 | phone: (850) 644-0069

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2