On Oct 22, 12:23 am, Joe Pfeiffer wrote:
> Unruh writes:
> > Joe Pfeiffer writes:
>
> >>Jagadeesh writes:
>
> >>> its a lame question, but let me ask you Gurus,
>
> >>> I have task of downloading files sized little over 1GB using rsync. It
> >>> is taking lot of time because of my lousy network speed. Just to test
> >>> I want to try rsync over UDP if possible, I am sure someone should
> >>> have done this trial, Can you share ideas about how to do it.
>
> >>Of course, 1GB files will take a long time with pretty darn good
> >>network speed (I for one would regard 5Mbit/sec as pretty good. 1GB
> >>at 5Mb/sec = 2000 seconds = half an hour). Why do you suspect network
> >>speed that would be improved by using UDP? Slow is slow, after
> >>all... FWIW, if there's a UDP-based protocol that rsync can use, I
> >>don't know of it.
>
> > Well, I would say that it depends on the connection. If he has Gb network
> > between the machines, rsync over ssh should give about 30MB/s. with the
> > encryption probably being the bottleneck.
> > If it is over 100Mb link, 10MB/s is achievable.
> > If it is a long distance transfer then an hour would not be out of place.
>
> All good points -- I was starting from the assumption that when he
> said "lousy network speed" he was on something slower than LAN speeds,
> and pointing out that pretty good long distance transfer will still be
> taking quite a while.
>
> > Why he thinks UDP will change his "lousy network speed" I have no idea.
>
> I was wondering that too.

Well. I was talking about slow wan

But good helpful replies.

Thanks

Unruh writes:
>
> No idea. All I know is that port 873 is defined as rsync for both TCP and
> UDP. I have no idea if and how it uses it.

Near as I can tell from reading rsync's man page, it doesn't. It only
actually uses tcp.

Jagadeesh writes:

>On Oct 22, 12:23 am, Joe Pfeiffer wrote:
>> Unruh writes:
>> > Joe Pfeiffer writes:
>>
>> >>Jagadeesh writes:
>>
>> >>> its a lame question, but let me ask you Gurus,
>>
>> >>> I have task of downloading files sized little over 1GB using rsync. It
>> >>> is taking lot of time because of my lousy network speed. Just to test
>> >>> I want to try rsync over UDP if possible, I am sure someone should
>> >>> have done this trial, Can you share ideas about how to do it.
>>
>> >>Of course, 1GB files will take a long time with pretty darn good
>> >>network speed (I for one would regard 5Mbit/sec as pretty good. 1GB
>> >>at 5Mb/sec = 2000 seconds = half an hour). Why do you suspect network
>> >>speed that would be improved by using UDP? Slow is slow, after
>> >>all... FWIW, if there's a UDP-based protocol that rsync can use, I
>> >>don't know of it.
>>
>> > Well, I would say that it depends on the connection. If he has Gb network
>> > between the machines, rsync over ssh should give about 30MB/s. with the
>> > encryption probably being the bottleneck.
>> > If it is over 100Mb link, 10MB/s is achievable.
>> > If it is a long distance transfer then an hour would not be out of place.
>>
>> All good points -- I was starting from the assumption that when he
>> said "lousy network speed" he was on something slower than LAN speeds,
>> and pointing out that pretty good long distance transfer will still be
>> taking quite a while.
>>
>> > Why he thinks UDP will change his "lousy network speed" I have no idea.
>>
>> I was wondering that too.

>Well. I was talking about slow wan

That of course tells me absolutely no more than did "lousy network speeds".
And I still do not know why UDP would make a difference.

Maxwell Lol wrote:
> Rick Jones writes:

> > Maxwell Lol wrote:
> >> *ALL* networks are unreliable. That's why TCP was developed - to
> >> provide reliable transfer of data over a network.
> >
> > Might it be more accurate to say that all IP networks are unreliable?

> Sure. Also XNS, DECNET, SMB/NETBIOS and many others.

> I'm not sure if you are implying that other networks are reliable.

I cannot recall the specifics, but IIRC X.25 has some degree of
"reliability" along the lines of retransmission and whatnot.

rick jones
--
oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Chris Davies wrote:
> Maxwell Lol wrote:
> > *ALL* networks are unreliable [...]

> Rick Jones wrote:
> > Might it be more accurate to say that all IP networks are
> > unreliable?

> Consider two computers linked by {whatever}. I can't see a situation
> where you couldn't (theoretically) break the linkage described by
> {whatever} by some means.

Sure, the question is whether or not there is a network-layer (ish)
protocol that provides reliability a la TCP or what not - ie one with
retransmissions and flow control etc, such that a ULP (Upper Layer
Protocol) sitting on top of it might presume reliability the way an
application using TCP does.

One can "break" a TCP connection but we still refer to TCP as a
"reliable" protocol - even though the only "guarantee" that TCP makes
is that it will tell you that it was _probably_ unable to get some of
your data to the remote destination.

rick jones
--
The glass is neither half-empty nor half-full. The glass has a leak.
The real question is "Can it be patched?"
these opinions are mine, all mine; HP might not want them anyway...
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Thank you very much all for you time and helping me.
I can get high speed connection. That's my limitation.

But all your replies saved my time that I would have spent on R&D
without getting much success.
Thank you again.

Jagadeesh

Jagadeesh writes:

>Thank you very much all for you time and helping me.
>I can get high speed connection. That's my limitation.

I think something went wrong witht he last two sentences.
What doe you define as "high speed".

>But all your replies saved my time that I would have spent on R&D
>without getting much success.
>Thank you again.

>Jagadeesh

>>> rsync has two modes-- one is over ssh (or rsh.) in that mode the local
>>> rsync makes an ssh connection to the remote machine and starts up a version
>>> of rsync on that remote machine to connect the two together.
>>> In the other mode, when the local rsync contacts the remote machines on
>>> port 873, the remote (x)inted starts up a version of rsync and intiates the
>>> transfer. That can be either tcp or udp.(Not sure how it decides which to
>>> use).

I have confirmed that UDP is NEVER used.
As as extra validation, the manual page says

>There are two different ways for rsync to contact a remote system:
>using a remote-shell program as the transport (such as ssh or rsh) or
>contacting an rsync daemon directly via TCP.

>>>That mode is NOT encrypted ( as rsync over ssh is) and can be about
>>> 50% faster depending on your CPU , network,...

I suspect it's because encryption isn't used.

Chris Davies writes:

> Maxwell Lol wrote:
>> If rsync works over UDP, what is the command option? I don't see one.
>
> You could try something like this, I suppose:
>
> RSYNC_RSH='ssh -f %h nc -u -l 60873; nc -u %h 60873'
> rsync -avHP local/files remotehost:/remote/path

Since nc sends a one-way stream, I don't see how anything can be
negotiated between the two systems. Which is the point of
rsync......

Nevertheless, I tried
RSYNC_RSH='ssh -f 192.168.1.50 nc -u -l -p 60873; nc -u 192.168.1.50 -p 60873'
export RSYNC_RSH

and was not able to get it to work.







>
> Chris

Jagadeesh writes:

> Thank you very much all for you time and helping me.
> I can get high speed connection. That's my limitation.

Assuming you mean you cannot get a high speed connection,
there may be other problems.

Encryption does slow things down. Do you want security or speed?

Is rsync being done over a wide area network? There are ways to tune this.

Do you have to use rsync? (instead of compressing a tar file).

Chris Davies writes:
> You could try something like this, I suppose:
> RSYNC_RSH='ssh -f %h nc -u -l 60873; nc -u %h 60873'

Maxwell Lol wrote:
> Since nc sends a one-way stream, I don't see how anything can be
> negotiated between the two systems. Which is the point of
> rsync......

thishost$ nc -u otherhost 60873
otherhost$ nc -u -l -p 60873

The two terminal sessions in which I ran these command can now send
packets between each other. Bidirectionally. (Granted, I missed a
"-p". Or perhaps there are different versions of nc. I can't remember.)


> RSYNC_RSH='ssh -f 192.168.1.50 nc -u -l -p 60873; nc -u 192.168.1.50 -p 60873'
> and was not able to get it to work.

It probably needs some more work, and since it was an illustration of
futility (rsync over udp? bletch!) I hope you don't mind if I don't
spend any more time on it.

Chris

Maxwell Lol writes:

>Jagadeesh writes:

>> Thank you very much all for you time and helping me.
>> I can get high speed connection. That's my limitation.

>Assuming you mean you cannot get a high speed connection,
>there may be other problems.

>Encryption does slow things down. Do you want security or speed?

>Is rsync being done over a wide area network? There are ways to tune this.

>Do you have to use rsync? (instead of compressing a tar file).

He refuses to tell us anything about his system. "Doctor I am sick". "What
appears to be the problem" "Doctor I am sick, do you thing a lobotomy would
help?"

On Tue, 21 Oct 2008 22:47:56 GMT, Unruh wrote:
> Allen Kistler writes:
>
> >Unruh wrote:
> >> Joe Pfeiffer writes:
>
> >FWIW, I've found that rsync over ssh over a 1 Gbit link has a bottleneck
> >at the platter, not the cpu. Of course, that's with
> >electrically-connected drives (SAS, SATA).
>
> It depends on the computer. I ran tests with a SATA drive on a 2GH machine
> and Gb ethernet, and it was ssh that was the bottleneck. With a faster
> machine
> it is possible that disk access would be the bottleneck. ( If I ran rsync
> straight-- ie rsync->rsync daemon which does not go through ssh) I got
> about 50-80% higher throughput.

As another data point, when I've compared sending a large file (e.g.
knoppix iso image) via rsync/ssh and netcat, netcat usually took half
as long. That was done on pata/sata attached drives in my office on
either 100M or gig ethernet.


Mike

--
Michael Zawrotny
Institute of Molecular Biophysics
Florida State University | email: zawrotny@sb.fsu.edu
Tallahassee, FL 32306-4380 | phone: (850) 644-0069