new to rsync - SCO

This is a discussion on new to rsync - SCO ; Steve M. Fabac, Jr. wrote: > > To believe that rsync is only sending the changes in the files and > magically patching the file at the receiving end That sounds skeptical. I don't think Andrew Tridgell faked his PhD ...

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2
Results 21 to 36 of 36

Thread: new to rsync

  1. Re: new to rsync


    Steve M. Fabac, Jr. wrote:

    >
    > To believe that rsync is only sending the changes in the files and
    > magically patching the file at the receiving end


    That sounds skeptical. I don't think Andrew Tridgell faked his PhD
    Thesis! http://samba.org/~tridge/phd_thesis.pdf

    > So in the above, I'd want to see "Total bytes sent: 2,681,520
    > (approximately
    > 10% of the 26,733,444 byte count for the 133 files updated.
    >
    > Is the confusion caused by using rsync in place of cpio -pmvd to copy
    > files from one file system to another file system on the same machine?
    >
    > "Find ./usr1 -mtime -1 | cpio -pmvd /util/backup" works ok but is not as
    > fine grained as rsync and copies the whole file at 10:00, 14:00, 17:00,
    > and 22:00.
    > Where rsync will notice a file that was changed at some time before 10:00
    > and include it in the 10:00 run but omit it from the 14:00 and later runs
    > if it has not been modified after 10:00.
    >
    > What command options do I need to give rsync to "diff and patch" files in
    > different file systems on the same machine rather then just moving the
    > entire
    > file ala find | cpio?
    >


    I suspect you need an rsync client talking to an rsync server.

    Perhaps rsync only uses diffs when it reduces overall time and network
    usage. On a single machine, reading both files, computing differences,
    and writing the new file is presumably more I/O than just copying the
    whole changed file. On two machines separated by a LAN (or WAN) disk I/O
    is probably not the bottleneck and computing diffs becomes advantageous?

    So rsync does whatever is fastest?


    A test:
    I download Rsync for Win32.
    I create a large text file.

    ---------------------------------------------------------------
    C:\Data>exiftool -r photos > photos-exif.txt

    C:\Data>dir photos-exif.txt
    05/10/2008 14:41 71,727,734 photos-exif.txt
    ---------------------------------------------------------------

    I have a USB drive as J: so I use Rsync to copy the file

    ------------------------------------------------------------
    C:\Data>rsync -v photos*.txt /cygdrive/j
    photos-exif.txt

    sent 71736566 bytes received 31 bytes 4099234.11 bytes/sec
    total size is 71727734 speedup is 1.00
    ------------------------------------------------------------

    I made a one byte change using vim and resent it, it sent the whole file
    (same byte counts and speedup factor as before)

    Then I set up an rsync daemon on a nearby ubuntu desktop PC
    I created an rsyncd.conf
    [rgb]
    path = /home/rgb
    list = true
    use chroot = false
    readonly = false
    and started the daemon using
    `rsync --daemon -v -4 --port=1111`


    -------------------------------------------------------------
    C:\Data>rsync -v --port=1111 photos*.txt 192.168.0.4::rgb
    photos-exif.txt

    sent 71736578 bytes received 38 bytes 4347673.70 bytes/sec
    total size is 71727734 speedup is 1.00

    C:\Data>
    -------------------------------------------------------------

    I made a one-byte change in the txt file and retried

    -------------------------------------------------------------
    C:\Data>rsync -v --port=1111 photos*.txt 192.168.0.4::rgb
    photos-exif.txt

    sent 42452 bytes received 59363 bytes 18511.82 bytes/sec
    total size is 71727734 speedup is 704.49
    -------------------------------------------------------------

    One file updated, less than the whole file transferred. I assume the
    42452 includes rsync protocol overheads: filenames, checksums for each
    block and so on.

    Conclusion: Tridge didn't lie in his PhD thesis :-)

    --
    RGB

  2. Re: new to rsync

    RedGrittyBrick wrote:
    >
    > Steve M. Fabac, Jr. wrote:
    >
    >>
    >> To believe that rsync is only sending the changes in the files and
    >> magically patching the file at the receiving end

    >
    > That sounds skeptical. I don't think Andrew Tridgell faked his PhD
    > Thesis! http://samba.org/~tridge/phd_thesis.pdf


    Never meant to imply that he did. Just that my experience to date has not
    been as expected based upon commonly understood operation.

    >
    >> So in the above, I'd want to see "Total bytes sent: 2,681,520
    >> (approximately
    >> 10% of the 26,733,444 byte count for the 133 files updated.
    >>
    >> Is the confusion caused by using rsync in place of cpio -pmvd to copy
    >> files from one file system to another file system on the same machine?
    >>
    >> "Find ./usr1 -mtime -1 | cpio -pmvd /util/backup" works ok but is not as
    >> fine grained as rsync and copies the whole file at 10:00, 14:00,
    >> 17:00, and 22:00.
    >> Where rsync will notice a file that was changed at some time before 10:00
    >> and include it in the 10:00 run but omit it from the 14:00 and later runs
    >> if it has not been modified after 10:00.
    >>
    >> What command options do I need to give rsync to "diff and patch" files in
    >> different file systems on the same machine rather then just moving the
    >> entire
    >> file ala find | cpio?
    >>

    >
    > I suspect you need an rsync client talking to an rsync server.
    >
    > Perhaps rsync only uses diffs when it reduces overall time and network
    > usage. On a single machine, reading both files, computing differences,
    > and writing the new file is presumably more I/O than just copying the
    > whole changed file. On two machines separated by a LAN (or WAN) disk I/O
    > is probably not the bottleneck and computing diffs becomes advantageous?
    >
    > So rsync does whatever is fastest?
    >
    >
    > A test:
    > I download Rsync for Win32.
    > I create a large text file.
    >
    > ---------------------------------------------------------------
    > C:\Data>exiftool -r photos > photos-exif.txt
    >
    > C:\Data>dir photos-exif.txt
    > 05/10/2008 14:41 71,727,734 photos-exif.txt
    > ---------------------------------------------------------------
    >
    > I have a USB drive as J: so I use Rsync to copy the file
    >
    > ------------------------------------------------------------
    > C:\Data>rsync -v photos*.txt /cygdrive/j
    > photos-exif.txt
    >
    > sent 71736566 bytes received 31 bytes 4099234.11 bytes/sec
    > total size is 71727734 speedup is 1.00
    > ------------------------------------------------------------
    >
    > I made a one byte change using vim and resent it, it sent the whole file
    > (same byte counts and speedup factor as before)


    That confirms what I am seeing on my client's system: Rsync sends the whole file
    for file system to file system copies on the same machine.

    On the client's system with 2.1GHZ Xeon 4 core processor, SCO 5.0.7 with MP5,
    Adaptec 2130SLP U320 RAID controller with two U320 disks in RAID1; system is
    idle as all users have been booted off; and, where it is theoretically possible
    to read and checksum both files at the same time by reading data on source and
    target files simultaneously from disk0 or disk1 in the RAID1 pair and locking
    a forked rsync child process to run on one core for the receiving process,
    rsync defaults to performing as it does on a Windows 32 system without
    preemptive multitasking and job scheduling. Rsync may be doing the best that
    it can (or thinks it can).

    >
    > Then I set up an rsync daemon on a nearby ubuntu desktop PC
    > I created an rsyncd.conf
    > [rgb]
    > path = /home/rgb
    > list = true
    > use chroot = false
    > readonly = false
    > and started the daemon using
    > `rsync --daemon -v -4 --port=1111`
    >
    >
    > -------------------------------------------------------------
    > C:\Data>rsync -v --port=1111 photos*.txt 192.168.0.4::rgb
    > photos-exif.txt
    >
    > sent 71736578 bytes received 38 bytes 4347673.70 bytes/sec
    > total size is 71727734 speedup is 1.00
    >
    > C:\Data>
    > -------------------------------------------------------------
    >
    > I made a one-byte change in the txt file and retried
    >
    > -------------------------------------------------------------
    > C:\Data>rsync -v --port=1111 photos*.txt 192.168.0.4::rgb
    > photos-exif.txt
    >
    > sent 42452 bytes received 59363 bytes 18511.82 bytes/sec
    > total size is 71727734 speedup is 704.49
    > -------------------------------------------------------------
    >
    > One file updated, less than the whole file transferred. I assume the
    > 42452 includes rsync protocol overheads: filenames, checksums for each
    > block and so on.


    Can we do the client-server thing on one machine? Perhaps rsync -v --port=1111
    /usr1 127.0.0.1:here's where I get lost: Tell it to copy to /util/backup/usr1,
    preserve permissions, owner, mtime, atime etc...)?

    >
    > Conclusion: Tridge didn't lie in his PhD thesis :-)
    >


    Again, never said that he did. Just trying to understand and use his tool
    to get the fastest file system to file system copy on the same machine
    (and ultimately, from the on-site server to the remote backup server).

    Thank you for your testing efforts to try to illuminate the actions
    of rsync.

    Can I ask you to repeat the tests (both disk to usb and machine to machine)
    after wrapping your rsync command in a batch file that records the start
    and completion times so that we can compare the real-world time to complete
    the operation?

    I would find it instructive to see how the time varies from run to run:
    Original file copy and file with one bite change in your two test cases
    and between local copy and remote system copy.

    --
    Steve Fabac
    S.M. Fabac & Associates
    816/765-1670

  3. Re: new to rsync

    On Sat, Oct 04, 2008, Brian K. White wrote:
    >
    >----- Original Message -----
    >From: "Steve M. Fabac, Jr."
    >Newsgroups: comp.unix.sco.misc
    >To:
    >Sent: Saturday, October 04, 2008 4:20 PM
    >Subject: Re: new to rsync
    >
    >
    >> Bill Campbell wrote:

    ....
    >> NOTE: I got the -avp from an example somewhere that showed typical rsync usage
    >> not from reading the man pages and selecting options (as the subject says: new
    >> to rsync).


    >Basically -a does all the things you would normally want and expect in a
    >"sane" copy.


    The options I normally use for backups are:

    --delete --delete-excluded -aHrxF

    -a Archive, preserve ownership, timestamps, and permissions.

    -H Preserver hard links (not done above)

    -r Recursive

    -x Do not cross file systems.

    --delete
    Delete stuff on destination that does not exist on source.
    I generally use this weekly, preserving deleted files for
    a while in case its necessary to grab something that has
    gone missing.

    --delete-excluded
    Delete files on destination system that are specified in
    .rsync-exclude files (see -F below).

    -F Filter using .rsync-filter files. This allows one to specify
    files that should not be copied. These files may be in any
    directory and apply recursively to subdirectories. As an example,
    here's mine from a /var directory so it won't copy files that
    are useless on backup recovery. I think this was new in
    rsync-3.x and above.

    # start of file
    - *.pid
    - *.lock
    - *.sock
    - postgresql/run/.s.PGSQL*
    # sockets
    - sasl/saslauthd/mux
    - mysql/mysql.sock
    - whoson/run/whoson.s
    - whoson/run/whoson.d
    - postfix/private/tlsmgr
    - postfix/private/rewrite
    - postfix/private/bounce
    - postfix/private/defer
    - postfix/private/trace
    - postfix/private/verify
    - postfix/private/proxymap
    - postfix/private/smtp
    - postfix/private/relay
    - postfix/private/error
    - postfix/private/local
    - postfix/private/virtual
    - postfix/private/lmtp
    - postfix/private/anvil
    - postfix/private/scache
    - postfix/private/smtp-amavis
    - postfix/public/flush
    - postfix/public/showq
    - postfix/public/pre-cleanup
    - postfix/public/cleanup
    - postgresql/run/.s.PGSQL.5432
    - clamav/clamd.sock
    - courier-authlib/spool/authdaemon/socket
    - zope/var/zopectlsock
    # FIFOs
    - postfix/public/pickup
    - postfix/public/qmgr
    - hylafax/FIFO
    # end of file

    >That is, it does the sensible thing wrt symlinks, permissions, ownerships,
    >etc. The one thing you might want different is, by default it will try to
    >convert uid's on the fly, so that files that were owned by user filepro, or
    >user vsifax, etc, on the source system, are still owned by filepro and
    >vsifax on the target system, even if they have different uid numbers on the
    >different systems. Because of exactly those examples, where files need to
    >be owned by special users in order for their applications to work, I think
    >it makes the most sense to do this remapping 99% of the time, even though
    >technically it constitutes a change. If I go to pull the copy back from a
    >backup server onto a live server, th same conversion will happen on the way
    >back so I am left with a local restore that is as if the change never
    >happened. Yet meanwhile, the data "works" on the remote machine instead of
    >being merely a hunk of data on the remote machine.


    >Some of the behaviour is also affected by options in the server side config
    >file, but that's only of you are talking to an rsync daemon directly vs
    >running rsync via ssh or rcmd.


    We always use rsync modules to an rsync server daemon for backups
    as it permits one to restrict systems to specified directories and
    restrict access by IP address or net block. These modules must
    have the option ``use chroot = yes'' or rsync does Strange Things(tm)
    to symbolic links on the destination system (e.g. it strips the
    leasing ``/'' from the symlink as gnu-tar does without the ``p''
    option). This makes life very difficult when restoring.

    We also set ``uid = root'' and ``gid = root (or wheel on BSDish
    systems)'' which preserves ownership on the destination system.

    ....
    Bill
    --
    INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC
    URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way
    Voice: (206) 236-1676 Mercer Island, WA 98040-0820
    Fax: (206) 232-9186

    It's time to feed the hogs
    -- Unintended Consequences

  4. Re: new to rsync


    Steve M. Fabac, Jr. wrote:
    > Can I ask you to repeat the tests (both disk to usb and machine to machine)
    > after wrapping your rsync command in a batch file that records the start
    > and completion times so that we can compare the real-world time to complete
    > the operation?
    >
    > I would find it instructive to see how the time varies from run to run:
    > Original file copy and file with one bite change in your two test cases
    > and between local copy and remote system copy.
    >


    --------------------------------------------------------------
    C:\Data>type rsync.pl
    #!perl
    use strict;
    use warnings;
    print scalar gmtime(), "\n";
    system 'rsync.exe -av photos-exif.txt /cygdrive/j';
    print scalar gmtime(), "\n";
    system 'rsync.exe -av --port=1111 photos-exif.txt 192.168.0.4::rgb';
    print scalar gmtime(), "\n";
    --------------------------------------------------------------
    C:\Data>perl rsync.pl
    Sun Oct 5 22:54:52 2008photos-exif.txt
    sent 71736566 bytes received 31 bytes 3877653.89 bytes/sec
    total size is 71727734 speedup is 1.00
    Sun Oct 5 22:55:10 2008
    photos-exif.txt
    sent 71736578 bytes received 38 bytes 6832058.67 bytes/sec
    total size is 71727734 speedup is 1.00
    Sun Oct 5 22:55:20 2008
    --------------------------------------------------------------
    One byte modified.
    --------------------------------------------------------------
    C:\Data>perl rsync.pl
    Sun Oct 5 23:00:04 2008
    sending incremental file list
    photos-exif.txt
    sent 71736586 bytes received 31 bytes 4347673.76 bytes/sec
    total size is 71727734 speedup is 1.00
    Sun Oct 5 23:00:20 2008
    photos-exif.txt
    sent 42489 bytes received 59363 bytes 29100.57 bytes/sec
    total size is 71727734 speedup is 704.23
    Sun Oct 5 23:00:23 2008
    --------------------------------------------------------------

    It probably needs a greater number of files to eliminate the skewing of
    timings by rsync startup houskeeping etc. A longer delay between runs
    might help eliminate caching effects. Though perhaps my USB drive would
    spin down or something. In other words, I wouldn't read too much into
    the exact timings. YMMV.

    --
    RGB

  5. Re: new to rsync


    ----- Original Message -----
    From: "RedGrittyBrick"
    Newsgroups: comp.unix.sco.misc
    To:
    Sent: Sunday, October 05, 2008 11:01 AM
    Subject: Re: new to rsync


    >
    > Steve M. Fabac, Jr. wrote:
    >
    >>
    >> To believe that rsync is only sending the changes in the files and
    >> magically patching the file at the receiving end

    >
    > That sounds skeptical. I don't think Andrew Tridgell faked his PhD Thesis!
    > http://samba.org/~tridge/phd_thesis.pdf
    >
    >> So in the above, I'd want to see "Total bytes sent: 2,681,520
    >> (approximately
    >> 10% of the 26,733,444 byte count for the 133 files updated.
    >>
    >> Is the confusion caused by using rsync in place of cpio -pmvd to copy
    >> files from one file system to another file system on the same machine?
    >>
    >> "Find ./usr1 -mtime -1 | cpio -pmvd /util/backup" works ok but is not as
    >> fine grained as rsync and copies the whole file at 10:00, 14:00, 17:00,
    >> and 22:00.
    >> Where rsync will notice a file that was changed at some time before 10:00
    >> and include it in the 10:00 run but omit it from the 14:00 and later runs
    >> if it has not been modified after 10:00.
    >>
    >> What command options do I need to give rsync to "diff and patch" files in
    >> different file systems on the same machine rather then just moving the
    >> entire
    >> file ala find | cpio?
    >>

    >
    > I suspect you need an rsync client talking to an rsync server.
    >
    > Perhaps rsync only uses diffs when it reduces overall time and network
    > usage. On a single machine, reading both files, computing differences, and
    > writing the new file is presumably more I/O than just copying the whole
    > changed file. On two machines separated by a LAN (or WAN) disk I/O is
    > probably not the bottleneck and computing diffs becomes advantageous?
    >
    > So rsync does whatever is fastest?
    >
    >
    > A test:
    > I download Rsync for Win32.
    > I create a large text file.
    >
    > ---------------------------------------------------------------
    > C:\Data>exiftool -r photos > photos-exif.txt
    >
    > C:\Data>dir photos-exif.txt
    > 05/10/2008 14:41 71,727,734 photos-exif.txt
    > ---------------------------------------------------------------
    >
    > I have a USB drive as J: so I use Rsync to copy the file
    >
    > ------------------------------------------------------------
    > C:\Data>rsync -v photos*.txt /cygdrive/j
    > photos-exif.txt
    >
    > sent 71736566 bytes received 31 bytes 4099234.11 bytes/sec
    > total size is 71727734 speedup is 1.00
    > ------------------------------------------------------------
    >
    > I made a one byte change using vim and resent it, it sent the whole file
    > (same byte counts and speedup factor as before)


    I just confirmed, when rsyncing from a local file to a local file, rsync
    sends the entire file, if the file is different at all.
    If the target file is already the same, it doesn't re-send it.
    ie:
    time rsync -avv /path/to/file /tmp

    creates /tmp/file

    then I edit one record in /path/to/file
    re-run the exact same time/rsync command, and it sends the entire file
    again, and, takes the same amount of time.

    I was able to force deltas, between the same two paths on the same machine,
    by using the local rsync damon for the target, and a module that would go to
    the same target directory:
    rsync -avv /path/to/file localhost::tmp/file

    I wondered if perhaps there is no or only a little advantage to be gained in
    the local to local case? since both the local and the remote machine are in
    fact the same machine? And so the machine would end up having to do about
    the same disk i/o in some for or other anyways, and so might as well just do
    the simpler copy?

    But that's not true because when I run the same test via the rsync daemon
    running locally, there is a significant speedup, even though I'm forcing it
    to go via tcp when it could use direct file i/o much faster, though at
    least, the tcp is just localhost, still not actually over any wire.

    I didn't realise rsync did that, but, I did say to test by sending a file
    over a network connection. If you had done that you would have seen what we
    were claiming, as advertised.

    And speaking of "as advertised":
    I shouldn't have had to do this test anyways. This whole discussion was
    unjustified because rsync actually told you what it was doing and why.
    The man page is ambiguous about local file behaviour, but the run-time
    messages are clear enough:
    ----------
    nj2:~ # time rsync -avv --stats /u/global/appl/filepro/gocust/key /tmp
    sending incremental file list
    delta-transmission disabled for local transfer or --whole-file
    key
    ----------

    Mystery solved.

    --
    Brian K. White brian@aljex.com http://www.myspace.com/KEYofR
    +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
    filePro BBx Linux SCO FreeBSD #callahans Satriani Filk!


  6. Re: new to rsync

    Brian K. White typed (on Mon, Oct 06, 2008 at 03:15:46AM -0400):
    |
    | ----- Original Message -----
    | From: "RedGrittyBrick"
    | Newsgroups: comp.unix.sco.misc
    | To:
    | Sent: Sunday, October 05, 2008 11:01 AM
    | Subject: Re: new to rsync
    |
    |
    | >
    | > Steve M. Fabac, Jr. wrote:
    | >
    | >>
    | >> To believe that rsync is only sending the changes in the files and
    | >> magically patching the file at the receiving end
    | >
    | > That sounds skeptical. I don't think Andrew Tridgell faked his PhD Thesis!
    | > http://samba.org/~tridge/phd_thesis.pdf
    | >
    | >> So in the above, I'd want to see "Total bytes sent: 2,681,520
    | >> (approximately
    | >> 10% of the 26,733,444 byte count for the 133 files updated.
    | >>
    | >> Is the confusion caused by using rsync in place of cpio -pmvd to copy
    | >> files from one file system to another file system on the same machine?
    | >>
    | >> "Find ./usr1 -mtime -1 | cpio -pmvd /util/backup" works ok but is not as
    | >> fine grained as rsync and copies the whole file at 10:00, 14:00, 17:00,
    | >> and 22:00.
    | >> Where rsync will notice a file that was changed at some time before 10:00
    | >> and include it in the 10:00 run but omit it from the 14:00 and later runs
    | >> if it has not been modified after 10:00.
    | >>
    | >> What command options do I need to give rsync to "diff and patch" files in
    | >> different file systems on the same machine rather then just moving the
    | >> entire
    | >> file ala find | cpio?
    | >>
    | >
    | > I suspect you need an rsync client talking to an rsync server.
    | >
    | > Perhaps rsync only uses diffs when it reduces overall time and network
    | > usage. On a single machine, reading both files, computing differences, and
    | > writing the new file is presumably more I/O than just copying the whole
    | > changed file. On two machines separated by a LAN (or WAN) disk I/O is
    | > probably not the bottleneck and computing diffs becomes advantageous?
    | >
    | > So rsync does whatever is fastest?
    | >
    | >
    | > A test:
    | > I download Rsync for Win32.
    | > I create a large text file.
    | >
    | > ---------------------------------------------------------------
    | > C:\Data>exiftool -r photos > photos-exif.txt
    | >
    | > C:\Data>dir photos-exif.txt
    | > 05/10/2008 14:41 71,727,734 photos-exif.txt
    | > ---------------------------------------------------------------
    | >
    | > I have a USB drive as J: so I use Rsync to copy the file
    | >
    | > ------------------------------------------------------------
    | > C:\Data>rsync -v photos*.txt /cygdrive/j
    | > photos-exif.txt
    | >
    | > sent 71736566 bytes received 31 bytes 4099234.11 bytes/sec
    | > total size is 71727734 speedup is 1.00
    | > ------------------------------------------------------------
    | >
    | > I made a one byte change using vim and resent it, it sent the whole file
    | > (same byte counts and speedup factor as before)
    |
    | I just confirmed, when rsyncing from a local file to a local file, rsync
    | sends the entire file, if the file is different at all.
    | If the target file is already the same, it doesn't re-send it.
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^

    Is this not condisered and Incremental ???
    - Jeff H


    | ie:
    | time rsync -avv /path/to/file /tmp
    |
    | creates /tmp/file
    |
    | then I edit one record in /path/to/file
    | re-run the exact same time/rsync command, and it sends the entire file
    | again, and, takes the same amount of time.
    |
    | I was able to force deltas, between the same two paths on the same machine,
    | by using the local rsync damon for the target, and a module that would go to
    | the same target directory:
    | rsync -avv /path/to/file localhost::tmp/file
    |
    | I wondered if perhaps there is no or only a little advantage to be gained in
    | the local to local case? since both the local and the remote machine are in
    | fact the same machine? And so the machine would end up having to do about
    | the same disk i/o in some for or other anyways, and so might as well just do
    | the simpler copy?
    |
    | But that's not true because when I run the same test via the rsync daemon
    | running locally, there is a significant speedup, even though I'm forcing it
    | to go via tcp when it could use direct file i/o much faster, though at
    | least, the tcp is just localhost, still not actually over any wire.
    |
    | I didn't realise rsync did that, but, I did say to test by sending a file
    | over a network connection. If you had done that you would have seen what we
    | were claiming, as advertised.
    |
    | And speaking of "as advertised":
    | I shouldn't have had to do this test anyways. This whole discussion was
    | unjustified because rsync actually told you what it was doing and why.
    | The man page is ambiguous about local file behaviour, but the run-time
    | messages are clear enough:
    | ----------
    | nj2:~ # time rsync -avv --stats /u/global/appl/filepro/gocust/key /tmp
    | sending incremental file list
    | delta-transmission disabled for local transfer or --whole-file
    | key
    | ----------
    |
    | Mystery solved.
    |
    | --
    | Brian K. White brian@aljex.com http://www.myspace.com/KEYofR
    | +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
    | filePro BBx Linux SCO FreeBSD #callahans Satriani Filk!


  7. Re: new to rsync

    Brian K. White wrote:
    > ----- Original Message -----
    > From: "RedGrittyBrick"
    > Newsgroups: comp.unix.sco.misc
    > To:
    > Sent: Sunday, October 05, 2008 11:01 AM
    > Subject: Re: new to rsync
    >
    >
    >> Steve M. Fabac, Jr. wrote:
    >>
    >>> To believe that rsync is only sending the changes in the files and
    >>> magically patching the file at the receiving end

    >> That sounds skeptical. I don't think Andrew Tridgell faked his PhD Thesis!
    >> http://samba.org/~tridge/phd_thesis.pdf
    >>
    >>> So in the above, I'd want to see "Total bytes sent: 2,681,520
    >>> (approximately
    >>> 10% of the 26,733,444 byte count for the 133 files updated.
    >>>
    >>> Is the confusion caused by using rsync in place of cpio -pmvd to copy
    >>> files from one file system to another file system on the same machine?
    >>>
    >>> "Find ./usr1 -mtime -1 | cpio -pmvd /util/backup" works ok but is not as
    >>> fine grained as rsync and copies the whole file at 10:00, 14:00, 17:00,
    >>> and 22:00.
    >>> Where rsync will notice a file that was changed at some time before 10:00
    >>> and include it in the 10:00 run but omit it from the 14:00 and later runs
    >>> if it has not been modified after 10:00.
    >>>
    >>> What command options do I need to give rsync to "diff and patch" files in
    >>> different file systems on the same machine rather then just moving the
    >>> entire
    >>> file ala find | cpio?
    >>>

    >> I suspect you need an rsync client talking to an rsync server.
    >>
    >> Perhaps rsync only uses diffs when it reduces overall time and network
    >> usage. On a single machine, reading both files, computing differences, and
    >> writing the new file is presumably more I/O than just copying the whole
    >> changed file. On two machines separated by a LAN (or WAN) disk I/O is
    >> probably not the bottleneck and computing diffs becomes advantageous?
    >>
    >> So rsync does whatever is fastest?
    >>
    >>
    >> A test:
    >> I download Rsync for Win32.
    >> I create a large text file.
    >>
    >> ---------------------------------------------------------------
    >> C:\Data>exiftool -r photos > photos-exif.txt
    >>
    >> C:\Data>dir photos-exif.txt
    >> 05/10/2008 14:41 71,727,734 photos-exif.txt
    >> ---------------------------------------------------------------
    >>
    >> I have a USB drive as J: so I use Rsync to copy the file
    >>
    >> ------------------------------------------------------------
    >> C:\Data>rsync -v photos*.txt /cygdrive/j
    >> photos-exif.txt
    >>
    >> sent 71736566 bytes received 31 bytes 4099234.11 bytes/sec
    >> total size is 71727734 speedup is 1.00
    >> ------------------------------------------------------------
    >>
    >> I made a one byte change using vim and resent it, it sent the whole file
    >> (same byte counts and speedup factor as before)

    >
    > I just confirmed, when rsyncing from a local file to a local file, rsync
    > sends the entire file, if the file is different at all.
    > If the target file is already the same, it doesn't re-send it.
    > ie:
    > time rsync -avv /path/to/file /tmp
    >
    > creates /tmp/file
    >
    > then I edit one record in /path/to/file
    > re-run the exact same time/rsync command, and it sends the entire file
    > again, and, takes the same amount of time.
    >
    > I was able to force deltas, between the same two paths on the same machine,
    > by using the local rsync damon for the target, and a module that would go to
    > the same target directory:
    > rsync -avv /path/to/file localhost::tmp/file
    >
    > I wondered if perhaps there is no or only a little advantage to be gained in
    > the local to local case? since both the local and the remote machine are in
    > fact the same machine? And so the machine would end up having to do about
    > the same disk i/o in some for or other anyways, and so might as well just do
    > the simpler copy?
    >
    > But that's not true because when I run the same test via the rsync daemon
    > running locally, there is a significant speedup, even though I'm forcing it
    > to go via tcp when it could use direct file i/o much faster, though at
    > least, the tcp is just localhost, still not actually over any wire.
    >
    > I didn't realise rsync did that, but, I did say to test by sending a file
    > over a network connection. If you had done that you would have seen what we
    > were claiming, as advertised.


    Can I be blamed for my confusion when someone acknowledged as a guru (and
    I don't think that anyone familiar with your contributions to this news group
    will dispute that honorific) and who has used and loved rsync for its
    capabilities didn't realize that rsync copies whole files locally?

    >
    > And speaking of "as advertised":
    > I shouldn't have had to do this test anyways. This whole discussion was
    > unjustified because rsync actually told you what it was doing and why.
    > The man page is ambiguous about local file behaviour, but the run-time
    > messages are clear enough:
    > ----------
    > nj2:~ # time rsync -avv --stats /u/global/appl/filepro/gocust/key /tmp
    > sending incremental file list
    > delta-transmission disabled for local transfer or --whole-file
    > key
    > ----------
    >
    > Mystery solved.


    Brian, thank you for your attention to this issue. However, in my defense,
    there is no messages concerning "delta-transmission disabled for local
    transfer" provided by rsync 3.0.0 when I used the command:

    /usr/local/bin/rsync -avp --stats /usr1 /util/backup >> /tmp/util.$$

    Since I am copying the whole file system and not individual files,
    I never received the message that should have explained the "problem."

    I plead guilty to adopting a tool to get a specific task accomplished for
    a client (creating a way to do snapshot backups of an active application
    four times a day with minimum interruption for the computer users) to
    insure they did not loose a whole day's production when the mysterious
    power problems they were having (that destroyed 15+ SCSI disk, two UNIX
    servers - both the primary and on-site backup server at the same time,
    an MS exchange server, and a web server over a four week period) struck
    again.

    (The power problem was finally traced to incoming AC mains where the power
    company was dropping and restoring one phase of the three phase power
    after hours. The power company replaced faulty equipment at the
    substation - problem solved.)

    Perhaps if there were a specific "--enable-local-delta" option in the long
    list of command options, I might have found it and saved everyone the
    trouble of following this "unjustified" discussion.

    Again, the subject says it all: "new to rsync." This was my first attempt
    to use (and learn) rsync and I stumbled when my results did not conform with
    what I expected from the common reputation of its performance.

    >


    --
    Steve Fabac
    S.M. Fabac & Associates
    816/765-1670

  8. Re: new to rsync

    > | > I made a one byte change using vim and resent it, it sent the whole file
    > | > (same byte counts and speedup factor as before)
    > |
    > | I just confirmed, when rsyncing from a local file to a local file, rsync
    > | sends the entire file, if the file is different at all.
    > | If the target file is already the same, it doesn't re-send it.
    > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^
    >
    > Is this not condisered and Incremental ???


    Only when using rsync from a local drive to a local drive, yes I guess it could be said to be similar to an incrimental backup, if there were such a thing as an incrimental backup that somehow updated the master tape so that you are always left with a single up to the minute master tape, not a master plus one or more small incrimental tapes.
    ( A master that doesn't fill the media with some incrimentals that happen to fit in the remaining space on the media does not count. that is still master + incrimentals, not updated master. )

    And again, this is only when making a local copy, which is not how rsync is most often used.
    The main point of rsync is to make or synchronize a _remote_ copy, and to do so using the theoretically least network traffic necessary, which means sending deltas not whole files. When using rsync over a network it does not behave as above.

    It IS actually handy to use rsync to make local copies too, and usually whenever I need to make a local copy of some tree (instead of one or a few files) I do use it for that too, just because the command line syntax is convenient, but it's not the main reason rsync exists and it's not the main reason people use it, so perhaps they just considered it generally unimportant to use deltas in that mode.

    Consider, generally, when do you want to make a local copy of some big tree of stuff?
    It's usually to perform some one time operation like prepare a modified copy for sending to a customer or make a backup copy before doing something risky to the live copy, or make a milestone copy and compress it for archiving, etc... What you generally do not want to do is actually maintain perpetually two identical copies of something on the same local drive.

    Maintining a copy on a different, but still local drive, like a firewire/usb/e-sata enclosure would be useful and seems like it should benefit from using deltas just about as much as a network. Don't know why rsync defaults to simple copies for local drives in that case. Maybe they just figure most of the time when you might actually want to maintain two indentical copies of something locally, you could probably use raid for that, and the remaining scenarios are too few to worry about, you can use network to localhost if you really need it.

    --
    Brian K. White brian@aljex.com http://www.myspace.com/KEYofR
    +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
    filePro BBx Linux SCO FreeBSD #callahans Satriani Filk!


  9. Re: new to rsync

    Thanks Brian... and all the replied.
    - Jeff H


    Brian K. White typed (on Mon, Oct 06, 2008 at 04:01:54PM -0400):
    | > | > I made a one byte change using vim and resent it, it sent the whole file
    | > | > (same byte counts and speedup factor as before)
    | > |
    | > | I just confirmed, when rsyncing from a local file to a local file, rsync
    | > | sends the entire file, if the file is different at all.
    | > | If the target file is already the same, it doesn't re-send it.
    | > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^
    | >
    | > Is this not condisered and Incremental ???
    |
    | Only when using rsync from a local drive to a local drive, yes I guess it could be said to be similar to an incrimental backup, if there were such a thing as an incrimental backup that somehow updated the master tape so that you are always left with a single up to the minute master tape, not a master plus one or more small incrimental tapes.
    | ( A master that doesn't fill the media with some incrimentals that happen to fit in the remaining space on the media does not count. that is still master + incrimentals, not updated master. )
    |
    | And again, this is only when making a local copy, which is not how rsync is most often used.
    | The main point of rsync is to make or synchronize a _remote_ copy, and to do so using the theoretically least network traffic necessary, which means sending deltas not whole files. When using rsync over a network it does not behave as above.
    |
    | It IS actually handy to use rsync to make local copies too, and usually whenever I need to make a local copy of some tree (instead of one or a few files) I do use it for that too, just because the command line syntax is convenient, but it's not the main reason rsync exists and it's not the main reason people use it, so perhaps they just considered it generally unimportant to use deltas in that mode.
    |
    | Consider, generally, when do you want to make a local copy of some big tree of stuff?
    | It's usually to perform some one time operation like prepare a modified copy for sending to a customer or make a backup copy before doing something risky to the live copy, or make a milestone copy and compress it for archiving, etc... What you generally do not want to do is actually maintain perpetually two identical copies of something on the same local drive.
    |
    | Maintining a copy on a different, but still local drive, like a firewire/usb/e-sata enclosure would be useful and seems like it should benefit from using deltas just about as much as a network. Don't know why rsync defaults to simple copies for local drives in that case. Maybe they just figure most of the time when you might actually want to maintain two indentical copies of something locally, you could probably use raid for that, and the remaining scenarios are too few to worry about, you can use network to localhost if you really need it.
    |
    | --
    | Brian K. White brian@aljex.com http://www.myspace.com/KEYofR
    | +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
    | filePro BBx Linux SCO FreeBSD #callahans Satriani Filk!

    --

  10. Re: new to rsync

    On 2 Oct, 23:17, "Steve M. Fabac, Jr." wrote:
    > How do I tell if rsync is only sending file differences and not the whole
    > file. And if it is sending the whole file, what considerations are
    > needed to make it send differences?
    >
    > Copying directories on the same machine?
    >
    > Copying directories from machine to machine?


    Out of curiousity on this old thread: Why do you care? Are you
    attempting to establish the performance benefit of rsync, or simply
    determine which files have been transferred, and which have only had
    differences sent, and which are untouched? Or what?

  11. Re: new to rsync

    Nico Kadel-Garcia wrote:
    > On 2 Oct, 23:17, "Steve M. Fabac, Jr." wrote:
    >> How do I tell if rsync is only sending file differences and not the whole
    >> file. And if it is sending the whole file, what considerations are
    >> needed to make it send differences?
    >>
    >> Copying directories on the same machine?
    >>
    >> Copying directories from machine to machine?

    >
    > Out of curiousity on this old thread: Why do you care? Are you
    > attempting to establish the performance benefit of rsync, or simply
    > determine which files have been transferred, and which have only had
    > differences sent, and which are untouched? Or what?
    >
    >


    Read the thread, it is all explained therein. But if you don't have access
    to the thread:

    > I plead guilty to adopting a tool to get a specific task accomplished for
    > a client (creating a way to do snapshot backups of an active application
    > four times a day with minimum interruption for the computer users) to
    > insure they did not loose a whole day's production when the mysterious
    > power problems they were having (that destroyed 15+ SCSI disk, two UNIX
    > servers - both the primary and on-site backup server at the same time,
    > an MS exchange server, and a web server over a four week period) struck
    > again.
    >
    > (The power problem was finally traced to incoming AC mains where the power
    > company was dropping and restoring one phase of the three phase power
    > after hours. The power company replaced faulty equipment at the
    > substation - problem solved.)


    And

    >
    > I had previously installed rsync on a client's system and have it set up
    > to copy three file systems (/usr1, /usr2, /usr3) and the /tmp directory to
    > /util/backup on the same system. The purpose is to create a "snapshot" of the
    > applications working directories under /util/backup at 10:00, 14:00, 17:00,
    > 22:00, and 01:00 then allow users back into the application while Backup Edge is used to
    > create a differential backup of /util/backup tree to an ftp resource. It
    > looks to me like rsync is copying all files where the time stamp of the file
    > has been updated due to file activity of the application.


    And

    >
    > Is the confusion caused by using rsync in place of cpio -pmvd to copy files from
    > one file system to another file system on the same machine?
    >
    > "Find ./usr1 -mtime -1 | cpio -pmvd /util/backup" works ok but is not as
    > fine grained as rsync and copies the whole file at 10:00, 14:00, 17:00, and 22:00.
    > Where rsync will notice a file that was changed at some time before 10:00
    > and include it in the 10:00 run but omit it from the 14:00 and later runs
    > if it has not been modified after 10:00.
    >
    > What command options do I need to give rsync to "diff and patch" files in
    > different file systems on the same machine rather then just moving the entire
    > file ala find | cpio?


    And

    > That confirms what I am seeing on my client's system: Rsync sends the whole file
    > for file system to file system copies on the same machine.
    >
    > On the client's system with 2.1GHZ Xeon 4 core processor, SCO 5.0.7 with MP5,
    > Adaptec 2130SLP U320 RAID controller with two U320 disks in RAID1; system is
    > idle as all users have been booted off; and, where it is theoretically possible
    > to read and checksum both files at the same time by reading data on source and
    > target files simultaneously from disk0 or disk1 in the RAID1 pair and locking
    > a forked rsync child process to run on one core for the receiving process,
    > rsync defaults to performing as it does on a Windows 32 system without
    > preemptive multitasking and job scheduling. Rsync may be doing the best that
    > it can (or thinks it can).


    So to date we have learned that rsync defaults to copying 100% of all files
    needing to be copied for file system to file system copies on the same machine.

    Prior to trying rsync, I had been using "find ./usr1 ./usr2 ./usr3 -mtime -1 | cpio ..."
    to copy the data to the /util/backup file system. The logs for the cpio based
    copy showed that it was taking 8-10 minutes while production workers were idle
    (forced off the system to insure data integrity of the copy). I then replaced
    find and cpio with rsync and the time for the copy was reduced to 4-6 minutes.
    But 4-6 minutes is still idle time and the client wanted the original 14:00
    copy point increased to 10:00, 14:00, 17:00, and 22:00 so there was significant
    lost production during the idle time.

    Brian has posted that he was able to force rsync to operate in its normally
    expected mode by running the daemon process on the machine and specifying the
    local host as the target machine in the rsync command line.

    I have verified that on my test system and will be modifying the client's system
    after my tests are completed (tests to develop the necessary rsyncd.conf and cron
    scripts).

    I am testing on my system for the basic reason that I am new to rsync and am attempting
    to use it in a non-standard way.

    Although rsync is trusted in its operation there's no reason to believe that I can't
    botch the implementation on the client's production system and run the risk screwing
    up the client's data or depleting free disk space by having the copied files end up
    in an un-intended file system.


    --
    Steve Fabac
    S.M. Fabac & Associates
    816/765-1670

  12. Re: new to rsync

    Steve M. Fabac, Jr. wrote:

    > Brian has posted that he was able to force rsync to operate in its normally
    > expected mode by running the daemon process on the machine and
    > specifying the
    > local host as the target machine in the rsync command line.
    >
    > I have verified that on my test system and will be modifying the
    > client's system
    > after my tests are completed (tests to develop the necessary rsyncd.conf
    > and cron
    > scripts).



    AHHH. You need to investigate 'rsnapshot', a wrapper for rsync that is
    incredibly useful for managing multiple, time-stamped snapshots. I use it with
    SCO OpenServer myself, and have been using it with Linux for years.


    > I am testing on my system for the basic reason that I am new to rsync
    > and am attempting
    > to use it in a non-standard way.
    >
    > Although rsync is trusted in its operation there's no reason to believe
    > that I can't
    > botch the implementation on the client's production system and run the
    > risk screwing
    > up the client's data or depleting free disk space by having the copied
    > files end up
    > in an un-intended file system.


    Yeah, it's an issue. It's safer to use rsync, in that case, as a daemon that
    provides only read access to designated clients, but it can cause serious
    security issues to provide such access, much like providing NFS shares of your
    / file system.

  13. RE: new to rsync

    Sorry to start a new question on this topic.

    What would be the syntax on making sure when an rsync takes place it would
    overwrite an older file with the newer on with the same name? If the is
    possible..

    Thanks in advance.

    Brandt

    -----Original Message-----
    From: sco-misc-bounces+beppler300=verizon.net@lists.celestial.com
    [mailto:sco-misc-bounces+beppler300=verizon.net@lists.celestial.com] On
    Behalf Of Nico Kadel-Garcia
    Sent: Tuesday, October 07, 2008 8:57 AM
    To: sco-misc@lists.celestial.com
    Subject: Re: new to rsync

    On 2 Oct, 23:17, "Steve M. Fabac, Jr." wrote:
    > How do I tell if rsync is only sending file differences and not the whole
    > file. And if it is sending the whole file, what considerations are
    > needed to make it send differences?
    >
    > Copying directories on the same machine?
    >
    > Copying directories from machine to machine?


    Out of curiousity on this old thread: Why do you care? Are you
    attempting to establish the performance benefit of rsync, or simply
    determine which files have been transferred, and which have only had
    differences sent, and which are untouched? Or what?


  14. Re: new to rsync


    ----- Original Message -----
    From: "Brandt Eppler"
    Newsgroups: comp.unix.sco.misc
    To: "'Nico Kadel-Garcia'" ;
    Sent: Wednesday, October 08, 2008 1:51 PM
    Subject: RE: new to rsync


    > Sorry to start a new question on this topic.
    >
    > What would be the syntax on making sure when an rsync takes place it would
    > overwrite an older file with the newer on with the same name? If the is
    > possible..
    >
    > Thanks in advance.
    >
    > Brandt


    I don't understand the question.
    It always does this.

    --
    Brian K. White brian@aljex.com http://www.myspace.com/KEYofR
    +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
    filePro BBx Linux SCO FreeBSD #callahans Satriani Filk!


  15. Re: new to rsync

    Brandt Eppler wrote:
    > Sorry to start a new question on this topic.
    >
    > What would be the syntax on making sure when an rsync takes place it would
    > overwrite an older file with the newer on with the same name? If the is
    > possible..
    >
    > Thanks in advance.
    >
    > Brandt
    >

    It's built into rsynd; 'rsync -aH' does a great job of transferring
    material, and hardlinks as well, with the timestamps and contents correct.

  16. Re: new to rsync

    On Wed, Oct 08, 2008, Brandt Eppler wrote:
    >Sorry to start a new question on this topic.
    >
    >What would be the syntax on making sure when an rsync takes place it would
    >overwrite an older file with the newer on with the same name? If the is
    >possible..


    That is the default action. The --update option tells rsync
    *NOT* to copy file from the source over a newer file on the
    destination machine. By default it will replace newer files with
    older ones from the source.

    Bill
    --
    INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC
    URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way
    Voice: (206) 236-1676 Mercer Island, WA 98040-0820
    Fax: (206) 232-9186

    A child can go only so far in life without potty training. It is not
    mere coincidence that six of the last seven presidents were potty
    trained, not to mention nearly half of the nation's state legislators.
    -- Dave Barry

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2