new to rsync - SCO

This is a discussion on new to rsync - SCO ; How do I tell if rsync is only sending file differences and not the whole file. And if it is sending the whole file, what considerations are needed to make it send differences? Copying directories on the same machine? Copying ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 36

Thread: new to rsync

  1. new to rsync

    How do I tell if rsync is only sending file differences and not the whole
    file. And if it is sending the whole file, what considerations are
    needed to make it send differences?

    Copying directories on the same machine?

    Copying directories from machine to machine?
    --
    Steve Fabac
    S.M. Fabac & Associates
    816/765-1670

  2. Re: new to rsync

    Steve M. Fabac, Jr. typed (on Thu, Oct 02, 2008 at 05:17:43PM -0500):
    > How do I tell if rsync is only sending file differences and not the whole
    > file. And if it is sending the whole file, what considerations are
    > needed to make it send differences?
    >
    > Copying directories on the same machine?
    >
    > Copying directories from machine to machine?


    Try the same job using rdist and then rsync.
    The latter should be quite a bit swifter.

    I've always been under the impression that "only sending file
    differences" is rsync's default and only mode of operation.

    Have I been deluded?

    --
    JP

  3. Re: new to rsync

    Jean-Pierre Radley wrote:
    > Steve M. Fabac, Jr. typed (on Thu, Oct 02, 2008 at 05:17:43PM -0500):
    >> How do I tell if rsync is only sending file differences and not the whole
    >> file. And if it is sending the whole file, what considerations are
    >> needed to make it send differences?
    >>
    >> Copying directories on the same machine?
    >>
    >> Copying directories from machine to machine?

    >
    > Try the same job using rdist and then rsync.
    > The latter should be quite a bit swifter.
    >
    > I've always been under the impression that "only sending file
    > differences" is rsync's default and only mode of operation.
    >
    > Have I been deluded?
    >

    No, that's how it works - although it does have a "there are too many
    differences, I'll just send the whole lot" point, but it is not
    configurable (and I can't recall what it is, the OP should just consult
    the source, or ask Tridge).

    Cheers,
    Gary B-)

    --
    __________________________________________________ ____________________________
    Armful of chairs: Something some people would not know
    whether you were up them with or not
    - Barry Humphries

  4. Re: new to rsync

    Bill Campbell wrote:
    > On Thu, Oct 02, 2008, Jean-Pierre Radley wrote:
    >> Steve M. Fabac, Jr. typed (on Thu, Oct 02, 2008 at 05:17:43PM -0500):
    >>> How do I tell if rsync is only sending file differences and not the whole
    >>> file. And if it is sending the whole file, what considerations are
    >>> needed to make it send differences?
    >>>
    >>> Copying directories on the same machine?
    >>>
    >>> Copying directories from machine to machine?

    >> Try the same job using rdist and then rsync.
    >> The latter should be quite a bit swifter.
    >>
    >> I've always been under the impression that "only sending file
    >> differences" is rsync's default and only mode of operation.
    >>
    >> Have I been deluded?

    >
    > I don't think so.
    >
    > Add ``--stats'' to the command line causes rsync to generate a
    > fair amount of detail about the transfer.
    >
    > The newer versions of rsync 3.x are considerably more efficient
    > than the older versions. In particular they examine directories
    > and start transferring immediately instead of building the entire
    > file list first.
    >
    > Here is output from the nightly backup of one of our systems (host names
    > obvuscated to protect the guilty :-).
    >
    > 2008-10-02 01:38:12 start backtape.mnt
    > rsync --stats --delete-excluded -aHrxF / backs.example.com::backup_mysys/rootfs/
    >
    > Number of files: 146238
    > Number of files transferred: 61
    > Total file size: 3219449996 bytes
    > Total transferred file size: 18106073 bytes
    > Literal data: 5373676 bytes
    > Matched data: 12732397 bytes
    > File list size: 2950865
    > File list generation time: 0.001 seconds
    > File list transfer time: 0.000 seconds
    > Total bytes sent: 8380274
    > Total bytes received: 84134
    >
    > sent 8380274 bytes received 84134 bytes 129227.60 bytes/sec
    > total size is 3219449996 speedup is 380.35


    The above looks good. While learning about rsync, I used it to copy /stand to /w
    to see the effect. One the copy was competed, I unmounted /dev/boot and executed
    touch unix.re.Z so that it is "newer" then the original unix.re.Z. With the --stats
    added I see:

    # touch unix.re.Z
    # cd /
    # umount /dev/boot
    # mount -r /dev/boot
    # rsync -avp --stats /stand /w
    sending incremental file list
    stand/unix.re.Z

    Number of files: 17
    Number of files transferred: 1
    Total file size: 12454466 bytes
    Total transferred file size: 1370181 bytes
    Literal data: 1370181 bytes
    Matched data: 0 bytes
    File list size: 347
    File list generation time: 0.001 seconds
    File list transfer time: 0.000 seconds
    Total bytes sent: 1370752
    Total bytes received: 35

    sent 1370752 bytes received 35 bytes 2741574.00 bytes/sec
    total size is 12454466 speedup is 9.09

    So a file that did not change other then its time stamp was sent in
    total by rsync when in theory it needed only to sync the date stamp
    of the file:

    # ls -l /stand/unix.re.Z /w/stand/unix.re.Z
    -r--r----- 1 bin mem 1370181 Oct 3 00:45 /stand/unix.re.Z
    -r--r----- 1 bin mem 1370181 Oct 3 00:45 /w/stand/unix.re.Z
    #

    # rsync --version
    rsync version 3.0.0 protocol version 30
    Copyright (C) 1996-2007 by Andrew Tridgell, Wayne Davison, and others.
    Web site: http://rsync.samba.org/
    Capabilities:
    32-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, no IPv6, batchfiles, inplace,
    append, no ACLs, no xattrs, iconv, no symtimes

    To be fair, a 1.3M file is probably below the threshold for bothering to
    check for differences.

    I had previously installed rsync on a client's system and have it set up
    to copy three file systems (/usr1, /usr2, /usr3) and the /tmp directory to
    /util/backup on the same system. The purpose is to create a "snapshot" of the
    applications working directories under /util/backup at 10:00, 14:00, 17:00,
    22:00, and 01:00 then allow users back into the application while Backup Edge is used to
    create a differential backup of /util/backup tree to an ftp resource. It
    looks to me like rsync is copying all files where the time stamp of the file
    has been updated due to file activity of the application.


    This is the rsync section of the cp_to_util.backup script on the client's system:

    echo "copy to util/backup started: \c" > /tmp/util.$$
    date >> /tmp/util.$$
    /usr/local/bin/rsync -avp /usr1 /util/backup >> /tmp/util.$$
    /usr/local/bin/rsync -avp /usr2 /util/backup >> /tmp/util.$$
    /usr/local/bin/rsync -avp /usr3 /util/backup >> /tmp/util.$$
    /usr/local/bin/rsync -avp /tmp /util/backup >> /tmp/util.$$
    echo "copy to util/backup complete: \c" >> /tmp/util.$$
    touch /usr1/cp_to_util_backup.ran


    And the result in /tmp/util.last (Note, I used the listed files
    to execute ls -lds /util/backup/file to get the file sizes and the
    block sizes):


    copy to util/backup started: Fri Oct 3 00:59:10 CDT 2008
    sending incremental file list
    0 -rw------- 1 root sys 0
    6 drwxrwxr-x 2 jennie group 2560
    0 -r-------- 1 jennie auth 0
    146 -rw-rw-r-- 1 jennie group 73728
    52 -rw-rw-r-- 1 jennie group 24902
    2 -rw-rw-r-- 1 jennie group 7
    1722 -rw-rw-r-- 1 craig group 876544
    16548 -rwxrwx--- 1 craig group 8437760
    2 drwxrwxr-x 2 test1 group 1024
    114 -rw------- 1 rgroup group 57344
    0 -rw------- 1 rgroup group 0
    0 -rw------- 1 rgroup group 0
    4 -rw-rw-r-- 1 test1 group 1802
    8 -rw-rw-r-- 1 test1 group 3878
    2 -rw-rw-r-- 1 test1 group 684

    sent 9585095 bytes received 548 bytes 3834257.20 bytes/sec
    total size is 1592346638 speedup is 166.12
    sending incremental file list
    3392 -rwxrwx--- 1 cii group 1728512
    146 -rwxrwx--- 1 cii group 73728
    34 -rwxrwx--- 1 cii group 16384
    4516 -rwxrwx--- 1 cii group 2301952
    16 -rwxrwx--- 1 cii group 8192
    18588 -rw-rw-rw- 1 cii group 9478144
    494544 -rwxrwx--- 1 cii group 252215296
    5914 -rw-rw-r-- 1 cii group 3014656
    1315638 -rwxrwx--- 1 cii group 670973952
    243162 -rwxrwx--- 1 cii group 124010496
    9720 -rwxrwx--- 1 cii group 4956160
    59290 -rw-rw-r-- 1 cii group 30236672

    sent 1099167202 bytes received 256 bytes 11882891.44 bytes/sec
    total size is 4942209559 speedup is 4.50
    sending incremental file list

    ** no files listed for /usr3 **

    sent 142756 bytes received 69 bytes 57130.00 bytes/sec
    total size is 1295823111 speedup is 9072.80
    sending incremental file list
    38 drwxrwxrwt 9 sys sys 17920
    436 -rw------- 1 rgroup group 221617
    126 -rw------- 1 rgroup group 62497
    0 -rw-r--r-- 1 root sys 0
    4 -rw------- 1 root sys 1440
    30 -rw------- 1 root sys 13937

    sent 318025 bytes received 165 bytes 212126.67 bytes/sec
    total size is 331376082 speedup is 1041.44
    copy to util/backup complete: Fri Oct 3 01:00:48 CDT 2008

    So I see speedup ranging from 5.28 to 9111.54 for the three file
    systems and the /tmp directory. And the total time to copy the
    files for the three file systems is 01:39 (m:s). This run was
    at 01:00 just prior to performing a master backup of /util/backup
    to the FTP resource. There is very little work done after 22:00
    so the amount of data moved is less (01:39 elapsed time) vs
    the common times of 6-7 minutes for cp_to_util.backup to complete
    at 10:00, 14:00, and 17:00.

    I've added the --stats directive to the rsync lines in cp_to_util.backup
    and then we'll see what the added information indicates.

    >
    > mysys.example.com: backup / complete
    >
    > rsync --stats --delete-excluded -aHrxF /home/ backs.example.com::backup_mysys/home/
    >
    > Number of files: 5731789
    > Number of files transferred: 1284
    > Total file size: 68589758993 bytes
    > Total transferred file size: 1267868485 bytes
    > Literal data: 328408964 bytes
    > Matched data: 939473525 bytes
    > File list size: 268051084
    > File list generation time: 0.064 seconds
    > File list transfer time: 0.000 seconds
    > Total bytes sent: 598584274
    > Total bytes received: 2724559
    >
    > sent 598584274 bytes received 2724559 bytes 212815.02 bytes/sec
    > total size is 68589758993 speedup is 114.07
    >
    > mysys.example.com: backup /home complete
    >
    > 2008-10-02 02:26:22 complete backtape.mnt
    >
    > Bill


    --
    Steve Fabac
    S.M. Fabac & Associates
    816/765-1670

  5. Re: new to rsync


    ----- Original Message -----
    From: "Jean-Pierre Radley"
    Newsgroups: comp.unix.sco.misc
    To:
    Sent: Thursday, October 02, 2008 6:27 PM
    Subject: Re: new to rsync


    > Steve M. Fabac, Jr. typed (on Thu, Oct 02, 2008 at 05:17:43PM -0500):
    >> How do I tell if rsync is only sending file differences and not the whole
    >> file. And if it is sending the whole file, what considerations are
    >> needed to make it send differences?

    >
    > I've always been under the impression that "only sending file
    > differences" is rsync's default and only mode of operation.
    >
    > Have I been deluded?


    That's the default but not only.

    You can force it to send the whole file with the -W option.

    Brian K. White brian@aljex.com http://www.myspace.com/KEYofR
    +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
    filePro BBx Linux SCO FreeBSD #callahans Satriani Filk!


  6. Re: new to rsync

    Brian K. White typed (on Fri, Oct 03, 2008 at 04:56:20AM -0400):
    |
    | ----- Original Message -----
    | From: "Jean-Pierre Radley"
    | Newsgroups: comp.unix.sco.misc
    | To:
    | Sent: Thursday, October 02, 2008 6:27 PM
    | Subject: Re: new to rsync
    |
    |
    | > Steve M. Fabac, Jr. typed (on Thu, Oct 02, 2008 at 05:17:43PM -0500):
    | >> How do I tell if rsync is only sending file differences and not the whole
    | >> file. And if it is sending the whole file, what considerations are
    | >> needed to make it send differences?
    | >
    | > I've always been under the impression that "only sending file
    | > differences" is rsync's default and only mode of operation.
    | >
    | > Have I been deluded?
    |
    | That's the default but not only.
    |
    | You can force it to send the whole file with the -W option.
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^

    Other then time and allocated HD space, is there a reason
    to NOT use the '-W' flag? In the event of a crash, what good
    would it be to only have a "portion" of an important data file?

    - Jeff H


  7. Re: new to rsync


    Jeff Hyman wrote:
    > Brian K. White typed (on Fri, Oct 03, 2008 at 04:56:20AM -0400):
    > |
    > | ----- Original Message -----
    > | From: "Jean-Pierre Radley"
    > | Newsgroups: comp.unix.sco.misc
    > | To:
    > | Sent: Thursday, October 02, 2008 6:27 PM
    > | Subject: Re: new to rsync
    > |
    > |
    > | > Steve M. Fabac, Jr. typed (on Thu, Oct 02, 2008 at 05:17:43PM -0500):
    > | >> How do I tell if rsync is only sending file differences and not the whole
    > | >> file. And if it is sending the whole file, what considerations are
    > | >> needed to make it send differences?
    > | >
    > | > I've always been under the impression that "only sending file
    > | > differences" is rsync's default and only mode of operation.
    > | >
    > | > Have I been deluded?
    > |
    > | That's the default but not only.
    > |
    > | You can force it to send the whole file with the -W option.
    > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^
    >
    > Other then time and allocated HD space, is there a reason
    > to NOT use the '-W' flag? In the event of a crash, what good
    > would it be to only have a "portion" of an important data file?
    >
    > - Jeff H
    >


    You always "have" the whole data file. If the files doesn't exist at the
    other end rsync sends the whole file. If you subsequently change a small
    bit of the same data file, rsync sends a diff and patches the file at
    the other end (except it doesn't use diff and patch of course).



    --
    RGB

  8. Re: new to rsync

    RedGrittyBrick typed (on Fri, Oct 03, 2008 at 05:15:54PM +0100):
    >
    > Jeff Hyman wrote:
    >> Brian K. White typed (on Fri, Oct 03, 2008 at 04:56:20AM -0400):
    >> | | ----- Original Message ----- | From: "Jean-Pierre Radley"
    >>
    >> | Newsgroups: comp.unix.sco.misc
    >> | To:
    >> | Sent: Thursday, October 02, 2008 6:27 PM
    >> | Subject: Re: new to rsync
    >> | | | > Steve M. Fabac, Jr. typed (on Thu, Oct 02, 2008 at 05:17:43PM
    >> -0500):
    >> | >> How do I tell if rsync is only sending file differences and not the whole
    >> | >> file. And if it is sending the whole file, what considerations are
    >> | >> needed to make it send differences?
    >> | > | > I've always been under the impression that "only sending file
    >> | > differences" is rsync's default and only mode of operation.
    >> | > | > Have I been deluded?
    >> | | That's the default but not only.
    >> | | You can force it to send the whole file with the -W option.
    >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^
    >>
    >> Other then time and allocated HD space, is there a reason
    >> to NOT use the '-W' flag? In the event of a crash, what good
    >> would it be to only have a "portion" of an important data file?
    >>
    >> - Jeff H
    >>

    >
    > You always "have" the whole data file. If the files doesn't exist at the
    > other end rsync sends the whole file. If you subsequently change a small
    > bit of the same data file, rsync sends a diff and patches the file at the
    > other end (except it doesn't use diff and patch of course).
    > --
    > RGB


    Very cool indeed.
    To be able to do this, I assume its a block-by-block approach.
    What is the method of verification that all data made it across the pipe?
    cksum? bit-level? and is there any issues with file-locking? as in
    someone has changed a phone number to a clients name in a table while
    that file is being digested by rsync?

    TIA!!!!
    Jeff H

  9. Re: new to rsync

    On Fri, Oct 03, 2008, Jeff Hyman wrote:
    >RedGrittyBrick typed (on Fri, Oct 03, 2008 at 05:15:54PM +0100):
    >>
    >> Jeff Hyman wrote:
    >>> Brian K. White typed (on Fri, Oct 03, 2008 at 04:56:20AM -0400):
    >>> | | ----- Original Message ----- | From: "Jean-Pierre Radley"
    >>>
    >>> | Newsgroups: comp.unix.sco.misc
    >>> | To:
    >>> | Sent: Thursday, October 02, 2008 6:27 PM
    >>> | Subject: Re: new to rsync
    >>> | | | > Steve M. Fabac, Jr. typed (on Thu, Oct 02, 2008 at 05:17:43PM
    >>> -0500):
    >>> | >> How do I tell if rsync is only sending file differences and not the whole
    >>> | >> file. And if it is sending the whole file, what considerations are
    >>> | >> needed to make it send differences?
    >>> | > | > I've always been under the impression that "only sending file
    >>> | > differences" is rsync's default and only mode of operation.
    >>> | > | > Have I been deluded?
    >>> | | That's the default but not only.
    >>> | | You can force it to send the whole file with the -W option.
    >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^
    >>>
    >>> Other then time and allocated HD space, is there a reason
    >>> to NOT use the '-W' flag? In the event of a crash, what good
    >>> would it be to only have a "portion" of an important data file?
    >>>
    >>> - Jeff H
    >>>

    >>
    >> You always "have" the whole data file. If the files doesn't exist at the
    >> other end rsync sends the whole file. If you subsequently change a small
    >> bit of the same data file, rsync sends a diff and patches the file at the
    >> other end (except it doesn't use diff and patch of course).
    >> --
    >> RGB

    >
    >Very cool indeed.
    >To be able to do this, I assume its a block-by-block approach.
    >What is the method of verification that all data made it across the pipe?
    >cksum? bit-level? and is there any issues with file-locking? as in
    >someone has changed a phone number to a clients name in a table while
    >that file is being digested by rsync?


    The method rsync uses is quite neat. Andrew Tridgell, author of
    rsync ans samba, did a presentation to the Seattle Unix group
    where he described the method rsync uses which was far smarter
    than I could understand. Amongst other things rsync looks at
    chunks of the files using things like md5 or sha1 digests to
    identify identical blocks, and can even pick up some reordering
    of the files. It does build a new copy of the file which is
    moved to replace the original when the new one is ready. This
    does require that there be sufficient disk space available on the
    destination file system for two copies of the largest file.

    http://rsync.samba.org/

    Rsync and Larry Wall's ``patch'' program are two programs which
    are basically magic. Andrew and Larry are responsible for
    several of the most useful FOSS software today, Andrew with rsync
    and samba, Larry with patch and perl.

    FWIW, Several members of the Samba core team testified at length
    in the EU's investigation of Microsoft, and were instrumental in
    the EU's understanding of Microsoft's obfuscation of networking
    protocols and ``Embrace and Extend'' activities.

    Bill
    --
    INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC
    URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way
    Voice: (206) 236-1676 Mercer Island, WA 98040-0820
    Fax: (206) 232-9186

    Never blame a legislative body for not doing something. When they do
    nothing, that don't hurt anybody. When they do something is when they
    become dangerous. -- Will Rogers

  10. Re: new to rsync


    Jeff Hyman wrote:
    > RedGrittyBrick typed (on Fri, Oct 03, 2008 at 05:15:54PM +0100):
    >> Jeff Hyman wrote:
    >>> Brian K. White typed (on Fri, Oct 03, 2008 at 04:56:20AM -0400):
    >>> | | ----- Original Message ----- | From: "Jean-Pierre Radley"
    >>>
    >>> | Newsgroups: comp.unix.sco.misc
    >>> | To:
    >>> | Sent: Thursday, October 02, 2008 6:27 PM
    >>> | Subject: Re: new to rsync
    >>> | | | > Steve M. Fabac, Jr. typed (on Thu, Oct 02, 2008 at 05:17:43PM
    >>> -0500):
    >>> | >> How do I tell if rsync is only sending file differences and not the whole
    >>> | >> file. And if it is sending the whole file, what considerations are
    >>> | >> needed to make it send differences?
    >>> | > | > I've always been under the impression that "only sending file
    >>> | > differences" is rsync's default and only mode of operation.
    >>> | > | > Have I been deluded?
    >>> | | That's the default but not only.
    >>> | | You can force it to send the whole file with the -W option.
    >>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^
    >>>
    >>> Other then time and allocated HD space, is there a reason
    >>> to NOT use the '-W' flag? In the event of a crash, what good
    >>> would it be to only have a "portion" of an important data file?
    >>>
    >>> - Jeff H
    >>>

    >> You always "have" the whole data file. If the files doesn't exist at the
    >> other end rsync sends the whole file. If you subsequently change a small
    >> bit of the same data file, rsync sends a diff and patches the file at the
    >> other end (except it doesn't use diff and patch of course).
    >> --
    >> RGB

    >
    > Very cool indeed.
    > To be able to do this, I assume its a block-by-block approach.
    > What is the method of verification that all data made it across the pipe?
    > cksum? bit-level?


    I don't know the details but Wikipedia describes the algorithm a little
    http://en.wikipedia.org/wiki/Rsync

    > and is there any issues with file-locking? as in
    > someone has changed a phone number to a clients name in a table while
    > that file is being digested by rsync?


    IIRC rsync emits an error message when it detects that a file has been
    changed whilst rsync was transmitting it. It carries on with the rainder
    of the job.

    -- later --

    "Rsync does no locking. Which means that: if you are modifying a file
    while it's being transferred, then probably the checksum will fail and
    it'll go round again. And if it goes around twice, and it still fails,
    then it prints a message saying; Error, checksum failed, file changed
    during transfer?"

    http://olstrans.sourceforge.net/rele...000-rsync.html

    --
    RGB

  11. Re: new to rsync

    RedGrittyBrick typed (on Fri, Oct 03, 2008 at 07:22:41PM +0100):
    >
    > Jeff Hyman wrote:
    >> RedGrittyBrick typed (on Fri, Oct 03, 2008 at 05:15:54PM +0100):
    >>> Jeff Hyman wrote:
    >>>> Brian K. White typed (on Fri, Oct 03, 2008 at 04:56:20AM -0400):
    >>>> | | ----- Original Message ----- | From: "Jean-Pierre Radley"
    >>>>
    >>>> | Newsgroups: comp.unix.sco.misc
    >>>> | To:
    >>>> | Sent: Thursday, October 02, 2008 6:27 PM
    >>>> | Subject: Re: new to rsync
    >>>> | | | > Steve M. Fabac, Jr. typed (on Thu, Oct 02, 2008 at 05:17:43PM
    >>>> -0500):
    >>>> | >> How do I tell if rsync is only sending file differences and not the whole
    >>>> | >> file. And if it is sending the whole file, what considerations are
    >>>> | >> needed to make it send differences?
    >>>> | > | > I've always been under the impression that "only sending file
    >>>> | > differences" is rsync's default and only mode of operation.
    >>>> | > | > Have I been deluded?
    >>>> | | That's the default but not only.
    >>>> | | You can force it to send the whole file with the -W option.
    >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^
    >>>>
    >>>> Other then time and allocated HD space, is there a reason
    >>>> to NOT use the '-W' flag? In the event of a crash, what good
    >>>> would it be to only have a "portion" of an important data file?
    >>>>
    >>>> - Jeff H
    >>>>
    >>> You always "have" the whole data file. If the files doesn't exist at the
    >>> other end rsync sends the whole file. If you subsequently change a small
    >>> bit of the same data file, rsync sends a diff and patches the file at the
    >>> other end (except it doesn't use diff and patch of course).
    >>> --
    >>> RGB

    >>
    >> Very cool indeed.
    >> To be able to do this, I assume its a block-by-block approach.
    >> What is the method of verification that all data made it across the pipe?
    >> cksum? bit-level?

    >
    > I don't know the details but Wikipedia describes the algorithm a little
    > http://en.wikipedia.org/wiki/Rsync
    >
    >> and is there any issues with file-locking? as in someone has changed a
    >> phone number to a clients name in a table while
    >> that file is being digested by rsync?

    >
    > IIRC rsync emits an error message when it detects that a file has been
    > changed whilst rsync was transmitting it. It carries on with the rainder of
    > the job.
    >
    > -- later --
    >
    > "Rsync does no locking. Which means that: if you are modifying a file while
    > it's being transferred, then probably the checksum will fail and it'll go
    > round again. And if it goes around twice, and it still fails, then it
    > prints a message saying; Error, checksum failed, file changed during
    > transfer?"
    >
    > http://olstrans.sourceforge.net/rele...000-rsync.html
    >
    > --
    > RGB


    Well guys... some great feedback and info.
    With this said, how are the Databases handled?
    ie: Oracle, Progress, Informix, etc. that have some real
    file locking issues (and solutions) and they
    get put into a "roll-over" mode. Is this something
    rsync gets involved with, or doesn't care about...
    as it's a chore that should be addressed prior to
    upload? Bottom line, what if a company wants a real-time
    "snap-shot" of a database, so if a recovery from a crash
    and all the pieces of data are in sync?


    - Jeff H

  12. Re: new to rsync

    Jeff Hyman wrote:
    [SNIP]
    Simple ad short - rsync knows *nothing* about locks.

    If a file is set to use mandatory locking, and something else opens it,
    rsync will *not* be able to open it, and the transfer of that file will
    fail.

    If access to a file is controlled by cooperative locking, rsync won't
    even notice and will transfer the file.

    If you are trying to copy around database files, it is best to quiesce
    the system before you do it, just to simplify matters.

    Rsync sort of does this:
    cp sys1:source_file sys2:.target_file.$PID
    mv sys2:.target_file.$PID sys2:source_file

    Where the "cp" only copies changed "bits" and pulls in unchanged bits
    from the target system. Again, this is "sort of", you can read up on
    the site:

    And it checks things all over the place.

    If the source file changes during the transfer, it will throw an error
    and stop that transfer. If it is a multi-file transfer it will remember
    the failure and come back at the end and re-do it.

    And this sort of failure is often a good indicator that the entire
    transfer should be re-done, as other files before the failure may have
    changed.

    That sentence doesn't make sense, I'll try to illustrate:
    rsync file1 target
    rsync file2 target (While sync'ing file2, both it and file1 are changed)
    rsync tells you that file2 has failed, goes on
    rsync file3 target
    ...
    rsync filen target
    rsync file2 target
    But file1, which has changed, will not be changed on the target.

    It *pays* to read the output of an rsync command. I know, I mirror a
    few hundred gig from across the pond using rsync, and when you miss one
    little error, it can bugger up a whole load of stuff.

    Cheers,
    Gary B-)

    --
    __________________________________________________ ____________________________
    Armful of chairs: Something some people would not know
    whether you were up them with or not
    - Barry Humphries

  13. Re: new to rsync

    Bill Campbell wrote:
    > On Fri, Oct 03, 2008, Jeff Hyman wrote:
    >> RedGrittyBrick typed (on Fri, Oct 03, 2008 at 05:15:54PM +0100):
    >>> Jeff Hyman wrote:
    >>>> Brian K. White typed (on Fri, Oct 03, 2008 at 04:56:20AM -0400):
    >>>> | | ----- Original Message ----- | From: "Jean-Pierre Radley"
    >>>>
    >>>> | Newsgroups: comp.unix.sco.misc
    >>>> | To:
    >>>> | Sent: Thursday, October 02, 2008 6:27 PM
    >>>> | Subject: Re: new to rsync
    >>>> | | | > Steve M. Fabac, Jr. typed (on Thu, Oct 02, 2008 at 05:17:43PM
    >>>> -0500):
    >>>> | >> How do I tell if rsync is only sending file differences and not the whole
    >>>> | >> file. And if it is sending the whole file, what considerations are
    >>>> | >> needed to make it send differences?
    >>>> | > | > I've always been under the impression that "only sending file
    >>>> | > differences" is rsync's default and only mode of operation.
    >>>> | > | > Have I been deluded?
    >>>> | | That's the default but not only.
    >>>> | | You can force it to send the whole file with the -W option.
    >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^
    >>>>
    >>>> Other then time and allocated HD space, is there a reason
    >>>> to NOT use the '-W' flag? In the event of a crash, what good
    >>>> would it be to only have a "portion" of an important data file?
    >>>>
    >>>> - Jeff H
    >>>>
    >>> You always "have" the whole data file. If the files doesn't exist at the
    >>> other end rsync sends the whole file. If you subsequently change a small
    >>> bit of the same data file, rsync sends a diff and patches the file at the
    >>> other end (except it doesn't use diff and patch of course).
    >>> --
    >>> RGB

    >> Very cool indeed.
    >> To be able to do this, I assume its a block-by-block approach.
    >> What is the method of verification that all data made it across the pipe?
    >> cksum? bit-level? and is there any issues with file-locking? as in
    >> someone has changed a phone number to a clients name in a table while
    >> that file is being digested by rsync?

    >
    > The method rsync uses is quite neat. Andrew Tridgell, author of
    > rsync ans samba, did a presentation to the Seattle Unix group
    > where he described the method rsync uses which was far smarter
    > than I could understand. Amongst other things rsync looks at
    > chunks of the files using things like md5 or sha1 digests to
    > identify identical blocks, and can even pick up some reordering
    > of the files. It does build a new copy of the file which is
    > moved to replace the original when the new one is ready. This
    > does require that there be sufficient disk space available on the
    > destination file system for two copies of the largest file.
    >
    > http://rsync.samba.org/
    >
    > Rsync and Larry Wall's ``patch'' program are two programs which
    > are basically magic. Andrew and Larry are responsible for
    > several of the most useful FOSS software today, Andrew with rsync
    > and samba, Larry with patch and perl.
    >
    > FWIW, Several members of the Samba core team testified at length
    > in the EU's investigation of Microsoft, and were instrumental in
    > the EU's understanding of Microsoft's obfuscation of networking
    > protocols and ``Embrace and Extend'' activities.
    >
    > Bill


    Bill,

    I don't see it. This is the log of the copy of /usr1 /usr2 /usr3 /tmp to
    /util/backup/usr1 ... /usr2 .../usr3 .../tmp at 10:00 on the client's active
    system. It looks like rsync is transferring over 100% of the bytes represented
    by the accumulated file sizes. If so, then I am doing something wrong with
    the command options I am using:

    echo "copy to util/backup started: \c" > /tmp/util.$$
    date >> /tmp/util.$$
    /usr/local/bin/rsync -avp --stats /usr1 /util/backup >> /tmp/util.$$
    /usr/local/bin/rsync -avp --stats /usr2 /util/backup >> /tmp/util.$$
    /usr/local/bin/rsync -avp --stats /usr3 /util/backup >> /tmp/util.$$
    /usr/local/bin/rsync -avp --stats /tmp /util/backup >> /tmp/util.$$
    echo "copy to util/backup complete: \c" >> /tmp/util.$$
    touch /usr1/cp_to_util_backup.ran

    From util.last at 10:20 (NOTE: I converted the list of files transfered to
    a script "ls -lsd path/file" in /util.last, then marked the start of the list
    and the end of the list and executed ":'a,'b!sh" to replace the list with the
    results of ls. I edited the line 'a,'bs/Oct.*//, then ran the command
    'a,'b!gawk ' { printf"\%s \%15d\n", $0, count+= $6 } ' to create the running
    total to the right of the file size listed by ls -lds I then added the comma's
    in the output of gawk and --stats reports to make them easer to read **now is see -h**):

    copy to util/backup started: Fri Oct 3 10:00:10 CDT 2008
    *START /usr1 <-- Manual mark
    sending incremental file list
    48 -rw------- 1 root sys 22563 22563
    0 -rw------- 1 root sys 0 22563
    2 drwxr-xr-x 2 charlene group 512 23075
    0 -r-------- 1 charlene auth 0 23075
    16 -rw-rw-r-- 1 charlene group 8192 31267
    2 -rw-rw-r-- 1 charlene group 9 31276
    2 drwxr-xr-x 2 cpdebbie group 512 31788
    0 -r-------- 1 cpdebbie auth 0 31788
    2 -rw-rw-r-- 1 cpdebbie group 9 31797
    ....
    726 -rwxrwx--- 1 craig group 368640 4716280
    9142 -rw-rw-r-- 1 craig group 4661248 9377528
    1722 -rw-rw-r-- 1 craig group 876544 10254072
    16548 -rwxrwx--- 1 craig group 8437760 18691832
    1464 -rw-rw-r-- 1 test1 group 745472 19437304
    114 -rw-rw-rw- 1 group group 57344 19494648
    ....
    4180 -rw------- 1 rgroup group 2129920 26434816
    582 -rw------- 1 rgroup group 294912 26729728
    0 -rw------- 1 rgroup group 0 26729728
    0 -rw------- 1 rgroup group 0 26729728
    6 -rwxrwxrwx 1 rgroup group 2961 26732689
    2 -rw-rw-r-- 1 test1 group 755 26,733,444

    Number of files: 5757
    Number of files transferred: 133
    Total file size: 1,592,929,997 bytes
    Total transferred file size: 26,699,652 bytes
    Literal data: 26699652 bytes
    Matched data: 0 bytes
    File list size: 106441
    File list generation time: 0.001 seconds
    File list transfer time: 0.000 seconds
    Total bytes sent: 26,815,208 == 100.305%
    Total bytes received: 2930

    sent 26815208 bytes received 2930 bytes 4125867.38 bytes/sec
    total size is 1592929997 speedup is 59.40
    *END /usr1 <-- Manual mark

    *START /usr2
    sending incremental file list
    16 -rw-rw-r-- 1 craig group 8192 8192
    178 -rw-rw-r-- 1 craig group 90112 98304
    16 -rw-rw-r-- 1 craig group 8192 106496
    34 -rw-rw-r-- 1 craig group 16384 122880
    50 -rw-rw-r-- 1 craig group 24576 147456
    34 -rw-rw-r-- 1 craig group 16384 163840
    ....
    52142 -rw-rw-r-- 1 test1 group 26591232 1730723840
    1315766 -rwxrwx--- 1 cii group 671039488 2401763328
    243162 -rwxrwx--- 1 cii group 124010496 2525773824
    9720 -rwxrwx--- 1 cii group 4956160 2530729984
    59290 -rw-rw-r-- 1 cii group 30236672 2560966656
    130 -rwxrwx--- 1 cii group 65536 2561032192
    194 -rwxrwx--- 1 cii group 98304 2561130496
    66 -rwxrwx--- 1 cii group 32768 2561163264
    162 -rw-rw-r-- 1 cii group 81920 2561245184
    1256 -rwxrwx--- 1 cii group 638976 2,561,884,160

    Number of files: 927
    Number of files transferred: 65
    Total file size: 4,942,537,239 bytes
    Total transferred file size: 2,561,884,160 bytes
    Literal data: 2,561,884,160 bytes
    Matched data: 0 bytes
    File list size: 18393
    File list generation time: 0.001 seconds
    File list transfer time: 0.000 seconds
    Total bytes sent: 2,562,217,967 == 100.013% of original file sizes
    Total bytes received: 1261

    sent 2,562,217,967 bytes received 1261 bytes 11567581.16 bytes/sec
    total size is 4,942,537,239 speedup is 1.93
    *END /usr2

    *START /usr3
    sending incremental file list

    Number of files: 8206
    Number of files transferred: 0
    Total file size: 1,295,823,111 bytes
    Total transferred file size: 0 bytes
    Literal data: 0 bytes
    Matched data: 0 bytes
    File list size: 142686
    File list generation time: 0.001 seconds
    File list transfer time: 0.000 seconds
    Total bytes sent: 142756
    Total bytes received: 69

    sent 142,756 bytes received 69 bytes 57130.00 bytes/sec
    total size is 1,295,823,111 speedup is 9072.80
    *END /usr3

    *START /tmp
    sending incremental file list
    38 drwxrwxrwt 9 sys sys 18432 18432
    1408 -rw-rw-r-- 1 donna group 715981 734413
    48 -rw-rw-r-- 1 debbie group 22654 757067
    192 -rw-rw-r-- 1 melanie group 96933 854000
    2600 -rw------- 1 rgroup group 1323482 2177482
    2566 -rw------- 1 rgroup group 1307067 3484549
    48 -rw-rw-r-- 1 test1 group 22765 3507314
    ....
    480 -rw------- 1 rgroup group 244341 15387673
    480 -rw------- 1 rgroup group 244341 15632014
    0 -rw-r--r-- 1 root sys 0 15632014
    54 -rw------- 1 root sys 25667 15657681
    2 -rw-rw-r-- 1 root sys 963 15658644
    18 -rw------- 1 root sys 9130 15667774
    4 -rw------- 1 root sys 1631 15669405
    6 -rw-rw-r-- 1 root sys 2284 15,671,689

    Number of files: 914
    Number of files transferred: 34
    Total file size: 331,920,054 bytes
    Total transferred file size: 15,652,585 bytes
    Literal data: 15,653,257 bytes
    Matched data: 0 bytes
    File list size: 18294
    File list generation time: 0.026 seconds
    File list transfer time: 0.000 seconds
    Total bytes sent: 15,674,928 == 100.020% of original file sizes
    Total bytes received: 716

    sent 15,674,928 bytes received 716 bytes 1492918.48 bytes/sec
    total size is 331,920,054 speedup is 21.17
    *END /tmp
    copy to util/backup complete: Fri Oct 3 10:04:10 CDT 2008

    Everybody responding to this thread is saying that the default is to send only the
    differences (unless -w is set) but I don't see that happening in the above
    listing.

    Tell me where I am going wrong and what I need to do the get the "default" action
    of sending only the differences: rsync -avp --stats does not do it.

    NOTE: I got the -avp from an example somewhere that showed typical rsync usage
    not from reading the man pages and selecting options (as the subject says: new
    to rsync).
    --
    Steve Fabac
    S.M. Fabac & Associates
    816/765-1670

  14. Re: new to rsync

    On Sat, Oct 04, 2008, Steve M. Fabac, Jr. wrote:
    >Bill Campbell wrote:

    ....
    >Bill,
    >
    >I don't see it. This is the log of the copy of /usr1 /usr2 /usr3 /tmp to
    >/util/backup/usr1 ... /usr2 .../usr3 .../tmp at 10:00 on the client's active
    >system. It looks like rsync is transferring over 100% of the bytes represented
    >by the accumulated file sizes. If so, then I am doing something wrong with
    >the command options I am using:


    Look at the numbers where it says ``speedup is xxx.xx'' for an
    indication of the time savings over an initial rsync when the
    speedup time is likely to be something like 0.98.

    ....
    >Number of files: 5757
    >Number of files transferred: 133
    >Total file size: 1,592,929,997 bytes
    >Total transferred file size: 26,699,652 bytes
    >Literal data: 26699652 bytes
    >Matched data: 0 bytes
    >File list size: 106441
    >File list generation time: 0.001 seconds
    >File list transfer time: 0.000 seconds
    >Total bytes sent: 26,815,208 == 100.305%
    >Total bytes received: 2930
    >
    >sent 26815208 bytes received 2930 bytes 4125867.38 bytes/sec
    >total size is 1592929997 speedup is 59.40


    The total sice is 1.6GB, and it send 26.7MB of data. That's a
    fair saving.

    Bill
    --
    INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC
    URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way
    Voice: (206) 236-1676 Mercer Island, WA 98040-0820
    Fax: (206) 232-9186

    That rifle on the wall of the labourer's cottage or working class flat is
    the symbol of democracy. It is our job to see that it stays there.
    --GEORGE ORWELL

  15. Re: new to rsync


    ----- Original Message -----
    From: "Steve M. Fabac, Jr."
    Newsgroups: comp.unix.sco.misc
    To:
    Sent: Saturday, October 04, 2008 4:20 PM
    Subject: Re: new to rsync


    > Bill Campbell wrote:
    >> On Fri, Oct 03, 2008, Jeff Hyman wrote:
    >>> RedGrittyBrick typed (on Fri, Oct 03, 2008 at 05:15:54PM +0100):
    >>>> Jeff Hyman wrote:
    >>>>> Brian K. White typed (on Fri, Oct 03, 2008 at 04:56:20AM -0400):
    >>>>> | | ----- Original Message ----- | From: "Jean-Pierre Radley"
    >>>>>
    >>>>> | Newsgroups: comp.unix.sco.misc
    >>>>> | To:
    >>>>> | Sent: Thursday, October 02, 2008 6:27 PM
    >>>>> | Subject: Re: new to rsync
    >>>>> | | | > Steve M. Fabac, Jr. typed (on Thu, Oct 02, 2008 at 05:17:43PM
    >>>>> -0500):
    >>>>> | >> How do I tell if rsync is only sending file differences and not the whole
    >>>>> | >> file. And if it is sending the whole file, what considerations are
    >>>>> | >> needed to make it send differences?
    >>>>> | > | > I've always been under the impression that "only sending file
    >>>>> | > differences" is rsync's default and only mode of operation.
    >>>>> | > | > Have I been deluded?
    >>>>> | | That's the default but not only.
    >>>>> | | You can force it to send the whole file with the -W option.
    >>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^
    >>>>>
    >>>>> Other then time and allocated HD space, is there a reason
    >>>>> to NOT use the '-W' flag? In the event of a crash, what good
    >>>>> would it be to only have a "portion" of an important data file?
    >>>>>
    >>>>> - Jeff H
    >>>>>
    >>>> You always "have" the whole data file. If the files doesn't exist at the
    >>>> other end rsync sends the whole file. If you subsequently change a small
    >>>> bit of the same data file, rsync sends a diff and patches the file at the
    >>>> other end (except it doesn't use diff and patch of course).
    >>>> --
    >>>> RGB
    >>> Very cool indeed.
    >>> To be able to do this, I assume its a block-by-block approach.
    >>> What is the method of verification that all data made it across the pipe?
    >>> cksum? bit-level? and is there any issues with file-locking? as in
    >>> someone has changed a phone number to a clients name in a table while
    >>> that file is being digested by rsync?

    >>
    >> The method rsync uses is quite neat. Andrew Tridgell, author of
    >> rsync ans samba, did a presentation to the Seattle Unix group
    >> where he described the method rsync uses which was far smarter
    >> than I could understand. Amongst other things rsync looks at
    >> chunks of the files using things like md5 or sha1 digests to
    >> identify identical blocks, and can even pick up some reordering
    >> of the files. It does build a new copy of the file which is
    >> moved to replace the original when the new one is ready. This
    >> does require that there be sufficient disk space available on the
    >> destination file system for two copies of the largest file.
    >>
    >> http://rsync.samba.org/
    >>
    >> Rsync and Larry Wall's ``patch'' program are two programs which
    >> are basically magic. Andrew and Larry are responsible for
    >> several of the most useful FOSS software today, Andrew with rsync
    >> and samba, Larry with patch and perl.
    >>
    >> FWIW, Several members of the Samba core team testified at length
    >> in the EU's investigation of Microsoft, and were instrumental in
    >> the EU's understanding of Microsoft's obfuscation of networking
    >> protocols and ``Embrace and Extend'' activities.
    >>
    >> Bill

    >
    > Bill,
    >
    > I don't see it. This is the log of the copy of /usr1 /usr2 /usr3 /tmp to
    > /util/backup/usr1 ... /usr2 .../usr3 .../tmp at 10:00 on the client's active
    > system. It looks like rsync is transferring over 100% of the bytes represented
    > by the accumulated file sizes. If so, then I am doing something wrong with
    > the command options I am using:
    >
    > echo "copy to util/backup started: \c" > /tmp/util.$$
    > date >> /tmp/util.$$
    > /usr/local/bin/rsync -avp --stats /usr1 /util/backup >> /tmp/util.$$
    > /usr/local/bin/rsync -avp --stats /usr2 /util/backup >> /tmp/util.$$
    > /usr/local/bin/rsync -avp --stats /usr3 /util/backup >> /tmp/util.$$
    > /usr/local/bin/rsync -avp --stats /tmp /util/backup >> /tmp/util.$$
    > echo "copy to util/backup complete: \c" >> /tmp/util.$$
    > touch /usr1/cp_to_util_backup.ran
    >
    > From util.last at 10:20 (NOTE: I converted the list of files transfered to
    > a script "ls -lsd path/file" in /util.last, then marked the start of the list
    > and the end of the list and executed ":'a,'b!sh" to replace the list with the
    > results of ls. I edited the line 'a,'bs/Oct.*//, then ran the command
    > 'a,'b!gawk ' { printf"\%s \%15d\n", $0, count+= $6 } ' to create the running
    > total to the right of the file size listed by ls -lds I then added the comma's
    > in the output of gawk and --stats reports to make them easer to read **now is see -h**):
    >
    > copy to util/backup started: Fri Oct 3 10:00:10 CDT 2008
    > *START /usr1 <-- Manual mark
    > sending incremental file list
    > 48 -rw------- 1 root sys 22563 22563
    > 0 -rw------- 1 root sys 0 22563
    > 2 drwxr-xr-x 2 charlene group 512 23075
    > 0 -r-------- 1 charlene auth 0 23075
    > 16 -rw-rw-r-- 1 charlene group 8192 31267
    > 2 -rw-rw-r-- 1 charlene group 9 31276
    > 2 drwxr-xr-x 2 cpdebbie group 512 31788
    > 0 -r-------- 1 cpdebbie auth 0 31788
    > 2 -rw-rw-r-- 1 cpdebbie group 9 31797
    > ...
    > 726 -rwxrwx--- 1 craig group 368640 4716280
    > 9142 -rw-rw-r-- 1 craig group 4661248 9377528
    > 1722 -rw-rw-r-- 1 craig group 876544 10254072
    > 16548 -rwxrwx--- 1 craig group 8437760 18691832
    > 1464 -rw-rw-r-- 1 test1 group 745472 19437304
    > 114 -rw-rw-rw- 1 group group 57344 19494648
    > ...
    > 4180 -rw------- 1 rgroup group 2129920 26434816
    > 582 -rw------- 1 rgroup group 294912 26729728
    > 0 -rw------- 1 rgroup group 0 26729728
    > 0 -rw------- 1 rgroup group 0 26729728
    > 6 -rwxrwxrwx 1 rgroup group 2961 26732689
    > 2 -rw-rw-r-- 1 test1 group 755 26,733,444
    >
    > Number of files: 5757
    > Number of files transferred: 133
    > Total file size: 1,592,929,997 bytes
    > Total transferred file size: 26,699,652 bytes
    > Literal data: 26699652 bytes
    > Matched data: 0 bytes
    > File list size: 106441
    > File list generation time: 0.001 seconds
    > File list transfer time: 0.000 seconds
    > Total bytes sent: 26,815,208 == 100.305%
    > Total bytes received: 2930
    >
    > sent 26815208 bytes received 2930 bytes 4125867.38 bytes/sec
    > total size is 1592929997 speedup is 59.40
    > *END /usr1 <-- Manual mark
    >
    > *START /usr2
    > sending incremental file list
    > 16 -rw-rw-r-- 1 craig group 8192 8192
    > 178 -rw-rw-r-- 1 craig group 90112 98304
    > 16 -rw-rw-r-- 1 craig group 8192 106496
    > 34 -rw-rw-r-- 1 craig group 16384 122880
    > 50 -rw-rw-r-- 1 craig group 24576 147456
    > 34 -rw-rw-r-- 1 craig group 16384 163840
    > ...
    > 52142 -rw-rw-r-- 1 test1 group 26591232 1730723840
    > 1315766 -rwxrwx--- 1 cii group 671039488 2401763328
    > 243162 -rwxrwx--- 1 cii group 124010496 2525773824
    > 9720 -rwxrwx--- 1 cii group 4956160 2530729984
    > 59290 -rw-rw-r-- 1 cii group 30236672 2560966656
    > 130 -rwxrwx--- 1 cii group 65536 2561032192
    > 194 -rwxrwx--- 1 cii group 98304 2561130496
    > 66 -rwxrwx--- 1 cii group 32768 2561163264
    > 162 -rw-rw-r-- 1 cii group 81920 2561245184
    > 1256 -rwxrwx--- 1 cii group 638976 2,561,884,160
    >
    > Number of files: 927
    > Number of files transferred: 65
    > Total file size: 4,942,537,239 bytes
    > Total transferred file size: 2,561,884,160 bytes
    > Literal data: 2,561,884,160 bytes
    > Matched data: 0 bytes
    > File list size: 18393
    > File list generation time: 0.001 seconds
    > File list transfer time: 0.000 seconds
    > Total bytes sent: 2,562,217,967 == 100.013% of original file sizes
    > Total bytes received: 1261
    >
    > sent 2,562,217,967 bytes received 1261 bytes 11567581.16 bytes/sec
    > total size is 4,942,537,239 speedup is 1.93
    > *END /usr2
    >
    > *START /usr3
    > sending incremental file list
    >
    > Number of files: 8206
    > Number of files transferred: 0
    > Total file size: 1,295,823,111 bytes
    > Total transferred file size: 0 bytes
    > Literal data: 0 bytes
    > Matched data: 0 bytes
    > File list size: 142686
    > File list generation time: 0.001 seconds
    > File list transfer time: 0.000 seconds
    > Total bytes sent: 142756
    > Total bytes received: 69
    >
    > sent 142,756 bytes received 69 bytes 57130.00 bytes/sec
    > total size is 1,295,823,111 speedup is 9072.80
    > *END /usr3
    >
    > *START /tmp
    > sending incremental file list
    > 38 drwxrwxrwt 9 sys sys 18432 18432
    > 1408 -rw-rw-r-- 1 donna group 715981 734413
    > 48 -rw-rw-r-- 1 debbie group 22654 757067
    > 192 -rw-rw-r-- 1 melanie group 96933 854000
    > 2600 -rw------- 1 rgroup group 1323482 2177482
    > 2566 -rw------- 1 rgroup group 1307067 3484549
    > 48 -rw-rw-r-- 1 test1 group 22765 3507314
    > ...
    > 480 -rw------- 1 rgroup group 244341 15387673
    > 480 -rw------- 1 rgroup group 244341 15632014
    > 0 -rw-r--r-- 1 root sys 0 15632014
    > 54 -rw------- 1 root sys 25667 15657681
    > 2 -rw-rw-r-- 1 root sys 963 15658644
    > 18 -rw------- 1 root sys 9130 15667774
    > 4 -rw------- 1 root sys 1631 15669405
    > 6 -rw-rw-r-- 1 root sys 2284 15,671,689
    >
    > Number of files: 914
    > Number of files transferred: 34
    > Total file size: 331,920,054 bytes
    > Total transferred file size: 15,652,585 bytes
    > Literal data: 15,653,257 bytes
    > Matched data: 0 bytes
    > File list size: 18294
    > File list generation time: 0.026 seconds
    > File list transfer time: 0.000 seconds
    > Total bytes sent: 15,674,928 == 100.020% of original file sizes
    > Total bytes received: 716
    >
    > sent 15,674,928 bytes received 716 bytes 1492918.48 bytes/sec
    > total size is 331,920,054 speedup is 21.17
    > *END /tmp
    > copy to util/backup complete: Fri Oct 3 10:04:10 CDT 2008
    >
    > Everybody responding to this thread is saying that the default is to send only the
    > differences (unless -w is set) but I don't see that happening in the above
    > listing.
    >
    > Tell me where I am going wrong and what I need to do the get the "default" action
    > of sending only the differences: rsync -avp --stats does not do it.
    >
    > NOTE: I got the -avp from an example somewhere that showed typical rsync usage
    > not from reading the man pages and selecting options (as the subject says: new
    > to rsync).



    Basically -a does all the things you would normally want and expect in a "sane" copy.
    That is, it does the sensible thing wrt symlinks, permissions, ownerships, etc. The one thing you might want different is, by default it will try to convert uid's on the fly, so that files that were owned by user filepro, or user vsifax, etc, on the source system, are still owned by filepro and vsifax on the target system, even if they have different uid numbers on the different systems. Because of exactly those examples, where files need to be owned by special users in order for their applications to work, I think it makes the most sense to do this remapping 99% of the time, even though technically it constitutes a change. If I go to pull the copy back from a backup server onto a live server, th same conversion will happen on the way back so I am left with a local restore that is as if the change never happened. Yet meanwhile, the data "works" on the remote machine instead of being merely a hunk of data on the remote machine.

    Some of the behaviour is also affected by options in the server side config file, but that's only of you are talking to an rsync daemon directly vs running rsync via ssh or rcmd.

    then, the most common additions or modifications to -a are:

    -n dry run, don't actually modify any files, but go through all the motions and show what would have been done.

    -v verbose, -vv, -vvv, -vvvv, --progress, --stats more & more verbose

    -z on the fly compression, these days it's basically always good to turn this on, even on a gigabit lan.
    since 3.0.0 or possibly a little earlier, rsync even knows how to not bother compressing some common files that are already compressed so no cpu waste even if syncing an big tree that is 90% .png files etc.

    --del delete files/dirs on the target if they don't exist on the source. generally, though it sounds bad, you want this. leaving thigs in a copy that should have been deleted is basically just a form of data corruption.

    There are a _lot_ of options in rsync. It's very finely tunable. But I almost never have to fiddle with anything beyond the options above. And as you can see, the only one of those options that actually modifies the results is --del.
    Most of my numerous rsync scripts all just have "-az --del" and variables that might add -v for interactive use or remove it for cron. Some of my fancier stcripts use filters which are a way to include or exclude various files from a job instead of having to accept the whole directory tree.
    That allows me to update application code on a production system without touching any data, or to copy/move just one qualifier of data from a system to another, leaving all other data and all code alone.

    Sometimes when doing an initial transfer where I already know that none of the files exists on the remote end, I'll add -W and --inplace. Those don't affect the end result data at all, they just tell rsync to send the whole file rather than try to scan for deltas, and to create the file on the remote end in-place. Normally rsync builds up a temp file on the target and then when the temp file is done it renames it over top of the original file. If I'm trasferring 100 gigs of 100kbyte files, thats a lot of files and in a case like that the renames are actually a measurable overhead. So, in the special case of an initial copy, it's slightly more efficient to tell it to not bother scanning for deltas.

    And trust me, without -W it is very definitely only sending differential deltas of files regardless how you're interpreting the stats. Create a 2 gig database file, rsync it to another machine via some slow internet link. Now update a single record in the file and re-run the exact same rsync command, then you'll see in a way that can't be mistaken. The 2nd rsync will take a teeny tiny fraction of the time. That is it's whole point of existence, not merely it's default mode. Without that, it's just rcp which is much simpler and already existed for years before rsync. And by now, rsync itself is very mature and has been used in very heavy production by unholy gobs of IT types and the tiniest most rare bugs have been found and removed and it's quite solid and dependable now, has been for years actually.

    If you run:

    rsync -avz --del source dest

    .... and don't get any errors, then the two copies _are_ the same.

    It's quite central to our own operations for a few different operations.
    Not just backups but also for broadcasting application code and data updates from a development server out to a growing list of production servers where users and their data actually live, and for moving individual data sets between servers, and we've never had a problem caused by rsync failing to do it's job as advertised.

    When we need a 100% clean copy, like moving a set of users from one box to another, we do one rsync while the system is live, this allows users to work while the time-consuming initial copy is made, then we quiesce both systems as far as those particular users/data are concerned, other users and processes are fine, we just disable that qualifier (filepro term for a data set) which ends up disabling all processes that would try to touch any of the data belonging to that qualifier.
    Then manually kick out any currently logged in users in that qualifier. Then repeate the same rsync, this time it only has a tiny amount of changes to update and the rsync goes in a flash, update a dns record that those users use to get to "their" system, test the new copy ourselves, then enable the qualifier on the remote end which allows the users to get back in. Total end user down time, 5 minutes maybe. Total changes the end user needs to know about or perform, none.
    Anyone who's dns doesn't update right away for some reason or they wrongly hard coded an ip in their terminal emulator, the qualifier is still disabled on the old system so no harm. They can't accidentally do work on the wrong box.

    --
    Brian K. White brian@aljex.com http://www.myspace.com/KEYofR
    +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
    filePro BBx Linux SCO FreeBSD #callahans Satriani Filk!


  16. Re: new to rsync

    Steve M. Fabac, Jr. wrote:
    [SNIP]
    > sent 26815208 bytes received 2930 bytes 4125867.38 bytes/sec
    > total size is 1592929997 speedup is 59.40

    [SNIP]
    > sent 2,562,217,967 bytes received 1261 bytes 11567581.16 bytes/sec
    > total size is 4,942,537,239 speedup is 1.93

    [SNIP]
    > sent 142,756 bytes received 69 bytes 57130.00 bytes/sec
    > total size is 1,295,823,111 speedup is 9072.80

    [SNIP]
    > sent 15,674,928 bytes received 716 bytes 1492918.48 bytes/sec
    > total size is 331,920,054 speedup is 21.17

    [SNIP]
    Pretty good, I'd say.

    Cheers,
    Gary B-)

    P.S. RTFM, please.

    --
    __________________________________________________ ____________________________
    Armful of chairs: Something some people would not know
    whether you were up them with or not
    - Barry Humphries

  17. Re: new to rsync

    Bill Campbell wrote:
    > On Sat, Oct 04, 2008, Steve M. Fabac, Jr. wrote:
    >> Bill Campbell wrote:

    > ...
    >> Bill,
    >>
    >> I don't see it. This is the log of the copy of /usr1 /usr2 /usr3 /tmp to
    >> /util/backup/usr1 ... /usr2 .../usr3 .../tmp at 10:00 on the client's active
    >> system. It looks like rsync is transferring over 100% of the bytes represented
    >> by the accumulated file sizes. If so, then I am doing something wrong with
    >> the command options I am using:

    >
    > Look at the numbers where it says ``speedup is xxx.xx'' for an
    > indication of the time savings over an initial rsync when the
    > speedup time is likely to be something like 0.98.
    >
    > ...
    >> Number of files: 5757
    >> Number of files transferred: 133
    >> Total file size: 1,592,929,997 bytes
    >> Total transferred file size: 26,699,652 bytes
    >> Literal data: 26699652 bytes
    >> Matched data: 0 bytes
    >> File list size: 106441
    >> File list generation time: 0.001 seconds
    >> File list transfer time: 0.000 seconds
    >> Total bytes sent: 26,815,208 == 100.305%
    >> Total bytes received: 2930
    >>
    >> sent 26815208 bytes received 2930 bytes 4125867.38 bytes/sec
    >> total size is 1592929997 speedup is 59.40

    >
    > The total sice is 1.6GB, and it send 26.7MB of data. That's a
    > fair saving.


    A fair savings in not having to send all 5757 files (1.6G) agreed.

    But still sending 26.8MB for 133 files actually sent where the sum
    of the files sent (ls -lds files_sent) adds up to 26.7MB.

    Before I tried rsync, I used "find ./usr1 -mtime -1 | cpio -pmvd"
    that moves 100% of files changed in the last 24 hours. Still comes
    up to 26MB for 133 files.

    >
    > Bill


    Sorry Bill but I still don't see it. You dropped some of the post above so
    I'll put it back and ask again:
    >
    > 0 -rw------- 1 rgroup group 0 26729728
    > 6 -rwxrwxrwx 1 rgroup group 2961 26732689
    > 2 -rw-rw-r-- 1 test1 group 755 26,733,444


    The 26,733,444 above is the accumulated addition of the file sizes returned
    by ls -l file_name for the files listed as transfered by the -v and --stats.
    I.E. out of 5757 files in the file system, only 133 were updated by rsync.
    When I check the size of the files listed and sum the sizes, I get the
    26,733,444 byte count.
    >
    > Number of files: 5757 <- rsync checks 5757 files in /usr1
    > Number of files transferred: 133 <- rsync finds that it must copy 133 files
    > Total file size: 1,592,929,997 bytes <- Total of all files in /usr1
    > Total transferred file size: 26,699,652 bytes <- rsync indicates that 133 files add to 26,599.952 bytes
    > Literal data: 26699652 bytes
    > Matched data: 0 bytes
    > File list size: 106441
    > File list generation time: 0.001 seconds
    > File list transfer time: 0.000 seconds
    > Total bytes sent: 26,815,208 == 100.305% <- rsync says "I sent 26,815,208 bytes to update these 133 files"


    The above number 26,815,208 that rsync reports as having been sent
    is 100.305% of the size of the files rsync listed as having been sent!!

    > Total bytes received: 2930
    >
    > sent 26815208 bytes received 2930 bytes 4125867.38 bytes/sec
    > total size is 1592929997 speedup is 59.40



    To believe that rsync is only sending the changes in the files and
    magically patching the file at the receiving end then I'd expect
    rsync to list the files it sent and then indicate that the
    "Total bytes sent: XXXXXXXX" would be some percentage below 10%
    (assuming application data files where the changed data represents
    additional order records).

    So in the above, I'd want to see "Total bytes sent: 2,681,520 (approximately
    10% of the 26,733,444 byte count for the 133 files updated.

    Is the confusion caused by using rsync in place of cpio -pmvd to copy files from
    one file system to another file system on the same machine?

    "Find ./usr1 -mtime -1 | cpio -pmvd /util/backup" works ok but is not as
    fine grained as rsync and copies the whole file at 10:00, 14:00, 17:00, and 22:00.
    Where rsync will notice a file that was changed at some time before 10:00
    and include it in the 10:00 run but omit it from the 14:00 and later runs
    if it has not been modified after 10:00.

    What command options do I need to give rsync to "diff and patch" files in
    different file systems on the same machine rather then just moving the entire
    file ala find | cpio?

    --
    Steve Fabac
    S.M. Fabac & Associates
    816/765-1670

  18. Re: new to rsync

    Brian K. White wrote:
    > ----- Original Message -----
    > From: "Steve M. Fabac, Jr."
    > Newsgroups: comp.unix.sco.misc
    > To:
    > Sent: Saturday, October 04, 2008 4:20 PM
    > Subject: Re: new to rsync
    >
    >
    >> Bill Campbell wrote:
    >>> On Fri, Oct 03, 2008, Jeff Hyman wrote:
    >>>> RedGrittyBrick typed (on Fri, Oct 03, 2008 at 05:15:54PM +0100):
    >>>>> Jeff Hyman wrote:
    >>>>>> Brian K. White typed (on Fri, Oct 03, 2008 at 04:56:20AM -0400):
    >>>>>> | | ----- Original Message ----- | From: "Jean-Pierre Radley"
    >>>>>>
    >>>>>> | Newsgroups: comp.unix.sco.misc
    >>>>>> | To:
    >>>>>> | Sent: Thursday, October 02, 2008 6:27 PM
    >>>>>> | Subject: Re: new to rsync
    >>>>>> | | | > Steve M. Fabac, Jr. typed (on Thu, Oct 02, 2008 at 05:17:43PM
    >>>>>> -0500):
    >>>>>> | >> How do I tell if rsync is only sending file differences and not the whole
    >>>>>> | >> file. And if it is sending the whole file, what considerations are
    >>>>>> | >> needed to make it send differences?
    >>>>>> | > | > I've always been under the impression that "only sending file
    >>>>>> | > differences" is rsync's default and only mode of operation.
    >>>>>> | > | > Have I been deluded?
    >>>>>> | | That's the default but not only.
    >>>>>> | | You can force it to send the whole file with the -W option.
    >>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^
    >>>>>>
    >>>>>> Other then time and allocated HD space, is there a reason
    >>>>>> to NOT use the '-W' flag? In the event of a crash, what good
    >>>>>> would it be to only have a "portion" of an important data file?
    >>>>>>
    >>>>>> - Jeff H
    >>>>>>
    >>>>> You always "have" the whole data file. If the files doesn't exist at the
    >>>>> other end rsync sends the whole file. If you subsequently change a small
    >>>>> bit of the same data file, rsync sends a diff and patches the file at the
    >>>>> other end (except it doesn't use diff and patch of course).
    >>>>> --
    >>>>> RGB
    >>>> Very cool indeed.
    >>>> To be able to do this, I assume its a block-by-block approach.
    >>>> What is the method of verification that all data made it across the pipe?
    >>>> cksum? bit-level? and is there any issues with file-locking? as in
    >>>> someone has changed a phone number to a clients name in a table while
    >>>> that file is being digested by rsync?
    >>> The method rsync uses is quite neat. Andrew Tridgell, author of
    >>> rsync ans samba, did a presentation to the Seattle Unix group
    >>> where he described the method rsync uses which was far smarter
    >>> than I could understand. Amongst other things rsync looks at
    >>> chunks of the files using things like md5 or sha1 digests to
    >>> identify identical blocks, and can even pick up some reordering
    >>> of the files. It does build a new copy of the file which is
    >>> moved to replace the original when the new one is ready. This
    >>> does require that there be sufficient disk space available on the
    >>> destination file system for two copies of the largest file.
    >>>
    >>> http://rsync.samba.org/
    >>>
    >>> Rsync and Larry Wall's ``patch'' program are two programs which
    >>> are basically magic. Andrew and Larry are responsible for
    >>> several of the most useful FOSS software today, Andrew with rsync
    >>> and samba, Larry with patch and perl.
    >>>
    >>> FWIW, Several members of the Samba core team testified at length
    >>> in the EU's investigation of Microsoft, and were instrumental in
    >>> the EU's understanding of Microsoft's obfuscation of networking
    >>> protocols and ``Embrace and Extend'' activities.
    >>>
    >>> Bill

    >> Bill,
    >>
    >> I don't see it. This is the log of the copy of /usr1 /usr2 /usr3 /tmp to
    >> /util/backup/usr1 ... /usr2 .../usr3 .../tmp at 10:00 on the client's active
    >> system. It looks like rsync is transferring over 100% of the bytes represented
    >> by the accumulated file sizes. If so, then I am doing something wrong with
    >> the command options I am using:
    >>
    >> echo "copy to util/backup started: \c" > /tmp/util.$$
    >> date >> /tmp/util.$$
    >> /usr/local/bin/rsync -avp --stats /usr1 /util/backup >> /tmp/util.$$
    >> /usr/local/bin/rsync -avp --stats /usr2 /util/backup >> /tmp/util.$$
    >> /usr/local/bin/rsync -avp --stats /usr3 /util/backup >> /tmp/util.$$
    >> /usr/local/bin/rsync -avp --stats /tmp /util/backup >> /tmp/util.$$
    >> echo "copy to util/backup complete: \c" >> /tmp/util.$$
    >> touch /usr1/cp_to_util_backup.ran
    >>
    >> From util.last at 10:20 (NOTE: I converted the list of files transfered to
    >> a script "ls -lsd path/file" in /util.last, then marked the start of the list
    >> and the end of the list and executed ":'a,'b!sh" to replace the list with the
    >> results of ls. I edited the line 'a,'bs/Oct.*//, then ran the command
    >> 'a,'b!gawk ' { printf"\%s \%15d\n", $0, count+= $6 } ' to create the running
    >> total to the right of the file size listed by ls -lds I then added the comma's
    >> in the output of gawk and --stats reports to make them easer to read **now is see -h**):
    >>
    >> copy to util/backup started: Fri Oct 3 10:00:10 CDT 2008
    >> *START /usr1 <-- Manual mark
    >> sending incremental file list
    >> 48 -rw------- 1 root sys 22563 22563
    >> 0 -rw------- 1 root sys 0 22563
    >> 2 drwxr-xr-x 2 charlene group 512 23075
    >> 0 -r-------- 1 charlene auth 0 23075
    >> 16 -rw-rw-r-- 1 charlene group 8192 31267
    >> 2 -rw-rw-r-- 1 charlene group 9 31276
    >> 2 drwxr-xr-x 2 cpdebbie group 512 31788
    >> 0 -r-------- 1 cpdebbie auth 0 31788
    >> 2 -rw-rw-r-- 1 cpdebbie group 9 31797
    >> ...
    >> 726 -rwxrwx--- 1 craig group 368640 4716280
    >> 9142 -rw-rw-r-- 1 craig group 4661248 9377528
    >> 1722 -rw-rw-r-- 1 craig group 876544 10254072
    >> 16548 -rwxrwx--- 1 craig group 8437760 18691832
    >> 1464 -rw-rw-r-- 1 test1 group 745472 19437304
    >> 114 -rw-rw-rw- 1 group group 57344 19494648
    >> ...
    >> 4180 -rw------- 1 rgroup group 2129920 26434816
    >> 582 -rw------- 1 rgroup group 294912 26729728
    >> 0 -rw------- 1 rgroup group 0 26729728
    >> 0 -rw------- 1 rgroup group 0 26729728
    >> 6 -rwxrwxrwx 1 rgroup group 2961 26732689
    >> 2 -rw-rw-r-- 1 test1 group 755 26,733,444
    >>
    >> Number of files: 5757
    >> Number of files transferred: 133
    >> Total file size: 1,592,929,997 bytes
    >> Total transferred file size: 26,699,652 bytes
    >> Literal data: 26699652 bytes
    >> Matched data: 0 bytes
    >> File list size: 106441
    >> File list generation time: 0.001 seconds
    >> File list transfer time: 0.000 seconds
    >> Total bytes sent: 26,815,208 == 100.305%
    >> Total bytes received: 2930
    >>
    >> sent 26815208 bytes received 2930 bytes 4125867.38 bytes/sec
    >> total size is 1592929997 speedup is 59.40
    >> *END /usr1 <-- Manual mark
    >>
    >> *START /usr2
    >> sending incremental file list
    >> 16 -rw-rw-r-- 1 craig group 8192 8192
    >> 178 -rw-rw-r-- 1 craig group 90112 98304
    >> 16 -rw-rw-r-- 1 craig group 8192 106496
    >> 34 -rw-rw-r-- 1 craig group 16384 122880
    >> 50 -rw-rw-r-- 1 craig group 24576 147456
    >> 34 -rw-rw-r-- 1 craig group 16384 163840
    >> ...
    >> 52142 -rw-rw-r-- 1 test1 group 26591232 1730723840
    >> 1315766 -rwxrwx--- 1 cii group 671039488 2401763328
    >> 243162 -rwxrwx--- 1 cii group 124010496 2525773824
    >> 9720 -rwxrwx--- 1 cii group 4956160 2530729984
    >> 59290 -rw-rw-r-- 1 cii group 30236672 2560966656
    >> 130 -rwxrwx--- 1 cii group 65536 2561032192
    >> 194 -rwxrwx--- 1 cii group 98304 2561130496
    >> 66 -rwxrwx--- 1 cii group 32768 2561163264
    >> 162 -rw-rw-r-- 1 cii group 81920 2561245184
    >> 1256 -rwxrwx--- 1 cii group 638976 2,561,884,160
    >>
    >> Number of files: 927
    >> Number of files transferred: 65
    >> Total file size: 4,942,537,239 bytes
    >> Total transferred file size: 2,561,884,160 bytes
    >> Literal data: 2,561,884,160 bytes
    >> Matched data: 0 bytes
    >> File list size: 18393
    >> File list generation time: 0.001 seconds
    >> File list transfer time: 0.000 seconds
    >> Total bytes sent: 2,562,217,967 == 100.013% of original file sizes
    >> Total bytes received: 1261
    >>
    >> sent 2,562,217,967 bytes received 1261 bytes 11567581.16 bytes/sec
    >> total size is 4,942,537,239 speedup is 1.93
    >> *END /usr2
    >>
    >> *START /usr3
    >> sending incremental file list
    >>
    >> Number of files: 8206
    >> Number of files transferred: 0
    >> Total file size: 1,295,823,111 bytes
    >> Total transferred file size: 0 bytes
    >> Literal data: 0 bytes
    >> Matched data: 0 bytes
    >> File list size: 142686
    >> File list generation time: 0.001 seconds
    >> File list transfer time: 0.000 seconds
    >> Total bytes sent: 142756
    >> Total bytes received: 69
    >>
    >> sent 142,756 bytes received 69 bytes 57130.00 bytes/sec
    >> total size is 1,295,823,111 speedup is 9072.80
    >> *END /usr3
    >>
    >> *START /tmp
    >> sending incremental file list
    >> 38 drwxrwxrwt 9 sys sys 18432 18432
    >> 1408 -rw-rw-r-- 1 donna group 715981 734413
    >> 48 -rw-rw-r-- 1 debbie group 22654 757067
    >> 192 -rw-rw-r-- 1 melanie group 96933 854000
    >> 2600 -rw------- 1 rgroup group 1323482 2177482
    >> 2566 -rw------- 1 rgroup group 1307067 3484549
    >> 48 -rw-rw-r-- 1 test1 group 22765 3507314
    >> ...
    >> 480 -rw------- 1 rgroup group 244341 15387673
    >> 480 -rw------- 1 rgroup group 244341 15632014
    >> 0 -rw-r--r-- 1 root sys 0 15632014
    >> 54 -rw------- 1 root sys 25667 15657681
    >> 2 -rw-rw-r-- 1 root sys 963 15658644
    >> 18 -rw------- 1 root sys 9130 15667774
    >> 4 -rw------- 1 root sys 1631 15669405
    >> 6 -rw-rw-r-- 1 root sys 2284 15,671,689
    >>
    >> Number of files: 914
    >> Number of files transferred: 34
    >> Total file size: 331,920,054 bytes
    >> Total transferred file size: 15,652,585 bytes
    >> Literal data: 15,653,257 bytes
    >> Matched data: 0 bytes
    >> File list size: 18294
    >> File list generation time: 0.026 seconds
    >> File list transfer time: 0.000 seconds
    >> Total bytes sent: 15,674,928 == 100.020% of original file sizes
    >> Total bytes received: 716
    >>
    >> sent 15,674,928 bytes received 716 bytes 1492918.48 bytes/sec
    >> total size is 331,920,054 speedup is 21.17
    >> *END /tmp
    >> copy to util/backup complete: Fri Oct 3 10:04:10 CDT 2008
    >>
    >> Everybody responding to this thread is saying that the default is to send only the
    >> differences (unless -w is set) but I don't see that happening in the above
    >> listing.
    >>
    >> Tell me where I am going wrong and what I need to do the get the "default" action
    >> of sending only the differences: rsync -avp --stats does not do it.
    >>
    >> NOTE: I got the -avp from an example somewhere that showed typical rsync usage
    >> not from reading the man pages and selecting options (as the subject says: new
    >> to rsync).

    >
    >
    > Basically -a does all the things you would normally want and expect in a "sane" copy.
    > That is, it does the sensible thing wrt symlinks, permissions, ownerships, etc. The one thing you might want different is, by default it will try to convert uid's on the fly, so that files that were owned by user filepro, or user vsifax, etc, on the source system, are still owned by filepro and vsifax on the target system, even if they have different uid numbers on the different systems. Because of exactly those examples, where files need to be owned by special users in order for their applications to work, I think it makes the most sense to do this remapping 99% of the time, even though technically it constitutes a change. If I go to pull the copy back from a backup server onto a live server, th same conversion will happen on the way back so I am left with a local restore that is as if the change never happened. Yet meanwhile, the data "works" on the remote machine instead of being merely a hunk of data on the remote machine.
    >
    > Some of the behaviour is also affected by options in the server side config file, but that's only of you are talking to an rsync daemon directly vs running rsync via ssh or rcmd.
    >
    > then, the most common additions or modifications to -a are:
    >
    > -n dry run, don't actually modify any files, but go through all the motions and show what would have been done.
    >
    > -v verbose, -vv, -vvv, -vvvv, --progress, --stats more & more verbose
    >
    > -z on the fly compression, these days it's basically always good to turn this on, even on a gigabit lan.
    > since 3.0.0 or possibly a little earlier, rsync even knows how to not bother compressing some common files that are already compressed so no cpu waste even if syncing an big tree that is 90% .png files etc.
    >
    > --del delete files/dirs on the target if they don't exist on the source. generally, though it sounds bad, you want this. leaving thigs in a copy that should have been deleted is basically just a form of data corruption.
    >
    > There are a _lot_ of options in rsync. It's very finely tunable. But I almost never have to fiddle with anything beyond the options above. And as you can see, the only one of those options that actually modifies the results is --del.
    > Most of my numerous rsync scripts all just have "-az --del" and variables that might add -v for interactive use or remove it for cron. Some of my fancier stcripts use filters which are a way to include or exclude various files from a job instead of having to accept the whole directory tree.
    > That allows me to update application code on a production system without touching any data, or to copy/move just one qualifier of data from a system to another, leaving all other data and all code alone.
    >
    > Sometimes when doing an initial transfer where I already know that none of the files exists on the remote end, I'll add -W and --inplace. Those don't affect the end result data at all, they just tell rsync to send the whole file rather than try to scan for deltas, and to create the file on the remote end in-place. Normally rsync builds up a temp file on the target and then when the temp file is done it renames it over top of the original file. If I'm trasferring 100 gigs of 100kbyte files, thats a lot of files and in a case like that the renames are actually a measurable overhead. So, in the special case of an initial copy, it's slightly more efficient to tell it to not bother scanning for deltas.
    >
    > And trust me, without -W it is very definitely only sending differential deltas of files regardless how you're interpreting the stats. Create a 2 gig database file, rsync it to another machine via some slow internet link. Now update a single record in the file and re-run the exact same rsync command, then you'll see in a way that can't be mistaken. The 2nd rsync will take a teeny tiny fraction of the time. That is it's whole point of existence, not merely it's default mode. Without that, it's just rcp which is much simpler and already existed for years before rsync. And by now, rsync itself is very mature and has been used in very heavy production by unholy gobs of IT types and the tiniest most rare bugs have been found and removed and it's quite solid and dependable now, has been for years actually.
    >

    Brian,

    The log I posted is from running rsync on a COBOL application receiving orders,
    generating picking tickets, printing invoices, etc. I expect that it meets your
    test above to change one record. Yet adding the size of the files reported by
    rsync as transfered (generated by ls -ls file_name) shows that rsync is saying that
    it is copying the entire file:

    >> 2 -rw-rw-r-- 1 root sys 963 15658644
    >> 18 -rw------- 1 root sys 9130 15667774
    >> 4 -rw------- 1 root sys 1631 15669405
    >> 6 -rw-rw-r-- 1 root sys 2284 15671689 = 15,671,689


    >> Total bytes sent: 15,674,928 == 100.020% of original file sizes


    Note that I am copying from one file system to a second file system on the same
    machine.

    We will be using rsync in the future to copy from the live on-site production
    system to a off-site backup system and will then likely use the -z

    > If you run:
    >
    > rsync -avz --del source dest
    >
    > ... and don't get any errors, then the two copies _are_ the same.
    >
    > It's quite central to our own operations for a few different operations.
    > Not just backups but also for broadcasting application code and data updates from a development server out to a growing list of production servers where users and their data actually live, and for moving individual data sets between servers, and we've never had a problem caused by rsync failing to do it's job as advertised.
    >
    > When we need a 100% clean copy, like moving a set of users from one box to another, we do one rsync while the system is live, this allows users to work while the time-consuming initial copy is made, then we quiesce both systems as far as those particular users/data are concerned, other users and processes are fine, we just disable that qualifier (filepro term for a data set) which ends up disabling all processes that would try to touch any of the data belonging to that qualifier.
    > Then manually kick out any currently logged in users in that qualifier. Then repeate the same rsync, this time it only has a tiny amount of changes to update and the rsync goes in a flash, update a dns record that those users use to get to "their" system, test the new copy ourselves, then enable the qualifier on the remote end which allows the users to get back in. Total end user down time, 5 minutes maybe. Total changes the end user needs to know about or perform, none.
    > Anyone who's dns doesn't update right away for some reason or they wrongly hard coded an ip in their terminal emulator, the qualifier is still disabled on the old system so no harm. They can't accidentally do work on the wrong box.
    >


    --
    Steve Fabac
    S.M. Fabac & Associates
    816/765-1670

  19. Re: new to rsync

    Gary R. Schmidt wrote:
    > Steve M. Fabac, Jr. wrote:
    > [SNIP]
    >> sent 26815208 bytes received 2930 bytes 4125867.38 bytes/sec
    >> total size is 1592929997 speedup is 59.40

    > [SNIP]
    >> sent 2,562,217,967 bytes received 1261 bytes 11567581.16 bytes/sec
    >> total size is 4,942,537,239 speedup is 1.93

    > [SNIP]
    >> sent 142,756 bytes received 69 bytes 57130.00 bytes/sec
    >> total size is 1,295,823,111 speedup is 9072.80

    > [SNIP]
    >> sent 15,674,928 bytes received 716 bytes 1492918.48 bytes/sec
    >> total size is 331,920,054 speedup is 21.17

    > [SNIP]
    > Pretty good, I'd say.
    >
    > Cheers,
    > Gary B-)
    >
    > P.S. RTFM, please.
    >


    RTFP, please.

    Gary, you gloss over the independent verification I've attempted to present
    of the aggregate size of the files "updated" by rsync being smaller then
    the number of bytes rsync reports that it sent:

    [snip]
    0 -rw------- 1 rgroup group + 0 = 26729728
    6 -rwxrwxrwx 1 rgroup group + 2961 = 26732689
    2 -rw-rw-r-- 1 test1 group + 755 = 26733444 = 26,733,444
    ....
    Total bytes sent: 26,815,208 == 100.305% @ 26,815,208 / 26,733,444
    Total bytes received: 2930

    sent 26815208 bytes received 2930 bytes 4125867.38 bytes/sec
    sent 26,815,208 bytes received 2930 bytes 4125867.38 bytes/sec
    total size is 1,592,929,997 speedup is 59.40 <- Total including files it did not
    need to send because they had not changed in any way!

    In each of the file systems copied with rsync, the reported "Total bytes sent"
    is always larger then the accumulated sum of the size of the files
    sent.

    As the subject says: "new to rsync." Too many options available = much confusion
    in selecting the ones needed.

    I posted my results with the rsync command and the options used to generate them,
    hoping that someone with experience using rsync will see the problem with
    the options used.

    The people who wrote the rsync manual, being fine individuals and
    knowlegable about their subject, did not (in the established tradition
    of UNIX documentation) try to warn novice users away from command
    options that might be inappropriate to combine. Check Yes or No

    Riddle me this: Why is a program, that everyone says sends only the
    differences and magically patches the destination file(s) by default
    (the desired action), sending the entire file every time it is run?

    Thank you for your contribution.

    --
    Steve Fabac
    S.M. Fabac & Associates
    816/765-1670

  20. Re: new to rsync

    Steve M. Fabac, Jr. wrote:
    [SNIP]
    > In each of the file systems copied with rsync, the reported "Total bytes
    > sent"
    > is always larger then the accumulated sum of the size of the files
    > sent.

    One thing you are failing to take into account the overhead
    transmissions: file lists, checksums, block lists, and so on.

    Without grovelling about in the source, it'd be something like this:
    file name - let's pick 11 bytes as an average
    time stamp - 4 bytes
    checksum - let's say 4 bytes, but it could be more.
    So, each file costs ~20 bytes, just to be mentioned.

    Just did a couple of rsync's of the laptop to the backup, 0 files
    transferred gives:
    sent 114708 bytes received 152 bytes 25524.44 bytes/sec
    total size is 1406743354 speedup is 12247.46
    And that is scanning ~6000 files.

    114708/6000 => 19.11 (Now, isn't *that* a surprise!!)

    The command used is:
    rsync -avz -e /bin/ssh.exe . server:workstation

    And, thinking about it, for 99% of the uses I've had for rsync that has
    been the command line I've used, the major variations that spring to
    mind are:
    dropping or modifying the -e option
    -H - preserver hard links
    -b - backup changed files
    --exclude-from=exclusions - files not to sync
    and, very rarely
    --delete-after - delete no longer extant files after transfer

    The idea behind rsync (and other, similar programs) is to reduce the
    amount of *redundant* data sent, not necessarily *total* data sent. So
    it can never send 0 bytes. (Well, it can, if there's an early enough
    error, I suppose.)

    Rsync does close to as good a job as is possible, in a situation where
    it cannot tell in advance just how much should be transferred.

    Cheers,
    Gary B-)

    --
    __________________________________________________ ____________________________
    Armful of chairs: Something some people would not know
    whether you were up them with or not
    - Barry Humphries

+ Reply to Thread
Page 1 of 2 1 2 LastLast