Rsync of LVM Snapshots copies whole file - Tools

This is a discussion on Rsync of LVM Snapshots copies whole file - Tools ; Hello, anybody has a clue why this happens? The scenario: Each night I create LVM snapshot of my volume which contains some virtual disk images (VirtualBox VDIs). Then I rsync the content of one of the snapshot directories to a ...

+ Reply to Thread
Results 1 to 9 of 9

Thread: Rsync of LVM Snapshots copies whole file

  1. Rsync of LVM Snapshots copies whole file

    Hello, anybody has a clue why this happens?

    The scenario:
    Each night I create LVM snapshot of my volume which contains some
    virtual disk images (VirtualBox VDIs). Then I rsync the content of one
    of the snapshot directories to a backup server. Each file size is
    between 1.3 GB and 6GB.
    I run the following commmand:
    sudo -u rsyncbackup rsync -avzP --exclude-from
    /home/rsyncbackup/excludefile -e ssh
    "192.168.0.3:/mnt/vdi_snapshot/VDI/mit_backup"
    "/mnt/crypteddevice/linuxX2/mit_backup"
    The source directory is part of the LVM snapshot.
    The destination directory is a truecrypt device which is already mounted.

    What I would expect:
    Only the differences of the virtual disk files should be transfered.

    What happens:
    I discover that always the whole file is transfered. Even if the
    original virtual machine is idle (so only some log files should have
    been enhaned in the virtual disk images).
    I do the following:
    - take LVM snapshot
    - Rsync the desired VDI file (2GB are transferred), it takes 10min
    - destroy LVM snapshot
    - take new LVM snapshot with same parameters as before
    - Rsync the same file (2GB are transferred), it takes 10min

    Am I doing something wrong?

    Best regards
    Peter


    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  2. Re: Rsync of LVM Snapshots copies whole file

    The problem with disk image as databases is that data change a lot
    into the file.
    When you run a system on a disk image, thre is not only new data
    appending to the end of the file, but the entire disk image is changing.
    Same thing would happen with large Outlook .pst databases.
    Rsync nor any other partial data transfer could deal with it as those
    files are "black boxes" seen from the backup tool. They only make
    sense from inside the program that create them.
    The best way to solve this problem is to not do the backup of the disk
    image, but backup from the system inside the disk image. Then, rsync
    can see file changes easily.

    Best regards,

    Vitorio

    Le 15 août 08 à 20:41, Peter P GMX a écrit :

    > Hello, anybody has a clue why this happens?
    >
    > The scenario:
    > Each night I create LVM snapshot of my volume which contains some
    > virtual disk images (VirtualBox VDIs). Then I rsync the content of
    > one of the snapshot directories to a backup server. Each file size
    > is between 1.3 GB and 6GB.
    > I run the following commmand:
    > sudo -u rsyncbackup rsync -avzP --exclude-from /home/rsyncbackup/
    > excludefile -e ssh "192.168.0.3:/mnt/vdi_snapshot/VDI/mit_backup" "/
    > mnt/crypteddevice/linuxX2/mit_backup"
    > The source directory is part of the LVM snapshot.
    > The destination directory is a truecrypt device which is already
    > mounted.
    >
    > What I would expect:
    > Only the differences of the virtual disk files should be transfered.
    >
    > What happens:
    > I discover that always the whole file is transfered. Even if the
    > original virtual machine is idle (so only some log files should have
    > been enhaned in the virtual disk images).
    > I do the following:
    > - take LVM snapshot
    > - Rsync the desired VDI file (2GB are transferred), it takes 10min
    > - destroy LVM snapshot
    > - take new LVM snapshot with same parameters as before
    > - Rsync the same file (2GB are transferred), it takes 10min
    >
    > Am I doing something wrong?
    >
    > Best regards
    > Peter
    >
    >
    > --
    > Please use reply-all for most replies to avoid omitting the mailing
    > list.
    > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  3. Re: Rsync of LVM Snapshots copies whole file

    On Fri 15 Aug 2008, Peter P GMX wrote:
    >
    > What happens:
    > I discover that always the whole file is transfered. Even if the
    > original virtual machine is idle (so only some log files should have
    > been enhaned in the virtual disk images).
    > I do the following:
    > - take LVM snapshot
    > - Rsync the desired VDI file (2GB are transferred), it takes 10min
    > - destroy LVM snapshot
    > - take new LVM snapshot with same parameters as before
    > - Rsync the same file (2GB are transferred), it takes 10min


    It may appear that the whole file is being transferred.
    What does the --stats output show?

    The whole of both the original and the copy need to be read through to
    find what blocks have changed, and that can take a lot of time. It may
    even be faster to use --whole-file to stop the searching of changed data
    and simply copy the data (like you suspect is happening now). That would
    save 2GB of reads. Rsync tries to optimize the network traffic; if
    you're doing a local transfer (no network involved) rsync will
    automatically do --whole-file. If your network is fast, --whole-file
    might be the better option as well. As you give no example of how you're
    invoking rsync, I'm left guessing as to your situation.


    Paul Slootman
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  4. Re: Rsync of LVM Snapshots copies whole file

    On Wed 20 Aug 2008, Peter P GMX wrote:

    > Hallo Paul,


    please keep replies to list messages on the list, others may also be
    able to help.

    > here is the call and the protocol for the transfer of one file. For me
    > it seems that the whole file is transferred, right?


    Nothing in the output indicates that all the file data is being
    transferred; only that the file is indeed being transferred.
    The output of --stats will help in showing how much file data is
    actually transferred.


    Paul Slootman


    > sudo -u rsyncbackup rsync -avzP --exclude-from
    > /home/rsyncbackup/excludefile -e ssh
    > "192.168.0.3:/mnt/vdi_snapshot/VDI/mit_backup"
    > "/mnt/crypteddevice/linuxX2/mit_backup"
    > receiving file list ...
    > 14 files to consider
    > mit_backup/
    > mit_backup/Debian40_XXXX.vdi
    > 0 0% 0.00kB/s 0:00:00 10235877 0% 9.69MB/s 0:02:19 21075684 1% 10.02MB/s
    > 0:02:13 27933044 2% 8.86MB/s 0:02:30 32490259 2% 7.72MB/s 0:02:52
    > 39455369 2% 6.95MB/s 0:03:10 45837776 3% 5.89MB/s 0:03:43 50797900 3%
    > 5.44MB/s 0:04:01 53961592 3% 5.12MB/s 0:04:16 61553523 4% 5.27MB/s
    > 0:04:07 67043585 4% 5.05MB/s 0:04:16 75011647 5% 5.60MB/s 0:03:50
    > 84224736 6% 7.00MB/s 0:03:03 89720101 6% 6.51MB/s 0:03:15 99630301 7%
    > 7.53MB/s 0:02:47 107071219 7% 7.63MB/s 0:02:44 113938846 8% 7.07MB/s
    > 0:02:56 125802777 9% 8.53MB/s 0:02:25 132129910 9% 7.57MB/s 0:02:42
    > 139575721 10% 7.58MB/s 0:02:41 144926021 10% 7.23MB/s 0:02:49 149718979
    > 10% 5.61MB/s 0:03:36 154452981 11% 5.32MB/s 0:03:47 160614584 11%
    > 5.00MB/s 0:04:01 165958612 11% 5.00MB/s 0:04:00 170811085 12% 5.01MB/s
    > 0:03:58 175120144 12% 4.82MB/s 0:04:07 183661136 13% 5.39MB/s 0:03:39
    > 189571426 13% 5.52MB/s 0:03:33 195568688 14% 5.78MB/s 0:03:22 202404006
    > 14% 6.49MB/s 0:02:59 210410776 15% 6.35MB/s 0:03:02 216901117 15%
    > 6.48MB/s 0:02:57 225794756 16% 6.94MB/s 0:02:44 233657909 16% 7.19MB/s
    > 0:02:37 239352121 17% 6.67MB/s 0:02:49 244896643 17% 6.44MB/s 0:02:54
    > 249979132 17% 5.75MB/s 0:03:14 255669742 18% 5.23MB/s 0:03:32 262874393
    > 18% 5.58MB/s 0:03:18 275524040 19% 7.27MB/s 0:02:30 294537718 21%
    > 10.58MB/s 0:01:41 306682448 21% 12.06MB/s 0:01:28 316879483 22%
    > 12.74MB/s 0:01:22 322410223 23% 11.06MB/s 0:01:34 331382172 23% 8.38MB/s
    > 0:02:03 344124151 24% 8.36MB/s 0:02:02 356277012 25% 8.84MB/s 0:01:54
    > 368731867 26% 10.30MB/s 0:01:37 376902647 27% 10.49MB/s 0:01:34
    > 382055316 27% 8.96MB/s 0:01:50 385099184 27% 6.81MB/s 0:02:24 394514192
    > 28% 6.14MB/s 0:02:39 399166976 28% 5.31MB/s 0:03:03 409033762 29%
    > 6.42MB/s 0:02:29 419680637 30% 8.02MB/s 0:01:58 429225330 30% 7.96MB/s
    > 0:01:58 443550036 31% 10.15MB/s 0:01:31 452116589 32% 9.86MB/s 0:01:33
    > 457430449 32% 8.85MB/s 0:01:43 465762610 33% 8.68MB/s 0:01:44 471632053
    > 33% 6.66MB/s 0:02:15 477048104 34% 5.91MB/s 0:02:31 483430930 34%
    > 5.93MB/s 0:02:30 490941562 35% 5.74MB/s 0:02:33 496540662 35% 5.70MB/s
    > 0:02:34 501713859 35% 5.64MB/s 0:02:34 508037197 36% 5.83MB/s 0:02:28
    > 518535763 37% 6.53MB/s 0:02:11 526524427 37% 7.08MB/s 0:01:59 538465877
    > 38% 8.46MB/s 0:01:38 552112154 39% 10.15MB/s 0:01:21 563067426 40%
    > 10.23MB/s 0:01:19 573336231 41% 10.73MB/s 0:01:14 574650941 41% 8.50MB/s
    > 0:01:34 585401475 41% 7.67MB/s 0:01:43 604962094 43% 9.70MB/s 0:01:19
    > 614156492 44% 9.44MB/s 0:01:20 616733228 44% 8.78MB/s 0:01:26 623854469
    > 44% 8.19MB/s 0:01:32 634887177 45% 6.37MB/s 0:01:56 643660294 46%
    > 6.31MB/s 0:01:56 651766969 46% 8.34MB/s 0:01:27 652307889 46% 6.71MB/s
    > 0:01:48 666193629 47% 7.38MB/s 0:01:36 675491703 48% 7.49MB/s 0:01:33
    > 684601774 49% 7.60MB/s 0:01:31 693070083 49% 9.52MB/s 0:01:12 698081645
    > 50% 7.45MB/s 0:01:31 704963446 50% 6.90MB/s 0:01:37 707735864 50%
    > 5.29MB/s 0:02:07 724318532 51% 7.15MB/s 0:01:31 743526578 53% 10.03MB/s
    > 0:01:03 754724412 54% 10.71MB/s 0:00:58 761665355 54% 12.06MB/s 0:00:51
    > 771483521 55% 10.55MB/s 0:00:57 778832521 55% 8.18MB/s 0:01:13 786776637
    > 56% 7.51MB/s 0:01:19 794628478 56% 7.73MB/s 0:01:15 800825574 57%
    > 6.87MB/s 0:01:24 807411368 57% 6.69MB/s 0:01:25 813464143 58% 6.34MB/s
    > 0:01:29 822882968 58% 6.54MB/s 0:01:25 836248436 59% 8.16MB/s 0:01:06
    > 842835485 60% 8.15MB/s 0:01:06 850384164 60% 8.52MB/s 0:01:02 862409333
    > 61% 9.33MB/s 0:00:55 872067584 62% 8.51MB/s 0:01:00 877114368 62%
    > 8.14MB/s 0:01:02 882553071 63% 7.58MB/s 0:01:06 891031774 63% 6.75MB/s
    > 0:01:12 900651352 64% 6.72MB/s 0:01:11 912028399 65% 8.22MB/s 0:00:57
    > 923130092 66% 9.63MB/s 0:00:47 930375419 66% 9.34MB/s 0:00:48 945088136
    > 67% 10.23MB/s 0:00:43 953608062 68% 9.57MB/s 0:00:45 960536527 68%
    > 8.51MB/s 0:00:49 971018224 69% 9.25MB/s 0:00:44 977655293 70% 7.66MB/s
    > 0:00:53 982496601 70% 6.80MB/s 0:00:59 990341428 70% 7.09MB/s 0:00:55
    > 996521787 71% 6.06MB/s 0:01:04 1005523756 72% 6.63MB/s 0:00:57
    > 1011459653 72% 6.88MB/s 0:00:54 1017321634 72% 6.41MB/s 0:00:57
    > 1024275542 73% 6.59MB/s 0:00:55 1032295357 73% 6.36MB/s 0:00:55
    > 1035005098 74% 5.12MB/s 0:01:08 1051640925 75% 7.47MB/s 0:00:44
    > 1058476439 75% 7.45MB/s 0:00:44 1068116134 76% 7.79MB/s 0:00:41
    > 1072174076 76% 8.72MB/s 0:00:36 1076894984 77% 5.90MB/s 0:00:52
    > 1086364486 77% 6.52MB/s 0:00:46 1093933058 78% 6.04MB/s 0:00:48
    > 1100717102 78% 6.75MB/s 0:00:42 1104505412 79% 6.53MB/s 0:00:43
    > 1109563535 79% 5.38MB/s 0:00:51 1127916628 80% 7.85MB/s 0:00:33
    > 1134948712 81% 7.92MB/s 0:00:32 1140284922 81% 8.33MB/s 0:00:29
    > 1148466121 82% 8.89MB/s 0:00:27 1148596687 82% 4.18MB/s 0:00:57
    > 1160963510 83% 5.26MB/s 0:00:43 1168681452 83% 5.74MB/s 0:00:38
    > 1175945190 84% 5.74MB/s 0:00:37 1178665232 84% 6.46MB/s 0:00:32
    > 1189338749 85% 6.09MB/s 0:00:33 1198811725 85% 6.47MB/s 0:00:29
    > 1207430133 86% 6.76MB/s 0:00:27 1216592101 87% 8.89MB/s 0:00:19
    > 1223955783 87% 8.06MB/s 0:00:20 1229789052 88% 7.19MB/s 0:00:22
    > 1255105302 89% 10.81MB/s 0:00:12 1268802266 90% 12.01MB/s 0:00:10
    > 1279289970 91% 12.82MB/s 0:00:08 1304782634 93% 17.29MB/s 0:00:05
    > 1325786214 94% 16.68MB/s 0:00:04 1353009354 96% 19.89MB/s 0:00:02
    > 1361234818 97% 19.35MB/s 0:00:01 1369458218 98% 15.27MB/s 0:00:01
    > 1379158338 98% 12.09MB/s 0:00:01 1395660800 100% 7.94MB/s 0:02:47
    > (xfer#1, to-check=12/14)

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  5. Re: Rsync of LVM Snapshots copies whole file

    Hello Paul,

    here is the status of the transfer.
    If I understand it right, about 3% of the data was changed (literal data
    against matched data). It took about 1h40min to transfer ~250MB of data.
    But to be honest, I do not get a real clue out of this status report.
    E.g. Laptop_D_10GB.vdi takes around 25 minutes to be transferred with
    6,74MBytes/sec. If I divide 10G by 6,74MBytes/sec (for a complete file
    transfer) I end up at 40 minutes. So almost half of the complete data
    transfer time of this volume is needed, although this drive is rather
    passive and there were almost no changes on the drive between the last
    sync. (The machines are a AMD X2 3800 on one originating side and a Atom
    1,6GHz on the destination side, Atom pulls from AMDs side).

    Best regards
    Peter

    receiving file list ...
    13 files to consider
    mit_backup/Debian40.vdi
    1395660800 100% 5.66MB/s 0:03:55 (xfer#1, to-check=11/13)
    mit_backup/Laptop_D_10GB.vdi
    10734313984 100% 6.74MB/s 0:25:19 (xfer#2, to-check=10/13)
    mit_backup/RubyServer_1_9GB.vdi
    2039489024 100% 8.15MB/s 0:03:58 (xfer#3, to-check=8/13)
    mit_backup/XXXX_web.vdi
    2874171904 100% 6.17MB/s 0:07:24 (xfer#4, to-check=7/13)
    mit_backup/Ubuntu_8_04_Development.vdi
    4367340032 100% 6.80MB/s 0:10:12 (xfer#5, to-check=6/13)
    mit_backup/Ubuntu_XXXX_3GB.vdi
    3221238272 100% 5.61MB/s 0:09:07 (xfer#6, to-check=5/13)
    mit_backup/W2000_6GB.vdi
    6442476032 100% 7.49MB/s 0:13:39 (xfer#7, to-check=4/13)
    mit_backup/XXXXXXXX.vdi
    4294984192 100% 8.05MB/s 0:08:28 (xfer#8, to-check=3/13)

    Number of files: 13
    Number of files transferred: 8
    Total file size: 47.710.416.177 bytes
    Total transferred file size: 35.369.674.240 bytes
    Literal data: 1.181.132.240 bytes
    Matched data: 34.188.542.000 bytes
    File list size: 530
    File list generation time: 0.001 seconds
    File list transfer time: 0.000 seconds
    Total bytes sent: 3.984.972
    Total bytes received: 249.776.322

    sent 3.984.972 bytes received 249.776.322 bytes 41515.14 bytes/sec
    total size is 47.710.416.177 speedup is 188.01



    Paul Slootman schrieb:
    > On Wed 20 Aug 2008, Peter P GMX wrote:
    >
    >
    >> Hallo Paul,
    >>

    >
    > please keep replies to list messages on the list, others may also be
    > able to help.
    >
    >
    >> here is the call and the protocol for the transfer of one file. For me
    >> it seems that the whole file is transferred, right?
    >>

    >
    > Nothing in the output indicates that all the file data is being
    > transferred; only that the file is indeed being transferred.
    > The output of --stats will help in showing how much file data is
    > actually transferred.
    >
    >
    > Paul Slootman
    >
    >
    >
    >> sudo -u rsyncbackup rsync -avzP --exclude-from
    >> /home/rsyncbackup/excludefile -e ssh
    >> "192.168.0.3:/mnt/vdi_snapshot/VDI/mit_backup"
    >> "/mnt/crypteddevice/linuxX2/mit_backup"
    >> receiving file list ...
    >> 14 files to consider
    >> mit_backup/
    >> mit_backup/Debian40_XXXX.vdi
    >> 0 0% 0.00kB/s 0:00:00 10235877 0% 9.69MB/s 0:02:19 21075684 1% 10.02MB/s
    >> 0:02:13 27933044 2% 8.86MB/s 0:02:30 32490259 2% 7.72MB/s 0:02:52
    >> 39455369 2% 6.95MB/s 0:03:10 45837776 3% 5.89MB/s 0:03:43 50797900 3%
    >> 5.44MB/s 0:04:01 53961592 3% 5.12MB/s 0:04:16 61553523 4% 5.27MB/s
    >> 0:04:07 67043585 4% 5.05MB/s 0:04:16 75011647 5% 5.60MB/s 0:03:50
    >> 84224736 6% 7.00MB/s 0:03:03 89720101 6% 6.51MB/s 0:03:15 99630301 7%
    >> 7.53MB/s 0:02:47 107071219 7% 7.63MB/s 0:02:44 113938846 8% 7.07MB/s
    >> 0:02:56 125802777 9% 8.53MB/s 0:02:25 132129910 9% 7.57MB/s 0:02:42
    >> 139575721 10% 7.58MB/s 0:02:41 144926021 10% 7.23MB/s 0:02:49 149718979
    >> 10% 5.61MB/s 0:03:36 154452981 11% 5.32MB/s 0:03:47 160614584 11%
    >> 5.00MB/s 0:04:01 165958612 11% 5.00MB/s 0:04:00 170811085 12% 5.01MB/s
    >> 0:03:58 175120144 12% 4.82MB/s 0:04:07 183661136 13% 5.39MB/s 0:03:39
    >> 189571426 13% 5.52MB/s 0:03:33 195568688 14% 5.78MB/s 0:03:22 202404006
    >> 14% 6.49MB/s 0:02:59 210410776 15% 6.35MB/s 0:03:02 216901117 15%
    >> 6.48MB/s 0:02:57 225794756 16% 6.94MB/s 0:02:44 233657909 16% 7.19MB/s
    >> 0:02:37 239352121 17% 6.67MB/s 0:02:49 244896643 17% 6.44MB/s 0:02:54
    >> 249979132 17% 5.75MB/s 0:03:14 255669742 18% 5.23MB/s 0:03:32 262874393
    >> 18% 5.58MB/s 0:03:18 275524040 19% 7.27MB/s 0:02:30 294537718 21%
    >> 10.58MB/s 0:01:41 306682448 21% 12.06MB/s 0:01:28 316879483 22%
    >> 12.74MB/s 0:01:22 322410223 23% 11.06MB/s 0:01:34 331382172 23% 8.38MB/s
    >> 0:02:03 344124151 24% 8.36MB/s 0:02:02 356277012 25% 8.84MB/s 0:01:54
    >> 368731867 26% 10.30MB/s 0:01:37 376902647 27% 10.49MB/s 0:01:34
    >> 382055316 27% 8.96MB/s 0:01:50 385099184 27% 6.81MB/s 0:02:24 394514192
    >> 28% 6.14MB/s 0:02:39 399166976 28% 5.31MB/s 0:03:03 409033762 29%
    >> 6.42MB/s 0:02:29 419680637 30% 8.02MB/s 0:01:58 429225330 30% 7.96MB/s
    >> 0:01:58 443550036 31% 10.15MB/s 0:01:31 452116589 32% 9.86MB/s 0:01:33
    >> 457430449 32% 8.85MB/s 0:01:43 465762610 33% 8.68MB/s 0:01:44 471632053
    >> 33% 6.66MB/s 0:02:15 477048104 34% 5.91MB/s 0:02:31 483430930 34%
    >> 5.93MB/s 0:02:30 490941562 35% 5.74MB/s 0:02:33 496540662 35% 5.70MB/s
    >> 0:02:34 501713859 35% 5.64MB/s 0:02:34 508037197 36% 5.83MB/s 0:02:28
    >> 518535763 37% 6.53MB/s 0:02:11 526524427 37% 7.08MB/s 0:01:59 538465877
    >> 38% 8.46MB/s 0:01:38 552112154 39% 10.15MB/s 0:01:21 563067426 40%
    >> 10.23MB/s 0:01:19 573336231 41% 10.73MB/s 0:01:14 574650941 41% 8.50MB/s
    >> 0:01:34 585401475 41% 7.67MB/s 0:01:43 604962094 43% 9.70MB/s 0:01:19
    >> 614156492 44% 9.44MB/s 0:01:20 616733228 44% 8.78MB/s 0:01:26 623854469
    >> 44% 8.19MB/s 0:01:32 634887177 45% 6.37MB/s 0:01:56 643660294 46%
    >> 6.31MB/s 0:01:56 651766969 46% 8.34MB/s 0:01:27 652307889 46% 6.71MB/s
    >> 0:01:48 666193629 47% 7.38MB/s 0:01:36 675491703 48% 7.49MB/s 0:01:33
    >> 684601774 49% 7.60MB/s 0:01:31 693070083 49% 9.52MB/s 0:01:12 698081645
    >> 50% 7.45MB/s 0:01:31 704963446 50% 6.90MB/s 0:01:37 707735864 50%
    >> 5.29MB/s 0:02:07 724318532 51% 7.15MB/s 0:01:31 743526578 53% 10.03MB/s
    >> 0:01:03 754724412 54% 10.71MB/s 0:00:58 761665355 54% 12.06MB/s 0:00:51
    >> 771483521 55% 10.55MB/s 0:00:57 778832521 55% 8.18MB/s 0:01:13 786776637
    >> 56% 7.51MB/s 0:01:19 794628478 56% 7.73MB/s 0:01:15 800825574 57%
    >> 6.87MB/s 0:01:24 807411368 57% 6.69MB/s 0:01:25 813464143 58% 6.34MB/s
    >> 0:01:29 822882968 58% 6.54MB/s 0:01:25 836248436 59% 8.16MB/s 0:01:06
    >> 842835485 60% 8.15MB/s 0:01:06 850384164 60% 8.52MB/s 0:01:02 862409333
    >> 61% 9.33MB/s 0:00:55 872067584 62% 8.51MB/s 0:01:00 877114368 62%
    >> 8.14MB/s 0:01:02 882553071 63% 7.58MB/s 0:01:06 891031774 63% 6.75MB/s
    >> 0:01:12 900651352 64% 6.72MB/s 0:01:11 912028399 65% 8.22MB/s 0:00:57
    >> 923130092 66% 9.63MB/s 0:00:47 930375419 66% 9.34MB/s 0:00:48 945088136
    >> 67% 10.23MB/s 0:00:43 953608062 68% 9.57MB/s 0:00:45 960536527 68%
    >> 8.51MB/s 0:00:49 971018224 69% 9.25MB/s 0:00:44 977655293 70% 7.66MB/s
    >> 0:00:53 982496601 70% 6.80MB/s 0:00:59 990341428 70% 7.09MB/s 0:00:55
    >> 996521787 71% 6.06MB/s 0:01:04 1005523756 72% 6.63MB/s 0:00:57
    >> 1011459653 72% 6.88MB/s 0:00:54 1017321634 72% 6.41MB/s 0:00:57
    >> 1024275542 73% 6.59MB/s 0:00:55 1032295357 73% 6.36MB/s 0:00:55
    >> 1035005098 74% 5.12MB/s 0:01:08 1051640925 75% 7.47MB/s 0:00:44
    >> 1058476439 75% 7.45MB/s 0:00:44 1068116134 76% 7.79MB/s 0:00:41
    >> 1072174076 76% 8.72MB/s 0:00:36 1076894984 77% 5.90MB/s 0:00:52
    >> 1086364486 77% 6.52MB/s 0:00:46 1093933058 78% 6.04MB/s 0:00:48
    >> 1100717102 78% 6.75MB/s 0:00:42 1104505412 79% 6.53MB/s 0:00:43
    >> 1109563535 79% 5.38MB/s 0:00:51 1127916628 80% 7.85MB/s 0:00:33
    >> 1134948712 81% 7.92MB/s 0:00:32 1140284922 81% 8.33MB/s 0:00:29
    >> 1148466121 82% 8.89MB/s 0:00:27 1148596687 82% 4.18MB/s 0:00:57
    >> 1160963510 83% 5.26MB/s 0:00:43 1168681452 83% 5.74MB/s 0:00:38
    >> 1175945190 84% 5.74MB/s 0:00:37 1178665232 84% 6.46MB/s 0:00:32
    >> 1189338749 85% 6.09MB/s 0:00:33 1198811725 85% 6.47MB/s 0:00:29
    >> 1207430133 86% 6.76MB/s 0:00:27 1216592101 87% 8.89MB/s 0:00:19
    >> 1223955783 87% 8.06MB/s 0:00:20 1229789052 88% 7.19MB/s 0:00:22
    >> 1255105302 89% 10.81MB/s 0:00:12 1268802266 90% 12.01MB/s 0:00:10
    >> 1279289970 91% 12.82MB/s 0:00:08 1304782634 93% 17.29MB/s 0:00:05
    >> 1325786214 94% 16.68MB/s 0:00:04 1353009354 96% 19.89MB/s 0:00:02
    >> 1361234818 97% 19.35MB/s 0:00:01 1369458218 98% 15.27MB/s 0:00:01
    >> 1379158338 98% 12.09MB/s 0:00:01 1395660800 100% 7.94MB/s 0:02:47
    >> (xfer#1, to-check=12/14)
    >>

    >
    >

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  6. Re: Rsync of LVM Snapshots copies whole file

    On Wed 20 Aug 2008, Peter P GMX wrote:
    >
    > here is the status of the transfer.
    > If I understand it right, about 3% of the data was changed (literal data
    > against matched data). It took about 1h40min to transfer ~250MB of data.


    The searching for matching data blocks is apparently taking a lot of
    time. However, it is clear that not the whole file is being transferred,
    the speedup is reported as 188.01 while a whole file transfer would be
    "1".

    > But to be honest, I do not get a real clue out of this status report.
    > E.g. Laptop_D_10GB.vdi takes around 25 minutes to be transferred with
    > 6,74MBytes/sec. If I divide 10G by 6,74MBytes/sec (for a complete file
    > transfer) I end up at 40 minutes. So almost half of the complete data
    > transfer time of this volume is needed, although this drive is rather
    > passive and there were almost no changes on the drive between the last
    > sync. (The machines are a AMD X2 3800 on one originating side and a Atom
    > 1,6GHz on the destination side, Atom pulls from AMDs side).


    When updating files, rsync reads the source data, reads the existing
    target data, and creates a new copy of the target data. If the disk on
    the destination is not so fast, the seeking to and from the old and new
    copy can impact performance. In that case using --whole-file could
    actually speed up the process... which is exactly the reverse of what
    you thought at the beginning :-)

    If you don't mind that the destination copy is invalid for some time
    (e.g. if it's just used for backup), _and_ you know that data won't be
    moved, only updated at random places, you might try --inplace. That way
    the existing copy is updated, instead of copying the data to a new file.


    Paul
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  7. Re: Rsync of LVM Snapshots copies whole file

    Paul Slootman wrote:
    > If you don't mind that the destination copy is invalid for some time
    > (e.g. if it's just used for backup), _and_ you know that data won't be
    > moved, only updated at random places, you might try --inplace. That way
    > the existing copy is updated, instead of copying the data to a new file.


    This reminds me. When using --sparse to copy an 80GB disk image, I
    noticed that rsync sends an awful lot of network data, given that the
    image is mostly zeros. It appeared that --sparse was being applied at
    the writing end, not optimising the block checksum transmission at all.

    -- Jamie
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  8. Re: Rsync of LVM Snapshots copies whole file

    Hello

    does nobody has a clue about this?

    Best regards
    Peter

    Peter P GMX schrieb:
    > Hello Paul,
    >
    > here is the status of the transfer.
    > If I understand it right, about 3% of the data was changed (literal
    > data against matched data). It took about 1h40min to transfer ~250MB
    > of data.
    > But to be honest, I do not get a real clue out of this status report.
    > E.g. Laptop_D_10GB.vdi takes around 25 minutes to be transferred with
    > 6,74MBytes/sec. If I divide 10G by 6,74MBytes/sec (for a complete file
    > transfer) I end up at 40 minutes. So almost half of the complete data
    > transfer time of this volume is needed, although this drive is rather
    > passive and there were almost no changes on the drive between the last
    > sync. (The machines are a AMD X2 3800 on one originating side and a
    > Atom 1,6GHz on the destination side, Atom pulls from AMDs side).
    >
    > Best regards
    > Peter
    >
    > receiving file list ...
    > 13 files to consider
    > mit_backup/Debian40.vdi
    > 1395660800 100% 5.66MB/s 0:03:55 (xfer#1, to-check=11/13)
    > mit_backup/Laptop_D_10GB.vdi
    > 10734313984 100% 6.74MB/s 0:25:19 (xfer#2, to-check=10/13)
    > mit_backup/RubyServer_1_9GB.vdi
    > 2039489024 100% 8.15MB/s 0:03:58 (xfer#3, to-check=8/13)
    > mit_backup/XXXX_web.vdi
    > 2874171904 100% 6.17MB/s 0:07:24 (xfer#4, to-check=7/13)
    > mit_backup/Ubuntu_8_04_Development.vdi
    > 4367340032 100% 6.80MB/s 0:10:12 (xfer#5, to-check=6/13)
    > mit_backup/Ubuntu_XXXX_3GB.vdi
    > 3221238272 100% 5.61MB/s 0:09:07 (xfer#6, to-check=5/13)
    > mit_backup/W2000_6GB.vdi
    > 6442476032 100% 7.49MB/s 0:13:39 (xfer#7, to-check=4/13)
    > mit_backup/XXXXXXXX.vdi
    > 4294984192 100% 8.05MB/s 0:08:28 (xfer#8, to-check=3/13)
    >
    > Number of files: 13
    > Number of files transferred: 8
    > Total file size: 47.710.416.177 bytes
    > Total transferred file size: 35.369.674.240 bytes
    > Literal data: 1.181.132.240 bytes
    > Matched data: 34.188.542.000 bytes
    > File list size: 530
    > File list generation time: 0.001 seconds
    > File list transfer time: 0.000 seconds
    > Total bytes sent: 3.984.972
    > Total bytes received: 249.776.322
    >
    > sent 3.984.972 bytes received 249.776.322 bytes 41515.14 bytes/sec
    > total size is 47.710.416.177 speedup is 188.01
    >
    >
    >
    > Paul Slootman schrieb:
    >> On Wed 20 Aug 2008, Peter P GMX wrote:
    >>
    >>
    >>> Hallo Paul,
    >>>

    >>
    >> please keep replies to list messages on the list, others may also be
    >> able to help.
    >>
    >>
    >>> here is the call and the protocol for the transfer of one file. For
    >>> me it seems that the whole file is transferred, right?
    >>>

    >>
    >> Nothing in the output indicates that all the file data is being
    >> transferred; only that the file is indeed being transferred.
    >> The output of --stats will help in showing how much file data is
    >> actually transferred.
    >>
    >>
    >> Paul Slootman
    >>
    >>
    >>
    >>> sudo -u rsyncbackup rsync -avzP --exclude-from
    >>> /home/rsyncbackup/excludefile -e ssh
    >>> "192.168.0.3:/mnt/vdi_snapshot/VDI/mit_backup"
    >>> "/mnt/crypteddevice/linuxX2/mit_backup"
    >>> receiving file list ...
    >>> 14 files to consider
    >>> mit_backup/
    >>> mit_backup/Debian40_XXXX.vdi
    >>> 0 0% 0.00kB/s 0:00:00 10235877 0% 9.69MB/s 0:02:19 21075684 1%
    >>> 10.02MB/s 0:02:13 27933044 2% 8.86MB/s 0:02:30 32490259 2% 7.72MB/s
    >>> 0:02:52 39455369 2% 6.95MB/s 0:03:10 45837776 3% 5.89MB/s 0:03:43
    >>> 50797900 3% 5.44MB/s 0:04:01 53961592 3% 5.12MB/s 0:04:16 61553523
    >>> 4% 5.27MB/s 0:04:07 67043585 4% 5.05MB/s 0:04:16 75011647 5%
    >>> 5.60MB/s 0:03:50 84224736 6% 7.00MB/s 0:03:03 89720101 6% 6.51MB/s
    >>> 0:03:15 99630301 7% 7.53MB/s 0:02:47 107071219 7% 7.63MB/s 0:02:44
    >>> 113938846 8% 7.07MB/s 0:02:56 125802777 9% 8.53MB/s 0:02:25
    >>> 132129910 9% 7.57MB/s 0:02:42 139575721 10% 7.58MB/s 0:02:41
    >>> 144926021 10% 7.23MB/s 0:02:49 149718979 10% 5.61MB/s 0:03:36
    >>> 154452981 11% 5.32MB/s 0:03:47 160614584 11% 5.00MB/s 0:04:01
    >>> 165958612 11% 5.00MB/s 0:04:00 170811085 12% 5.01MB/s 0:03:58
    >>> 175120144 12% 4.82MB/s 0:04:07 183661136 13% 5.39MB/s 0:03:39
    >>> 189571426 13% 5.52MB/s 0:03:33 195568688 14% 5.78MB/s 0:03:22
    >>> 202404006 14% 6.49MB/s 0:02:59 210410776 15% 6.35MB/s 0:03:02
    >>> 216901117 15% 6.48MB/s 0:02:57 225794756 16% 6.94MB/s 0:02:44
    >>> 233657909 16% 7.19MB/s 0:02:37 239352121 17% 6.67MB/s 0:02:49
    >>> 244896643 17% 6.44MB/s 0:02:54 249979132 17% 5.75MB/s 0:03:14
    >>> 255669742 18% 5.23MB/s 0:03:32 262874393 18% 5.58MB/s 0:03:18
    >>> 275524040 19% 7.27MB/s 0:02:30 294537718 21% 10.58MB/s 0:01:41
    >>> 306682448 21% 12.06MB/s 0:01:28 316879483 22% 12.74MB/s 0:01:22
    >>> 322410223 23% 11.06MB/s 0:01:34 331382172 23% 8.38MB/s 0:02:03
    >>> 344124151 24% 8.36MB/s 0:02:02 356277012 25% 8.84MB/s 0:01:54
    >>> 368731867 26% 10.30MB/s 0:01:37 376902647 27% 10.49MB/s 0:01:34
    >>> 382055316 27% 8.96MB/s 0:01:50 385099184 27% 6.81MB/s 0:02:24
    >>> 394514192 28% 6.14MB/s 0:02:39 399166976 28% 5.31MB/s 0:03:03
    >>> 409033762 29% 6.42MB/s 0:02:29 419680637 30% 8.02MB/s 0:01:58
    >>> 429225330 30% 7.96MB/s 0:01:58 443550036 31% 10.15MB/s 0:01:31
    >>> 452116589 32% 9.86MB/s 0:01:33 457430449 32% 8.85MB/s 0:01:43
    >>> 465762610 33% 8.68MB/s 0:01:44 471632053 33% 6.66MB/s 0:02:15
    >>> 477048104 34% 5.91MB/s 0:02:31 483430930 34% 5.93MB/s 0:02:30
    >>> 490941562 35% 5.74MB/s 0:02:33 496540662 35% 5.70MB/s 0:02:34
    >>> 501713859 35% 5.64MB/s 0:02:34 508037197 36% 5.83MB/s 0:02:28
    >>> 518535763 37% 6.53MB/s 0:02:11 526524427 37% 7.08MB/s 0:01:59
    >>> 538465877 38% 8.46MB/s 0:01:38 552112154 39% 10.15MB/s 0:01:21
    >>> 563067426 40% 10.23MB/s 0:01:19 573336231 41% 10.73MB/s 0:01:14
    >>> 574650941 41% 8.50MB/s 0:01:34 585401475 41% 7.67MB/s 0:01:43
    >>> 604962094 43% 9.70MB/s 0:01:19 614156492 44% 9.44MB/s 0:01:20
    >>> 616733228 44% 8.78MB/s 0:01:26 623854469 44% 8.19MB/s 0:01:32
    >>> 634887177 45% 6.37MB/s 0:01:56 643660294 46% 6.31MB/s 0:01:56
    >>> 651766969 46% 8.34MB/s 0:01:27 652307889 46% 6.71MB/s 0:01:48
    >>> 666193629 47% 7.38MB/s 0:01:36 675491703 48% 7.49MB/s 0:01:33
    >>> 684601774 49% 7.60MB/s 0:01:31 693070083 49% 9.52MB/s 0:01:12
    >>> 698081645 50% 7.45MB/s 0:01:31 704963446 50% 6.90MB/s 0:01:37
    >>> 707735864 50% 5.29MB/s 0:02:07 724318532 51% 7.15MB/s 0:01:31
    >>> 743526578 53% 10.03MB/s 0:01:03 754724412 54% 10.71MB/s 0:00:58
    >>> 761665355 54% 12.06MB/s 0:00:51 771483521 55% 10.55MB/s 0:00:57
    >>> 778832521 55% 8.18MB/s 0:01:13 786776637 56% 7.51MB/s 0:01:19
    >>> 794628478 56% 7.73MB/s 0:01:15 800825574 57% 6.87MB/s 0:01:24
    >>> 807411368 57% 6.69MB/s 0:01:25 813464143 58% 6.34MB/s 0:01:29
    >>> 822882968 58% 6.54MB/s 0:01:25 836248436 59% 8.16MB/s 0:01:06
    >>> 842835485 60% 8.15MB/s 0:01:06 850384164 60% 8.52MB/s 0:01:02
    >>> 862409333 61% 9.33MB/s 0:00:55 872067584 62% 8.51MB/s 0:01:00
    >>> 877114368 62% 8.14MB/s 0:01:02 882553071 63% 7.58MB/s 0:01:06
    >>> 891031774 63% 6.75MB/s 0:01:12 900651352 64% 6.72MB/s 0:01:11
    >>> 912028399 65% 8.22MB/s 0:00:57 923130092 66% 9.63MB/s 0:00:47
    >>> 930375419 66% 9.34MB/s 0:00:48 945088136 67% 10.23MB/s 0:00:43
    >>> 953608062 68% 9.57MB/s 0:00:45 960536527 68% 8.51MB/s 0:00:49
    >>> 971018224 69% 9.25MB/s 0:00:44 977655293 70% 7.66MB/s 0:00:53
    >>> 982496601 70% 6.80MB/s 0:00:59 990341428 70% 7.09MB/s 0:00:55
    >>> 996521787 71% 6.06MB/s 0:01:04 1005523756 72% 6.63MB/s 0:00:57
    >>> 1011459653 72% 6.88MB/s 0:00:54 1017321634 72% 6.41MB/s 0:00:57
    >>> 1024275542 73% 6.59MB/s 0:00:55 1032295357 73% 6.36MB/s 0:00:55
    >>> 1035005098 74% 5.12MB/s 0:01:08 1051640925 75% 7.47MB/s 0:00:44
    >>> 1058476439 75% 7.45MB/s 0:00:44 1068116134 76% 7.79MB/s 0:00:41
    >>> 1072174076 76% 8.72MB/s 0:00:36 1076894984 77% 5.90MB/s 0:00:52
    >>> 1086364486 77% 6.52MB/s 0:00:46 1093933058 78% 6.04MB/s 0:00:48
    >>> 1100717102 78% 6.75MB/s 0:00:42 1104505412 79% 6.53MB/s 0:00:43
    >>> 1109563535 79% 5.38MB/s 0:00:51 1127916628 80% 7.85MB/s 0:00:33
    >>> 1134948712 81% 7.92MB/s 0:00:32 1140284922 81% 8.33MB/s 0:00:29
    >>> 1148466121 82% 8.89MB/s 0:00:27 1148596687 82% 4.18MB/s 0:00:57
    >>> 1160963510 83% 5.26MB/s 0:00:43 1168681452 83% 5.74MB/s 0:00:38
    >>> 1175945190 84% 5.74MB/s 0:00:37 1178665232 84% 6.46MB/s 0:00:32
    >>> 1189338749 85% 6.09MB/s 0:00:33 1198811725 85% 6.47MB/s 0:00:29
    >>> 1207430133 86% 6.76MB/s 0:00:27 1216592101 87% 8.89MB/s 0:00:19
    >>> 1223955783 87% 8.06MB/s 0:00:20 1229789052 88% 7.19MB/s 0:00:22
    >>> 1255105302 89% 10.81MB/s 0:00:12 1268802266 90% 12.01MB/s 0:00:10
    >>> 1279289970 91% 12.82MB/s 0:00:08 1304782634 93% 17.29MB/s 0:00:05
    >>> 1325786214 94% 16.68MB/s 0:00:04 1353009354 96% 19.89MB/s 0:00:02
    >>> 1361234818 97% 19.35MB/s 0:00:01 1369458218 98% 15.27MB/s 0:00:01
    >>> 1379158338 98% 12.09MB/s 0:00:01 1395660800 100% 7.94MB/s 0:02:47
    >>> (xfer#1, to-check=12/14)
    >>>

    >>
    >>

    >

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  9. Re: Rsync of LVM Snapshots copies whole file

    On Tue, 2008-08-26 at 15:02 +0200, Peter P GMX wrote:
    > does nobody has a clue about this?


    As Paul said, the bottleneck appears to be disk I/O rather than network
    bandwidth, so you should not expect any speedup from the delta-transfer
    algorithm. (Even though only 3% of the data differs, rsync needs to
    read the entire source and destination images from disk to determine
    that.) --inplace and/or --whole-file might help reduce the disk I/O.

    Matt

    > Peter P GMX schrieb:
    > > Hello Paul,
    > >
    > > here is the status of the transfer.
    > > If I understand it right, about 3% of the data was changed (literal
    > > data against matched data). It took about 1h40min to transfer ~250MB
    > > of data.
    > > But to be honest, I do not get a real clue out of this status report.
    > > E.g. Laptop_D_10GB.vdi takes around 25 minutes to be transferred with
    > > 6,74MBytes/sec. If I divide 10G by 6,74MBytes/sec (for a complete file
    > > transfer) I end up at 40 minutes. So almost half of the complete data
    > > transfer time of this volume is needed, although this drive is rather
    > > passive and there were almost no changes on the drive between the last
    > > sync. (The machines are a AMD X2 3800 on one originating side and a
    > > Atom 1,6GHz on the destination side, Atom pulls from AMDs side).
    > >
    > > Best regards
    > > Peter
    > >
    > > receiving file list ...
    > > 13 files to consider
    > > mit_backup/Debian40.vdi
    > > 1395660800 100% 5.66MB/s 0:03:55 (xfer#1, to-check=11/13)
    > > mit_backup/Laptop_D_10GB.vdi
    > > 10734313984 100% 6.74MB/s 0:25:19 (xfer#2, to-check=10/13)
    > > mit_backup/RubyServer_1_9GB.vdi
    > > 2039489024 100% 8.15MB/s 0:03:58 (xfer#3, to-check=8/13)
    > > mit_backup/XXXX_web.vdi
    > > 2874171904 100% 6.17MB/s 0:07:24 (xfer#4, to-check=7/13)
    > > mit_backup/Ubuntu_8_04_Development.vdi
    > > 4367340032 100% 6.80MB/s 0:10:12 (xfer#5, to-check=6/13)
    > > mit_backup/Ubuntu_XXXX_3GB.vdi
    > > 3221238272 100% 5.61MB/s 0:09:07 (xfer#6, to-check=5/13)
    > > mit_backup/W2000_6GB.vdi
    > > 6442476032 100% 7.49MB/s 0:13:39 (xfer#7, to-check=4/13)
    > > mit_backup/XXXXXXXX.vdi
    > > 4294984192 100% 8.05MB/s 0:08:28 (xfer#8, to-check=3/13)
    > >
    > > Number of files: 13
    > > Number of files transferred: 8
    > > Total file size: 47.710.416.177 bytes
    > > Total transferred file size: 35.369.674.240 bytes
    > > Literal data: 1.181.132.240 bytes
    > > Matched data: 34.188.542.000 bytes
    > > File list size: 530
    > > File list generation time: 0.001 seconds
    > > File list transfer time: 0.000 seconds
    > > Total bytes sent: 3.984.972
    > > Total bytes received: 249.776.322
    > >
    > > sent 3.984.972 bytes received 249.776.322 bytes 41515.14 bytes/sec
    > > total size is 47.710.416.177 speedup is 188.01


    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEABECAAYFAki0jIQACgkQC+xSYN/RlfvjLgCgihM0peDHB9LrKLm7/RwMIHnI
    8sQAniVq/s36umLIo+TMUbYIdlb9NyiH
    =xZeE
    -----END PGP SIGNATURE-----


+ Reply to Thread