simple incremental backup woes. solutions? - Tools

This is a discussion on simple incremental backup woes. solutions? - Tools ; Hi all, I've finally hit rsync limit. As someone already addressed the problem in this list the problem regards the impossibility for rsync to rename directories. The situation is as follow: every night I backup my samba dir with this ...

+ Reply to Thread
Results 1 to 12 of 12

Thread: simple incremental backup woes. solutions?

  1. simple incremental backup woes. solutions?

    Hi all,
    I've finally hit rsync limit. As someone already addressed the problem in this list the problem regards the impossibility for rsync to rename directories.
    The situation is as follow:
    every night I backup my samba dir with this command:

    rsync -ab --suffix=-`date +%F-%H%M` --filter="protect *-[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9][0-9][0-9]" --delete --numeric-ids --stats -h /home/samba/ /mnt/usbdisk

    as you can see, I simply rename the files deleted with the date they've been found deleted, hence providing a protect filter for the files with a date at the end (as you might imagine the machine serves windows clients).
    Obviously (!) when a directory is deleted on the source dir, all the files in the destination will be renamed, but the dir will not, and rsync tries to delete it ending with a "cannot delete a non-empty dir".

    I'm asking here because I need even a dirty solution to this problem.
    I've been addressed to rsnapshot, but seems hard to replicate the same backup type I've setup, so I gave it up.

    Any suggestion, solutions and such will be very much appreciated.

    Thank you in advance,


    --
    Matteo 'Peach' Pescarin
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  2. Moved files

    Not sure if this has ever been covered, but I can't seem to find anything on
    Google:

    Is it possible to perform the following for moved files (i.e. files that
    already exist on destination, but are moved into a different or new dir on
    the source):

    - generate filelist including deleted files (without deleting)
    - during transfer, for each *new* file, check against above list for
    deletions.
    - If there is a match, and size is similar, try using that file as a basis
    for transfer.
    - finally delete all... (like --delete-after)

    All this would be sort of related to the --fuzzy option, but for moved files
    instead of renamed. This would be especially useful for scripts that deal
    with a few large files that sometimes change place, are dumped into archive
    dirs etc...

    I'm not sure if this can already be done with some combination of options,
    if not I think it would be a cool feature.

    Cheers
    Julian




    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  3. Re: Moved files

    On Thu, 2008-09-04 at 17:52 +0200, Julian Pace Ross wrote:
    > Is it possible to perform the following for moved files (i.e. files that
    > already exist on destination, but are moved into a different or new dir on
    > the source):
    >
    > - generate filelist including deleted files (without deleting)
    > - during transfer, for each *new* file, check against above list for
    > deletions.
    > - If there is a match, and size is similar, try using that file as a basis
    > for transfer.
    > - finally delete all... (like --delete-after)
    >
    > All this would be sort of related to the --fuzzy option, but for moved files
    > instead of renamed. This would be especially useful for scripts that deal
    > with a few large files that sometimes change place, are dumped into archive
    > dirs etc...


    The --detect-renamed option added by detect-renamed.diff does
    essentially this, except that it checks each deleted file against the
    list of new files rather than vice versa. This technique was probably
    easier to implement before incremental recursion because a full list of
    the source was available, but it fails for moves into directories that
    haven't yet been created and doesn't extend nicely to detecting copies.

    I would like to have an implementation that checks each new file against
    the list of existing destination files; I'm not sure whether it would be
    better to generate the list first (as you described) or postpone new
    transfers to a second pass after the whole destination is scanned (as
    the current implementation does). Neither technique would interfere
    with the use of incremental recursion on the source.

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEABECAAYFAkjAFX8ACgkQC+xSYN/Rlfs3XQCfeUCYVzK8L4Iu//zOLJ8EYQES
    onYAnA1j2bPhjUdElNzCH2ReEFFf7XfQ
    =2HKa
    -----END PGP SIGNATURE-----


  4. Re: simple incremental backup woes. solutions?

    On Wed, 2008-09-03 at 17:59 +0200, The Peach wrote:
    > I've finally hit rsync limit. As someone already addressed the
    > problem in this list the problem regards the impossibility for rsync
    > to rename directories.
    > The situation is as follow:
    > every night I backup my samba dir with this command:
    >
    > rsync -ab --suffix=-`date +%F-%H%M` --filter="protect *-[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9][0-9][0-9]" --delete --numeric-ids --stats -h /home/samba/ /mnt/usbdisk
    >
    > as you can see, I simply rename the files deleted with the date
    > they've been found deleted, hence providing a protect filter for the
    > files with a date at the end (as you might imagine the machine serves
    > windows clients).
    > Obviously (!) when a directory is deleted on the source dir, all the
    > files in the destination will be renamed, but the dir will not, and
    > rsync tries to delete it ending with a "cannot delete a non-empty
    > dir".
    >
    > I'm asking here because I need even a dirty solution to this problem.


    It's not clear to me what the problem is, since the "cannot delete a
    non-empty dir" message is pretty harmless. Do you want the directories
    to be renamed? If so, should the individual files inside be renamed
    too? A naive implementation of either behavior that works in the
    absence of nonperishable protect filters would be quite easy. Note that
    the current behavior has the advantage or disadvantage (depending on
    your point of view) that all versions of the file at a given path are
    kept together even if an ancestor directory is deleted and recreated.

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEABECAAYFAkjAZaoACgkQC+xSYN/RlfsJEgCgm0cEM0JpgXbyxnZ+1w9JsQzW
    /bUAoJX5yal86jlS+862d2eiu9p7/8J3
    =RQQl
    -----END PGP SIGNATURE-----


  5. Re: simple incremental backup woes. solutions?

    On Thu, 04 Sep 2008 22:48:28 +0000
    Matt wrote:

    > It's not clear to me what the problem is, since the "cannot delete a
    > non-empty dir" message is pretty harmless. Do you want the directories
    > to be renamed? If so, should the individual files inside be renamed
    > too?


    The situation with my rsync config, as stated in my first post, is this:
    1) In the source I've got a directory with several files inside it.
    2) When first executed, rsync copies everything correctly.
    3) On the source, I delete the directory.
    4) On next execution, rsync renames all the files inside the directory
    as I told it to do so. But the directory will not be renamed like it
    did with the files. Instead rsync tries to delete the directory
    finishing with an obvious warning of "cannot delete non-empty
    directory".

    is it clear? As long as I've searched, this seems the normal behaviour
    of rsync, and I have nothing against it. I'm just trying to achieve my
    personal results either with a combination of options or in some other
    way (i.e. looking at rsync output with a bash script or whatever), or
    even asking here if this can be implemented.

    First question: if a --suffix-dir option can be implemented, the
    suffixed dirs will be exluded by --filter option? It should, as long as
    the directory is a file itself on unix systems, but I didn't tried it
    already.

    Thanks for your replies.
    regards,

    --
    Matteo 'Peach' Pescarin
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  6. Re: simple incremental backup woes. solutions?

    On Fri, 2008-09-05 at 13:15 +0200, The Peach wrote:
    > On Thu, 04 Sep 2008 22:48:28 +0000
    > Matt wrote:
    >
    > > It's not clear to me what the problem is, since the "cannot delete a
    > > non-empty dir" message is pretty harmless. Do you want the directories
    > > to be renamed? If so, should the individual files inside be renamed
    > > too?

    >
    > The situation with my rsync config, as stated in my first post, is this:
    > 1) In the source I've got a directory with several files inside it.
    > 2) When first executed, rsync copies everything correctly.
    > 3) On the source, I delete the directory.
    > 4) On next execution, rsync renames all the files inside the directory
    > as I told it to do so. But the directory will not be renamed like it
    > did with the files. Instead rsync tries to delete the directory
    > finishing with an obvious warning of "cannot delete non-empty
    > directory".
    >
    > is it clear?


    I understand what rsync is doing now. My question which you haven't
    answered is, what exactly do you *want* rsync to do? Do you want it to
    just add the suffix to deleted destination directories, or to both the
    directories and the files inside, or what?

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEABECAAYFAkjBSs0ACgkQC+xSYN/RlfsoFACggDMKTHvV1Ht5I/hSutomvNXo
    MKUAoLDcwRAHeRpJWmH9oypjHqi0qZzb
    =i+9B
    -----END PGP SIGNATURE-----


  7. Re: simple incremental backup woes. solutions?

    On Fri, 05 Sep 2008 11:05:52 -0400
    Matt wrote:

    > o you want it to
    > just add the suffix to deleted destination directories, or to both the
    > directories and the files inside, or what?


    Sorry for this misunderstanding.

    Right now:
    if the directory in the source is deleted, in the destination the files
    contained in the directory are renamed, while the directory itself
    is not.

    So... is it possible to have both renamed instead of only the files?


    Regards,
    --
    Matteo 'Peach' Pescarin
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  8. Re: simple incremental backup woes. solutions?

    On Fri, 2008-09-05 at 20:29 +0200, The Peach wrote:
    > Right now:
    > if the directory in the source is deleted, in the destination the files
    > contained in the directory are renamed, while the directory itself
    > is not.
    >
    > So... is it possible to have both renamed instead of only the files?


    Yes. I hacked up a patch to do this for you with a --backup-whole-dirs
    option; the patch is attached. The patched version of rsync is also
    available in branch hacks/backup-whole-dirs of my repository.

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

  9. Re: simple incremental backup woes. solutions?

    On Fri, 05 Sep 2008 15:44:43 -0400
    Matt wrote:

    > Yes. I hacked up a patch to do this for you with a --backup-whole-dirs
    > option; the patch is attached. The patched version of rsync is also
    > available in branch hacks/backup-whole-dirs of my repository.
    >
    > Matt


    thank you a lot.
    I'm looking forward to test this.
    I'll report in asap.

    thanks again,

    --
    Matteo 'Peach' Pescarin
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  10. Re: Moved files


  11. Re: simple incremental backup woes. solutions?

    On Fri, 05 Sep 2008 15:44:43 -0400
    Matt wrote:

    > I hacked up a patch to do this for you with a --backup-whole-dirs
    > option; the patch is attached.


    sorry if I report in too late, but I finally got time to test the patch and - I never had a suspect about it - it works perfectly!

    thanks a lot again!

    Just an info: will this patch be merged in the stable branch of rsync?
    or should I patch any new versions if I want to upgrade?

    ciao

    --
    Matteo 'Peach' Pescarin
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  12. Re: simple incremental backup woes. solutions?

    On Mon, 2008-09-22 at 11:46 +0200, The Peach wrote:
    > Just an info: will this patch be merged in the stable branch of rsync?


    Most likely not, as it's a quick hack.

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


+ Reply to Thread