Yet another rsync incremental thread - Tools

This is a discussion on Yet another rsync incremental thread - Tools ; Hello all, Since the rsync on Panther many things changed in my professional life. This project is abandoned although it should work. But this you already know. What's new? On my new job I have several servers to administrate. Servers ...

+ Reply to Thread
Results 1 to 5 of 5

Thread: Yet another rsync incremental thread

  1. Yet another rsync incremental thread

    Hello all,

    Since the rsync on Panther many things changed in my professional
    life. This project is abandoned although it should work. But this you
    already know. What's new? On my new job I have several servers to
    administrate. Servers that aren't backed up (sic). So, there's why I'm
    back to rsync.

    The backup plan I would like:

    1) Client side: PCs running rsync (or cwrsync with UTF-8 mod for long
    names for windows ones). They run rsync (I don't need data encryption
    so no need to ssh unless it's simpler) and pull data they want to the
    backup server. This side is OK.

    2) Server side: A standard PC with FreeNAS FreeBSD distrib, RAID-1
    disks and rsync server. Here I would like the backups to be
    incremental and rotated, with a sort of time stamp. Here is where
    things get complicated.

    Solutions I've found:

    A - Simpler at the end but would need modifications to rsync: Go
    further on the backup module of rsync, building in the rotate mv/rm
    commands. Why simpler? Because a rsync client would only do one
    command to the rsync server saying "there is it", and the server do
    it's own internal cook with the stuff. No need to additional scripts.
    I know that rsync isn't an incremental backup tool at the basis but a
    syncing tool. But as it already do some of the work, why not doing it
    better?

    B - As I said before, additional scripts. Do the client open an ssh
    connection, do the pre_backup.sh then close ssh (as I understood even
    using ssh for rsync, it uses it own tunnel, and not an already opened
    one, no?) run rsync, then rerun ssh, do the post_backup.sh. That
    should work, but why scripting many times (I'm sure I'm not the only
    one in this case) when it could be done once for all?

    C - Between A and B solutions, rsync could, in a first time, have an
    option to run pre and post scripts when running on ssh. In this manner
    there would not be needed to open and close ssh 3 times.

    D - I looked at rsnapshot, but it wouldn't help on this case of
    rotating server side folders. I also found complicated scripts for
    rotating folders from the server, but the client would need putting a
    flag on the server saying syncing is done. All of this don't seem to
    be the good approach for me. To much complicated to respond to some
    lines of code missing on rsync (OK, missing for me but I know that
    this isn't the original goal of rsync and it does lovely what it was
    designed for).

    Well, I'm not an rsync expert, so please correct my exposé if I'm
    wrong. I think solution C would be easy to code, solution A would be
    the ideal for me but requires some more functions. I'm ready to help
    coding it. What do you think? Am I on the right path? Would this
    addition being integrated to the main rsync (after testing, of
    course)? Any other remarks or suggestions?

    Best regards,

    Vitorio--
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  2. Re: Yet another rsync incremental thread

    On Fri, 2008-07-18 at 12:40 +0200, macuserfr wrote:
    > What's new? On my new job I have several servers to
    > administrate. Servers that aren't backed up (sic). So, there's why I'm
    > back to rsync.
    >
    > The backup plan I would like:
    >
    > 1) Client side: PCs running rsync (or cwrsync with UTF-8 mod for long
    > names for windows ones). They run rsync (I don't need data encryption
    > so no need to ssh unless it's simpler) and pull data they want to the
    > backup server. This side is OK.


    I think you mean "push".

    > 2) Server side: A standard PC with FreeNAS FreeBSD distrib, RAID-1
    > disks and rsync server. Here I would like the backups to be
    > incremental and rotated, with a sort of time stamp. Here is where
    > things get complicated.


    Just use an rsync daemon wired up to an rsnapshot installation in
    sync_first mode, where the daemon module points to the .sync dir in the
    snapshot root and the "post-xfer exec" command calls "rsnapshot
    LOWEST-INTERVAL". You can use a listening daemon or a single-use daemon
    over ssh as you prefer. This is the approach I've been recommending for
    pushing hard-linked backups for a while; it achieves everything you want
    without modifying any of the tools. I helped Eric Johansson with a
    similar setup a while ago:

    http://lists.samba.org/archive/rsync...er/019470.html

    If you have multiple clients pushing backups, it's simplest to have a
    separate module and snapshot root for each client. To keep things
    manageable, you can write a script to automatically generate the
    individual rsnapshot.conf files by substituting the necessary values
    into a template.

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEABECAAYFAkiA8GIACgkQC+xSYN/RlfsNNACfd5IMMPqiAw0pW65F6ebtMmtF
    3mUAmwcIp+0ajpxSMaH1k4RCWMLshlCD
    =M8Js
    -----END PGP SIGNATURE-----


  3. Re: Yet another rsync incremental thread

    Thanks Matt!! I'm always amazed of the quick and precise answers you
    give!

    Yes, sorry, I meant push (English isn't my native langage and I always
    swap push and pull).
    For rsnapshot, OK, I understood now how to connect them. The "post-
    xfer exec" I was looking for don't exist on the rsync man page, but
    on the rsyncd.conf which I've didn't look at (my fault :/ ). I think I
    will not use rsnapshot because I don't like to use to many software
    but only a small rotating script home made.

    Didn't finished the script yet, but it will do something like this:
    1) Expected first state:

    backup (rsync transfer folder), lastest (previous transfer folder) and
    directories (old backup)

    2) Execution of the script (pseudo code):

    get date
    get free disk space
    get backup size

    mv lastest ->
    mv backup -> lastest

    while(free space < backup size) #disk almost full, need deleting some
    old stuff
    get oldest backup folder name
    rm it

    That's all. The only thing I haven't coded yet is the while loop, but
    I will do it next week.
    What could rsnapshot give me in addition to this? Would it manage free
    space for many concurrent backups?
    Using my script for 2 or more backups, if the other backup takes
    almost all disk, it won't be smart enough to delete backups from the
    other backup. There are some other cases I should care about. But I
    still have the impression rsnapshot would be too big stuff for only
    this need.

    In another hand, don't you think we could add the content of this
    script (revised, of course) to make rsync a complete incremental
    backup solution without needing 3rd part software? Isn't a good idea?

    Best regards,

    Vitorio

    Le 18 juil. 08 à 21:35, Matt McCutchen a écrit :

    > On Fri, 2008-07-18 at 12:40 +0200, macuserfr wrote:
    >> What's new? On my new job I have several servers to
    >> administrate. Servers that aren't backed up (sic). So, there's why
    >> I'm
    >> back to rsync.
    >>
    >> The backup plan I would like:
    >>
    >> 1) Client side: PCs running rsync (or cwrsync with UTF-8 mod for long
    >> names for windows ones). They run rsync (I don't need data encryption
    >> so no need to ssh unless it's simpler) and pull data they want to the
    >> backup server. This side is OK.

    >
    > I think you mean "push".
    >
    >> 2) Server side: A standard PC with FreeNAS FreeBSD distrib, RAID-1
    >> disks and rsync server. Here I would like the backups to be
    >> incremental and rotated, with a sort of time stamp. Here is where
    >> things get complicated.

    >
    > Just use an rsync daemon wired up to an rsnapshot installation in
    > sync_first mode, where the daemon module points to the .sync dir in
    > the
    > snapshot root and the "post-xfer exec" command calls "rsnapshot
    > LOWEST-INTERVAL". You can use a listening daemon or a single-use
    > daemon
    > over ssh as you prefer. This is the approach I've been recommending
    > for
    > pushing hard-linked backups for a while; it achieves everything you
    > want
    > without modifying any of the tools. I helped Eric Johansson with a
    > similar setup a while ago:
    >
    > http://lists.samba.org/archive/rsync...er/019470.html
    >
    > If you have multiple clients pushing backups, it's simplest to have a
    > separate module and snapshot root for each client. To keep things
    > manageable, you can write a script to automatically generate the
    > individual rsnapshot.conf files by substituting the necessary values
    > into a template.
    >
    > Matt


    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  4. Re: Yet another rsync incremental thread

    On Sat, 2008-07-19 at 15:45 +0200, Mac User FR wrote:
    > For rsnapshot, OK, I understood now how to connect them. The "post-
    > xfer exec" I was looking for don't exist on the rsync man page, but
    > on the rsyncd.conf which I've didn't look at (my fault :/ ). I think I
    > will not use rsnapshot because I don't like to use to many software
    > but only a small rotating script home made.
    >
    > Didn't finished the script yet, but it will do something like this:
    > 1) Expected first state:
    >
    > backup (rsync transfer folder), lastest (previous transfer folder) and
    > directories (old backup)
    >
    > 2) Execution of the script (pseudo code):
    >
    > get date
    > get free disk space
    > get backup size
    >
    > mv lastest ->
    > mv backup -> lastest
    >
    > while(free space < backup size) #disk almost full, need deleting some
    > old stuff
    > get oldest backup folder name
    > rm it
    >
    > That's all. The only thing I haven't coded yet is the while loop, but
    > I will do it next week.
    > What could rsnapshot give me in addition to this? Would it manage free
    > space for many concurrent backups?
    > Using my script for 2 or more backups, if the other backup takes
    > almost all disk, it won't be smart enough to delete backups from the
    > other backup. There are some other cases I should care about. But I
    > still have the impression rsnapshot would be too big stuff for only
    > this need.


    Rsnapshot is similar to your script except that it retains a specified
    fixed number of backups. It has no feature to delete as many backups as
    necessary to free the proper amount of disk space, but that could be
    done pretty easily in a script that is invoked before (or after)
    rsnapshot. With multiple clients, it would not be hard to make the
    script consider all the snapshot roots together when choosing the backup
    to delete. I think rsnapshot is not as heavy-weight as you believe, but
    go ahead and use your own script if you find it easier to understand.

    > In another hand, don't you think we could add the content of this
    > script (revised, of course) to make rsync a complete incremental
    > backup solution without needing 3rd part software? Isn't a good idea?


    Why? The goal of rsync is to copy files, not to be a complete
    incremental backup solution by itself, even though several of its
    options are motivated by this use case. I think the current state of
    affairs with a wide variety of available rsync-based backup solutions
    (rsnapshot, dirvish, ccollect, various homegrown scripts including
    yours, ...) is just fine. Different users prefer different solutions,
    so I don't think arbitrarily choosing one to include in rsync would be
    appropriate.

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEABECAAYFAkiCLuIACgkQC+xSYN/RlftPegCghLPW5K4vksgOR8F1XIY3RVhw
    HfEAn0BTGdI1OcjYMc3ZTXIBmXA7Wv5b
    =Kj2l
    -----END PGP SIGNATURE-----


  5. Re: Yet another rsync incremental thread


    Le 19 juil. 08 à 20:13, Matt McCutchen a écrit :

    > On Sat, 2008-07-19 at 15:45 +0200, Mac User FR wrote:
    >>
    >> In another hand, don't you think we could add the content of this
    >> script (revised, of course) to make rsync a complete incremental
    >> backup solution without needing 3rd part software? Isn't a good idea?

    >
    > Why? The goal of rsync is to copy files, not to be a complete
    > incremental backup solution by itself, even though several of its
    > options are motivated by this use case. I think the current state of
    > affairs with a wide variety of available rsync-based backup solutions
    > (rsnapshot, dirvish, ccollect, various homegrown scripts including
    > yours, ...) is just fine. Different users prefer different solutions,
    > so I don't think arbitrarily choosing one to include in rsync would be
    > appropriate.
    >
    > Matt


    OK, I though you though this way. I'm not wanting to impose my script
    particularly, just thinking about people with less computer skills
    than ours. Making things easier for them. This is not my case, I'm not
    afraid about coding some stuff like I did for Panther being able to
    save resource forks or now putting hands to work to get an incremental
    backup. But I'm sure some people would enjoy a free, open source
    solution to make those incremental backups out of the box.

    I understand that this isn't the goal of rsync. I'm only pointing this
    because the adaptation would only take few code lines and I think
    rsync worth this, as an option --incremental or something like that
    that people are not obligated to use by default. If you still think
    rsync don't need it, the solution would be giving another name for the
    moded rsync. Unfortunately I can't help going this way. I could have
    some time to code this feature inside rsync, but maintaining a
    parallel program takes too much resources for me.

    Best regards,

    Vitorio--
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


+ Reply to Thread