Best way to back up directories? - Slackware

This is a discussion on Best way to back up directories? - Slackware ; Hello Fellow Slackers, I am using Slackware 12.0 and I have several directories that I would like to backup incrementally every day to an external USB hard drive.. As per the tar manual, tar --update is not a good option ...

+ Reply to Thread
Results 1 to 11 of 11

Thread: Best way to back up directories?

  1. Best way to back up directories?

    Hello Fellow Slackers,

    I am using Slackware 12.0 and I have several directories that I would
    like to backup incrementally every day to an external USB hard drive..
    As per the tar manual, tar --update is not a good option for this.

    What's the best way to do this? Everything I've found on the net is for
    people looking to clone or back up an entire system. I just want to
    preserve my few precious data directories, all on my /home partition.

    I've also tried using /usr/sbin/backup included with tar, but that is
    specifically designed to control tape drives, and hasn't worked yet with
    this HD.

    Thanks for any suggestions,
    Joel
    --
    Joel J. Adamson
    Biostatistician
    Pediatric Psychopharmacology Research Unit
    Massachusetts General Hospital
    Boston, MA 02114
    (617) 643-1432
    (303) 880-3109

  2. Re: Best way to back up directories?

    Joel J. Adamson wrote:
    > What's the best way to do this? Everything I've found on the net is for
    > people looking to clone or back up an entire system. I just want to
    > preserve my few precious data directories, all on my /home partition.


    rsync i think would be the standard utility. you may also check out unison
    . unison is really meant for
    two-way synchronisation, so in actual fact more than what you want, but it
    has a GUI (which is optional: you can also run it from the command line or
    in a script) which may make it a bit easier to use. (you can set up the
    synchronisation through the GUI, then run the CLI version every day through
    cron, for example.)


    --
    Joost Kremers joostkremers@yahoo.com
    Selbst in die Unterwelt dringt durch Spalten Licht
    EN:SiS(9)

  3. Re: Best way to back up directories?

    On 2007-12-11, Joel J. Adamson wrote:
    >
    > I am using Slackware 12.0 and I have several directories that I would
    > like to backup incrementally every day to an external USB hard drive..
    > As per the tar manual, tar --update is not a good option for this.
    >
    > What's the best way to do this? Everything I've found on the net is for
    > people looking to clone or back up an entire system. I just want to
    > preserve my few precious data directories, all on my /home partition.


    I use a Perl script called dobackup.pl for this purpose:

    http://www.webcon.ca/opensource/dobackup/

    It's pretty easy to set up and run from cron, and (for your needs) easy
    to configure to back up only certain subdirectories of your filesystems.

    --keith

    --
    kkeller-usenet@wombat.san-francisco.ca.us
    (try just my userid to email me)
    AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt
    see X- headers for PGP signature information


  4. Re: Best way to back up directories?

    On Tue, 11 Dec 2007 16:52:16 -0500, jadamson@partners.org (Joel J. Adamson) wrote:

    >Hello Fellow Slackers,
    >
    >I am using Slackware 12.0 and I have several directories that I would
    >like to backup incrementally every day to an external USB hard drive..
    >As per the tar manual, tar --update is not a good option for this.
    >
    >What's the best way to do this? Everything I've found on the net is for
    >people looking to clone or back up an entire system. I just want to
    >preserve my few precious data directories, all on my /home partition.


    I use this script: http://bugsplatter.mine.nu/misc/backup-manager for
    a while now to keep a regular rsync snapshot backup of selected areas.

    Perhaps you could roll-your-own based on this?

    Grant.
    --
    http://bugsplatter.mine.nu/

  5. Re: Best way to back up directories?

    On Wed, 11 Dec 2007, Joost Kremers wrote:

    >
    > Joel J. Adamson wrote:
    >> What's the best way to do this? Everything I've found on the net is for
    >> people looking to clone or back up an entire system. I just want to
    >> preserve my few precious data directories, all on my /home partition.

    >
    > rsync i think would be the standard utility.



    I use rsync to fully sync all machines weekly, I do a 'rolling 7 day'
    daily and a weekly TAR of /etc /home and a few other critical dirs
    depending on what server/desktop it is and config files such as linux
    src's .config, sendmail.mc and so on, so backing up is a cp -a
    of the disk and extract the most recent tar of configs and home etc ...

    If you have databases, I do a 'rolling 24 hour by 7 days' (yup 168
    backups) mysqldump backup [ warning: requires huge disks unless your
    DB's are small... ] I also do a daily tar backup of the
    /var/lib/mysql/mysql directory, all the sqldumps in the world wont help
    you much if you screwup your mysql access Well ok its easy enough to
    recover, but I find it faster to do it this way... I also keep a copy of
    all this on the database server as well (saves times and hassle by
    avoiding logging into and copying the dump to the db server then loading
    it, ok, so it might only save 15 seconds, but those seconds can mean
    missed business for customers.

    Although rsync is fantastic and has saved my ass on more times that I
    care to remember with failed disks, it only takes you to go away for the
    weekend, and miss one 'run' (or 2 if you use the backup option) and you
    can't recover as the runs that occured when you weren't there, has synced
    the corrupted file(s) or deleted the missing one(s).

    All backup scripts are in perl and use Mail::Sendmail to let me know if
    somthing fails.



    --
    Cheers
    Res

    mysql> update auth set Framed-IP-Address='127.0.0.127' where user= 'troll';
    ~# radzap troll


  6. Re: Best way to back up directories?

    Hallo, Joel,

    Du meintest am 11.12.07:

    > I am using Slackware 12.0 and I have several directories that I would
    > like to backup incrementally every day to an external USB hard
    > drive.. As per the tar manual, tar --update is not a good option for
    > this.


    > What's the best way to do this?


    http://arktur.de/Wiki/Zusatzprogramme:rsnapshot
    http://www.rsnapshot.org/

    Viele Gruesse
    Helmut

    "Ubuntu" - an African word, meaning "Slackware is too hard for me".


  7. Re: Best way to back up directories?

    On 2007-12-12, Helmut Hullen wrote:

    > Viele Gruesse
    > Helmut
    >
    > "Ubuntu" - an African word, meaning "Slackware is too hard for me".


    If you're gonna use my signature, you should at least learn how to
    properly use a signature. Hint: "delimiter".


    --
    "Ubuntu" - an African word meaning "Slackware is too hard for me".


  8. Re: Best way to back up directories?

    On Tue, 11 Dec 2007 16:52:16 -0500, Joel J. Adamson wrote:

    > Hello Fellow Slackers,
    >
    > I am using Slackware 12.0 and I have several directories that I would
    > like to backup incrementally every day to an external USB hard drive..
    > As per the tar manual, tar --update is not a good option for this.
    >
    > What's the best way to do this? Everything I've found on the net is for
    > people looking to clone or back up an entire system. I just want to
    > preserve my few precious data directories, all on my /home partition.
    >
    > I've also tried using /usr/sbin/backup included with tar, but that is
    > specifically designed to control tape drives, and hasn't worked yet with
    > this HD.
    >
    > Thanks for any suggestions,
    > Joel
    >

    I have been using a home-grown system to backup directories. It has been a
    good choice for me because it concentrates most of the backup effort on
    actual file changes, and ignores a large set of static files. The
    usual caveats should be stated in advance, though. Backing up does not
    lend itselt to a one-size-fits-all approach. Pick the method which
    fits your needs. IME, it never hurts to know the fundamentals of the
    tools which you use, and to practice using them (both backing up and
    restoring) whatever method is selected.

    The method that is explained below sets up a simple "CVS" system for the
    directory targetted for backup. This method allows any previous saved
    state to be restored. For example, it's easy to retrieve files saved last
    Thursday, or last July. This method may be overkill for you specific
    application. If so, then another tool which might be used to do something
    very similar is rsync, secifically, using rsync's "--only-write-batch"
    mode.

    Here is an overview of the method that I use:

    1. Create a full snapshot of the directory to be backed up using the
    method of your choice; IMO, tar works fine. If you need extended
    attributes (acl and attr), then save them separately (man getfacl, man
    getfattr)

    2. Schedule incremental daily backup jobs with "at" or as a cron job at
    the appropriate time interval for your requirements. When this backup job
    runs it obtains a current listing of the target directory, and compares
    this new listing to the listing as obtained by the last backup job. This
    comparison yields four reports:

    report 1: files which match exactly with the previous day.
    report 2: files which have changed
    report 3: files which have been deleted
    report 4: files which have been added

    Files in report 2 and report 4 should be backed up. Again, tar works fine
    for this job using the "-T" switch. Also, a compression stage (gzip,
    bzip2) can significantly reduce the size of incremental jobs.

    Along with the archive, save the following:
    a. The directory listing when the incremental backup was saved
    b. File selections in this backup. This makes it easier to lookup what
    has been backed up (and which versions are available for restore.)
    c. File extended attributes. Some features may not be captured by tar, or
    other archive tools.

    Warning: "locked" and active files and databases may require special
    consideration in order for them to be saved in a consistent state.

    3. Review daily backup log. "at" jobs will send email back to the
    user. Review the message to verify that all files which were selected for
    backup could be opened and were written into the daily backup job.

    4. Repeat to step 2. To start over with a new clean backup set, repeat to
    step 1.

    There are various options for restoring. The simplest is to restore in
    reverse chronological order (use the "-k" switch with tar.)

    Notice: this method yields a "superset" of all files which were backed up
    in the past. This is good from a "CVS" point of view, but can have some
    side effects, also. For example, the space necessary to perform a restore
    operation can be much larger than the space in use, especially if the
    directory structure has been rearranged. Another side effect is that a
    restore operation is not complete until the "superset" (as restored) is
    "pruned" to account for file deletions on the interval.

    Notice: daily incremental backups can be restored separately to generate
    larger relative backups. For example, daily backups can be recombined into
    weekly, or monthly relative backups. This can simplify the restore
    operation, while potentially sacrificing some granularity.


    --
    Douglas Mayne

  9. Re: Best way to back up directories?

    On Tue, 11 Dec 2007 16:52:16 -0500, Joel J. Adamson wrote:

    > Hello Fellow Slackers,
    >
    > I am using Slackware 12.0 and I have several directories that I would
    > like to backup incrementally every day to an external USB hard drive..
    > As per the tar manual, tar --update is not a good option for this.
    >
    > What's the best way to do this? Everything I've found on the net is for
    > people looking to clone or back up an entire system. I just want to
    > preserve my few precious data directories, all on my /home partition.
    >
    > I've also tried using /usr/sbin/backup included with tar, but that is
    > specifically designed to control tape drives, and hasn't worked yet with
    > this HD.
    >


    Here is a link to a nice article on using rsync to do backups, including
    incremental backups.

    http://www.mikerubel.org/computers/r...ots/index.html

    Cheers,
    Mike


  10. Re: Best way to back up directories?

    Joel J. Adamson wrote:
    > Hello Fellow Slackers,


    > I am using Slackware 12.0 and I have several directories that I would
    > like to backup incrementally every day to an external USB hard drive..
    > As per the tar manual, tar --update is not a good option for this.


    [snip]

    Joel,

    It sounds to me like you want to copy the directories to a location on
    your USB hard drive.

    So make a destination directory on the USB hard drive and mount the USB
    hard drive. For example, your destination directory when mounted could
    be

    /mnt/backups/071214aa

    The directory /mnt/backups is the mount point, and the dir 071214aa is
    the dir that you made on the USB hard drive. The name 071214aa means
    "the aa backup (or first backup) on 12-14-07". The second backup on
    that date would be "ab" and so on, however I don't know if you are
    likely to do more than one in one day.

    Then use (probably as root):

    cp -Rpdi path-to-source-directory /mnt/backups/071214aa

    repeat for each source directory that you wish to backup

    Here's what the options -Rpdi do

    -R Copy directories recursively. This gets all the files *and*
    subdirectories, and so on.

    -p Preserve the original files' owner, group, permissions.

    -d Copy symbolic links as symbolic links rather than copying
    the files that they point to.

    -i Prompts you if anything already exists at the target with the
    same name. This is not going to happen if the target dir is
    new, but it is a nice habbit to use -i. You might someday
    avoid overwriting something this way.

    The idea is that you can do backups quite adequately (despite all the
    complex advice you received from others) by doing a simple copy. Each
    time (each new backup date) use a new target directory.

    After a while you can start deleting the old target directories. For
    example, to delete /mnt/backups/071214aa do this (probably as root):

    rm -R /mnt/backups/071214aa

    If you backups are really large, then you may want to look at the data
    you are backing up. Have you mixed short files that change frequently
    with longer files (like sound, video or picture files) that never
    change? I keep sound, video picture and various other kinds of files
    in separate directories. Because the files are large, they fill up a
    lot of space in a hurry. Because they don't ever change there is no
    need to back them up repeatedly. So I occasionally "archive" them by
    burning them to a cd, then I delete the dirs from my hard drive. Or
    you could just make the dirs read only.

    The idea is that it pays to curtail the size of the directories that you
    are backing up. If you do this effectively then it is practical to
    backup everything each time by copying the whole directory as I showed
    above. A copy of a directory has the benefit that it is easy to refer
    to later.

    Or maybe your directories are small enough that size isn't an issue.

    Do you write bash scripts? A little scripting to make some tools for
    your own use would make the above use of mount, cp and rm easier and
    more consistant.

    -Joe Rosevear

  11. Re: Best way to back up directories?

    On 2007-12-11, Joel J. Adamson wrote:
    > Hello Fellow Slackers,
    >
    > I am using Slackware 12.0 and I have several directories that I would
    > like to backup incrementally every day to an external USB hard drive..
    > As per the tar manual, tar --update is not a good option for this.
    >
    > What's the best way to do this? Everything I've found on the net is for
    > people looking to clone or back up an entire system. I just want to
    > preserve my few precious data directories, all on my /home partition.
    >
    > I've also tried using /usr/sbin/backup included with tar, but that is
    > specifically designed to control tape drives, and hasn't worked yet with
    > this HD.



    You've gotten several response now, so this might be too little and
    too late, but...

    Erik Jan Tromp (alphageek) has a "hardlink" script he wrote to 'merge'
    source trees that contain many identical files - it's here:
    http://alphageek.dyndns.org/linux/script-feh/hardlink
    Something along those lines coupled with rsync's -H (preserve hardlinks)
    option might be what you're after to keep incremental backups without
    using nearly as much space as that normally incurs.

    --RW

+ Reply to Thread