Ready-to-roll System Backup - Setup

This is a discussion on Ready-to-roll System Backup - Setup ; Greetings, I'm trying to come up with a backup solution that will allow me to incrementally clone the entire system (Box A) each night to another machine on the network (Box B) so that if Box A goes down, Box ...

+ Reply to Thread
Results 1 to 11 of 11

Thread: Ready-to-roll System Backup

  1. Ready-to-roll System Backup

    Greetings,

    I'm trying to come up with a backup solution that will allow me to
    incrementally clone the entire system (Box A) each night to another
    machine on the network (Box B) so that if Box A goes down, Box B will
    be an exact replica of Box A, ready to boot up and run.

    What's the best way to achieve this? I've looked at tar solutions,
    rsync solutions, dd solutions...I'm just not sure which method would
    be best for this situation.

    Thanks in advance!

    Alan


  2. Re: Ready-to-roll System Backup

    On Thu, 01 Feb 2007 07:34:05 -0800, AlanM wrote:

    > Greetings,
    >
    > I'm trying to come up with a backup solution that will allow me to
    > incrementally clone the entire system (Box A) each night to another
    > machine on the network (Box B) so that if Box A goes down, Box B will
    > be an exact replica of Box A, ready to boot up and run.
    >
    > What's the best way to achieve this? I've looked at tar solutions,
    > rsync solutions, dd solutions...I'm just not sure which method would
    > be best for this situation.
    >
    > Thanks in advance!
    >
    > Alan
    >

    I guess it depends on how exact the backup needs to be. Most backup
    techniques are like taking photographs, and photography is a
    discrete process. That is, if you were using photographs to rank the
    finishers at a horse race, then one photograph might establish the
    winner, but that may not be enough to fix "place" and "show." The
    horses keep moving; likewise, your filesystem is in continuous motion, too.

    The first step is to establish you requirements in finer detail. Here are
    some items which might be included for consideration:

    possible failure modes,
    risks associated with downtime,
    expense of hardware,
    discrete or continuous backup strategy (is losing transactions permitted),
    the chosen solution's complexity.

    Here are some possibilities, rated from lower to higher complexity:

    1. LVMv2 (device mapper) snapshots. I wrote about this here:
    http://groups.google.com/group/comp....9e22af15298610

    A discrete snapshot can be restored to an earlier state. Some downtime is
    to be expected.

    2. Two boxes, which are hosts for a virtual machine which is stored as a
    virtual machine image on a SAN.

    SAN
    VM Host 1 ----------+-------------VM Host 2


    If hardware fails one VM Host, then the other can take over while it is
    repaired with very little downtime. This solution assumes no failure in
    the SAN and the both VM host boxes are not disabled during the same time
    interval or maintenance window.

    3. Live, "on the wire" clustering. Each box in the cluster does
    its own calculations. If the primary cluster box fails, then another
    can take over using some failover mechanism. This is beyond my experience.
    I know it exists, though.

    --
    Douglas Mayne

  3. Re: Ready-to-roll System Backup

    Thanks for the response. It really doesn't have to be exact down to
    the second. The backup would only be run once each night. Basically, I
    just need the backup server to be a duplicate of the main server so
    that if the main server goes down, we can just call networking and
    have the DNS pointed to the backup until the main server is fixed (or
    replaced).

    I suppose rsync would do the trick but i was hoping to do an entire
    system clone in one fell swoop. I guess one way of doing it might be
    to clone the basic system, then do an rsync each night just for all
    the data that needs to be backed up. (It's a web server so just
    rsyncing the web docs would suffice, don't you think?)

    Alan

    On Feb 1, 11:30 am, Douglas Mayne wrote:
    > On Thu, 01 Feb 2007 07:34:05 -0800, AlanM wrote:
    > > Greetings,

    >
    > > I'm trying to come up with a backup solution that will allow me to
    > > incrementally clone the entire system (Box A) each night to another
    > > machine on the network (Box B) so that if Box A goes down, Box B will
    > > be an exact replica of Box A, ready to boot up and run.

    >
    > > What's the best way to achieve this? I've looked at tar solutions,
    > > rsync solutions, dd solutions...I'm just not sure which method would
    > > be best for this situation.

    >
    > > Thanks in advance!

    >
    > > Alan

    >
    > I guess it depends on how exact the backup needs to be. Most backup
    > techniques are like taking photographs, and photography is a
    > discrete process. That is, if you were using photographs to rank the
    > finishers at a horse race, then one photograph might establish the
    > winner, but that may not be enough to fix "place" and "show." The
    > horses keep moving; likewise, your filesystem is in continuous motion, too.
    >
    > The first step is to establish you requirements in finer detail. Here are
    > some items which might be included for consideration:
    >
    > possible failure modes,
    > risks associated with downtime,
    > expense of hardware,
    > discrete or continuous backup strategy (is losing transactions permitted),
    > the chosen solution's complexity.
    >
    > Here are some possibilities, rated from lower to higher complexity:
    >
    > 1. LVMv2 (device mapper) snapshots. I wrote about this here:http://groups.google.com/group/comp....9e22af15298610
    >
    > A discrete snapshot can be restored to an earlier state. Some downtime is
    > to be expected.
    >
    > 2. Two boxes, which are hosts for a virtual machine which is stored as a
    > virtual machine image on a SAN.
    >
    > SAN
    > VM Host 1 ----------+-------------VM Host 2
    >
    > If hardware fails one VM Host, then the other can take over while it is
    > repaired with very little downtime. This solution assumes no failure in
    > the SAN and the both VM host boxes are not disabled during the same time
    > interval or maintenance window.
    >
    > 3. Live, "on the wire" clustering. Each box in the cluster does
    > its own calculations. If the primary cluster box fails, then another
    > can take over using some failover mechanism. This is beyond my experience.
    > I know it exists, though.
    >
    > --
    > Douglas Mayne




  4. Re: Ready-to-roll System Backup

    On Thu, 01 Feb 2007 09:13:29 -0800, AlanM wrote:

    > Thanks for the response. It really doesn't have to be exact down to
    > the second. The backup would only be run once each night. Basically, I
    > just need the backup server to be a duplicate of the main server so
    > that if the main server goes down, we can just call networking and
    > have the DNS pointed to the backup until the main server is fixed (or
    > replaced).
    >
    > I suppose rsync would do the trick but i was hoping to do an entire
    > system clone in one fell swoop. I guess one way of doing it might be
    > to clone the basic system, then do an rsync each night just for all
    > the data that needs to be backed up. (It's a web server so just
    > rsyncing the web docs would suffice, don't you think?)
    >

    Idea: rsync based solution using a device mapper snapshot as its source

    Steps (in general):
    0. Have a box which you would like to mirror (call it box 1).
    1. Build a box (box 2) with nearly identical hardware to box 1 (disc
    capacity, controller, etc.)

    2. Plan your backup strategy, with attention to box 2's
    bootstrap. Box 2 probably needs to be either a dual boot system (with a
    separte disk), or bootable from a live CD. The goal is a platform where
    networking and rsync is available to perform the backup of box 1. An
    independant boot allows the backup to run without interferring with the
    running system. A live CD based solution may be preferable because it is
    less intrusive. With a little planning, both boxes can be used as either
    the primary (live) or secondary (on backup) system.

    3. Install an initial image on box 2. Probably, restore a backup of
    box 1 to box 2 and fix bootloader, etc.

    4. On box 1: schedule a task to create a device mapper snapshot at a
    certain time of day. Mount the "fixed" image (device mapper device) into
    the filespace.

    5. On box 2: schedule a task to update the image on box 2 using rsync,
    using the mount point reference above. Send a signal to box 1 when
    complete.

    6. On box 1: when the "complete" signal is noticed/arrives from box 2,
    then "tear down" the fixed image (umount, dmsetup remove, etc.).

    7. Rinse and repeat.

    If these instructions don't help you, then maybe they'll help me because I
    think this strategy has some potential for simple backups. I am impressed
    with the capabilities of the device mapper facility.

    --
    Douglas Mayne

  5. Re: Ready-to-roll System Backup

    That sounds like a feasible plan. One question though. If I wanted to
    back up one of the system partitions.../usr, for example, couldn't I
    just tar the partition that it's on? For example,

    # mount /dev/sda5 /backup
    # tar -cvjpf sda5.tar.bz2 backup

    Would I be able to do this without booting from a live CD and still
    get an exact duplicate of that partition?

    Thanks!

    Alan

    On Feb 1, 1:15 pm, Douglas Mayne wrote:
    > On Thu, 01 Feb 2007 09:13:29 -0800, AlanM wrote:
    > > Thanks for the response. It really doesn't have to be exact down to
    > > the second. The backup would only be run once each night. Basically, I
    > > just need the backup server to be a duplicate of the main server so
    > > that if the main server goes down, we can just call networking and
    > > have the DNS pointed to the backup until the main server is fixed (or
    > > replaced).

    >
    > > I suppose rsync would do the trick but i was hoping to do an entire
    > > system clone in one fell swoop. I guess one way of doing it might be
    > > to clone the basic system, then do an rsync each night just for all
    > > the data that needs to be backed up. (It's a web server so just
    > > rsyncing the web docs would suffice, don't you think?)

    >
    > Idea: rsync based solution using a device mapper snapshot as its source
    >
    > Steps (in general):
    > 0. Have a box which you would like to mirror (call it box 1).
    > 1. Build a box (box 2) with nearly identical hardware to box 1 (disc
    > capacity, controller, etc.)
    >
    > 2. Plan your backup strategy, with attention to box 2's
    > bootstrap. Box 2 probably needs to be either a dual boot system (with a
    > separte disk), or bootable from a live CD. The goal is a platform where
    > networking and rsync is available to perform the backup of box 1. An
    > independant boot allows the backup to run without interferring with the
    > running system. A live CD based solution may be preferable because it is
    > less intrusive. With a little planning, both boxes can be used as either
    > the primary (live) or secondary (on backup) system.
    >
    > 3. Install an initial image on box 2. Probably, restore a backup of
    > box 1 to box 2 and fix bootloader, etc.
    >
    > 4. On box 1: schedule a task to create a device mapper snapshot at a
    > certain time of day. Mount the "fixed" image (device mapper device) into
    > the filespace.
    >
    > 5. On box 2: schedule a task to update the image on box 2 using rsync,
    > using the mount point reference above. Send a signal to box 1 when
    > complete.
    >
    > 6. On box 1: when the "complete" signal is noticed/arrives from box 2,
    > then "tear down" the fixed image (umount, dmsetup remove, etc.).
    >
    > 7. Rinse and repeat.
    >
    > If these instructions don't help you, then maybe they'll help me because I
    > think this strategy has some potential for simple backups. I am impressed
    > with the capabilities of the device mapper facility.
    >
    > --
    > Douglas Mayne




  6. Re: Ready-to-roll System Backup

    On Mon, 05 Feb 2007 07:09:47 -0800, AlanM wrote:

    >
    > On Feb 1, 1:15 pm, Douglas Mayne wrote:
    >> On Thu, 01 Feb 2007 09:13:29 -0800, AlanM wrote:
    >> > Thanks for the response. It really doesn't have to be exact down to
    >> > the second. The backup would only be run once each night. Basically, I
    >> > just need the backup server to be a duplicate of the main server so
    >> > that if the main server goes down, we can just call networking and
    >> > have the DNS pointed to the backup until the main server is fixed (or
    >> > replaced).

    >>
    >> > I suppose rsync would do the trick but i was hoping to do an entire
    >> > system clone in one fell swoop. I guess one way of doing it might be
    >> > to clone the basic system, then do an rsync each night just for all
    >> > the data that needs to be backed up. (It's a web server so just
    >> > rsyncing the web docs would suffice, don't you think?)

    >>
    >> Idea: rsync based solution using a device mapper snapshot as its source
    >>
    >> Steps (in general):
    >> 0. Have a box which you would like to mirror (call it box 1).
    >> 1. Build a box (box 2) with nearly identical hardware to box 1 (disc
    >> capacity, controller, etc.)
    >>
    >> 2. Plan your backup strategy, with attention to box 2's
    >> bootstrap. Box 2 probably needs to be either a dual boot system (with a
    >> separte disk), or bootable from a live CD. The goal is a platform where
    >> networking and rsync is available to perform the backup of box 1. An
    >> independant boot allows the backup to run without interferring with the
    >> running system. A live CD based solution may be preferable because it is
    >> less intrusive. With a little planning, both boxes can be used as either
    >> the primary (live) or secondary (on backup) system.


    Also, a live CD image can be installed to a hard disk. This eliminates
    the need to boot from a separate disk. The box's boot menu would include
    options for the mode the box would work in, primary (live), or secondary
    (on backup.)

    This post shows how to setup Slax on a HD:
    http://groups.google.com/group/comp....ab4e99488378dd

    >>
    >> 3. Install an initial image on box 2. Probably, restore a backup of box
    >> 1 to box 2 and fix bootloader, etc.
    >>
    >> 4. On box 1: schedule a task to create a device mapper snapshot at a
    >> certain time of day. Mount the "fixed" image (device mapper device)
    >> into the filespace.
    >>
    >> 5. On box 2: schedule a task to update the image on box 2 using rsync,
    >> using the mount point reference above. Send a signal to box 1 when
    >> complete.
    >>
    >> 6. On box 1: when the "complete" signal is noticed/arrives from box 2,
    >> then "tear down" the fixed image (umount, dmsetup remove, etc.).
    >>
    >> 7. Rinse and repeat.
    >>
    >> If these instructions don't help you, then maybe they'll help me
    >> because I think this strategy has some potential for simple backups. I
    >> am impressed with the capabilities of the device mapper facility.
    >>


    > That sounds like a feasible plan. One question though. If I wanted to
    > back up one of the system partitions.../usr, for example, couldn't I
    > just tar the partition that it's on? For example,
    >
    > # mount /dev/sda5 /backup


    What is the context of this mount? The previous posts discussed networking
    between two boxes.

    > # tar -cvjpf sda5.tar.bz2 backup
    >
    > Would I be able to do this without booting from a live CD and still get
    > an exact duplicate of that partition?
    >
    > Thanks!
    >

    Note: Some comments inline.

    Please, do not top post. I should have said that earlier. I have fixed the
    last reply to follow my response.

    Back to the subject. I am a little confused about the context (how will
    you mount your target partition?) in this folloup question, especially
    given the previous discussion. In any case, the key phrase is "exact
    duplicate." If you are attempting to backup the system which is booted
    (i.e, not a live CD), then you have to worry about open files and files
    subject to change during the time it takes to backup. Using a live CD, or
    a dual boot system, are easy ways to achieve a coherent state for the
    backup. Device mapper is a way to "fix" a snapshot. I suppose my
    previous instructions could be adapted to use device mapper on both boxes,
    but this seems somewhat less straight-forward. For one thing, it would be
    clear which mode a given box was working (either as the backup, or as the
    online server.) To change modes, a reboot is required, and the user
    selects the box's mode at reboot.) Again, both boxes could be setup to
    boot in either mode. Other approaches could be used to eliminate the need
    for a reboot, but I discussed the approach I thought most simple. You can
    adapt liberally to meet your specific needs.

    If you have questions at a more fundamental level, I recommend studying
    the hard disk upgrade howto:
    http://tldp.org/HOWTO/Hard-Disk-Upgrade/index.html

    It is definitely worth studying and practicing with that howto.
    Understanding the basics of filesystems and bootloaders at this level will
    definitely help you in your GNU/Linux career.

    --
    Douglas Mayne

  7. Re: Ready-to-roll System Backup

    I have some backup ideas related to using raid arrays. Not sure if
    there is anyway to create this effect with commands such as rsync.
    Maybe.

    After having a motherboard fail recently with a 2 month repair/
    warranty cycle I decided that this was no way to run a multimedia
    business. With multimedia linux work I find that getting my system
    installed, configured and tweaked is very time-consuming. And once
    the system is running well I become timid of playing with the
    installation and running the risk of corrupting it (managed to corrupt
    my FC4_64 multimedia setup Glibs trying to build ardour2 recently).

    All this means that having a backup of a home partition is far from
    adequate. The root partition and the hardware/software interactions
    are critical too. So I need clone systems. Here is my plan ; I
    haven't fully implemented this yet although I have built my cloned
    systems and worked most of the neccessary commands out ; My strategy
    doesn't involve networking. Feedback welcome.

    1. 2 machines with identical hardware - thus a supply of swaps for
    all parts. The second machine to be used by my wife unless needed in
    my backup strategy.

    2. One extra backup drive - a hotplugged sata backup that is actually
    foms the redundant half of a raid 1 array. You plug it in and let it
    sync up with the on-board disk then unplug and put it on the bookshelf
    safe away from lightning strikes and surges. While it is unplugged
    you are running the computer as a degraded raid-1 array.

    The advantage of the raid syncing process is that the drives will
    slowly come into sync but you still have processing power to keep
    working during the 'backup'. Providing you wait until the background
    raid re-syncing catches up you will have a bootable snapshot of the
    entire system at the point you decided to unplug. Very useful. Just
    plug it into your second hardware setup and go. You don't even have
    to trouble-shoot a hardware failure until you have time. Also if you
    corrupt your installation somehow and want to revert to the last
    system-snapshot you just sync the array in reverse (ie boot off the
    backup and then hotplug and sync the onboard disk.).

    I would really like this all to work without any manual intervention
    but don't quite have the knowledge. Also hotplugging is very new in
    the linux kernel. The idea would be just hotplug the drives in and
    out of the raid 1 array without needing to issue commands - the newly
    plugged drive would always sync to the other drive.

    Graham E.


  8. Re: Ready-to-roll System Backup

    "gray" wrote in news:1170857167.213319.88720
    @a75g2000cwd.googlegroups.com:

    >
    > 1. 2 machines with identical hardware - thus a supply of swaps for
    > all parts. The second machine to be used by my wife unless needed in
    > my backup strategy.
    >
    > 2. One extra backup drive - a hotplugged sata backup that is actually
    > foms the redundant half of a raid 1 array. You plug it in and let it
    > sync up with the on-board disk then unplug and put it on the bookshelf
    > safe away from lightning strikes and surges. While it is unplugged
    > you are running the computer as a degraded raid-1 array.
    >
    >


    I've just been playing with something similar on a Proliant box so taht
    I have a backup of the system when I do experimentation and something
    breaks. I want to be able to just boot from the image on another drive.

    The problem I'm having is that if I do a binary copy (using 'dd') then
    the system sometimes finds and mounts the partition of the backup drive
    instead of the boot drive.

    I can't solve this by constantly removing and replacing the backup hard
    drive as this entails shutting down before and after each backup.
    (Doing otherwise might break the backup somehow and replacing the drive
    seems to do the same.) And having to do so would be som cumbersome that
    I will end up not doing it regularly.

    This is a problem on a somewhat older SCSI RAID setup, but it might be
    the same problem you'll experience with SATA hotplug setup.

    You'd better test it out first.

    Boaz

  9. Re: Ready-to-roll System Backup

    In message <1170857167.213319.88720@a75g2000cwd.googlegroups.c om>
    "gray" wrote:

    > I have some backup ideas related to using raid arrays. Not sure if
    > there is anyway to create this effect with commands such as rsync.
    > Maybe.
    >


    > 2. One extra backup drive - a hotplugged sata backup that is actually
    > foms the redundant half of a raid 1 array. You plug it in and let it
    > sync up with the on-board disk then unplug and put it on the bookshelf
    > safe away from lightning strikes and surges. While it is unplugged
    > you are running the computer as a degraded raid-1 array.
    >
    > The advantage of the raid syncing process is that the drives will
    > slowly come into sync but you still have processing power to keep
    > working during the 'backup'. Providing you wait until the background
    > raid re-syncing catches up you will have a bootable snapshot of the
    > entire system at the point you decided to unplug. Very useful. Just


    Have you considered / is it possible to / have a 3-disk raid-1 set,
    and remove the member which you store as backup, leaving a fully
    redundant running system?

    --
    Alan Adams, from Northamptonshire
    alan.adams@orchard-way.freeserve.co.uk
    http://www.nckc.org.uk/

  10. Re: Ready-to-roll System Backup

    > The problem I'm having is that if I do a binary copy (using 'dd') then
    > the system sometimes finds and mounts the partition of the backup drive
    > instead of the boot drive.


    you bios should allow you to select which device to boot off but - if
    this is a problem there should be another solution...
    >
    > I can't solve this by constantly removing and replacing the backup hard
    > drive as this entails shutting down before and after each backup.
    > (Doing otherwise might break the backup somehow and replacing the drive
    > seems to do the same.) And having to do so would be som cumbersome that
    > I will end up not doing it regularly.


    If the system insists on booting off your backup drive it doesn't
    really matter - as long as the fstab entries point to the right drive
    then the right partitions will be mounted. Your problem is probably
    those new fangled volume labels like ROOT, HOME etc in your fstab
    device line instead of the old fashioned /dev/hda2, /dev/hda4 and the
    like. I have already run into problems with this and reverted the
    fstab file to the old fashioned device pointers.

    interested in hearing any other details of your system. I'm also
    interested in knowing if rsync is really capable of creating this sort
    of functionality over a network and if so how to implement it.

    Graham
    >
    > This is a problem on a somewhat older SCSI RAID setup, but it might be
    > the same problem you'll experience with SATA hotplug setup.
    >
    > You'd better test it out first.

    heh heh. Just one more 400MB drive and I'm ready to blast off...

    >
    > Boaz




  11. Re: Ready-to-roll System Backup

    >Have you considered / is it possible to / have a 3-disk raid-1 set,
    >and remove the member which you store as backup, leaving a fully
    >redundant running system?


    Having a backup drive on the shelf and connecting it into the array
    once a day is enough redundancy for me. I don't regularly have disks
    go down and when they do it would be no trouble to restore the array
    from the backup with a new drive. The only advantage I can see with
    your set up is protection against a simultaneous failure of the
    unplugged disk and the plugged in disk. Unlikely. And that always-on
    redundancy will cost cpu cycles (we are talking about software RAID
    here).

    Graham


+ Reply to Thread