Backup tape reuse strategies?

Hi all,

We have a centralized backup system of several dozen machines. My boss
is cracking down on tape costs, so we are trying to come up with a
strategy to recycle old backup tapes. The problem is that it's not
easy to do it because the backup software (Time Navigator) writes
files from different machines and from different dates onto the same
tape. Thus, if I want to keep a full backup only of a certain subset
of machines from a year ago, it's very time-consuming to track down
the tapes involved.

Obviously, the software allows to backup each host to a different set
of tapes, but managing all this becomes hard very quickly, and it
would waste tapes anyway. I can also set cut-off dates where I close
all tapes and start anew, say, each month, but this also adds to the
management time and tape wastage.

Has anyone run into this problem (ease-of-use vs precise
accountability) with backup tapes, especially regarding tape reuse?
This would probably arise with any kind of centralized backup
solution...

Thanks,
Simon Ilyushchenko

Raven and Peter Håkanson,

Thank you for your responses.
[color=blue]
> Can I ask if he has a valid point or is it just a case of penny pinching?
> How much value do you place on the tapes if you actually need to use them
> for a restore?[/color]

I never thought I'd regret I am not in corporate world. :) Here is a
bigger picture: we are a research institution. We *are* on a tight
budget, and doing monthly fulls for the available disk space without
recycling tapes would run into six figures (USD). Backups are
important mostly for disaster recovery and short-term data loss - I
have had virtually no requests for files over half a year old.
However, we have had cases in the past when years-old data had to be
dug up regarding some patent issues. On top of all, our attempts to
ask the backup policy on a higher level have not had any reply yet - I
guess nobody wants to be responsible for formulating it. (Looks like
this is the first thing to pursue - there are still even higher people
to consult.)
[color=blue]
> Could it be time to reconsider your whole strategy including the choice of
> backup software?[/color]

I am not averse to the idea - one of the reasons for the post is to
find out what other people are doing in similar situations.
[color=blue]
> How many tape units and locations? By the sounds of it you need to run
> separate jobs for each machine so as to keep groups of tapes for each.[/color]

Two locations, two tape units in each, for Windows and Unix servers
separately.
Close to a hundred servers in total with about 10 Tb of disks storage
(and growing), though, thankfully, it's only about 50% utilized so
far.

The problem with separate jobs is increased maintenance time. After
all, we are not only minimizing the tape cost, but the man-hours
spent, which can be more productively used elsewhere.
[color=blue]
> Agreed and agreed. Perhaps it's worth re-evaluating whether the cost of
> tapes under the current system is really that excessive.[/color]

From what I understand, the budgets are *really* tight. :)
[color=blue]
> How many tapes are we talking about in total? Surely the value of the tape
> is irrelevant if the backups are so critical to the operation?[/color]

Judging by the frequency of the requests, long-term backups are not
very critical, but as I mentioned, in the absence of the policy you
never know.
[color=blue]
> As far as accountability, if they really are tracking the tape costs
> themselves simply serial number each tape and do a semi-regular audit. Tapes
> that go bad get officially written off, tapes that go missing get
> questioned.[/color]

I did not mean just keeping track of tapes themselves, but rather of
what exactly is on each tape. And with centralized backup software,
it's non-trivial.

Simon Ilyushchenko

BTW, I forgot to ask: is anyone checking whether a tape is good before
using it? I mean the whole tape, not just a small part in the
beginning. My backup software does not do it.

Thanks,
Simon

In comp.unix.admin Simon <simonf@simonf.com> wrote:[color=blue]
> Raven and Peter Håkanson,[/color]
[color=blue]
> Thank you for your responses.[/color]

Is the tapes fully utilized ? If not ( and the contents are not compressed
on non-compressing drives) lots of space might be wasted.

You might also gain significant amount if you can separate things that
can be recovered from distributions ( OS and tools ) from the actual
data. This might include forcing data to "fileservers-only" where
regular backups are done. Creating schedules with full backups
intermixed with "levels" ( aka hanoi-tower) might spare a number
of tapes. ( I have had great success using freely avaliable amanda
in a network of machines). The perfect case is where you can say :
"We don't backup workstations, they are recreated via fresh install".

Now a good policy is essential to define the "acceptable service", without
that you are always on the loosing side whenever data is lost. If
management avoids this you could create one yourself and state "We will
use this policy until something else is decided".

This policy might stipulate that "backup is for disaster purposes and
limited to (xx ) months. Any archiving has to be done via other purposes.
( remember that backup and archiving is two totally different things,
using different tools and with differing goals)

--
Peter Håkanson
IPSec Sverige ( At Gothenburg Riverside )
Sorry about my e-mail address, but i'm trying to keep spam out,
remove "icke-reklam" if you feel for mailing me. Thanx.

On 29 Jul 2003 09:59:11 -0700, Simon wrote:[color=blue]
>
> I never thought I'd regret I am not in corporate world. :) Here is a
> bigger picture: we are a research institution. We *are* on a tight
> budget, and doing monthly fulls for the available disk space without
> recycling tapes would run into six figures (USD). Backups are
> important mostly for disaster recovery and short-term data loss - I
> have had virtually no requests for files over half a year old.[/color]

Yes, IRS auditors were looking into a million $ plus project
which we wrote off as development research several years ago.
I think their computer brought it up because time was about to expire
and they would not be able to look into it. :(

Our lawyer calls me up and asks for project plans, requirement/design
docs, memos, source. Glad my operations group have yearly backups
going way back.

What is even better, when the media changed from round reel to
cartridge, they copied all the data onto the new media.

We would had to have paid a bit of a penalty without the items.
Penalty would have wiped all those *savings*.

Penny wise can be pound foolish.

Hey site the risk, the example. Ask them to check with the lawyer
if Management is libel for the policy. I find when I put management
in harm's way, we can find the better solution. :)

In comp.unix.admin Bit Twister <BitTwister@localhost.localdomain> wrote:
[color=blue]
> Our lawyer calls me up and asks for project plans, requirement/design
> docs, memos, source. Glad my operations group have yearly backups
> going way back.
> What is even better, when the media changed from round reel to
> cartridge, they copied all the data onto the new media.[/color]

Similar situation: I used to work for $VERY_BIG_COMPANY who was being
sued for a (bogus) patent infringement claim. By going back to backup
tapes, again, 2 generations older than current technology, I was able to
retrieve the weekly "report to my boss what I was working on this week"
file from the key developer who had been on the project. Using these files,
we were able to show what, when, and by whom, the work had been done.

If we had reused those tapes, it would have been a finger-pointing contest,
at which point a jury could have very well decided to give the advantage
to the "little guy" fighting the very very big company. We kept all of our
full backups (done weekly) offsite, and pay an obscene amount of money to
store them (and for the tapes that are effectively "lost"), but that one
lawsuit could have more than wiped out any savings that we would have seen
by reusing those tapes.

If nothing else, decide on a policy...full backups have *x* months of
retention ('forever' might be the right answer), but incrementals have
just a few months before they're reused. Maybe you only want to keep one
set of full backups per month, but keep those forever; it depends on
how much risk you're willing to take, compared to the cost of tapes.

One other thing - if your software can be configured so that full and
incrementals get written to different tapes, do it. That way, only full
backups will be on the tapes you need to keep around.
[color=blue]
> Penny wise can be pound foolish.[/color]

Absolutely correct. Tape is cheaper than the work needed to generate the
data on it, almost always.

Dave Hinz

On Mon, 28 Jul 2003 19:21:51 -0500, "Kermit T Tensmeyer"
<kermit.tensmeyer@ti.com> wrote:

[ Snip ]
[color=blue]
> Most tape systems have a hard time figuring out that a tape is full, so it
>will use tape errors to identify that it can't write any more to the tape.
> The downside to this is that the system has a hard
> time detecting that the tape is worn out.[/color]

As a general rule, in the tape drive business we call tape systems
that behave as you describe "broken".

Reaching End-of-Tape is a specific non-error exception; unfortunately,
a lot of *software* fails to distinguish the case where a write
successfully wrote only part of the requested data because of an EOT
was reached (partial completion, no error), and the case where the
same thing happened because of an unrecoverable write error (partial
completion, write error).

[ Snip ]

Malc.

In comp.unix.admin Nicholas Bachmann <usenet@not-real.org> wrote:[color=blue]
> [email]davehinz@spamcop.net[/email] wrote:[color=green]
>>
>> Similar situation: I used to work for $VERY_BIG_COMPANY who was being
>> sued for a (bogus) patent infringement claim. By going back to backup[/color][/color]
[color=blue]
> Let me guess: some company nobody's ever heard of whose sole business is
> owning patents and filing lawsuits against big companies (like GE, IBM,
> or Microsoft) who will most likely settle quietly and finance the
> purchase of more stupid patents and more bogus lawsuits. Am I right?[/color]

I think you know I worked for GE, right? So, you're kind of right...
it was a case of "David vs. Goliath". To be honest, I'm not sure how
(or if) it finally was settled.

It was an MRI scanning technology that, yes, there was a patent that
vaguely described something like what GE was also doing, and the little
company claimed "We thought of it first, see, here." It was something
like "Instead of taking scans along just the X, Y, or Z axis, let's take
them at an angle" which, to my non-legal mind, falls into the category
of "blisteringly ****ing obvious application of the technology" - kind of
like patenting using the side of a crayon for coloring instead of just
the tip.

It did have one relevant (to this thread) effect, though...my backup
tape budget went _way_ up, and I was able to switch from 8mm
(boo, hiss) to DLT-7000 drives. About the same cost per gig for tape,
but much cheaper to pay for storage (they charge by the tub, not by
the byte).

It all depends on the priorities. If the OP's needs are more of a backup/
restore than an archiving, then that changes the answers somewhat. If the
decision comes down to "Do we pay for archive tapes, or continue research",
well, that seems like a pretty straightforward question to answer...making
the organization become history just to preserve the history would be silly.

Dave Hinz

First of all Thank God I don't have that kind of problem.

The problem you have is a little strange in that way that the cost of
tape compared to the cost of the time you spend on minimizing the tape
cost, it would be more easy, secure and cost less just to buy the
tapes you need.

In our small shop we use NetBackup Datacenter as backupsoftware (good
software, ok support, you can trust it and it has the features most
people need.
On the hardware side we use a StorageTek L-series library and
StorageTek 9940B tapedrives.

Basicly our tapemanagement is based on the standard features in
NetBackup (no fancy stuff at all)
In the different policies we configure how long to keep a given
backup, and when there is no longer a valid backup on the tape
NetBackup uses it again.
The only real planning on minimizing the numbers of tape we use are on
the number of days we keep the backups. I found out that we used less
tapes on a daily basis when we used the same period for all backups.
In the beginning it seemed a little strange to me, but basicly it's
because we didn't fill all the tapes.

We run around 250 backups a day (could be more since we don't count
them) and the total amount written to tape is in the area of 1 TB a
day.

Compared to what the backupsoftware, backuphardware and the people to
run the backups cost, tapecost is a minor issue (I think we pay less
then EUR 100 a tape)

If I wanted to keep some of my backups for a longer period I would
duplicate them to a seperate volume pool (standard feature in NBU but
I don't know about you software) or use vaulting and let NetBackup do
it automaticly)

It seems to me that you want to do some realy fancy backup stuff with
a low cost software and hardware installation. And the only way to get
that to work is to use a lot of manpower for it.

/johnny

On 28 Jul 2003 14:43:18 -0700, [email]simonf@simonf.com[/email] (Simon) wrote:
[color=blue]
>Hi all,
>
>We have a centralized backup system of several dozen machines. My boss
>is cracking down on tape costs, so we are trying to come up with a
>strategy to recycle old backup tapes. The problem is that it's not
>easy to do it because the backup software (Time Navigator) writes
>files from different machines and from different dates onto the same
>tape. Thus, if I want to keep a full backup only of a certain subset
>of machines from a year ago, it's very time-consuming to track down
>the tapes involved.
>
>Obviously, the software allows to backup each host to a different set
>of tapes, but managing all this becomes hard very quickly, and it
>would waste tapes anyway. I can also set cut-off dates where I close
>all tapes and start anew, say, each month, but this also adds to the
>management time and tape wastage.
>
>Has anyone run into this problem (ease-of-use vs precise
>accountability) with backup tapes, especially regarding tape reuse?
>This would probably arise with any kind of centralized backup
>solution...
>
>Thanks,
>Simon Ilyushchenko[/color]