Data pre-load/seed from an existing backup

[Petek]

Hi. Wonder if you can help.

 

We are utilising Retro Multi to backup remote sites to replace another application. We have the ability to restore the remote sites' backup locally or to another machine within the site the Retro server is located and need to pre-seed the backup for the remote site, as in one case its 250gig and the bandwidth between sites is far, far too low to do this from scratch across the wan.

 

I've tried to do this by restoring to a local folder (e.g. "C") , and pointing the backup job at this, then changing the backup source to the be the client on the remote server, but Retro does not match any of the files that are already stored. The times and dates were maintained from the previous restore, so every single one cannot have changed, therefore I would expect at least a good percentage to have matched, and only needed a catchup backup, plus system state etc.

 

I have "match source volumes to catalog file" and also "dont add duplicates to backup set" ticked in the client options.

 

Can anyone help please or offer any advise to get Retro to really match the files in the seed backup set to the live client?

 

Matching works fine for other backup sets created from scratch within Retro and the snapshot process works too.

 

TIA

 

[Ramon88]

I'm afraid you explanation is a bit confusing (and thus you won't get many replies).

 

Did you look at the "Duplication" feature of Retrospect? That might work better for you as it's much more straightforward when 'syncing' directories.

 

However Retrospect isn't really the best tool for what you want to do. You might need something like TortoiseSVN, which is more tailored to your specific needs: http://tortoisesvn.net/

[rhwalker]

Can anyone help please or offer any advise to get Retro to really match the files in the seed backup set to the live client?

The sources aren't the same.

 

Here's my suggestion. Hint:

 

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.

Tanenbaum, Computer Networks, p. 91 (4th ed.)

 

Temporarily set up Retrospect at the remote location, make a backup, carry the backup (tape or disk) back to the home location, do incrementals to that backup set from that point forward using the main Retrospect, using the remote client as the source, to that transported backup set.

 

Russ

[Petek]

Sorry it was un-clear. Previous backups at headoffice were made with a different system. We want to be able to restore these locally, backup into Retrospect and then point Retrospect at the remote server to perform a "catch up" backup in the normal fashion.

[Petek]

While fully appreciate what you say and understand it isn't feasible to send out 131 remote Retrospect servers to perform a local backup and then physically ship them and transfer the snapshots onto the master server, especially when we already have all the data in the same building as the master server and can restore it to a disk / other server etc for the pre-load.

 

Any other thoughts please?!

[rhwalker]

I think you misunderstand what I suggested.

 

The only way that I can suggest to have Retrospect do the incremental backups over your low-bandwidth channel using remote clients as the source is to have Retrospect make those backups to a backup set that was initially created using those clients.

 

Because you indicate that this initial backup cannot be made over your low-bandwidth channel, my suggestion instead was to temporarily install Retrospect at the remote location, perhaps with a trial license, and create the initial backup onto a disk or tape there, then bring that disk or tape backup set back to your main Retrospect location and update that initial backup set with incrementals through your slow channel.

 

There is nothing in that suggestion about "sending out 131 remote Retrospect servers" and then "physically ship them".

 

Sorry you misunderstood.

 

At your main location you only have the data, not the sources. Retrospect knows the source of the data, and is unable to make that match.

 

Perhaps someone else can suggest a different workaround with the current product. If none is possible, and if you don't see how my suggestion can be used, then you might make a suggestion for a feature addition to accommodate your special needs.

 

I was just trying to provide a suggestion, sorry it was misunderstood.

 

Regards,

 

Russ

[Petek]

Russ, I wasn't trying to be facesious in my statement, it was a statement of fact!

 

We presently have 131 remote sites, all of which we have local full backups for (courier and tape per site per week) and hence need the ability to restore these locally, and pre-seed the Retro server with then, then do the catch up incrementally.

 

I was enquiring if anyone new of a method, otherwise Retrospect just isn't the right product - pretty much every ISP type remote backu product has a seed/catch up mechanism and I presumed that (forgetting Exchange and system state and suchlike) that providing Retro with a local file source then changing the source would still perform the file matching.

 

I am trying the process again without NTFS permissions this time to see if there is any chance of it matching the fileset..

 

Cheers, Pete

[Ramon88]

So... is this correct?:

 

1) You have the data already at your server park. But it's in another backup solution.

 

2) You want to restore data from that other backup solution to a folder at your server park.

 

3) You want to use (from the point of view of the server park) a remote Retrospect application (so it only has a slow connection) to do -something- ...

 

That -something- isn't entirely clear to me... Can you explain that part? And did I have points 1 through 3 correct?

[Petek]

So... is this correct?:

 

1) You have the data already at your server park. But it's in another backup solution.

Yes

2) You want to restore data from that other backup solution to a folder at your server park.

Yes

3) You want to use (from the point of view of the server park) a remote Retrospect application (so it only has a slow connection) to do -something- ...

No, not quite. The data from the remote site (with the slow connection) is on tape at head office (i.e. 1) and to be restored locally (2) then once this is complete we want to point Retrospect at a client agent at the remote site to perform daily incremental/snapshots.

 

That -something- isn't entirely clear to me... Can you explain that part? And did I have points 1 through 3 correct?

 

Does that make it clear?

 

Thanks

 

[Ramon88]

And where in this equation is Retrospect situated? Is it separated from the source and destination by a slow connection?

[Petek]

No, Retrospect is at the "server park" along with all the backup tapes from the servers at the low bandwidth sites.

[Ramon88]

So, so do I understand correctly:

 

- Your source files are located at the remote server park.

- Your destination targets are located at the remote server park.

- Your Retrospect server is located at the remote server park.

 

So from the viewpoint of the Retrospect server, there are no slow connections between it and the source and destination?

[Petek]

So, so do I understand correctly:

No, I'm afraid not.

 

Retrospect Server plus backup tapes (from remote sites) are located at "head office" or "server park"

 

Remote sites have servers with slow connections.

 

[Ramon88]

For Retrospect to do a 'match' in a regular backup it needs to be able to read the -whole- source file and compare it with it's back-upped counterpart. So with a slow connection that takes (too much) time.

 

The only option you might want to try is to use Retrospects "Duplication" mode. That mode comes very close to "synchronising" two directories. But beware, if your source is erased, it will erase the target as well...

 

We have a similar situation, but have a Retrospect server at both our development site and our remote data centre. We also have a 100 Mbit full duplex leased line between both sites. On top of that we have an offsite backup site for storage, connected with 1 Gbit full duplex to our development site. Our web- application- and database servers at the remote data centre also have a Gigabit LAN to connect them together.

 

To make it work out for you, you need to have a Retrospect server at the data centre as well. And have that connected to your clients with a dedicated speedy LAN. Combined with offsite tape services of course.

 

I don't see this work in any other way, considering your low connection speed handicap at essential points.

[Petek]

For Retrospect to do a 'match' in a regular backup it needs to be able to read the -whole- source file and compare it with it's back-upped counterpart. So with a slow connection that takes (too much) time.

 

I will take a look at the Duplication options. However what you state above is not true.

 

We have a number of these backups that are running upstream over ADSL lines (at 400k and 800k) which are happily matching and snapshot backing up each night already. Ok, none are as big as the ones I'm now trying (perhaps 10gig or in one case 60gig) but these were completed with Retro on a laptop taken to the remote site, backed up locally and then the snapshots transferred back at headoffice. They happily match and backup changes including MS Exchange and SQL every night.

 

I still can't quite believe Retro will not match the fileset if provided to the server from a differing location (i.e. locally) and then swung over to hit a remote client install - that seems madness.

 

[Ramon88]

In my experience Retrospect always needs to read the whole file ONCE before it knows and can store away its checksum. If you do incremental backups, the first one takes a lot of time. The next is a lot quicker. Retrospect doesn't need to read all the files in that case.

 

I'm no sure, but when you move the files around, that can have a big influence on the way the original checksum is generated. Something like that is probably going on, and to Retrospect that means seeing different files.

[Petek]

Agreed. But we are maintaining the file/access times and dates, the archive bits and the security, so I aren't sure which bit its not matching on.

 

To be honest for what we are doing archive flag would probably do but there isn't an option to drop all the intelligent matching and just do the dopey stuff!

[Ramon88]

Retrospect sometimes moves in mysterious ways.

 

I have a feeling "Duplication" might just do the trick for you, as Retrospect doesn't use any of its own matching technology, but size/time stamps etc. At least as far as I know that is...

 

Be sure to test this really well, because it will be a irreversible process.

[rhwalker]

we are maintaining the file/access times and dates, the archive bits and the security, so I aren't sure which bit its not matching on.

Pete, I'm not trying to be difficult here, but perhaps I can give some insight that might help you solve your problem and to better understand why I made the suggestion that I did, even though you didn't find it helpful.

 

Retrospect is very particular when it does its matching, and is this careful so as to ensure that the "right" data is backed up again if needed. I'm not trying to defend Retrospect's algorithm, just to state the way it is.

 

Issues can develop (and have been seen) when the filesystem doesn't support exactly the same metadata model that Retrospect expects. This is why it is difficult to get "duplication" comparison of metadata to work when the source and destination have different filesystem models. For example, the source and destination filesystems might differ in the timestamp granularity or permission model supported. Again, I'm not trying to defend Retrospect's algorithm, just to state the way it is.

 

You may think that the timestamps are identical (by viewing a human-readable listing of the files) but, to Retrospect, looking at the filesystem data structures returned by syscalls, they might not be exactly the same.

 

I suspect that the problem is occurring because the locally restored files, coming via the foreign backup program, might not have exactly the same metadata that Retrospect's client sees when viewing the remote files. Retrospect is comparing its local backup of the restored files from the foreign backup program with what its client sees on the remote machine.

 

Retrospect is able to handle differences in the metadata model between the source and destinations when it does its own backup of the source, because all of that information goes into the Retrospect backup set "container" (database) rather than relying on the destination filesystem to preserve the metadata. I just point this out because a "duplicate" operation relies on the destination filesystem to preserve the metadata (as also happens with your local restore from the foreign backup program) rather than Retrospect's own backup set "container", if it had made the initial full backup of the client.

 

You have several opportunities for metadata mismatch to be introduced in the scenario you propose.

 

The first is whether the foreign backup system truly preserves all metadata from the remote client in the backup, and whether it truly restores that metadata when you do the local restore at your server site.

 

The second is whether the local restore destination's metadata model is the same as that seen at the remote client by the Retrospect client program.

 

The third is a bit more subtle, and is somewhat related to the second. Depending on the user authentication scope, you may have different user and group IDs (to use the Unix term; unsure what filesystem model you have) at the local and remote sites, and ownership credentials may appear different to a program than a human-readable listing shows.

 

I just toss these out as avenues for you to investigate as you experiment with Retrospect's matching options.

 

Regards,

 

Russ

[Ramon88]

Excellent explanation Russ! :thumbup:

 

We use duplication with Retrospect (for some tasks - I think it is rather dangerous to use as a 'backup method'), but in that case we transfer SQL database backups (generated by SQL itself) to an offsite location. There it gets a regular incremental backup treatment by another Retrospect machine. This has worked well for us.

 

But in Pete's scenario there is one more factor involved and that is that other backup program.

 

But does Pete have any other options using Duplication with Retrospect? That's why I suggested an SVN system instead, though that is involving to set up I guess. Still duplication might be the ticket to success, though as Russ has explained, might not be 100% reliable in Pete's case.