LTO1/LTO3 transfer to LTO4 tapes

[GMRMacBackup]

I've encountered an issue with a recent transfer of backup sets from an LTO3 library to a new LTO4 library. The first transfer appeared to work well, the logs indicated it moved nearly 6TB from a mix of 13 LTO1/LTO3 tapes to 7 LTO4 tapes. When performing a restore test I can see all of the Snapshots copied from the originals but trying to restore from the copies result in an error stating "No Snapshot found for session". I have performed a variety of transfers (with snapshots, without snapshots) and subsequent rebuilds, repairs and recatalogs and have not been able to recover anything from the copies.

 

Steps performed/[color:blue]results returned[/color]:

• Open 'Reports>Contents', select transferred backup set, select one of the snapshots in the list, click 'Browse'. [color:blue]Browser window opens and I am able to browse all snapshots.[/color]
  
• Open 'Immediate>Restore', select 'Restore files from backup', select transferred backup set. [color:blue]'Current Snapshots' window indicates 'Archive Local does not appear to contain Snapshots'.[/color]
  
• Select 'Add Snapshot...' button. Snapshot Retrieval window lists all transferred snapshots. Select desired snapshot and select 'Retrieve' button. [color:blue]'Retrieving Snapshots...' window appears, after several minutes a new window is displayed indicating 'No Snapshot found for session Archive, 11/7/2008 9:30 PM.'[/color]
  
• Open 'Tools>Repair', select 'Rebuild from tapes'. Window 'Media Selection' appears, manually move first transferred tape to LTO4 drive, click 'OK' button when ready. [color:blue]System progresses through each tape, recatalog operation takes 123 hours(!) and indicates the backup set contains 26.3TB of data. Subsequent restore operations cannot use backup set contents as there are dozens of references to the same files. [/color]
  
• View 'Operations Log'. [color:blue]Find hundreds of thousands of 'Backup set format inconsistency (4 at 2165502080)' errors.[/color]
  

 

There are 68 sessions on the original tapes, the only practical fix to this I can think of is to restore each session back to disk and then back it up to the new library. Unfortunately I do not have an unlimited time frame, the original library will be shipped off soon on a lease return so I will not be able to use the LTO1 tapes in the LTO4 library. Any assistance or suggestions would be gratefully accepted.

 

System specifics:

• Intel Xserve (10.5.6 Server, 2 x 2Ghz, Apple Raid card, 3 x 750GB Raid 5, 10GB RAM)
  
• Retrospect Backup version 6.1.230 (driver 6.1.15.101)
  
• Apple 4Ghz 4 Port FC Card (driver 2.02)
  
• Promise 610f 9TB (Raid 5, 4GB FC controller, connected to Apple FC card)
  
• HP StorageWorks 4048 LTO4 dual drive w/4GB FC controller (connected to Apple FC card)
  

 

[twickland]

Your description is admirably detailed, except for the absence of information about how you performed the transfer and where your backup set catalogs are stored.

 

Please confirm that you created a brand-new tape backup set for you LTO4 tapes, and that you used Tools> Copy> Transfer to transfer your data from the old backup set to the new.

 

Are the catalogs for both backup sets stored on the same volume? Is this the RAID volume or some other volume?

 

What happens if you try to rebuild the catalog from your original tapes? Are you able to do so? (There's probably no need to rebuild everything; just one or two members' worth.) Also, don't trash your original catalog; just move it somewhere safe and forget it in Retrospect before attempting the partial rebuild.

 

Are you able to create a brand-new LTO4 tape backup set, write some test data to it, and then successfully restore data from it?

[GMRMacBackup]

Thank you for the quick response.

 

I performed two different transfers, one with Snapshot copying and another without, the following steps were identical between both transfers...

• Prepped system by connected both libraries via fibre channel to independant controllers.
  
• Loaded original backup tapes in LTO3 library
  
• Loaded 10 fresh, blank LTO4 tapes in the new LTO4 library
  
• Confirmed access to both independent libraries with
  
• Selected 'Tools>Copy'.
  
• On next untitled window, selected 'Transfer' function.
  
• On window 'Backup Set Selection' window, selected original backup set 'Archive Local', clicked 'OK'.
  
• On second 'Backup Set Selection' window, selected 'New...' button.
  
• On 'Backup Set Creation' window, selected 'Tape', did not change Security options, left hardware data compression activated, provided a new name of 'LTO3Archive Local'. Clicked the 'New...' button.
  
• Saved the catalog to same location as existing catalog. Since catalogs were named differently I felt this was a safe option.
  
• Clicked 'OK' on the 'Backup Set Selection' window to start the transfer.
  
• On the 'Backup Set Transfer: Searching' window I left 'All Files' selected. Clicked 'OK'
  
• On next 'Backup Set Transfer' window, verified Source and Destination, Searching 'All Files', Files Chosen '567876 Files 6.0T'

 

Here is where I tried two different transfers, on the 'Options' button of the 'Backup Set Transfer' window I left the original setting of 'Copy Snapshots' for the first transfer. After it transferred the 6TB (approximately 47 hrs if I remember correctly) I tried a restore from one of the snapshots. The list of contents from the Snapshots appeared identical to the original, but when a restore was attempted it will search the new LTO4 tapes but report back that no Snapshots exist. The second transfer I disabled the 'Copy Snapshots' function, intending to build a fresh set of snapshots from a catalog rebuild from tapes. Unfortunately this is where the numerous errors ('Backup set format inconsistency (4 at 2165502080)' happened and I stopped the catalog rebuild. Many times I saw 'Resynchronizing slow' messages on the status screen but after 127 hours I felt it was not going to happen (i'm a very patient person). After reviewing the resulting snapshots it indicated 27.8T was contained on 13 LTO3 tapes and the file count was way higher than the 568K files listed in the original.

 

To answer your question about using the LTO4 successfully to restore data, the answer is 'Yes'. This problem appears to be limited to the transferred backup set, my weekend and weekday backups have both been used to restore from. Both the original catalog and new catalogs are being stored to an internal RAID 5 drive, this drive has been checked several times for reliability and consistency.

 

Your suggestion of rebuilding the original catalog sounds like a fresh perspective, I have never had a problem with the catalog but that doesn't mean that it is pristine. I will attempt to do that within the limited timeframe that I have left on this old library. Performing a Repair on the copied tapes did not produce usable snapshots.

 

 

[twickland]

I performed two different transfers, one with Snapshot copying and another without, the following steps were identical between both transfers...

OK, with the steps you list, everything should have gone fine.

 

I tried a restore from one of the snapshots. The list of contents from the Snapshots appeared identical to the original, but when a restore was attempted it will search the new LTO4 tapes but report back that no Snapshots exist.

So, something is wrong with the data on the LTO4 tape such that Retrospect is unable to recognize its snapshots.

 

The second transfer I disabled the 'Copy Snapshots' function, intending to build a fresh set of snapshots from a catalog rebuild from tapes.

Reconstructing snapshots from the backup data is something you cannot do. The snapshots are additional pieces of data about the status of the source volumes at the time of backup, and will not typically have a close connection to the files that were backed up in a particular backup session.

 

Unfortunately this is where the numerous errors ('Backup set format inconsistency (4 at 2165502080)' happened and I stopped the catalog rebuild. Many times I saw 'Resynchronizing slow' messages...

Ah, the dreaded "Resynchronizing (slow)" messages. In my experience, these either mean problems with the hardware, or corruption of a tape member.

 

To answer your question about using the LTO4 successfully to restore data, the answer is 'Yes'.

Good. That would seem to leave the new drive and its interface off the hook.

 

Both the original catalog and new catalogs are being stored to an internal RAID 5 drive, this drive has been checked several times for reliability and consistency.

Is this volume different from the Promise RAID you list above? If so, how is this drive connected to your XServe?

 

Your suggestion of rebuilding the original catalog sounds like a fresh perspective, I have never had a problem with the catalog but that doesn't mean that it is pristine.

I wasn't so much thinking of the condition of your catalog as of the tape members themselves. You can often restore data from a damaged tape if the catalog is good, provided that the file you're seeking does not reside on a damaged portion. However, transferring all the data, or rebuilding a catalog from the media, will require traversing any damaged areas on your tapes, which could cause trouble. This test is meant to eliminate the possibility of damaged tapes in your original backup set.

[GMRMacBackup]

After attempting a rebuild of the original tapes it appears they have their own problems. The rebuild took 72 hours and reported back 5,395 errors, unfortunately it was the type of 'Backup set format inconsistency (4 at 1939884352)' errors that did not indicate what snapshot was involved, what date the backup was done or anything I could use as a reference as to where the problem was located. There was a listing of 'Bad backup set header found (0x7cc16478 at 1,940,377,952)' that appeared to terminate the error string, not sure if there is a way to reverse transcribe that into usable info or not.

 

Your description of the expected results of my second transfer (copying with snapshots turned off) was a bit disappointing, especially considering how much time it takes to perform a transfer of this type. If turning snapshots off results in unusable tapes then why offer the option in the first place? I figured you were not involved with the original design of that function, it's odd that Retrospect would offer it in the first place.

 

To answer your question about where the catalogs are stored, the XServe has 3 internal, matched 750GB drives configured in a 1.34TB RAID 5 partition via the Apple XServe RAID card (this is a hardware RAID instead of a software RAID). This card attaches the drives directly to the main logic board and data bus within the XServe. Every 24 hours the catalog folder is copied to the 8.87TB Promise RAID volume, it is verified after the copy. The transfers were attempted from the original copy.

 

I think my next step is to restore each snapshot and immediately back up the results, I have about a week left on the lease of my original tape library before I am required to send it back. This will give me a chance to identify which of the 68 snapshots are valid. If the 5,395 errors correlate to a backup set that contained that many files I'll attempt to restore it first.

 

Thank you for taking the time to respond to this problem, I've used Retrospect since 1993 and have been generally satisfied with it's performance. I can't wait to jump into the new version and explore the new feature set.

 

 

[twickland]

After attempting a rebuild of the original tapes it appears they have their own problems. The rebuild took 72 hours and reported back 5,395 errors

OK, so your problem either lies with your original tapes, or with the older drive you're using to read them.

 

Your description of the expected results of my second transfer (copying with snapshots turned off) was a bit disappointing, especially considering how much time it takes to perform a transfer of this type. If turning snapshots off results in unusable tapes then why offer the option in the first place?

A backup set without snapshots is perfectly usable if all you intend to do for your restores is search for files. If, however, you wanted to restore a volume to the state it was in at the time of a given backup (Retrospect's true forté), it would be an immense chore without the snapshot.

 

I figured you were not involved with the original design of that function

Right-O, matey. This is a peer-to-peer forum; I'm a Retrospect user just like you, with absolutely no connection to EMC.

 

To answer your question about where the catalogs are stored, the XServe has 3 internal, matched 750GB drives configured in a 1.34TB RAID 5 partition via the Apple XServe RAID card

I was thinking there might be some slight chance of an issue with your fibre channel HBA, but your responses pretty much eliminate that possibility.