Retro 10.0.1 (105) with ISA missing large amounts of data during Backups

[earmarkmedia]

I am running Retrospect (and have been since 6.1) on an Xserve 3,1 running Mac OS X Server 10.6.8 with 12Gigs of RAM, 24TB of Active Storage backup disk space and a Quantum Scalar 50 LTO-4 Two Drive Fibre Channel tape library connected via a QLogic 5602 FC Switch with Zoning enabled. I have Advanced Tape Support as part of my licensing. I upgraded Retrospect on this Xserve about 2 weeks ago from 9.0.2 (107) Multi Server/Unlimited Clients to 10.0.1 (105) Multi Server/Unlimited clients. Everything appeared to be OK during the upgrade. Everything remained intact. After the upgrade I immediately started to notice the following day a change in the way Retro 10 scans files and accounts for them.

 

Background: I have a staged backup system in place so that, using Carbon Copy Cloner, I clone two source disks from two different Xserve with the same aforementioned specs. I clone to sync, removing any data on the destination which does not exist on the source. One source has about 5.3 TB and the other about 100GB. They are cloned every 4 hours onto one 12TB Active Storage LUN. From here I use a Retrospect backup to Tape script to perform one Full backup of the 12TB backup disk over the weekend and then incrementals nightly during the week. A new Media set is started during the Full weekend and Skip to New Member is used for incrementals since the tapes are removed offsite daily the following morning. This system has worked flawlessly for several years over several versions of Retrospect and countless Restores have been successfully performed both during DR drills and real emergencies.

 

After upgrading to version 10 I noticed that Retrospect was 'seeing' about 128GB of data during a newly instituted Full backup over the weekend. The Tape Media set showed 128GB in size for the Full backup. Normally the full 5.4 TB was scanned and backed up at the start of a Full backup to a new Media set and then the incrementals averaged about 20GB during the nightly backups during the week. So essentially Retrospect was missing 99% of the data. I even created a new script to make sure that it was not a corrupted script caused by the upgrade to 10. Same results. Retrospect was scanning about 128 GB for the Full backup and 24k or so for the following incrementals.

 

I noticed in the Logs for each backup a mention of the new Instant Scan process. Searching the documentation for 10 and then Forum topics related to Instant Scan (ISA), none of which seemed to mention my specific issue, I decided to disable the ISA process. Once I did that Retrospect performed as expected, 'seeing' all of the data on the source volume (5.4TB total) and backing up all of the data to tape during the Full backup and any changes during the incremental.

 

Am I missing something? Is what I described the proper functionality with the new ISA process? This seems to affect all scripts. I have another Disk to Disk script with the same symptoms when ISA is enabled.

 

Keeping ISA disabled for now.

 

Thanks for any insights...

[Lennart_T]

Am I missing something?

I think it says in the manual (User's Guide) that it takes about half an hour for RetroISA to catch up with the latest changes. And you (via Carbon Copy Cloner) throws a lot of changes onto the Retrospect source and perform a backup before RetroISA has caught up with the changes. So perhaps even half an hour isn't enough?

 

I would keep RetroISA OFF in your situation.

[earmarkmedia]

Thanks for responding.

 

Right. I read the documentation on Instant Scan. The CCC schedule is disabled during the nightly Retrospect backup to prevent any data creep. I probably did not describe my situation properly. Let me use an example of the behavior I am witnessing if I have Instant Scan (RetroISA) enabled.

 

Let's say I connect a new disk to this backup server. One which Retrospect has never 'seen'. This new volume is not connected in any way with a CCC process. Let's say it has 1TB of data on it. I create a brand new backup script to backup the data every night at 8PM. My media set is tape but it can be anything, disk, tape, etc. When the script runs at 8PM Retrospect (with Instant Scan enabled) will scan and backup only a small portion of this newly added hard drive. It's as though Retrospect is performing an incremental backup...during it's first pass. Retrospect misses most of the data during the scan and the subsequent backup. This occurs on any type of backup when Instant Scan is enabled. If I disable Instant Scan, Retrospect functions normally and scans all of the data (in this example, 1TB) and backups all of the data on the first backup.

 

The Retrospect software is behaving as though it's scanning and backing up all of the data without any indication (other than the log info which I parse) as to the discrepancy between the amount of data that needs to be backed up and what is actually backed up. I am not seeing any errors in the Retrospect master log (retroISA_log.utx) or the OS X syslogs during these backups with Instant Scan enabled.

 

This behavior occurs on a second Xserve which has another copy of Retrospect 10 installed. The server is the exact same technical spec as this one.

 

I am posting this info here to make sure that before I contact Retrospect support that I am not misunderstanding Retrospects functionality as it relates to Instant Scan.

 

Thanks.

[Lennart_T]

Just to make things clear: When backing up the "new" volume, did you also use a "new" (=empty) media set? Otherwise you will bring file de-duplication into the equation.

[earmarkmedia]

Correct. The test was done with every component being new, source, script (Media set, schedule, etc).

[Lennart_T]

One final question, or rather three:

1) How many files were there on the "new" drive? Just one huge file or millions of tiny files, or something in between?

2) How long time elapsed from the connection of the "new" drive until Retrospect was made aware of the drive? (Until the new drive/source was added to a script.)

3) How long time elapsed from the "new" source was added to a script until the script ran the first time?

[earmarkmedia]

1. Hundreds of thousands.

2. 4 hours. 2TB drive was re-formatted and sample data was copied to it before going into Retrospect and setting up the script.

3. 3 Hours. Setup was finished around 16:00 and script kicked off at 20:00.

[Lennart_T]

Right. It does NOT sound like a timing issue.

It does sound as a real bug.

[David W Lee]

Please do contact Support. Thanks.

[Don Lee]

If it is true that after "a bunch of data" is added to a volume, that Instant Scan has to "catch up", and if a backup is performed before the ISA process is complete it omits files, that is a serious bug. Retro should detect that ISA is not done, and use its old, slow, techniques to ensure that *every* backup backs up every file. Anything less is not behaving properly.