Discrepancy in amount of data transferred vs. verified

[egoff]

In Retrospect Windows Multi-Server 7.7, every time I run a script to transfer a complete backup set from one set to another set (recycle, with media verification on), after the job finishes the log always reports less data and fewer files verified than it says were transferred. (See attached screenshot.) It doesn't report any errors, though. Does this indicate a problem, or is it a bug in the logging...?

[Scillonian]

This is just speculation — could this be a result of deduplication if you have that option enable? For example, as I understand it, if the backup set contained three Windows 7 clients the operating system files for all three systems would be included in the total count of files but only one copy of a file exist in the backup set where it is the same on more than one system. When it comes to the verification there will only be the one file to verify instead of multiple copies.

[egoff]

Hmm, if it was deduplicating at a file level, wouldn't it deduplicate when it transfers, not later when it verifies what it already transferred? When I transfer from, say, Set A (3 tapes) to Set B, it always needs 3 tapes in Set B, and the log reports that it transferred an amount of data that should require 3 tapes; but then when it verifies, it only reports verifying an amount of data that should have fit on 2 tapes. 

[Scillonian]

I've never seen it published anywhere how Retrospect shows deduplication events. My assumption is that during the copy/transfer all the duplicate copies of file X would be shown as copied for snapshot purposes even though only one is physically copied. However when it comes to the verification only the one copy will be available.

 

I stopped using tape (DAT) a few years ago because of capacity, speed and hardware driver support constraints so my recent experience of transfers is with disk backup sets.

 

What options do you have selected for your transfer (e.g. Most recent snapshot, All snapshots, etc)?

 

Are you doing a full verification or an MD5 sum verification?

[egoff]

Transfer options: Copy Snapshots, Media Verification, Match source Catalog File to destination Catalog File, Don't add duplicates to Backup Set.

 

My transfer scripts are Recycle transfers, and the backups on the source set were created with "Don't add duplicates to Backup Set" turned on. The only difference between the original backup settings and the transfer settings is that the original backup has "Match only files in same location" turned on, while this option is off in the transfer. I guess this *should* result in less data being transferred than was originally backed up, because of the scenario you outlined, where OS files on different systems would not be copied multiple times, but what I still don't understand is why a transfer of a 3-tape set always requires 3 tapes in the destination set, even when the amount of data reported verified in the log should easily have fit on 2 tapes...

 

Operations log example: Under Transferring, it says "Completed: 3303458 files, 2,124.3 GB." Then under Verifying, it says "Completed: 2679494 files, 1,567.8 GB." The Verifying section has "Execution completed successfully," which is nice to see, but the very next line is "Remaining: 1 files, 281.8 MB," which doesn't exactly inspire confidence...it verifies all but the last file? Or all but the last 600,000+ files??

[egoff]

After looking at more logs, I'm sure something is wrong, either with the transfer/verification process or with the logging. I run a transfer job once a week, and it's the same data each time, just with incremental changes around the edges. But the log entries about verification are all over the place. Excerpts of logs of last 2 transfer jobs:

 

6/16/2013

Transferring... [from Set C to Set C Clone, Recycle]

Execution completed successfully

Completed: 3219299 files, 2,263.4 GB

Verifying...

Execution completed successfully

Remaining: 594 files, 1.2 GB

Completed: 3187573 files, 2,257.4 GB

 

6/23/2013

Transferring... [from Set A to Set A Clone, Recycle]

Execution completed successfully

Completed: 3303458 files, 2,124.3 GB

Verifying...

Execution completed successfully

Remaining: 1 files, 281.8 MB

Completed: 2679494 files, 1,567.8 GB

[Scillonian]

I'd noticed that remaining file entry in the verification on your original screenshot. As you are using media verification I can't see how that can happen.

 

I think you may have found a bug in the verification process. I checked through my backup history and discovered that the verification for one of my client laptops routinely reports verifying two, and occasionally three, more files than are backed-up.

 

I'll see if I can find some old logs for disk backup set transfers and see what happen during the verification. At the time they reported that the execution had completed successfully so I didn't pay any attention to the file and size counts.

[pronto]

In Retrospect Windows Multi-Server 7.7, every time I run a script to transfer a complete backup set from one set to another set (recycle, with media verification on), after the job finishes the log always reports less data and fewer files verified than it says were transferred. (See attached screenshot.) It doesn't report any errors, though. Does this indicate a problem, or is it a bug in the logging...?

 

We noticed a similiar issue a couple of years ago and it follows us til now. We run in a problem with a huge loss of data when we perform a recycle backup set transfer from disk to tape out of a script. If we recycle the tape manualy before we transfer the disk backup set, the transfer completed successfully with the full amount of data. We had a very long period of time contact to the EMC Retrospect support center without any results. Since that we changed the backup server hardware two times but this error remains in force.

 

HTH Tom