Jump to content
egoff

Discrepancy in amount of data transferred vs. verified

Recommended Posts

In Retrospect Windows Multi-Server 7.7, every time I run a script to transfer a complete backup set from one set to another set (recycle, with media verification on), after the job finishes the log always reports less data and fewer files verified than it says were transferred. (See attached screenshot.) It doesn't report any errors, though. Does this indicate a problem, or is it a bug in the logging...?

post-35750-0-46029200-1371655766_thumb.png

  • Like 1

Share this post


Link to post
Share on other sites

This is just speculation — could this be a result of deduplication if you have that option enable? For example, as I understand it, if the backup set contained three Windows 7 clients the operating system files for all three systems would be included in the total count of files but only one copy of a file exist in the backup set where it is the same on more than one system. When it comes to the verification there will only be the one file to verify instead of multiple copies.

Share this post


Link to post
Share on other sites

Hmm, if it was deduplicating at a file level, wouldn't it deduplicate when it transfers, not later when it verifies what it already transferred? When I transfer from, say, Set A (3 tapes) to Set B, it always needs 3 tapes in Set B, and the log reports that it transferred an amount of data that should require 3 tapes; but then when it verifies, it only reports verifying an amount of data that should have fit on 2 tapes. 

Share this post


Link to post
Share on other sites

I've never seen it published anywhere how Retrospect shows deduplication events. My assumption is that during the copy/transfer all the duplicate copies of file X would be shown as copied for snapshot purposes even though only one is physically copied. However when it comes to the verification only the one copy will be available.

 

I stopped using tape (DAT) a few years ago because of capacity, speed and hardware driver support constraints so my recent experience of transfers is with disk backup sets.

 

What options do you have selected for your transfer (e.g. Most recent snapshot, All snapshots, etc)?

 

Are you doing a full verification or an MD5 sum verification?

Share this post


Link to post
Share on other sites

Transfer options: Copy Snapshots, Media Verification, Match source Catalog File to destination Catalog File, Don't add duplicates to Backup Set.

 

My transfer scripts are Recycle transfers, and the backups on the source set were created with "Don't add duplicates to Backup Set" turned on. The only difference between the original backup settings and the transfer settings is that the original backup has "Match only files in same location" turned on, while this option is off in the transfer. I guess this *should* result in less data being transferred than was originally backed up, because of the scenario you outlined, where OS files on different systems would not be copied multiple times, but what I still don't understand is why a transfer of a 3-tape set always requires 3 tapes in the destination set, even when the amount of data reported verified in the log should easily have fit on 2 tapes...

 

Operations log example: Under Transferring, it says "Completed: 3303458 files, 2,124.3 GB." Then under Verifying, it says "Completed: 2679494 files, 1,567.8 GB." The Verifying section has "Execution completed successfully," which is nice to see, but the very next line is "Remaining: 1 files, 281.8 MB," which doesn't exactly inspire confidence...it verifies all but the last file? Or all but the last 600,000+ files??

Share this post


Link to post
Share on other sites

After looking at more logs, I'm sure something is wrong, either with the transfer/verification process or with the logging. I run a transfer job once a week, and it's the same data each time, just with incremental changes around the edges. But the log entries about verification are all over the place. Excerpts of logs of last 2 transfer jobs:

 

6/16/2013

Transferring... [from Set C to Set C Clone, Recycle]

Execution completed successfully

Completed: 3219299 files, 2,263.4 GB

Verifying...

Execution completed successfully

Remaining: 594 files, 1.2 GB

Completed: 3187573 files, 2,257.4 GB

 

6/23/2013

Transferring... [from Set A to Set A Clone, Recycle]

Execution completed successfully

Completed: 3303458 files, 2,124.3 GB

Verifying...

Execution completed successfully

Remaining: 1 files, 281.8 MB

Completed: 2679494 files, 1,567.8 GB

Share this post


Link to post
Share on other sites

I'd noticed that remaining file entry in the verification on your original screenshot. As you are using media verification I can't see how that can happen.

 

I think you may have found a bug in the verification process. I checked through my backup history and discovered that the verification for one of my client laptops routinely reports verifying two, and occasionally three, more files than are backed-up.

 

I'll see if I can find some old logs for disk backup set transfers and see what happen during the verification. At the time they reported that the execution had completed successfully so I didn't pay any attention to the file and size counts.

Share this post


Link to post
Share on other sites

In Retrospect Windows Multi-Server 7.7, every time I run a script to transfer a complete backup set from one set to another set (recycle, with media verification on), after the job finishes the log always reports less data and fewer files verified than it says were transferred. (See attached screenshot.) It doesn't report any errors, though. Does this indicate a problem, or is it a bug in the logging...?

 

We noticed a similiar issue a couple of years ago and it follows us til now. We run in a problem with a huge loss of data when we perform a recycle backup set transfer from disk to tape out of a script. If we recycle the tape manualy before we transfer the disk backup set, the transfer completed successfully with the full amount of data. We had a very long period of time contact to the EMC Retrospect support center without any results. Since that we changed the backup server hardware two times but this error remains in force.

 

HTH Tom

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×