Jump to content

Verify vs copy and data errors

Recommended Posts

A good piece of SW behaves correctly. A really great piece of SW behaves correctly even in the face of errors, and provides useful information when things go wrong. Retrospect 10.5 is not yet great.


I have an occasional problem with my external firewire disks. I dont know where the errors originate, but they are "sticky", in that they are not intermittent once they appear. I have taken to running a "verify" script regularly on the media sets to ensure that the underlying HW is functioning correctly.


These errors usually appear in clusters, affecting multiple media sets in a burst, and then staying "stable" between occurrences. In this case, out of 14 media sets, errors appeared on 7 of them. 6 of them are of the form:


(blah blah)

!Media Set format inconsistency (2 at 19357344)

!Media Set format inconsistency (2 at 19971744)

!Media Set format inconsistency (2 at 20586144)

!Media Set format inconsistency (2 at 21200544)

!Media Set format inconsistency (2 at 21814944)

!Media Set format inconsistency (2 at 22429344)

!Media Set format inconsistency (2 at 23043744)

!Media Set format inconsistency (2 at 23658144)

!Media Set format inconsistency (2 at 24272544)

!Media Set format inconsistency (2 at 24886944)

!Media Set format inconsistency (2 at 25501344)

!Media Set format inconsistency (2 at 26115744)

!Media Set format inconsistency (2 at 26132288)

Remaining: 550 files, 0 B

Completed: 3770 files, 34.6 GB


Note that there are 550 files "remaining", and the 3770 is less than the number expected (by 550 files).


The other type of error is:


- 12/14/14 10:30:50 AM: Verifying v9_Virtue2

!Generated MD5 digest for file "/Volumes/VirtueHD/private/var/servermgrd/sessions/root@4B101fgg23MYmjL4Hhq9pNXY9pN1Idb5-admin" does not match stored MD5 digest.

Completed: 467630 files, 29.3 GB

Performance: 3,860.5 MB/minute

Duration: 00:08:08 (00:00:22 idle/loading/preparing)


These "MD5" errors seem to be rarer.



I presume that these errors are some sort of data transfer problem with the disk, because when I read the disks in raw mode (or copy the files) no errors are detected in the disk operations. For some reason, data appears to have been lost in the transfer of the data from memory to the disk. Given the (lack of) data protection in the data path in essentially "consumer" HW, this is not so surprising.




What is surprising is that the VERIFY operation detects errors, and a media set COPY runs WITHOUT error.


What I have done (several times now) is copy the media set to another, and then copy it back. When I do this, I get the expected number of files reported by the verify, but *no errors*.


Something is not right.


1. What does retrospect do with those missing 550 files if I try to restore from that media set? I have never seen an error reported from a restore saying it tried to restore a file and it was "unavailable".


2. Why does verify detect errors that copy ignores? (can't see?)



Retrospect exists to protect my data. It needs to be explicit with me about what it fails to backup/preserve/restore.


What's up with this? Is it different in retro 11.x?

Link to comment
Share on other sites

I can see the contrast when I copy a media set with an MD5 error:


+ Executing acopy 2.5 at 12/14/14 (Activity Thread 2)

To Backup Set a_v9_virtue2...

- 12/14/14 2:58:15 PM: Transferring from v9_Virtue2

12/14/14 3:13:08 PM: Execution completed successfully

Completed: 467476 files, 28.7 GB

Performance: 2,026.7 MB/minute

Duration: 00:14:53 (00:00:22 idle/loading/preparing)


- 12/14/14 3:13:08 PM: Verifying a_v9_virtue2

!Generated MD5 digest for file "/Volumes/VirtueHD/private/var/servermgrd/sessions/root@4B101fgg23MYmjL4Hhq9pNXY9pN1Idb5-admin" does not match stored MD5 digest.

12/14/14 3:20:45 PM: 1 execution errors

Remaining: 1 files, 3.2 GB

Completed: 467475 files, 28.7 GB

Performance: 3,866.8 MB/minute

Duration: 00:07:36


Judging from the file count and the error, retro copies the (bad?) file, and detects the MD5 error in the verify pass.


When I copy these media set copies BACK to the originals, the copy operation only copies the number of files reported for the forward copy with the error. (If 500 files were missing, 500 fewer files are copied back, and no errors are reported)


I have not attempted to restore from these "reduced" media sets, but I will do a backup, and expect that the missing 550 files will be re-added.

Link to comment
Share on other sites

Another anomaly of note:


The MD5 error that I am seeing shows the file with the MD5 error being copied to the destination media set. (!) I copied original -> copy, recycled "original", and then copied copy -> original. The same file showed the MD5 error on the copy in each direction.


What does this mean? I have to conclude that if the MD5 digest does NOT match, then either the digest, or the data is corrupted. If the file is corrupted, I probably don't want it. I want to know about losing it, but I probably don't want it copied to the destination media set. If the file is still on the source, I want retro to toss it and re-copy it to the media set on the next backup.


If I restore from this set, does retro tell me that the data is bad? (I'll try it)

Link to comment
Share on other sites

I restored the file with the MD5 error on the copied media set. The resultant (zero length) file is named:




The original on the media set, according to the log, is:




The full path of the file can be seen in the log excerpt above.


The log of the restore shows no errors:



+ Executing Restore Assistant - 12/14/14 9:44 PM at 12/14/14 (Activity Thread 1)

12/14/14 9:48:08 PM: Connected to Witsend

To volume TEMP_RETRO on Witsend...

- 12/14/14 9:48:08 PM: Restoring from a_v9_virtue2, Snapshot /Volumes/VirtueHD, 12/8/14 2:15:48 AM

12/14/14 9:48:21 PM: Execution completed successfully

Completed: 33 files, 18 KB

Performance: 0.2 MB/minute

Duration: 00:00:13 (00:00:08 idle/loading/preparing)

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Create New...