Jump to content

MD5 digest calculated incorreectly??


Don Lee

Recommended Posts

I have seen this behavior since v9, and it seems to be rare, but about the same. You can see my post about this in:

 

http://forums.retrospect.com/index.php?/topic/151548-verify-vs-copy-and-data-errors/

( and http://forums.retrospect.com/index.php?/topic/151193-error-in-the-log-what-does-it-mean/

 and http://forums.retrospect.com/index.php?/topic/150803-what-does-verify-error-mean/ )

 

In short, I get MD5 errors in media sets that remain even after the media set is copied, and seem to be repeated if the same file is backed up again. It is as if there is something about the file that the MD5 algorithm can't handle correctly.

 

A verify of the media set will reliably complain of the MD5 error. A restore of file with the MD5 error will succeed without error, and if I copy the media set, the copy operation will see no MD5 error, but the resultant copied media set will show the MD5 error on the same file when I do a verify on the copied media set.

 

I would normally blame this on hardware - bad disk, bad cables, flaky hardware,  but I have seen this on several disks and several computers, and the problem does not appear to be hardware. The pattern appears to be a software problem in MD5 generation (or verification). The error usually happens on files with very long "garbage" file names. (generated by some sort of hash)

 

I can't pin this down, nor reliably reproduce it. If I can, I'll do so.

  • Like 4
Link to comment
Share on other sites

Example (from verify op):

- 7/30/2015 4:15:18 PM: Verifying UserSet

!Generated MD5 digest for file "/Volumes/BarryMiniHD/Users/barry/Library/Caches/TemporaryItems/dftmpIHPFNLBNnpnkkkkk--------" does not match stored MD5 digest.
Completed: 118155 files, 24.7 GB
Performance: 3669.9 MB/minute
Duration: 00:07:30 (00:00:36 idle/loading/preparing)

Link to comment
Share on other sites

  • 2 months later...

Yes, I have just seen it in two consecutive backups using Windows Professional Retrospect 9.5.

 

It is the same file both times, precisely.

 

I pulled the file from the backup, and tested using Microsoft's md5 hash tool myself.

 

Both the original and the backup are identical, give precisely the same hash.

 

But Retrospect has saved the wrong hash, and fails now to verify that file in the backup.

 

I would say this is a serious problem, even if it occurs only once in a while, and I hope Retrospect will be responsible, releasing a fix for it as an update each of their recent release levels.

 

 

Link to comment
Share on other sites

  • 4 months later...

I'm not so sure it's a fatal problem. It's a warning, and does not seem to affect the data when restored. I suspect that the hash that Retro generates includes the name and meta-data, so checking that hash against the "standard" tool is probably not valid.

 

That said, I agree it should be fixed.

Link to comment
Share on other sites

  • 4 months later...

Had similar issues and have ignored them, so far. However on restores it might not be so trivial. I had to use a backup for restoration which contained errors. On restore, I got an error as well on many of the same files, plus a few more. Here's an example of an error on the restore. If Retro cannot reliably back up and restore without error then that's not good. Luckily I also use an alternate file based backup system in case of such an eventuality (call it a lack of trust and an abundance of drive space...). This, if it truly botches files, is a truly discouraging situation if it proves to affect the files in question. Maybe an epic fail for me in the future as this issue has previously been discounted (literally, I've been told by support to ignore it!) by Retro support when it occurs during the backup. I've never seen it in restores before! 

 

 !An error occurred during the verification step. The MD5 digest for the file "/Volumes/LaCie 10/iPhoto Library_old.migratedphotolibrary/Masters/2009/0902006_Select lowrez/IMG_2006_LOWREZ.JPG" did not match, error -1,129 (MD5 digest mismatch)

 !An error occurred during the verification step. The MD5 digest for the file "/Volumes/LaCie 10/*MyPicturesLibrary/2009/2009-02-28/IMG_2059.CR2" did not match, error -1,129 (MD5 digest mismatch)

 
BTW, the history on this backup is that it started from the previous backup as a recycle event on 26 JUN 2016 and the next day finished another backup (27 JUN 2016). Thereafter, the source drive failed and I needed a restore to new media over the next days. Found the errors on restore yesterday and again today (tried again, just to be sure)
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...