MD5 Digest

[NBArchives]

I have been having issues with the MD5 Digest as it has been giving me errors. I have been looking on other posts on this fourm as to find a solution; however, there has been nothing in terms of solutions.

 

One thing I have encountered is that I can disable the MD5 Digest. My problem is that I don't have a solid idea on what the main purpose of it is. I have heard that it useful for incremental backups, but that's about it.

 

In my case I am not doing incremental updates, I am commiting the data to the tape, with a duplicate, and that data is wiped from the Hard Drive. From my understanding I could disable the MD5 Digest and hopefully avoid future errors and continue my backups. Is this a wise move, or am I going to be jepordize my data?

 

Running:

OS X 10.5.6

Retro 8.1.626

 

[rhwalker]

One thing I have encountered is that I can disable the MD5 Digest. My problem is that I don't have a solid idea on what the main purpose of it is. I have heard that it useful for incremental backups, but that's about it.

It's all explained in the manual. Oh, there is no manual. Well, the product has only been out for over a year, so perhaps that's a bit much to expect.

 

See MD5 in Retrospect Windows 7.5 or page 270 in the Retrospect 7.5 for Windows Users Guide

 

But here is a brief synopsis:

Modern backup programs (Retrospect included, now) generate an MD5 polynomial of the data for each file (and its metadata) that is backed up, and store the suite of MD5 calculations in an "MD5 Digest".

 

This permits the backup program to validate the backup data (Retrospect calls it a "media validation") against the MD5 digest (by reading the backup data and recalculating the MD5 values for the data in the, um, media set) even if the source files have changed or are missing. This solves the problem of a compare phase of the backup data against source files on the disk when the source files are changing during the backup window.

 

That's all it does. It has nothing to do with incremental backups, etc. It's just to solve the compare problem against changing files.

 

Clear?

 

Russ

[NBArchives]

Very. The first link you posted says it all. Without MD5 there's no guarantee that my one time backups will will be properly varified in the future, which negates the whole project.

 

Now comes the fun part of trying to figure out to what is causing the problem.

 

When I do my backups, in a set of Tape1A and Tape1B, I get the MD5 mismatch errors on different files, but on the same file types. I am backup up Video and uadio files that correlate to each other and the errors a generally on video files.

 

The funny thing about this is that the files I am backing up do not get opened, or modified at all. My only hunch is that maybe Final Cut Pro Studio may have something to do with it as I just updated it and that's when the errors started to occur. Anyone else run into this situation?

 

[rhwalker]

My only hunch is that maybe Final Cut Pro Studio may have something to do with it as I just updated it and that's when the errors started to occur.

Ahhh.... That might be the clue.

 

Now, because there is no manual, no one knows for sure what goes into the MD5 calculation, and the mismatch diagnostic messages are, shall we say, not fully informative.

 

But I believe that the MD5 calculation includes metadata (other than just the file itself) and, depending on the version of the Mac OS, might include file type/creator metadata used to control what program launches when the file is opened. If it doesn't, then there wouldn't be any way to validate the metadata in the backup store.

 

For a discussion of this can of worms on Snow Leopard, see Metadata madness.

 

So, the Final Cut Pro Studio update may have gone around and warmed over all the metadata for its files when it was installed.

 

The interesting test would be to try the same comparison against a read-only filesystem, or to restore an earlier snapshot (prior to the Final Cut Pro Studio update), do MD5 calculations on that and compare against the same calculation done on the current file.

 

Note that your comment that the files "do not get opened" is not true. Retrospect has to open them when the backup is done, and again if the file is read for another MD5 validation. Final Cut Pro may not have opened them, but Retrospect has opened them. And that changes the accessed time field for the metadata. Hmmm.... perhaps there's a bug in what metadata is used for MD5 calculation... Nah, couldn't be, Retrospect 8 doesn't have any bugs.

 

You might want to work with EMC Support to see what is going on here.

Contact EMC Retrospect support

 

Russ

[NBArchives]

Note that your comment that the files "do not get opened" is not true. Retrospect has to open them when the backup is done, and again if the file is read for another MD5 validation.

 

That I knew that was happening; what I was actually referring to was that I was not opening or modifying the files myself.

 

I have tried to use the backup without the MD5 digest, but it also gaves a comparison error.

 

So, I'm going with your logic Russ, I set the disk permissions to read only, and see if that works. If failing, go back with the pre-updated Final Cut and see how that pans out. If that fails, I'll stop being stubborn and talk to EMC support.

[NBArchives]

I was mulling around in the knowledger base and in the known issues page I found this.

 

During backups, Retrospect 8.0 will report comparison errors on its own Catalog files. These errors are expected and should be ignored. In fact, Retrospect should not report them.

 

Anyone know if this is refering to media sets?

 

 

 

On another note, I still get the md5 digest errors, then it foes into a bad media set header found, and media set format inconsistency.

 

Example from log:

*File "mc2073 - 083.mov": didn't compare

>

!An error occurred during the verification step. the MD5 digest for the file "mc2073 - 083.mov" did not match, error -1129 ( MD5 digest mismatch)

>

!Media Set format inconsistency (10 at 4505580)

>

!Bad Media Set header found (0x21b8bb1f at 610185677)

>

Trouble positioning: "1-mc2073_tape4b" (9728153), error 0 ( unknown)

 

and then it loops in the with the position error and the media set format inconsistency until the log file maxes out.

 

I'm assuming that the tape machine is getting slowed down by the digest error and goofing up or the tape has bit the dust. Either way I'm going to get a hold of EMC tomorrow, but any comments or help is much appreciated.

[rhwalker]

I was mulling around in the knowledger base and in the known issues page I found this.

 

During backups, Retrospect [color:red]8.0[/color] will report comparison errors on its own [color:red]Catalog files[/color]. These errors are expected and should be ignored. In fact, Retrospect should not report them.

 

Anyone know if this is refering to media sets?

No, it's referring to [color:red]Catalog files[/color]. Besides, what's the relevance?

Retrospect 8.0 Known Issues

 

That tech note is almost a year old, and you don't seem to be running Retrospect 8.0:

 

Running:

OS X 10.5.6

[color:red]Retro 8.1.626[/color]

That's why it's so important to get people to post their version numbers.

 

 

On another note, I still get the md5 digest errors, then it foes into a bad media set header found, and media set format inconsistency.

That's an entirely different issue. Robin indicates that yet another massive bug fix (over 200) is expected "real soon now":

Retrospect 8.1 bug fix update perhaps sometime this year

 

I'm assuming that the tape machine is getting slowed down by the digest error and goofing up or the tape has bit the dust. Either way I'm going to get a hold of EMC tomorrow, but any comments or help is much appreciated.

You are on the right track. There's really not much we can do to fix this bug in this user-to-user support forum.

 

Russ