Jump to content
dzeleznik

Verify reports "chain broken", how to fix?

Recommended Posts

I run a verify "on all backups not previously verified" monthly for each of my active backup sets just to keep tabs on their integrity. I have one backup set that just ran its monthly verify pass and reported 10 files with "chain broken" errors. This backup set has never (nor have any of my others) reported this error in the past. This backup set has been in use for 2 years and has cleanly verified every month until now. I upgraded from Retrospect 11 to 12.1.0.174 two weeks ago, so all previous clean verifies of this backup set were with the older version. Not sure if that is a coincidence or not.

 

Bottom line, how do I repair this error? I can find no help online via countless searches. Will rebuilding the catalog solve it? Or do I need to do something else.

 

Thanks!

Share this post


Link to post
Share on other sites

dzeleznik is not the only one getting this error on the latest version of Retrospect, even though that other person is a Retrospect Mac administrator.  See this post.

 

I found that post by clicking on the gear icon at the top right of the Web page, on the same line that says "IPS Community" on the left.  That gave me Advanced Search, and I typed "chain broken"—including the quotes—into the Find Words box.  I did not select anything in the Find in Forum dropdown, but just clicked the Search Now button.

 

My next post in this thread will be my boilerplate explanation of why and how to file a Support Case.  Since dzeleznik and iCompute are both getting the same error in similar circumstances on what is—under the hood—the same new version of Retrospect, one of the two had better file one.

Share this post


Link to post
Share on other sites

If you think this is a bug that should be fixed by Retrospect Inc., you will have to submit it as a Support Case.  For English speakers, that is done by going here http://www.retrospect.com/en/support/contact, and filling out the form (sorry, I don't know what the equivalent addresses are for non-English speakers, but they can figure it out from their appropriate Retrospect website address).  IMHO this is quite reasonable; obliging you to fill out the form provides Retrospect Inc. with useful details about your Retrospect installation that they would otherwise have to query you for.

 

As a result, Retrospect Inc. will pay no attention to your post in this forum.  On 12 December 2016, in response to a letter I snail-mailed to Mayoff,  I received an e-mail through a Mayoff account that was signed by JG Heithcock, CEO, Retrospect, Inc. http://www.retrospect.com/en/about#exec.  In it he says "From reading your letter, I think the main issue is that you view the forums as a good place to talk to us, Retrospect, Inc. But we view the audience of the forums as restricted to our customers [my emphasis]. The one caveat we have made on that is for feature requests, largely as we would like to see if other customers also agree on the desirability and feature set for these requests."

 

That means that the only audience for "Retrospect bug reports" in this forum will be other administrators of Retrospect.  Nevertheless, by posting in this forum you are providing a useful service to us fellow administrator peasants.  Thank you.

 

Please be aware that the "description of your issue" in the Support Case form is IME limited to about 2000 characters by the Support Case software.  If you go over that limit your "description" will be broken up into a "description" plus one or more "additional notes".  The same is true for any additional notes you may later post yourself.  I suggest that, to avoid the appearance of choppiness in your Support Case, you create your case in a post in this forum and then copy it paragraph-by-paragraph to your Support Case. 

 

Note that, despite the new dialogs in the Retrospect Inc. Support Case system urging you to sign up for Annual Support and Maintenance, Mayoff has verbally assured me that you don't need to be signed up for ASM to report a bug—only to get personal assistance with coping with it.

 

If this post sounds formulaic, that's because I intend it to be.  I intend to post it in every new thread that appears in this forum, unless the OP indicates that he/she has or will open a Support Case for the bug that the thread reports.  Of course, Mayoff could take 5 minutes of his time to post a slightly-more-polite version of this post as a  "sticky thread" that will always appear at the top of the forum.  I don't intend to hold my breath until that happens (insert appropriate smiley here).

 

 

Share this post


Link to post
Share on other sites

I have just rebuilt the catalog for the offending backup set and ran a complete verify. This took quite a while sandwiched in between other scheduled Retrospect activity because the backup set is one of my largest at ~3TB. The complete verify turned up additional "chain broken" errors further back in time. During the interim, the scheduled monthly incremental verifies on my other backup sets completed and each one of them also reported 4 - 30 chain broken errors.

 

For now, I am going to assume that the errors are real latent ones that were only just reported because v12's verification process is more thorough than v11's was. Since a catalog rebuild does not seem to solve the issue, I am now starting with my smallest backup set to save time (<300GB) and transferring it to create a duplicate. I will then run a complete verify on the dupe. If all is well, I will swap the new set into my scripts and forget the old one. I'll then run a backup and check that the files that were previously "broken" are freshly backed up. If *that* all goes well, I'll rinse and repeat for my other backup sets.

 

Definitely a PIA and will take quite a while, but hopefully will scrub my backup sets to be clean and dependable. I'll report back when I finish my testing on this small backup set. Fingers crossed...

Share this post


Link to post
Share on other sites

Latest status

  1. Original backup set created and maintained with v11, incrementally verified monthly with no errors until upgrade to v12. At first incremental verify after upgrade to v12, 6 chain broken errors were reported. Of special concern were several crucial application databases.
  2. Rebuilt the catalog and performed complete verify which reported 18 chain broken errors. If the catalog rebuild did not fix any of the errors, I would have expected the reported errors to be a superset of the errors reported by the incremental verify in step #1. This was not the case. Some of the errors reported during the incremental verify were not reported after the catalog rebuild and complete verify. In fact, none of my crucial db's are reported broken anymore, the only errors seem to be relatively minor log and misc. files. I cannot be sure if the catalog rebuild resolved some of the broken chain errors or whether the first incremental verify using v12 generates some bogus results. I will need to test this out on a different backup set.
  3. Transferred the backup set in it's entirety and ran complete verify on the new dupe. The same 18 chain broken errors were reported as for the original backup set in step #2.
  4. I am now in process of doing another transfer of the original backup set, but this time using a selector to exclude the 18 files that are reported to be broken.
  5. Stay tuned.....

My conclusion so far is that:

  • v12 reports chain broken verify errors on backup sets that have verified continuously clean using v11
  • Rebuilding the catalog *may* fix some of the chain broken errors, still TBD
  • Transferring the backup set in its entirety does not fix chain broken errors

Share this post


Link to post
Share on other sites

Very interesting, because the Release Notes for Retrospect Windows 12.1 include the Fixed entry "Fixed uncommon case of fast rebuild misreporting block-level incremental backup chains as broken and not restorable (#6668)".  It sounds to me as though, assuming dzeleznik was not using Fast Catalog Rebuild, the fix for bug #6668 may have broken something else.

 

In any case, IMHO dzeleznik should file a Support Case per post #3 in this thread.  He/she can easily adapt the contents of posts in this thread for the Description of Problem and Additional Notes.  He/she should also try to preserve log files etc., because IME Retrospect Support will ask for uploaded copies.

 

Here's a more controversial suggestion: Since dzeleznik evidently upgraded from Retrospect Windows 11 to Retrospect Windows 12.1 within the past 45 days, he/she should phone Retrospect Sales and ask for a refund of the upgrade fee.  That's a very good way to get Retrospect Sales to make Retrospect Support and Engineering pay attention to the evidently-new "chain broken" bug.  Of course the downside of this strategy is that Retrospect Sales might call dzeleznik's bluff on his/her threat.  dzeleznik should therefore consider whether reverting to Retrospect Windows 11.5 would deprive him/her of bug fixes that he/she really needs; IMHO most non-Cloud-using Retrospect administrators don't really need the New features or Improved features in Retrospect Windows 12.0 and 12.1.

 

I'll make a post in iCompute's thread in the "Retrospect 9 or higher for Macintosh" forum, suggesting that he look at the third paragraph in this post and consider also phoning Retrospect Sales.

Share this post


Link to post
Share on other sites

Apropos of what I wrote in the first paragraph of post #6 in this thread, I now notice that dzeleznik speaks in his/her Latest Status paragraph 2. in post #5 of "none of my crucial db's".  I assume that means that these are databases, and that dzeleznik has therefore enabled Retrospect's block-level incremental backup feature (pages 525-529 of the Retrospect Windows 12 User's Guide) for them.

 

Curiouser and curiouser (said Lewis Carroll's Alice, for non-English speakers).  I wonder if the fix for bug #6668, which was supposed to fix "uncommon case of fast rebuild [my emphasis] misreporting block-level incremental backup chains as broken", created misreporting for verify runs—which iCompute is also talking about in his thread.

Share this post


Link to post
Share on other sites

I have completed my testing of the situation using much a larger (hence the delay in reporting back) backup set:

  1. Original backup set created and maintained with v11, incrementally verified monthly with no errors until upgrade to v12. Backup set has been maintained with scripts with block level incremental backup enabled.
  2. First incremental verify after upgrade to v12, 18 chain broken errors were reported. Some files reported are large files that were supposed to be backed up using block level incremental, others are misc. smaller files where the only "chain" is a single session and it is missing.
  3. Performed complete verify which reported 33 chain broken errors. The errors reported are NOT a strict superset of the 18 reported by the incremental verify in step #2. The 33 files in error are again a mixture of large dbs that are backed up using block level incremental where 1 or more sessions are missing, and other smaller files that have a single session and it is missing.
  4. Rebuilt the catalog, log says it was a "fast rebuild", no errors reported
  5. Repeated a complete verify to determine if the rebuild solved any issues. Answer is no, the exact same 33 errors are reported as in step #3.
  6. Transfered the backup set to a new one using a selector to exclude the 33 files that are reported to be broken.
  7. Full verify on the new backup set is clean
  8. Next backup using the new backup set backs up the files (if they still exist on the client) that were excluded.

Conclusions:

  • My backup set had real latent errors that were never reported or detected by v11.
  • v12 incremental verify on a v11-maintained backup set reports a mixture of real errors and false positives
  • v12 full verify on a v11-maintained backup set reports the true set of errors
  • Rebuilding the catalog solves nothing, the errors are missing sessions in the backup set. If they are missing, all the cataloging in the world will not bring them back from the dead.
  • Transferring the backup set minus the files in error generates a clean starting point for future backups

I honestly can't say whether any of this is related to the bug 6668 mentioned by the previous poster. There does seem to be a bug related to false positives reported by incremental verify on a backup set that has real errors in it. I will report this to Retrospect. In the meantime, since all of my backup sets have some errors my strategy is to:

  • Run a full verify on each backup set to determine the true list of file errors
  • Transfer/dupe the backup set filtering the broken files to create a fresh starting point for each of my backup scripts
  • Run a full verify on the new backup sets and then continue with monthly incremental verifies

Share this post


Link to post
Share on other sites

Very neat job of testing and analysis, dzeleznik.  Sorry to hear that your "chain broken" errors are real.  I hope you can re-backup the files in error.

 

Obviously, the fact that the errors are real means you can forget about my "more controversial suggestion" in the third paragraph of post #6 of this thread.  My second paragraph of post #7 evidently is still applicable, but only to false positives when the "Verify only backups not previously verified" option (pages 375-376 of the Retrospect Windows 12 User's Guide) is used for Verification scripts.

 

The burning question now is whether Retrospect Inc. engineering knows what caused the "real latent errors" in Retrospect Windows 11 (and earlier?), and whether they have plans to fix them.  When you submit your Support Case, dzeleznik, please ask that question—possibly as an Additional Note.  If Jeff (probably Jeff McIntire) reads your Support Case, he may give you an answer.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×