Don Lee Posted April 23, 2012 Report Share Posted April 23, 2012 I am rebuilding the catalogs on some Retro 6.1 tapes to see if I can restore from them. When the first rebuild was done, I looked at the log: + Executing Rebuild at 4/22/12 (Activity Thread 1) To Backup Set MercyShare10 [001]... !Backup Set format inconsistency (4 at 317871024) !Backup Set format inconsistency (4 at 317768288) !Backup Set format inconsistency (4 at 317960128) !Backup Set format inconsistency (4 at 36464464) !Backup Set format inconsistency (4 at 317530656) !Backup Set format inconsistency (4 at 295383072) !Backup Set format inconsistency (4 at 296098384) !Backup Set format inconsistency (4 at 293092656) !Backup Set format inconsistency (4 at 295433920) ... ... a whole bunch of these..... ... !Backup Set format inconsistency (4 at 314986544) !Backup Set format inconsistency (4 at 315011504) !Backup Set format inconsistency (4 at 315020688) !Backup Set format inconsistency (4 at 315034864) !Backup Set format inconsistency (4 at 315057760) !Backup Set format inconsistency (4 at 315054256) !Backup Set format inconsistency (4 at 315097008) !Backup Set format inconsistency (4 at 315105376) 4/22/12 10:09:19 PM: Execution completed successfully Completed: 3173 files, 39.7 GB Performance: 150.3 MB/minute Duration: 07:16:45 (02:46:45 idle/loading/preparing) What is this "inconsistency" it is complaining about, and then deciding that the rebuild went just fine!?? There were three tapes in this set. 1 Quote Link to comment Share on other sites More sharing options...
Lennart_T Posted April 23, 2012 Report Share Posted April 23, 2012 I am rebuilding the catalogs on some Retro 6.1 tapes to see if I can restore from them. OK, how did the actual restore work out? Quote Link to comment Share on other sites More sharing options...
Don Lee Posted April 24, 2012 Author Report Share Posted April 24, 2012 (edited) Rebuilding tapes is not a quick operation. I have rebuilt 4 sets, and am starting the copy operations. All 4 tape sets ((9 tapes) have the "!Backup Set format inconsistency (x at yyyyyyyyyy)" messages. x is 2, or 4. yyyyyy is a large number. (hundreds of millions) Also of note: The last rebuild had quite a few of these messages, and came out "zero length", with no files in it, and zero "size' according to the "Media Set" display. The ones that rebuilt with content looked like this in the log: !Backup Set format inconsistency (4 at 315766848) !Backup Set format inconsistency (4 at 315762384) 4/23/12 2:58:15 PM: Execution completed successfully Completed: 3154 files, 34 GB Performance: 150.1 MB/minute Duration: 03:54:35 (00:03:02 idle/loading/preparing) The empty one looked like this: !Backup Set format inconsistency (2 at 316258912) !Backup Set format inconsistency (2 at 316258944) !Backup Set format inconsistency (2 at 316256624) 4/23/12 9:53:27 PM: Execution completed successfully Duration: 06:52:36 (02:52:09 idle/loading/preparing) Starting the reload/copy..... Edited April 24, 2012 by iCompute Quote Link to comment Share on other sites More sharing options...
Don Lee Posted April 30, 2012 Author Report Share Posted April 30, 2012 I have now rebuilt about 9 of these tape sets, and I'm getting 3 results. 1. The rebuild works. 2. The rebuild fails with a strange tape error, and puts crap in the log, but seems to recover, and the resultant catalog has about 80% of the files it is supposed to have. (Is this good or bad? Hard to say.) 3. The rebuild process ends up with quite a few (50-100+?) errors roughly of the form below. The catalog is empty (zero files) when the rebuild completes. Of the 8 or so sets I have re-built so far, I have seen 3 sets rebuild with no resultant files (#3, above). I have seen one of #2, (and suspect that its problem is tied to its tape, which generates tape drive errors. It may be going bad.) The rest were all #1, and went smoothly. The long-running operation (hours!) resulting in no output is a frustration, and I smell a bug. It appears that I can tell when this is going to happen, because the status window in "activity" will normally show its progress, as files are counted, and an estimate of the data volume catalogued is displayed (as well as the occasional file name). The telltale sign of failure is the "Backup Set format" message below, plus no counters in the status window. Note that the "Backup Set format inconsistency" is pretty common. I don't think I have more than one or two rebuilds that did not produce at least a few of these. -dgl- + Executing Rebuild at 4/29/12 (Activity Thread 1) To Backup Set iCompute11 [004]... !Backup Set format inconsistency (2 at 1884360320) !Backup Set format inconsistency (2 at 316812800) !Backup Set format inconsistency (2 at 316813904) !Backup Set format inconsistency (2 at 316809200) !Backup Set format inconsistency (2 at 316801808) !Backup Set format inconsistency (2 at 316805200) !Backup Set format inconsistency (2 at 316807792) !Backup Set format inconsistency (2 at 316802944) !Backup Set format inconsistency (2 at 316802976) !Backup Set format inconsistency (2 at 316808880) !Backup Set format inconsistency (2 at 316793568) !Backup Set format inconsistency (2 at 316796112) !Backup Set format inconsistency (2 at 316797808) !Backup Set format inconsistency (2 at 316799136) !Backup Set format inconsistency (2 at 316792976) !Backup Set format inconsistency (2 at 316794816) !Backup Set format inconsistency (2 at 316800336) !Backup Set format inconsistency (2 at 316800368) !Backup Set format inconsistency (2 at 316790576) !Backup Set format inconsistency (2 at 316789136) !Backup Set format inconsistency (2 at 316788320) !Backup Set format inconsistency (2 at 316785760) !Backup Set format inconsistency (2 at 316784672) !Backup Set format inconsistency (2 at 316775504) !Backup Set format inconsistency (2 at 316784240) !Backup Set format inconsistency (2 at 316778976) !Backup Set format inconsistency (2 at 316779008) !Backup Set format inconsistency (2 at 316775968) !Backup Set format inconsistency (2 at 316777776) !Backup Set format inconsistency (2 at 316778112) !Backup Set format inconsistency (2 at 316774160) !Backup Set format inconsistency (2 at 316773072) !Backup Set format inconsistency (2 at 316771968) !Backup Set format inconsistency (2 at 316770832) !Backup Set format inconsistency (2 at 316768720) !Backup Set format inconsistency (2 at 316768752) !Backup Set format inconsistency (2 at 316763680) !Backup Set format inconsistency (2 at 316765632) !Backup Set format inconsistency (2 at 316761776) !Backup Set format inconsistency (2 at 316762400) !Backup Set format inconsistency (2 at 316759728) !Backup Set format inconsistency (2 at 316760880) !Backup Set format inconsistency (2 at 316764416) !Backup Set format inconsistency (2 at 316756528) !Backup Set format inconsistency (2 at 316756560) !Backup Set format inconsistency (2 at 316754336) !Backup Set format inconsistency (2 at 316753280) !Backup Set format inconsistency (2 at 316752400) !Backup Set format inconsistency (2 at 316741104) !Backup Set format inconsistency (2 at 316747872) !Backup Set format inconsistency (2 at 316745216) !Backup Set format inconsistency (2 at 316749680) !Backup Set format inconsistency (2 at 316743488) !Backup Set format inconsistency (2 at 316743520) !Backup Set format inconsistency (2 at 316748960) !Backup Set format inconsistency (2 at 316740768) !Backup Set format inconsistency (2 at 316739888) !Backup Set format inconsistency (2 at 316737648) !Backup Set format inconsistency (2 at 316736560) !Backup Set format inconsistency (2 at 316735520) !Backup Set format inconsistency (2 at 316734400) !Backup Set format inconsistency (2 at 316728720) !Backup Set format inconsistency (2 at 316728752) !Backup Set format inconsistency (2 at 316729696) !Backup Set format inconsistency (2 at 316732640) !Backup Set format inconsistency (2 at 316730672) !Backup Set format inconsistency (2 at 316727248) !Backup Set format inconsistency (2 at 316731968) !Backup Set format inconsistency (2 at 316723888) !Backup Set format inconsistency (2 at 316722864) !Backup Set format inconsistency (2 at 316720960) !Backup Set format inconsistency (2 at 316720992) !Backup Set format inconsistency (2 at 316718624) !Backup Set format inconsistency (2 at 316714704) !Backup Set format inconsistency (2 at 316716768) !Backup Set format inconsistency (2 at 316715024) !Backup Set format inconsistency (2 at 316714000) !Backup Set format inconsistency (2 at 316708720) !Backup Set format inconsistency (2 at 316709952) !Backup Set format inconsistency (2 at 316711648) !Backup Set format inconsistency (2 at 316711680) !Backup Set format inconsistency (2 at 316706960) !Backup Set format inconsistency (2 at 316705904) !Backup Set format inconsistency (2 at 316705088) !Backup Set format inconsistency (2 at 316702800) !Backup Set format inconsistency (2 at 316701824) !Backup Set format inconsistency (2 at 316698080) !Backup Set format inconsistency (2 at 316701488) !Backup Set format inconsistency (2 at 316696496) !Backup Set format inconsistency (2 at 316696528) !Backup Set format inconsistency (2 at 316692544) !Backup Set format inconsistency (2 at 316693856) !Backup Set format inconsistency (2 at 316694160) !Backup Set format inconsistency (2 at 316689712) !Backup Set format inconsistency (2 at 316688720) !Backup Set format inconsistency (2 at 316687728) !Backup Set format inconsistency (2 at 316686608) !Backup Set format inconsistency (2 at 316684816) !Backup Set format inconsistency (2 at 316684848) !Backup Set format inconsistency (2 at 316678272) !Backup Set format inconsistency (2 at 316680736) !Backup Set format inconsistency (2 at 316681072) !Backup Set format inconsistency (2 at 316679296) !Backup Set format inconsistency (2 at 316676544) !Backup Set format inconsistency (2 at 316677696) !Backup Set format inconsistency (2 at 316681984) !Backup Set format inconsistency (2 at 316674416) !Backup Set format inconsistency (2 at 316674448) !Backup Set format inconsistency (2 at 316671776) !Backup Set format inconsistency (2 at 316672880) !Backup Set format inconsistency (2 at 316649152) !Backup Set format inconsistency (2 at 316651440) !Backup Set format inconsistency (2 at 316673760) !Backup Set format inconsistency (2 at 316648208) !Backup Set format inconsistency (2 at 316647120) !Backup Set format inconsistency (2 at 316645360) !Backup Set format inconsistency (2 at 316645392) !Backup Set format inconsistency (2 at 316642960) !Backup Set format inconsistency (2 at 316633616) !Backup Set format inconsistency (2 at 316632240) !Backup Set format inconsistency (2 at 316639088) !Backup Set format inconsistency (2 at 316638096) !Backup Set format inconsistency (2 at 316637584) !Backup Set format inconsistency (2 at 316634240) !Backup Set format inconsistency (2 at 316640896) !Backup Set format inconsistency (2 at 316640928) !Backup Set format inconsistency (2 at 316630528) !Backup Set format inconsistency (2 at 316629504) !Backup Set format inconsistency (2 at 316628640) !Backup Set format inconsistency (2 at 316625696) !Backup Set format inconsistency (2 at 316624768) !Backup Set format inconsistency (2 at 316617104) !Backup Set format inconsistency (2 at 316624432) !Backup Set format inconsistency (2 at 316619808) !Backup Set format inconsistency (2 at 316619840) !Backup Set format inconsistency (2 at 316617584) !Backup Set format inconsistency (2 at 316619296) !Backup Set format inconsistency (2 at 316623600) !Backup Set format inconsistency (2 at 316614640) !Backup Set format inconsistency (2 at 316613600) !Backup Set format inconsistency (2 at 316612512) !Backup Set format inconsistency (2 at 316611392) !Backup Set format inconsistency (2 at 316609536) !Backup Set format inconsistency (2 at 316609568) !Backup Set format inconsistency (2 at 316605104) !Backup Set format inconsistency (2 at 316606384) !Backup Set format inconsistency (2 at 316603216) !Backup Set format inconsistency (2 at 316603872) !Backup Set format inconsistency (2 at 316600528) !Backup Set format inconsistency (2 at 316602032) !Backup Set format inconsistency (2 at 316605808) !Backup Set format inconsistency (2 at 316596560) !Backup Set format inconsistency (2 at 316596592) !Backup Set format inconsistency (2 at 316594096) !Backup Set format inconsistency (2 at 316593008) !Backup Set format inconsistency (2 at 316592160) !Backup Set format inconsistency (2 at 316581024) !Backup Set format inconsistency (2 at 316590448) !Backup Set format inconsistency (2 at 316585376) !Backup Set format inconsistency (2 at 316587216) !Backup Set format inconsistency (2 at 316581856) !Backup Set format inconsistency (2 at 316581888) !Backup Set format inconsistency (2 at 316589952) !Backup Set format inconsistency (2 at 316580656) !Backup Set format inconsistency (2 at 316579728) !Backup Set format inconsistency (2 at 316577328) !Backup Set format inconsistency (2 at 316576272) !Backup Set format inconsistency (2 at 316575216) !Backup Set format inconsistency (2 at 316574176) !Backup Set format inconsistency (2 at 316570688) !Backup Set format inconsistency (2 at 316570720) !Backup Set format inconsistency (2 at 316567984) !Backup Set format inconsistency (2 at 316572352) !Backup Set format inconsistency (2 at 316569440) !Backup Set format inconsistency (2 at 316567360) !Backup Set format inconsistency (2 at 316571040) !Backup Set format inconsistency (2 at 316559296) !Backup Set format inconsistency (2 at 316559920) !Backup Set format inconsistency (2 at 316561536) !Backup Set format inconsistency (2 at 316561568) !Backup Set format inconsistency (2 at 316554864) !Backup Set format inconsistency (2 at 316556032) !Backup Set format inconsistency (2 at 316556880) !Backup Set format inconsistency (2 at 316553008) !Backup Set format inconsistency (2 at 316551632) !Backup Set format inconsistency (2 at 316550512) !Backup Set format inconsistency (2 at 316549072) !Backup Set format inconsistency (2 at 316547200) !Backup Set format inconsistency (2 at 316547232) !Backup Set format inconsistency (2 at 316540656) !Backup Set format inconsistency (2 at 316541312) !Backup Set format inconsistency (2 at 1884360320) !Backup Set format inconsistency (2 at 316543344) !Backup Set format inconsistency (2 at 316538304) !Backup Set format inconsistency (2 at 316539440) !Backup Set format inconsistency (2 at 316544144) !Backup Set format inconsistency (2 at 316536256) !Backup Set format inconsistency (2 at 316535136) !Backup Set format inconsistency (2 at 316533088) !Backup Set format inconsistency (2 at 316533120) !Backup Set format inconsistency (2 at 316530000) !Backup Set format inconsistency (2 at 316525264) !Backup Set format inconsistency (2 at 316527520) !Backup Set format inconsistency (2 at 316525504) !Backup Set format inconsistency (2 at 316523888) !Backup Set format inconsistency (2 at 316518592) !Backup Set format inconsistency (2 at 316519744) !Backup Set format inconsistency (2 at 316521584) !Backup Set format inconsistency (2 at 316521616) !Backup Set format inconsistency (2 at 316516976) !Backup Set format inconsistency (2 at 316515856) !Backup Set format inconsistency (2 at 316514976) !Backup Set format inconsistency (2 at 316512656) !Backup Set format inconsistency (2 at 316511648) !Backup Set format inconsistency (2 at 316508240) !Backup Set format inconsistency (2 at 316511248) !Backup Set format inconsistency (2 at 316507104) !Backup Set format inconsistency (2 at 316507136) !Backup Set format inconsistency (2 at 316503424) !Backup Set format inconsistency (2 at 316504736) !Backup Set format inconsistency (2 at 316505072) !Backup Set format inconsistency (2 at 316499248) !Backup Set format inconsistency (2 at 316496896) !Backup Set format inconsistency (2 at 316498864) !Backup Set format inconsistency (2 at 316500048) !Backup Set format inconsistency (2 at 316493472) !Backup Set format inconsistency (2 at 316493504) !Backup Set format inconsistency (2 at 316495744) !Backup Set format inconsistency (2 at 316492352) !Backup Set format inconsistency (2 at 316491600) !Backup Set format inconsistency (2 at 316489552) !Backup Set format inconsistency (2 at 316488416) !Backup Set format inconsistency (2 at 316487376) !Backup Set format inconsistency (2 at 316485232) !Backup Set format inconsistency (2 at 316483552) !Backup Set format inconsistency (2 at 316483584) !Backup Set format inconsistency (2 at 316481200) !Backup Set format inconsistency (2 at 316479520) !Backup Set format inconsistency (2 at 316478640) !Backup Set format inconsistency (2 at 316476256) !Backup Set format inconsistency (2 at 316475168) !Backup Set format inconsistency (2 at 316474032) !Backup Set format inconsistency (2 at 316472976) !Backup Set format inconsistency (2 at 316471184) !Backup Set format inconsistency (2 at 316471216) (still running) Quote Link to comment Share on other sites More sharing options...
Don Lee Posted April 30, 2012 Author Report Share Posted April 30, 2012 (edited) More news on this. Of 6 tape backup sets, I have now gotten one "poisoned" set, two successes, and 2 sets that get lots of "Backup set format inconsistency" errors, and zero-length catalogs, after appearing to read the entire set. I got suspicious when the 6th set rebuild started and immediately got a couple of "inconsistency" errors - within seconds of the start of the read of the tape. I stopped and restarted the Retro engine. I power cycled the tape drive. (HP Storage works DDS Dat-72 drive - USB) Powered up the drive. Started the Engine I re-started the re-build of the 6th tape set. THIS time, the summary window in "Activity" is showing progress - file counts, file names, etc. No more "inconsistency" errors - at least so far. It looks to me like there was something in the tape drive or engine that was "stuck. I note that the problems started when I got the errors on the "poisoned" tape ( poisoned 6.1 tape cannot be rebuilt ). If this happens again, I will try cycling *just* the engine, or *just* the drive to see where the "stuck" component was. Edited May 4, 2012 by iCompute Quote Link to comment Share on other sites More sharing options...
Don Lee Posted May 4, 2012 Author Report Share Posted May 4, 2012 I got 3 sets in a row that started out with no files "counted' in the summary, and some with "inconsistency" errors in the logs. I decided to see if this is a problem with the engine or tape drive, so I power cycled the drive, and re-read the tape. The second rebuild - same tape - worked. (!) (sort-of... see below) This tells me that power cycling the drive either produced different data afterwards, or something about the power cycling whacked the Retro engine up-side the head and made it happier. In any case, the set is rebuilding, and not showing the inconsistency errors now. I'm going to back up and re-do the other two to see if they now produce different data, too. I'm still wondering what this message means..... !Backup Set format inconsistency (2 at 316745216) I note that the end of the re-build is showing the "out of memory" problem. Here is the tail of the log: ... ... !Backup Set format inconsistency (4 at 315909536) !Backup Set format inconsistency (4 at 315905136) !Backup Set format inconsistency (4 at 315911616) SixOneTree:TAddAbstractFile failed mca_ns_replica.la !Not enough application memory 5/4/12 11:30:21 AM: Execution incomplete Completed: 529856 files, 43.9 GB Performance: 202.5 MB/minute Duration: 09:45:19 (06:03:40 idle/loading/preparing) This does not appear to be a "bad spot" on the tape. I was sitting here, and did not hear the tape drive repositioning, which would be expected if it were having trouble with the media. Quote Link to comment Share on other sites More sharing options...
Don Lee Posted May 7, 2012 Author Report Share Posted May 7, 2012 I have identified one cause for different results on different operations, the apparent race condition when loading the tape drive, as documented here. Quote Link to comment Share on other sites More sharing options...
Don Lee Posted May 22, 2012 Author Report Share Posted May 22, 2012 For the record, I still get lots of these errors ("!Backup Set format inconsistency (4 at 123123123)") when recataloging tapes. It would be nice to know what the error means. It can't be a _good_ thing... Quote Link to comment Share on other sites More sharing options...
twickland Posted May 22, 2012 Report Share Posted May 22, 2012 This error indicates that Retrospect detected problems reading the original backup tapes. Back in the Retro 6.1 and earlier days, it almost always meant problems with the physical tape member itself, though it could also indicate problems with the tape drive or communications bus. You seem to have uncovered evidence that suggests it might also be caused by issues in Retro 8/9, perhaps in communication with the tape drive but perhaps also in interpretation of the old 6.1 data. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.