Jump to content

8.2 catalog recovery


Recommended Posts

Hello,

I've been running Retrospect 8.x since last year, backing up to a Droboshare NAS - 7Tb. The performance has never been great - 2 weeks for the initial backup of 1Tb, but I put that down to the Droboshare throughput.

 

My backup systems are a macpro ( also running the retrospect engine) and 3 laptops.

 

Due to a system crash and accidently clicking on the 'lost' checkbox for my latest mediaset, I think my option is now limited to execute a rebuild a disk media set.

 

I initially tried this and waited for the add member dialog for 4 days! During this time, my Macpro was 100% busy on one of the CPUs and did very little in terms of network or disk IO.

 

Since then I have re-installed Retrospect and attached the disk locally to the macpro.

 

I have the following folder structure:

 

/Volumes/Drobo/Retrospect/Media Set A

containing

1-Media Set A containing 2405 .rdb files

3-Media Set A containing 4148 .rdb files

4-Media Set A containing 2 .rdb files

 

/Volumes/Drobo/Retrospect/Media Set B

containing

1-Media Set B containing 59 .rdb files

 

I also followed the thread and changed the nice value from 0 to -15.

 

Now, I click on rebuild, I see a list of disks. I click on the Drobo and select Retrospect. At this point the rebuild a Disk Media Set box reappears and 20 seconds later, the interface is unresponsive - retro engine is once again doing 100% CPU. Disk activity is a few KB per second.

 

The macpro has 5Gb ram - currently 2Gb free.

 

Obviously I need to get this working again. I have a couple of questions.

 

1 - is using a NAS or local raid and having many thousands of .rdb files a real problem - I did not want to start grooming backups as yet?

 

2 - What else could be causing this hanging dialog?

 

3 - Is there any way to view the retrospect log outside of the retrospect console ?

 

 

best regards

Steve Groom

 

Link to comment
Share on other sites

...clicking on the 'lost' checkbox for my latest mediaset...

 

I have the following folder structure:

 

/Volumes/Drobo/Retrospect/Media Set A

containing

1-Media Set A containing 2405 .rdb files

3-Media Set A containing 4148 .rdb files

4-Media Set A containing 2 .rdb files

 

 

The "lost" checkbox is for Members of Media Sets. Looking at the above, it appears you have multiple Members of the same Media Set stored on the same volume. This is unnecessary and inefficient. Additional Members are designed to be created as a way to infinitely increase your backup storage capability; keeping them on the same physical (or even logical) drive doesn't help with anything.

 

I is also unclear exactly where the "2-Media Set A" is; do you still have that folder? Were you storing that on the same volume as the others the whole time?

 

"Lost" is generally when the physical media is missing (tape run over by a truck, DVD scratched, hard drive crashed, etc). If it's really lost, rebuilding doesn't help with anything. It's only necessary to rebuild if you actually _find_ the Member that had been previously marked as Missing.

 

I don't know what's causing your hangup; Retrospect is quite finicky when displaying these sorts of dialogs. Since you have multiple Members of the same Media Set within the same Retrospect folder, if this were me I'd try breaking out the folder structure to see if Retrospect can deal with fewer items to scan through.

 

Perhaps I'd change my folder structure to:

 

/Volumes/Drobo/[color:red]Red[/color]/Retrospect/Media Set A/1-Media Set A

/Volumes/Drobo/White/Retrospect/Media Set A/3-Media Set A

/Volumes/Drobo/[color:blue]Blue[/color]/Retrospect/Media Set A/4-Media Set A

 

Then at the select Member dialog, I'd drill down to the /Red/Retrospect folder first, and see if it doesn't choke. Then add the other two when prompted ("Add Additional Members") and proceed with the rebuild.

 

Worth a shot.

 

 

Dave

Link to comment
Share on other sites

 

It is also unclear exactly where the "2-Media Set A" is; do you still have that folder? Were you storing that on the same volume as the others the whole time?

There was a crash and a recovery about a year ago and I could no longer add to 1-Media Set A, eventually I got things to work, but I already had created and deleted 2-Media Set A and started 3-Media Set A.

 

 

"Lost" is generally when the physical media is missing (tape run over by a truck, DVD scratched, hard drive crashed, etc). If it's really lost, rebuilding doesn't help with anything. It's only necessary to rebuild if you actually _find_ the Member that had been previously marked as Missing.

 

In this case I accidently clicked on Lost - against 3-Media Set A , where most of the backups are stored.

 

 

 

I don't know what's causing your hangup; Retrospect is quite finicky when displaying these sorts of dialogs. Since you have multiple Members of the same Media Set within the same Retrospect folder, if this were me I'd try breaking out the folder structure to see if Retrospect can deal with fewer items to scan through.

 

Perhaps I'd change my folder structure to:

 

/Volumes/Drobo/Red/Retrospect/Media Set A/1-Media Set A

/Volumes/Drobo/White/Retrospect/Media Set A/3-Media Set A

/Volumes/Drobo/Blue/Retrospect/Media Set A/4-Media Set A

 

Then at the select Member dialog, I'd drill down to the /Red/Retrospect folder first, and see if it doesn't choke. Then add the other two when prompted ("Add Additional Members") and proceed with the rebuild.

 

 

I have tried this - I created Drobo/One, Drobo/Three and Drobo/Four with the structure as you indicated.

 

When the select a disk member folder is displayed I cannot drill down past the first level, so I proceed with One, Two or Four.

 

In each case, Retrospect returns to the add member dialog box and then hangs again.

 

The drobo disk is formatted as HFS+, are there perhaps permissions that should be corrected for the rertrospect files and folders ?

 

regards

Steve

 

Link to comment
Share on other sites

A couple of updates.

 

When using repair or rebuild I can only browse to the first level folder names on the locally attached (HFS+ formatted) Drobo.

 

When I try to expand the /Volumes/Drobo/Retrospect folder, I see the following in the console:

 

10/10/2010 11:43:25 com.roxio.RetroEngine[1398] com.roxio.RetroEngine !Error: Scanning incomplete, error -1101 ( file/directory not found).

10/10/2010 11:43:25 Retrospect[1361] Retrospect ServerFolder::children exception: LiveVolumeTree::ScanFolder failed; error = -1101

 

And the dialog seems to show the folder is open but empty.

 

Here are the permissions:

macpro:Drobo steve$ ls -lae /Volumes/Drobo/Retrospect/

total 16

drwxr-xr-x 6 root admin 204 9 Oct 12:05 .

drwxr-xr-x 21 root wheel 782 9 Oct 23:19 ..

-rwxr--r--@ 1 root admin 6148 9 Oct 12:05 .DS_Store

-rwxr-xr-x 1 root admin 0 11 Jul 2009 EMC

drwxr-xr-x 3 root admin 102 9 Oct 23:23 Media Set A

 

 

If I copy a few of the .rdb files to another disk I can execute a rebuild - but of course I do not have another 7Tb disk lying around to do the complete job :-(

 

Hope this helps with finding a solution.

 

regards

Steve

 

Link to comment
Share on other sites

Real progress now.

 

Found this thread:

http://forums.dantz.com/printpost.php?tid/29862/

discussing the -1101 error.

 

In my case I found in the sources section a disk called drobo - the locally attached one (yellow icon) and another also called drobo referring the the disk when it was a NAS (white icon).

 

I unmounted the drobo, removed the /Volumes/Droboshare mount and removed the NAS source in Restrospect.

 

I then was able to use the repair dialog to select the three folders containing parts of Media Set A!!

 

I then tried a restore - this failed with the message

Reported missing by user: "3-Media Set A"

 

I guess I now have to do the complete catalog rebuild.

 

Should I take this opportunity to put all the .rdb files into a single folder 1-Media Set A to tidy up the Drobo ?

 

regards

Steve

Link to comment
Share on other sites

I've never done this (had multiple members for a media set), but...

 

are the .rdb files in the other member folders starting over with 0000000.rdb? Or do they continue with the numbering where member #1 ended?

 

If so, you might be able to move everything to one folder.

 

 

If that doesn't work -- and you have the space -- you could rebuild this set with all the members and then do a "copy media set" to copy everything to a new media set (with one member) if you want things a bit cleaner.

Link to comment
Share on other sites

Thanks' for the reply. The .rdb files are numbered sequentailly from 0 with no duplicates and only a couple missing. I have a catalog rebuild running at the moment - it should complete in around 30 hours for around 3.2Tb. I'll try to reorganise the files into a single folder after that if all goes well.

 

regards

Steve

 

 

Link to comment
Share on other sites

Status update:

 

Hello,

The rebuild of the catalog file has now been running for 99 hours.

 

The amount of data read is 3.6Tb, which is a little more the total backup disk size !!

 

I am getting concerned that the rebuild is looping!

 

I see the follow status info:

Building Catalog File

File

ipodUpdater

several jpg file names

then Resynchronising ( This may take some time)

 

after a couple of minutes, the sequence of file names seems to repeat.

 

I think this has been happening for a while.

 

In the mac console I see this repeated

14/10/2010 18:51:17 Retrospect[1531] Retrospect Script::verify exception: MacRequestor::Request: connection already closed

 

every 10 to 30 seconds

 

 

What can I do to further disgnose this ?

What should I do next?

 

regards

Steve

 

Link to comment
Share on other sites

I'd try rebuilding that catalog file again with advanced logging on:

 

Edit the "Retro.ini" file inside the RetrospectEngine.bundle so that the line says:

 

SetDevicesLogging=7

 

Then stop/restart the engine.

 

The next time you rebuild a catalog, it'll flood the log with a lot of step-by-step detail about the process of the catalog rebuild.

 

That will tell you specifically if it's looping or not.

Link to comment
Share on other sites

To conclude.

When I saw that the catalog rebuild was looping, I did not kill the recatalog process, rather I shut down the computer to give it a bit of a rest.

 

Imagine my surprise the following day, when I saw that retrospect was again trying to run a backup, but rather than prompting for media, it was executing the rest of the catalog rebuild. This does not appear as a separate job, but rather as the first step in the current backup. I checked and could see that it was really progressing, so I left it alone and now after a furter 50 hours, the rebuild is complete as are several backups.

 

Thanks to everyone for the helpful suggestions. I will try copying the backup to a bigger (faster) Drobo in a month or so.

 

best regards

Steve

Link to comment
Share on other sites

Well,

it seems that it did not really work. I have just started the backup of my main data hard disk - 1.5Tb of data where most of the content was backed up before this problem started.

The disk backup is now running and indicates that 958Gb still needs to be backed-up. I suspect that this means that it has failed to catalog all the .rdb files.

Is there a report that can show what backups exist in what parts of the catalog?

 

I'm now trying the restore -> restore from which backupsto try to recover a catalog from August but that is still scanning, does this normally take a long time ( >30mins so far still spinning).

 

hmm.

 

 

regards

Steve

Link to comment
Share on other sites

You never indicated above that you tried to rebuild the catalog with advanced logging on (or that you actually ever *had* a successful catalog rebuild.)

 

 

I'd still recommend you turn on the advanced logging and rebuild again and find out where the problem is.

 

Worse case? The logging will tell you where things start "looping" and you can just delete the possibly bad .rdb file, then recatalog again without it -- you'll just have lost the files backed up within that one .rdb file.

Link to comment
Share on other sites

Hello,

started over with the recatalog and now logging is switched on. It looks like recatlog is looping again.

 

From the activities menu I can see >19000 errors, 736000 files copied.

 

I can display the log, but I can't see very much in it - can you tell me where the log is located so that I can attach it to this thread?

 

Most of the lines are like these:

 

.....

 

devSwapHeaderCheck(1): curver 0x24b3f450

devSwapHeaderCheck(1): curver 0x24b3f720

devSwapHeaderCheck(1): curver 0x24b3f9f0

MacVLoc::vlocRDiskDoGetFInfo: >>>>>> Warning!!! file(AA006909.rdb) datapos(0) is lower so it's neglected!

devSwapHeaderCheck(1): curver 0x24b40320

MacVLoc::vlocRDiskDoGetFInfo: >>>>>> Warning!!! file(AA006910.rdb) datapos(1024704) is lower so it's neglected!

devSwapHeaderCheck(1): curver 0x24b40ca0

MacVLoc::vlocRDiskDoGetFInfo: >>>>>> Warning!!! file(AA006911.rdb) datapos(1178304) is lower so it's neglected!

devSwapHeaderCheck(1): curver 0x24b41620

.....

devSwapHeaderCheck(1): curver 0x24b7e990

devSwapHeaderCheck(1): curver 0x24b7ec60

devSwapHeaderCheck(1): curver 0x24b7ef30

devSwapHeaderCheck(1): curver 0x24b7f200

devSwapHeaderCheck(1): curver 0x24b7f4d0

.....

devSetExec: fully released devRef 0x7067a670

devPoll: change occurred, devRef 0x7067a670 loRef 0x7067a670

devSetExec: fully released devRef 0x7067a670

devSetExec: fully released devRef 0x7067a670

devSetExec: fully released devRef 0x7067a670

devSetExec: fully released devRef 0x7067a670

 

 

regards

Steve

Link to comment
Share on other sites

Logging like this is in the basic "operations_log.utx" file.

 

I have a couple of thoughts:

 

How many .rdb files are within your media set? Are the last three 6909-6911? Or do you have ones after that?

 

You might consider just moving those 3 .rdb files out and recatalogging again. If you don't need the data in them (based on the dates on the files), then just trash them.

 

If you *do* need the data on them, I believe you could try moving them to their own folder and attempting to recatalog a new media set just containing those files.

 

I had an error once after I groomed a media set where the log indicated an error in the set. Any subsequent recatalogging would continue to generate an error. Eventually, I turned on logging, found the "bad" file -- which was in the middle of all the .rdb files -- and just trashed it and recatalogged again without issue.

 

It depends on your data retention policy as to whether you feel comfortable doing this or not.

 

 

Link to comment
Share on other sites

Hello Maser,

I did not include all the errors from the log - I think there are two groups of files AA002403 to AA002999 and AA006909 to AA006999 with the neglected error message in the logs.

 

The files are currently organised into the following 8 folders:

 

/Volumes/Drobo/Retrospect_10/Retrospect/Media Set A/1-Media Set A

containing the first 1000 files.

 

2-Media Set A

containging the next 1000 files and so on up to

8-Media Set A containing AA007000 to AA007552.rdb

 

As the files were previously in rather a muddled set of folders, these are now better organised. I'd like to keep the number of .rdb files in each folder to 1000 as the Drobo seems to struggle when there are thousands of files in a directory, but am not sure if this is the best way to achieve this.

 

But it is odd that both the groups of errors ended with the last .rdb file in a particular folder.

 

There are 7550 .rdb files in the backup - no gaps and no grooming has ever run.

 

The backup contents are a mix of two Windows Laptops, two Macbooks and the Macpro that is also running Retrospect.

 

Waiting on your thoughts.

 

thanks and best regards

Steve

Link to comment
Share on other sites

I honestly have no experience with a Drobo, so I can't tell you if there are issues with the file system on it or not.

 

What you *might* try is seeing if you can rebuild a new media set for each set of 1000 files. If so, then maybe do a "copy media set" to put them all into one new set?

Link to comment
Share on other sites

Hello,

I think I have the catalog recovery bug as mentioned in this thread:

POST: RETROSPECT 7.7 CAN'T GROOM OR RECREATE CATALOGS FOR BACKUPS MADE BY 7.6 (TOPIC#34762)

 

But with 8.0 -> 8.2

 

Earlier I mentioned the recatalog had the same error on files

MacVLoc::vlocRDiskDoGetFInfo: >>>>>> Warning!!! file(AA002403.rdb) datapos(0) is lower so it's neglected!

devSwapHeaderCheck(1): curver 0x26a96560

through AA002999.rdb ( the last one in that folder ).

 

I tried to put all the .rdb files into a single folder and now, the recatalog has the above error against AA002403.rdb through AA005261.rdb and AA006909.rdb through AA007552.rdb - that is a huge chunk of all the backup.

 

These files were created over the past 16 months via Retrospect 8.0 through Retrospect 8.2.

 

It looks like I'll have to open a bug report w/ whomever owns Retrospect this week.

 

regards

Steve

 

Link to comment
Share on other sites

I think I have the catalog recovery bug as mentioned in this thread:

POST: RETROSPECT 7.7 CAN'T GROOM OR RECREATE CATALOGS FOR BACKUPS MADE BY 7.6 (TOPIC#34762)

 

But with 8.0 -> 8.2

Nope. That thread is for the Windows code base, which is different from the 8.x code base. They really haven't converged quite yet.

 

These files were created over the past 16 months via Retrospect 8.0 through Retrospect 8.2.

Actually, I suspect that might be the problem. Issues have been seen with bit rot carried forward from 8.0 into 8.2.

 

Let me suggest a different approach, if you can pull it (or a variant) off.

 

Start a new media set, try to transfer (oops, terminology change with 8.x - "copy") from the older media sets into the new media set. That will do a catalog creation on the fly for the new media set. It may be that Retrospect is confused about the old media sets because of their ancestry.

 

Get the mindset that you have a database (that's what the media sets are) with a possibly-corrupted database index (that's what the catalog is), and do database operations to move the data from one database to the other. Then, once you've got a good database with a good index, it might be possible to Groom, Steve.

 

Russ

Link to comment
Share on other sites

I don't think you can transfer from an old media set to a new media set if you don't have a valid catalog file for the old media set, can you?

 

I have media sets running that were created in 8.0 and beyond that I've rebuilt the catalog and groomed from without issue. I don't necessarily think it's expected "bit rot", but YMMV.

 

the fact that the same files keep complaining mean to me that there's something wrong with the files for some reason.

 

You could try just moving 2403 and 2404 to a new folder and rebuilding a catalog *just* from those two files.

 

My guess is that won't work, either, but it's worth a shot.

 

 

Also, have you confirmed you can *Finder copy* the "bad" files from your drive to another drive?

Edited by Guest
Link to comment
Share on other sites

I don't think you can transfer from an old media set to a new media set if you don't have a valid catalog file for the old media set, can you?

Nope. I was trying to push the OP into a direction where some of the members of the media set (the good ones) could be used to extract whatever was possible. It was also unclear to me, because this thread has gotten long, whether the problem was grooming caused - perhaps the catalog (or a saved catalog) worked well enough for a transfer (copy).

 

I have media sets running that were created in 8.0 and beyond that I've rebuilt the catalog and groomed from without issue. I don't necessarily think it's expected "bit rot", but YMMV.

From everything I have seen, if the set can be groomed, then the integrity of the catalog is OK. I was using the term "bit rot" to cover the mysterious and still unknown causes of catalog corruption that continue to happen. Who knows? AFAIK, it's mystery garbage blocks from space. I'm willing to chalk it up to bugs that have subsequently been fixed as 8.x has matured, and I'm not sure that it's productive (or even possible) to have a post-mortem analysis of what the cause was. At this point, I still treat 8.2 and its predecessors as an evolving alpha, beta series, not yet solid enough for my data, at least from my tests, and it takes a LOT of my time to shake out a release with regression testing. Perhaps with the 8.9 or 9.1 release... Hope springs eternal ...

 

the fact that the same files keep complaining mean to me that there's something wrong with the files for some reason.

 

You could try just moving 2403 and 2404 to a new folder and rebuilding a catalog *just* from those two files.

 

My guess is that won't work, either, but it's worth a shot.

Agreed, as long as it's not my time that's involved.

 

Also, have you confirmed you can *Finder copy* the "bad" files from your drive to another drive?

Well, that's something to try, but I haven't seen anything in this thread to indicate that the problem is filesystem related.

 

A concern to me for all of this testing is that some have become so accustomed to engine crashes (and, rarely, kernel panic caused by Retrospect, despite what Robin may think), that people may be hitting that "power reset" button, which upsets Unix a lot, even considering journaled filesystems. In the absence of a ZFS-type filesystem, garbage can occur anywhere in such a scenario, which is why I do my testing on an isolated testbed, away from my data.

 

Russ

Link to comment
Share on other sites

Recap:

Retrospect 8.x used with NAS storage ( Drobo ) for almost two years.

Backup volume is now 4.5Tb.

Retrospect updates installed without problems.

Now running 8.2

Catalog corruption occured - don't know the cause.

Grooming has never been enabled.

Backups are a mix of Windows and Macs

 

Catalog rebuild runs into a loop and the same files are mentioned over and over.

 

The catalog has been omitted from the backup for over a year! So recovering the last months data shows no useful catalog.

 

Opened an incident at Roxio support

Tkt: 1071619

Status:Open

Date: 10/27/2010

Issue Description: Technical Support | Retrospect for Mac 8.x | Catalog Rebuild

 

So far no replies at all - this is not support!! Tried telephone support but that only works in office hours and I am using Retrospect at home.

Also the phone support number costs GBP 1:00 per minute !! - I can't run that kind of charge through my bosses' phone!

 

Hopefully I'll get some movement on this soon. I'm getting nervous in not having a backup at the moment - starting over with a new NAS and a new set of disks will be rather expensive and time consuming. And I still need to recover files from time to time.

 

Q1: Can I rebuild the catalog based on the first 50 .rdb files and then extend it to cover the next ones and so on ?

 

Q2: Why would the catalog not be backed up anymore ?

 

Thank you all for your support and suggestions, I'll be trying again soon.

 

best regards

Steve

Link to comment
Share on other sites

A1: I believe you can remove *any* .rdb files from a disk media set and redo another catalog rebuild -- the rebuild will only cover the .rdb files it can find (and they can be non sequential).

 

A2: You can't back up a "live" catalog (IIRC) with Retrospect. You *do* have the Catalog file on your internal Engine computer, right? Not on the Drobo?

 

 

 

Way up the thread you stated:

 

There was a crash and a recovery about a year ago and I could no longer add to 1-Media Set A

 

 

This would seem to me that you have other potential problems with the Drobo that might be addressed by reformatting the drive. But that's just a guess at this point.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

×
×
  • Create New...