rebuild catalog fails-out of memory

[tgfn]

Hi

 

Backup Computer: G5 Dual, 2,5GHz, with 6.5GB Ram

System: Tiger 10.4.11

Retrospect 6.1

Scsi Interface: Atto Express PCI UL4S

Tape Drive: SDLT 600, Quantum (External)

 

since we had a powerfailure my backup catalog is not readable anymore (error -24205 if i remember right).

Now i am having big problems to rebuild the catalog from the tapes, since after maybe 20 GB of rebuilded data , the rebuild stops, giving me the chance to save the rebuild or to revert. log file says, that the computer run out of memory.

Watching the activity monitor, i could see, that retrospect stops after approching the use of 3GB Ram.

Tried the update trick mentioned earlier in the forum, to no avail. To me it seems, that retrospect can`t free the memory, after it writes the (rebuild) catalog to the disk.

Is anybody out there having the same problem, or even better, a solution to this problem?

 

 

 

Edited May 28, 2008 by Guest

[rhwalker]

Retrospect 6.1

Exactly what version of Retrospect? 6.1.what

 

What RDU are you using? (although I doubt that this is the issue)

[tgfn]

thanks for the quick response.

iám runnung retrospect 6.1.138

with rdu 6.1.14.101

[rhwalker]

Some have reported problems when the computer has more than 3 GB of memory. See if you can try on another computer, or on this one with less memory.

 

I've never seen this issue on our Xserve G5.

 

Russ

[tgfn]

ok, i`ll try that. next time the rebuild stops i`ll put some memory out. Really funny, if i think about it

[rhwalker]

Really funny, if i think about it

Well, I think it's related to the ancient codebase (and Carbon libraries against which it links) of the current version of Retrospect, ported from the MacOS 7/8/9 version. A new Universal Binary version of Retropect, based on the current Windows codebase, is supposedly going into beta later this year. Let us hope that this will fix all the issues with the aging codebase. It's a bit hard for me to laugh about the situation.

 

Russ

[tgfn]

that was pure sarcasm from my side. Anyway, with only 2GB of Ram the rebuild run for about 200GB, alas 10 times more as before (still 9TB to go). Don`t now, whether this was by accident or because of the ram.

I noticed, that i had in Dimm 0 and 1 256MB, the rest was 1GB each, but with different timing (CL 2,5, eg. CL3). Never cared to much about diffent ram dimms as long as i had identical ones in the corresponding channels, since memory controllers should be able to handle that.

Now i have two identical Dimms with the same timing, 1GB each.

I will make further trys, when i get back my spare tape drive. (This tape drive, which i have for emergencies actually caused the hole mess, since when i plugged that one in, we got a power-failure which in turn corrupted my catalogs).

I will hook this tapedrive to another G5 running Leopard and will see what happens with different Ram settings.

 

[CallMeDave]

This tape drive, which i have for emergencies actually caused the hole mess, since when i plugged that one in, we got a power-failure which in turn corrupted my catalogs.

 

Perhaps the mess was actually caused by the absence of a UPS of adequate capacity to allow for a graceful shutdown after a power failure?

 

Just sayin'

 

 

Dave

[tgfn]

strange things happen as my last post disappeared into the nirvana. hope there is no censorship of any kind here.

dave is right, part of the mess is my fault, since i did not take too many precautions.

the other part of the mess is obviously my fault as well since i trusted a software too much.

if a software offers a feature like rebuilding a catalog and then fails because it runs out of memory i can hardly say, that the programmers did a good job.

anyway doesn`t help to complain, i´am still looking for a way to get my data back.

the amount of ram in the computer doesnt seem to make any difference. the rebuild stops with a "not enough memory" message in the log, regardless whether i use 2GB, 3GB, 4GB or more ram.

i managed to almost get one tape back with the update catalog function (clicking on retrieve, when the restore stops and then just started over again, strange enough, the update process started, where it had stopped, and added more files to the catalog, instead of starting all over again).

still waiting for somebody who managed to restore a catalog file of about 20GB of size from about 10TB of data. Anybody out there and if so, who did you do it?

Edited June 1, 2008 by Guest

[Mayoff]

I guess you didn't see:

http://forums.dantz.com/showpost.php?post/107539/

 

[CallMeDave]

still waiting for somebody who managed to restore a catalog file of about 20GB of size from about 10TB of data

I can't say I've had a Backup Set with that much data. How many tapes does the Backup Set contain? 20? More?

 

A Backup Set's catalog file is valuable property, no different then your financial databases or mailstore. As such, managing it in advance helps prevent this sort of jam; minimizing the number of Members in your Backup Set so if you have to rebuild you do so from fewer tapes, keeping backup copies of the catalog so you can begin any rebuild or update from a reasonably current version, etc.

 

I note these things only so other readers will be reminded of this, not to minimize the problem you're having.

 

When you write:

 

"...after maybe 20 GB of rebuilded data , the rebuild stops..."

What does this mean? How many tapes have been read at this point?

 

I'm unclear on the exact steps you are doing. There are various ways to rebuild a catalog file from scratch. I think having an idea of how many tapes you're able to rebuild from before the failure would help us understand better.

[tgfn]

Hi Dave

 

thanks for your response.

part of the questions were answered in the post which disappeared, sorry for my paranoia.

I have the following situation:

corrupted catalog of about 20GB, about 20 tapes, roughly 10TB data on the tapes.

i have a saved the catalog for the first 10 Tapes, which consistes of the full backups of all my clients.

On the following tapes are the incremental backups.

So, when i start a new backup, i reuse the fullbackup tapes and catalog (the first ten tapes + saved catalog), so i don`t have to make the fullbackups again and erase the incremental ones (which would be tape 11-20).

Now what i want to do is to restore the catalog from the tapes.

Since i already have a catalog for the first 10 tapes, i start with this catalog and tape 11, doing the "update catalog" thing.

This update process usually stops at about 20GB of restored data.

Retrospect then gives me the chance to "revert" or to "save". If i use "save", the rest of the tape will not be updated any longer, instead retrospect would ask for the next tape.

if i click on "revert", retrospect stangly doesn`t start all over again with this tape, but starts where it stopped, cataloging additional 20GB or so.

By the way, i did a rebuild catalog (not update) as well, forgetting the first 10 tapes and starting with the 11 tape. i had the same result then, retrospect quit at some point with the known message.

Watching the activity-monitor i could see, that retrospect always used a little more than 3GB of physikal ram before it crashes.

logfile only said: not enough memory.

syslog says the folowing:

Retrospect (147, 0xa0b6a074) malloc:

***mmap (size=182034432) failed (error code=12)

error: can`t allocate region

***set a breakpoint in malloc_error_break to debug

 

which of course doesn`t help me much since i am no programmer.

Anyway it looks to me that retrospect has a problem freeing the memory again, but i am not sure.

 

Other posibility is, that retrospect runs into something on the tapes which it can`t handle out of some reason (a do verify and i do read the logfile, which didn´t show me anything unusual during the backup).

So that`s why i wonder, whether Retrospect is capable of restoring a catalog of this size and wheteher someone actually managed to rebuild a catalog of that size

cheers

tgfn

 

Edited June 2, 2008 by Guest

[CallMeDave]

I'm going to have to re-read the above a few times to best understand the information provided. But a few things pop out at me right away:

 

when i start a new backup, i reuse the fullbackup tapes and catalog (the first ten tapes + saved catalog), so i don't have to make the fullbackups again and erase the incremental ones...

This isn't how Retrospect is designed to be used, but I'm not at this point willing to suggest that it's not how Retrospect _can_ be used. I would think that erasing valid Members of a Backup Set would require some careful planning and handling.

 

Since i already have a catalog for the first 10 tapes, i start with this catalog and tape 11, doing the "update catalog" thing.

 

I don't understand. If you have tapes 1-10 and have a saved version of the Catalog file that was used to create those tapes (and was moved/renamed/archived without using it to catalog the subsequent tapes) then why is there any need to update anything? When that Catalog file is used as the basis of a Backup Set, Retrospect would only know about tapes 1-10.

 

Or, do you erase tapes 11-20 and then expect the Catalog file that was used for tapes 1-20 to be used, even though you've erased valid Members it expects to see?

 

Now what i want to do is to restore the catalog from the tapes.

 

"Restore" is data that you have backed up and you want to have again. "Rebuild" is when you re-create a Catalog file from the Media.

 

This update process usually stops at about 20GB of restored data.

 

Here's where the terminology being used gets hopelessly conflated. "Restored data" is a particular thing. "Update existing catalog file" is something different.

 

 

i did a rebuild catalog (not update) as well, forgetting the first 10 tapes and starting with the 11 tape. i had the same result then, retrospect quit at some point...

 

I continue to ask about how many tapes you are able to successfully use in the "Rebuild from tapes" process; that answer has not yet been given. Are you getting Media Request windows? Do you swap tapes successfully up to a point?

 

And now that I think about (but can't test, as I don't have any devices online to use with an appropriate Type of Backup Set) where do you "forget" tapes when you "Rebuild from tapes" in the Tools->Repair window? Doesn't Retrospect prompt you for what you know to be the first Member of the Backup Set that you have? "Forgetting" is something you do with a Catalog that already knows about the Members.

 

When Rebuilding from tapes, Retrospect doesn't know how much data your Backup Set contains, so the total size of all 20 doesn't really come into play in the beginning of the process.

[tgfn]

juggle,juggle. that`s what they already told me at school: be more accurate. even my wife complains once i want to explain something.

soory for that, i`ll try:

let`s say, i completly start a new backup. I will save every client just once, so that i have fullbackups from every client (this are tapes 1-10 in my case). I then save the catalog, take a new

tape (tape 11) and start my normal backup server script, which now will make incremental backups.

I leave this running until performance gets too low, which usually will be at around 20 tapes( it´s not the amount of tapes but the amount of files and sessions that slows everything down)

The catalog of this 20 tapes i will then move to another place. Then i take the saved catalog from Tape 1-10, move it to the retrospect folder and start the incremental backup again with a now erased tape 11 (i have more than one backupset running, i just take one for simplicity). So i have two catalog files. One for tape 1-10, the other for tape 1-20. This works without a problem, i did restores and did`t have any troubles.

"Now what i want to do is to restore the catalog from the tapes".

you are right

 

"This update process usually stops at about 20GB of restored data".

 

you are right again, what i mean is, that in the update window the process runs to about 20GB of data on the tape

 

 

"I continue to ask about how many tapes you are able to successfully use in the "Rebuild from tapes" process; that answer has not yet been given. Are you getting Media Request windows? Do you swap tapes successfully up to a point?"

 

none, since the rebuild did`t run to the point to request the next tape

 

"where do you "forget" tapes when you "Rebuild from tapes" in the Tools->Repair window"

 

when you dp a rebuild, retrospect will ask you for the first tape in the backupset. if another tape is inserted, it will ask you something like "where is tape 1". You will have the same window as for example when you erase a tape or retropsect requests a new tape.

In this window there is in the upper right corner a button called choices. There you can set the tape missing.

[tgfn]

update.

retrospect now actually asked me for the next tape.

as written before, i got stuck in doing the update catalog thing, since updating the catalog file stopped with the "not enough memory" message in the log file, giving me then the chance to either save the partial session or to retrieve.

At one point during this cycle i coppied the "updated catalog" to another location, because i wanted to try a few things out.

Now, after being stuck in this cycle, and the update process failed so often, i finally deleted the catalog file i am updating and reused the one i did copy to another location.

With this catalog file i am making far better progress as with the old one.

The updating process gives me the "not enough memory" message still every now and so often (150 -200GB), but after clicking the retrieve button goes on with updating the catalog file.

Knock on wood.

Still wondering about this behaviour, since the coppied catalog file is just that, a copy.

My assumption would be, that retrospect encountered something on the tapes it couldn`t handle. My problem in explaining the whole thing is, that with the copy, retrospect should have encountered the same thing on the tapes.

Anyway i can still give further updates, in case somebody is interested

 

 

 

[tgfn]

ok, i now managed to get the catalog updated again.

Problem is, that this does`t go as smooth as i expected.

As i wrote before, retrospect is not able to update a catalog-file when it exceeds a certain size. It will stop, give me the choices to revert or save. Clicking save, the remainder of the tape will be forgotten and the update-process will ask for the next tape.

Clicking revert, retrospect will run further with the update process, but after a few of this revert-clicks, retrospect will stop adding new data to the catalog.

Instead, it will at the same data to the catalog over and over again. At this point, retrospect is not able to add the new data to the catalog, it will instead forget everything what it just did. (as one can see for example from the size of the catalog file on the hard disk. During the update process the catalog file will enlarge for about 3GB, and after i click revert, it will diminish for this 3GB).

 

Out of unknown reason, retrospect doesn`t show this behavior, if one keeps the catalog compressed, and in my hands, this is the only way to update the catalog file.As one can imagin, this is very time-consuming.

So what we`ve got here is an obvious design bug in retrospect.

At one stage, retrospect is just not able to store in the catalog file the data it just updated unless one keeps the catalog compressed.

Needless to say that i could not find anything about this matter in the knowledge base.

If a software offers a given feature, this feature should work. And if it doesn`t work, i would expect the vendor to publish the limitations of this feature.