Complete Failure in Backup server - Memory Errors - catalog corruption - unable to do anything

[mkohara]

First:

 

Background

 

Chassis:

Dell Poweredge T410

Xeon E5620 @ 2.4 GHz

8GB RAM

5TB RAID 5 on a Dell PERC 6/i RAID controller

IBM ULTRIUM-HH5 Tape Drive

 

Software

Windows Server 2008 R2 Standard x64

Retrospect 7.7.620

 

 

--

 

It all started when I did a standard cleanup on the RAID and it wiped out the entire catalog set. So I dumped that destination, and started over. Ever since I have started having severe issues with the backup process. The screenshot attached says it all.

 

 

What ends up happening is even on a full reboot, as soon as it tries to backup any client on the network, Memory usage climbs to maximum and the backup either hangs, or fails eventually with the "Not Enough Application Memory" error. On the worst case, Retrospect throws a cpp error and dumps the process. in either case it damages the catalog file belonging to the destination it was accessing, Attempting to rebuild it, fails with the same error... and endless loops ensue. I am honestly at a loss here... I have not backed up the network since the 1st and I am beginning to look at other options. Help?!

[Mayoff]

Can you attach a copy of the assert_log.utx file for your crash?

 

What does the operations log say when you try the catalog rebuild? Are you saying that a catalog rebuild causes an assert error? Does it always stop at the same point during the catalog rebuild attempt?

 

You could have corrupt Retrospect preferences. You can try renaming the Retrospect folder in C:\Program Data to Retrospect Old and launch Retrospect to see if memory increases again.

[mkohara]

Update:

 

Tried a few things.

 

I did a re-install of retrospect (complete) and reloaded the scripts even changing out my script configs to backup files in smaller sets so the catalogs wouldnt be so large. Reset the backup sets, and started over essentially.

 

Fast forward a month or so, and All of a sudden the following happens:

 

I see a message in windows that a non interactive service has a message for me, and to click to see said message. I do so and am greeted with an error from Retrospect.

 

OS: Windows Vista version 6.1 (build 7601), Service Pack 1, (64 bit)

Application: C:\Program Files\Retrospect\Retrospect 7.7\retrospect.exe,

Exception occurred on 09/04/2013 at 1:15:07 PM

Error info: Assertion failure at "arc.cpp-1891"

 

Which of course means a hard crash from Retrospect.

 

Upon restart of Retrospect it starts throwing out of memory errors when attempting to run a backup. I can't gracefully close the program, or do any action to stop the running executions that doesn;t involve a hard termination of the App.

 

Catalog rebuilds throw this message:

 

+ Executing Recatalog at 11/04/2013 5:02 PM (Execution unit 1)

To Backup Set RAID...

Not enough application memory

11/04/2013 5:08:22 PM: Execution incomplete

Duration: 00:01:53

 

Backups to the same Catalog throw the same error eventually but also have the added bonus of causing Retrospect to have to be forcefully quit (as mentioned above)

 

I will attach the most recent assert_log.utx entry

 

PS: A renaming of the Retrospect ProgramData folder and starting that over does not cure the problem.

 

It also appears it is failing at the same point each time a rebuild is attemtped (right at the very begining of the rebuild - before it actually proccesses any files)

assert_log (recent).utx

Edited April 11, 2013 by mkohara