SIGABRT crashes constantly in 8.1.626.1 on 10.6.3

[salmedia]

I am a fairly new administrator for Retrospect, and I have configured Retrospect 8.1 (build 626) on a mac-mini running 10.6.3

 

Backups have been running successfully over to a mapped drive on a windows server via AFP

 

Several times I go check on the backups & I am consistently seeing crash reports. When I dig into the crash report I see this:

 

Exception Type: EXC_CRASH (SIGABRT)

Exception Codes: 0x0000000000000000, 0x0000000000000000

 

Upon inspecting the Console this also sticks out

 

[0x0-0x45b45b].com.emc.Retrospect[54069] job appears to have crashed: abort trap

 

Retrospect(54069,0xb0103000) malloc: *** mmap(size=16777216) failed (error code=12)

*** error: cant allocate region

*** set a breakpoint in malloc_error_break to debug

I called EMC earlier about this today and was advised to switch the Media Set backup location to the local drive for now - which I did. However, the scheduled backup isn't till 6pm, and I looked back on the mac-mini later and the crash occurred again.

 

Is this a Mac OS X issue or a Retrospect problem?

 

Can anyone help out please?

[Maser]

So, the engine crashed *sometime after* you modified where the Media Set is located?

 

Or *immediately after*?

 

There's a difference. If it was "sometime after" -- do you have other scripts running that might have caused the crash?

[salmedia]

Hi Maser. Thanks for responding.

 

I talked to EMC about 945am PDT today, made the change. I didn't immediately see the crash right then, as I closed my VNC connection. But an hour or 2 later, when I went back to research some more on the issue, I saw that Retrospect had stopped again generating a crash report.

 

No other scripts are running - I only have the (1) job configured, and it doesn't even kick off till 6pm PDT.

 

I did grab this from the operations_log.utx for yesterday and today:

 

$[//]- 4/7/2010 6:00:00 PM: Copying $[*!20613,,14,+3]Macintosh HD$[//21/- %D %.1T: Copying %s/]$[/D/30/2010-04-07T18:00:00.848Z-07:00/]$[/T/30/2010-04-07T18:00:00.848Z-07:00/]$[/s/138/$[//]$[*!20613,,14,+3]Macintosh HD$[//4/%s%S/]$[/s/60/$[//]$[*!20613,,14,+3]$[//15/$[*!%~d,,14,+3]/]$[/d/5/20613/]/]$[/s/12/Macintosh HD/]/]

$[//]4/7/2010 6:00:00 PM: No files need to be copied$[//35/%D %.1T: No files need to be copied/]$[/D/30/2010-04-07T18:00:00.848Z-07:00/]$[/T/30/2010-04-07T18:00:00.848Z-07:00/]

$[//]4/7/2010 6:11:40 PM: Snapshot stored, 131.9 MB$[//30/%D %.1T: Snapshot stored, %lKB/]$[/D/30/2010-04-07T18:11:40.580Z-07:00/]$[/T/30/2010-04-07T18:11:40.580Z-07:00/]$[/K/6/135038/]

$[//]4/7/2010 6:13:47 PM: Comparing $[*!20613,,14,+3]Macintosh HD$[//21/%D %.1T: Comparing %s/]$[/D/30/2010-04-07T18:13:47.365Z-07:00/]$[/T/30/2010-04-07T18:13:47.365Z-07:00/]$[/s/138/$[//]$[*!20613,,14,+3]Macintosh HD$[//4/%s%S/]$[/s/60/$[//]$[*!20613,,14,+3]$[//15/$[*!%~d,,14,+3]/]$[/d/5/20613/]/]$[/s/12/Macintosh HD/]/]

$[//]4/7/2010 6:15:05 PM: Execution completed successfully$[//11/%D %.1T: %s/]$[/D/30/2010-04-07T18:15:05.837Z-07:00/]$[/T/30/2010-04-07T18:15:05.837Z-07:00/]$[/s/89/$[//]Execution completed successfully$[//2/%s/]$[/s/32/Execution completed successfully/]/]

[prl]

I am a fairly new administrator for Retrospect, and I have configured Retrospect 8.1 (build 626) on a mac-mini running 10.6.3

 

Backups have been running successfully over to a mapped drive on a windows server via AFP

 

Several times I go check on the backups & I am consistently seeing crash reports. When I dig into the crash report I see this:

 

Exception Type: EXC_CRASH (SIGABRT)

Exception Codes: 0x0000000000000000, 0x0000000000000000

 

Upon inspecting the Console this also sticks out

 

[0x0-0x45b45b].com.emc.Retrospect[54069] job appears to have crashed: abort trap

 

Retrospect(54069,0xb0103000) malloc: *** mmap(size=16777216) failed (error code=12)

*** error: cant allocate region

*** set a breakpoint in malloc_error_break to debug

I called EMC earlier about this today and was advised to switch the Media Set backup location to the local drive for now - which I did. However, the scheduled backup isn't till 6pm, and I looked back on the mac-mini later and the crash occurred again.

 

Is this a Mac OS X issue or a Retrospect problem?

 

Can anyone help out please?

SIGABRT is a Unix signal (software interrupt, defaults to killing the program with a core dump) that's generetad by the abort() library call, and is usually used when some internal inconsistency is detected by a program.

 

The problem being flagged is that the malloc() library call (runtime memory allocation function) can't allocate any more memory on the Retrospect server's program heap, and so the Retrospect is deliberately crashed by the library call.

 

The most likely immediate causes are: not enough page space on the system disk on the Retrospect server (system boot disk full?), infinite (or very long) loop in Retrospect server that just keeps allocating memory, some actual call of malloc() in Retrospect server that's trying to allocate an unreasonably large amount of memory, or possibly corruption of malloc's internal data structures in a way that causes it to allocate an unreasonably large amount of memory even though it's been called correctly.

[Maser]

Hi Maser. Thanks for responding.

 

I talked to EMC about 945am PDT today, made the change. I didn't immediately see the crash right then, as I closed my VNC connection. But an hour or 2 later, when I went back to research some more on the issue, I saw that Retrospect had stopped again generating a crash report.

 

 

Was the crash report timestamped immediately after you made the change?

 

 

I'd probably suggest (as it won't hurt anything) doing a rebuild of the catalog file in case something got messed up with that.

[Mayoff]

It would be very helpful to get a copy of the entire crash log from the OS X console.

[salmedia]

I am a fairly new administrator for Retrospect' date=' and I have configured Retrospect 8.1 (build 626) on a mac-mini running 10.6.3

 

Backups have been running successfully over to a mapped drive on a windows server via AFP

 

Several times I go check on the backups & I am consistently seeing crash reports. When I dig into the crash report I see this:

 

[b']Exception Type: EXC_CRASH (SIGABRT)

Exception Codes: 0x0000000000000000, 0x0000000000000000[/b]

 

Upon inspecting the Console this also sticks out

 

[0x0-0x45b45b].com.emc.Retrospect[54069] job appears to have crashed: abort trap

 

Retrospect(54069,0xb0103000) malloc: *** mmap(size=16777216) failed (error code=12)

*** error: cant allocate region

*** set a breakpoint in malloc_error_break to debug

I called EMC earlier about this today and was advised to switch the Media Set backup location to the local drive for now - which I did. However, the scheduled backup isn't till 6pm, and I looked back on the mac-mini later and the crash occurred again.

 

Is this a Mac OS X issue or a Retrospect problem?

 

Can anyone help out please?

SIGABRT is a Unix signal (software interrupt, defaults to killing the program with a core dump) that's generetad by the abort() library call, and is usually used when some internal inconsistency is detected by a program.

 

The problem being flagged is that the malloc() library call (runtime memory allocation function) can't allocate any more memory on the Retrospect server's program heap, and so the Retrospect is deliberately crashed by the library call.

 

The most likely immediate causes are: not enough page space on the system disk on the Retrospect server (system boot disk full?), infinite (or very long) loop in Retrospect server that just keeps allocating memory, some actual call of malloc() in Retrospect server that's trying to allocate an unreasonably large amount of memory, or possibly corruption of malloc's internal data structures in a way that causes it to allocate an unreasonably large amount of memory even though it's been called correctly.

 

 

the system that is running the engine, the mac mini, has 4GB of RAM, and 100Gb of free space, with a 2.26Ghz intel core duo

 

the mapped drive location has 20Gb free - but even when I added a new Media Set pointing the location to local disk - it still crashed shortly after the change, and has crashed again last night.

 

However, the backups are still completing successfully, I just have to keep re-launching the application.

 

[salmedia]

Hi there. I have attached one of the first diagnostic crash reports I have, from 3/29, and then (2) subsequent crashes that occurred last night after the change over to Media Set B (which is local fixed disk vs mapped drive to a Windows share)

 

 

[salmedia]

OK, am rebuilding the catalog set now.

 

also added a backup policy for the catalog files themselves - which I wasn't doing

[Maser]

My only other suggestion is that you move up the backup to when you can watch the action in the console (I'm assuming you have this set to run when you've left for the day.)

 

 

Just to see *where* in the process of the backup things are crashing -- are they crashing "scanning" your source(s)? Is the crashing occuring in the "matching" or "backup" phase, etc?

 

 

Maybe there's some specific *file* on a specific *source* that causes the crashes? (I've recently submitted a bug with a specific folder with an odd set of characters in the name which will cause the program to barf if I "browse" this folder...)

 

It may just be that one of your sources has a similar bad file/folder and you'll have to hunt this down...

 

 

 

 

 

[salmedia]

OK, I have some updates.

 

I wound up essentially starting from scratch. I created new Media Sets, new backup jobs, the backups run successfully, but Retrospect still crashes. What I am seeing is that Retrospect crashes more specifically when there isn't any backup activity.

 

Yesterday I finally had the chance to run 2 backup jobs and watch their progress. I do a weekly Catalog backup & monthly media recycle of them (per the suggestion in a post on this site) and then I do a daily backup of the database files - which Retrospect was actually purchased for.

 

Both backup jobs completed successfully. Both Media Sets are still pointed to a network share and they run correctly.

 

I created another Backup job - where the source is the local disk and the Media Set is also the local disk. This just ran successfully as I was typing this.

 

The crashing that I am seeing specifically occurs AFTER backups have completed. Example - yesterday, the primary database job fired off at 4pm, it was done in under 5 mins. However, around 6:15pm is when the crash occurred.

 

Could I be looking at an OS X issue here and not specifically a Retrospect problem? All I know is I get the same error in the console logs over & over:

 

Exception Type: EXC_CRASH (SIGABRT)

Exception Codes: 0x0000000000000000, 0x0000000000000000

Crashed Thread: 8 Dispatch queue: com.apple.root.default-priority

 

Application Specific INformation:

abort() called

 

 

Thanks for any help in advance

 

[Maser]

So, these are strictly *backup* scripts (not proactive scripts)?

 

Can you elaborate on your schedule of the scripts?

 

Meaning, the "database backup" ran at 4 p.m. Was this scripted? Or did you do a manual backup?

 

Where there any other scripts scheduled to run *after* that (meaning at 6:15 when it crashed?)

 

 

Maybe post a screen shot of your list of activities?

[Maser]

Also, what happens if you move your media sets so that they are *not* on a "network share"?

 

I've never tested doing that and I'm wondering if that's your problem -- all my testing (and I'd think most other people?) have their media sets on a locally attached drive of some kind -- not an AFP share.

 

 

[salmedia]

So, these are strictly *backup* scripts (not proactive scripts)?

 

Can you elaborate on your schedule of the scripts?

 

Meaning, the "database backup" ran at 4 p.m. Was this scripted? Or did you do a manual backup?

 

Where there any other scripts scheduled to run *after* that (meaning at 6:15 when it crashed?)

 

 

Maybe post a screen shot of your list of activities?

 

 

These were strictly backup scripts - not proactive scripts - at least I dont think they were configured as proactive

 

The backup that ran at 4pm was scheduled. I'd rather have it ran after hours, but for testing purposes and to watch the backup, I scheduled it at 4pm, and let it kick off per the scheduled parameter.

 

No other scripts were configured other than a 4pm backup and then the 430pm Catalog backup

 

I have posted a screenshot of the Activities Pane. The job that you see failed from yesterday at 4:30pm was because when I reconfigured the job/script I used the wrong password for connecting to the AFP share.

[Maser]

Based on the screen shot unless the engine is having trouble keeping track of where your media set is located, I'm not sure why it would have crashed 90 minutes after the backup completed.

 

I'd still suggest doing a test of having your media sets be "local" and see if that causes the same crashes.

 

You may be doing something that's *possible*, but not expected (having the media sets be "on the network"...)

[salmedia]

Well I changed the Catalog backup, and the database backup to point towards a test Media Set that I have pointed towards the mac-mini's local hard-disk.

 

I configured them for 1:05pm, 1:15pm, and I have another simple test backup (going to the same Media Set) that just does a backup of some desktop items - this backup was configured for 1:30pm.

 

Right when the 1:05pm backup was scheduled to kick off - Retrospect simply closed and i was presented again with the same error message. i have attached the console error log that was generated

[Maser]

Do you still have the "network" media set listed under "Media Sets"?

 

It's possible the program regularly checks to see if the media set is there (even if it's not in use) and that's what it's having problems with.

 

Try this: Stop the engine. Make *copies* of your "config80.xxx" files, restart the engine and *remove* the network media set from "Media Sets".

 

Then check your scripts to make sure they point at your local sets and see what happens.

 

If you don't crash (and hopefully you won't), then you could try readding your network media set to see what happens after that.

 

 

If you *do* crash, I'd consider then just starting with a completely clean config80.dat file...

 

 

[Maser]

Wait...

 

From your screenshot.

 

Is the crash happening after the "Catalog backup" script?

 

Is that script actually backing up the *active* Retrospect catalogs?

 

Meaning the catalog file for the media set you are backing up to?

 

Maybe that's what's causing the crash? (Or maybe not?)

[bioheath]

I am not sure how it would work with AFP, but we have about 20 scripts pointed to SMB shares on network attached storage. These run proactively (mostly, a few backups) and they complete without causing the server any trouble. If the network attached storage ever vanishes unexpectedly it does cause the console to hang, but the server seems to continue backing up other scripts to other media sets.

[salmedia]

Actually I was incorrect about something - the connection is not over AFP, its connected via SMB.

 

@Maser - I will try your recommendations from above now.

[salmedia]

@Maser

 

The crash you saw in the Activities screenshot I provided was because of a bad password entered - once corrected the backup was successful. However, as noted, the crash occurred a good 2 hours after the backup had completed. But as you asked, yes the Catalog backup was of the actual catalog80.dat file - I saw that recommended in a post - but then I saw some debating the validity of that in another post. I can see the logic behind that, and perhaps backing up the catalog80.bak file would be better.

 

So I went ahead and did (5) local disk backups today and just completed a network backup successfully as well. I am leaving for the day now and will check on the application tomorrow morning to see if it crashed overnight.

 

There are no more scheduled jobs for this evening.

 

Thanks