7.7.203 memory leak?

[jwright]

We recently upgraded to 7.7.203 from 7.6. I noticed that our backups were not completing, and when I checked the server retrospect had crashed. I have been monitoring it for several days now, and it seems the problem is that Retrospect.exe slow consumes more and more memory until it dies. Has anyone else encountered such an issue?

 

J

[rhwalker]

Has anyone else encountered such an issue?

Memory Leak?

[jwright]

I should add that the retrospect.exe process continues to use more and more memory while doing any backup, regardless if it crashes the program or not. It did not exhibit this behavior in 7.6.

 

This is a show stopper. The software is unreliable at this point. We can't roll back to 7.6 now either because of the changes made in 7.7.

 

When I asked if anyone else is experiencing this problem, I mean the problem I am describing, of course. It was really directed toward retrospect support. Are you guys aware of this issue?

 

J

Edited December 22, 2009 by Guest

[rhwalker]

When I asked if anyone else is experiencing this problem, I mean the problem I am describing, of course. It was really directed toward retrospect support. Are you guys aware of this issue?

You are aware, aren't you, that these forums are user-to-user support? From the EMC Retrospect Forum Rules:

 

This forum is a community based self help tool for users of the Retrospect Backup Software and other EMC Insignia Products. ...

 

While this forum is monitored by members of the EMC Technical Support team, it is not possible to reply to all questions and threads. This forum is not an official method for contacting technical support. EMC employees are under no obligation to reply to individual forum posts. If you need immediate technical support, you can contact technical support directly at http://www.emcinsignia.com/contactsupport

We are just users like you are.

 

If you want to ask EMC Support about this issue, I suggest that you contact them using the information at the link above.

 

Russ

[jwright]

Ah...that explains the poor support I usually get here. Just kidding.

 

Here is the error I get:

"

Retrospect has encountered a serious error:

Assertion failure at "tmemory.cpp-283"

 

A log of this error has been written been written to the file "assert_log.utx".

 

Please tell EMC about this problem.

 

We have created an error report that can help us improve EMC Retrospect. We will treat this report as confidential and anonymous.

 

Send the error report?

 

Yes No

"

 

I have sent it along every time...probably 15-20 times now. Anyone seen this sort of error message before?

 

J

[jwright]

I just spoke with EMC support. They believe that my config files need to be replaced. So they want me to do a complete uninstall and reinstall! Doh.

 

Translation: Sometimes the upgrade from 7.6 to 7.7 does not work and you will need to reinstall from scratch.

 

I will post a follow up before the new year.

 

J

[jwright]

Just wanted to confirm, for anyone else experiencing this sort of issue, that corrupt configuration files can cause retrospect.exe to slowly consume memory to the point that it crashes retrospect.

 

In my case, I am running a Windows server that I upgraded from 7.6 to 7.7. The upgrade screwed up my configuration files, though there were no obvious signs of this in the application.

 

The resolution was to completely reinstall...reinstalling retrospect with the existing configuration files in place did not work, nor did uninstalling retrospect and reinstalling with the configuration files in place.

 

Here is what I did:

 

1. Open the License Manager in Retrospect and write down all of your license codes. You will need them to complete the reinstall.

2. Write down all of the details for your scripts, clients, preferences, backup sets, etc. This can be a huge task.

NOTE: Make sure you have all of your client passwords so that you can add them to the server.

3. Open "C:\Documents and Settings\All Users\Application Data" and rename the Retrospect folder to something like "Retrospect_OLD". This is where your configuration files live.

4. Uninstall Retrospect using Add/Remove Programs.

5. Reboot.

6. Install Retrospect using the latest installer.

7. Enter your license codes when you run Retrospect, add all of your clients, point the server to the old backup sets(catalog files), set up preferences, and rebuild your scripts.

 

Hope that helps someone.

 

J

[mwilliamson]

Hi J. I am having the exact same issues. I have been really hoping to avoid a complete reinstall because I have so many scripts and source groups to rebuild.

 

Can you verify the since the re-install the TMemory errors and assert crashes have completely gone away? I just want to be sure before I blow everything away. Thx.

 

-M

[jwright]

It worked for us. I have noticed that retrospect.exe still uses a good bit of ram when processing some scripts, but it never runs away. Keep in mind that I never monitored ram before either, because Retrospect "just worked".

 

Retrospect support suggested this as a first try -- just uninstall retrospect using the add/remove programs, reboot and reinstall. This keeps your configuration files intact but might fix the issue. I still recommend that you at least make a copy of the config files before you uninstall. Note that this method did not resolve the issue for us.

 

If you look in the configuration files folder (C:\Documents and Settings\All Users\Application Data\Retrospect), there are several dat files. Perhaps just renaming one of those might limit the amount of work you have to do. Just a shot in the dark though.

 

A reinstall is really a poor fix. If their upgrade were working properly, it would not be needed. You might just call Retrospect support directly. They were helpful and responsive at least. Retrospect support doesn't seem to document much these days, so it is possible that they have a better solution now.

 

I probably spent 20+ hours monitoring/troubleshooting this issue. Then another 15+ trying to fix it. I would be curious how much time this problem costs you.

 

Keep us posted.

 

Jeff

 

[mwilliamson]

I has already cost me many days worth of time. When I noticed the system hanging (after the upgrade), I thought I would try to rebuild my catalog files. Of course the rebuilds always fail with a TMemory error. I thought there was a problem with the files in the catalog so just deleted them and tried starting with fresh backup sets. Now I get the TMemory error and then an assert dump and Retrospect crash. This usually happens 6 - 12 hours into a backup session. So right now I am basically stuck with an unusable backup system and no backups. I do have support so I will give them a call. Thanks for all of the tips and great information.

 

-Matt

[jwright]

I didn't have support when I called. Can you let us all know what they recommend and what actually works for you?

 

Thanks,

 

Jeff

[mwilliamson]

Hi Jeff, just an update. I completely uninstalled and reinstalled per your post. Unfortunately I got the same error. I contacted EMC support and they said this is a known bug, for which there is no fix yet. I provided them with some log files. They did however suggest I roll back to v7.6 until there is a resolution for this issue. I am now stuck in a terrible position. I haven't had a good backup since this update came out in mid December. I have 40 production servers and 15 desktops unprotected right now. Rolling back will take days as I rebuild catalog files that were converted to 7.7. Or I could wait, hope, and pray they release a bug fix for this soon.

 

-Matt

[rhwalker]

Or you could restore the catalog files from a backup and then simply update them rather than doing a complete rebuild.

[jwright]

Matt:

 

I have been watching the memory consumption on my machine since you posted and it does still throttle up very slowly during a backup, but after the fix it is using far less ram than before. I can see that in some situations -- perhaps a lot of clients in a script or a lot of proactive scripts -- that Retrospect might not throttle down and just crash.

 

If it is a known bug, it is a show stopper. I wonder how wide-spread it is. Did they say in which scenarios this is problem? I mean, it can't be a problem for everyone or else we wouldn't be the only two people talking about it.

 

Rolling back is a lousy option. I would do some serious CYA if I were you. Document your efforts and your conversation with EMC support and follow up with them daily. If you are responsible for backups, you don't want to find yourself unable to restore files without a very good excuse.

 

Shouldn't the Read Me file for 7.7 contain known bugs?

 

http://kb.dantz.com/display/2n/index.asp?searchtype=allwords&searchstring=7.7 for Windows read me &searchby=keywords&tab=search&search=1

 

Shouldn't this be linked off the support page, like the 7.6 readme is?

 

Jeff

Edited January 7, 2010 by Guest

[robvil]

Maybe not that many people have upgraded yet as there seems to be massive issues with ver. 7.7

 

Regards

Robert

 

[Richy_Boy]

Oh balls.

 

I was relying on Retrospect to get 7.7 right as I need Win7 support after waiting and waiting and waiting... for it to be released.

 

Do I stay strong, or jump ship at this point as they always seem to mess every release up.

 

Rebuilding my entire backup script/client list is simply not an option, I might as well start fresh on a properly supported solution.

 

Sorry this is overly negative, but do they realise they supply software for businesses and not just home users!?

 

Rich

Edited January 8, 2010 by Guest

[robvil]

I know what you mean... I have been waiting for 7.7 as I rearly need support for Windows 2008 and AD backup.

 

Robert

 

[mwilliamson]

Hi all. I still haven't rolled back. Reinstalling the clients would be a big hassle for me as well. Something unusual has happened though. I have backup sessions that have now run for over 24 hours without failing. They have never gotten this far since the upgrade to 7.7. The only change I made was to set the priority for the Retrospect.exe process to "High" within Task Manager. I am keeping my fingers crossed that this change will allow the jobs to complete. During this period memory has been all over the place but I do have 3 scheduled and 1 proactive scripts running. Memory has been as high as 800MB and as low as 40MB.

 

I am not sure if this helps any of you but it may help to know we're all in the same boat. I haven't heard back from EMC support yet but have supplied them with more log files for review. I really hope they issue a fix soon because rolling back is not going to be any fun.

 

-Matt

[Dmorgan]

Hey there guys I just wanted to let you know that you are not alone here.

 

The first problem that I ran into was assertion errors in a number of Exchange inboxes. In total 3 inboxes ended up coming up with this error and every time it would completely crash Retrospect rather than simply skipping that inbox and moving to the next. A phone call to support was replied to with 'well there must be something wrong with those inboxes. you will have to delete and recreate them'. Well I didn't buy that and thought perhaps there was data in those inboxes causing the problem so I spent hours pulling everything out of one of them into a PST file. Well after doing that Retrospect would STILL crash on a completely empty inbox which to all other appearances was a perfectly functioning inbox. At that point I decided to simply skip them until I could deal with the problem further.

 

After sussing that out I thought everything was hunky dory but then started getting the same memory error crashes you have been talking about. After trying to work around it for a few days I decided to simply do a rollback to 7.6 which I was able to do with no problems. I had no valid 7.7 backups over that week to worry about so I simply recycled the backups and let 7.6 make fresh ones. I was also able to rollback all of the clients to 7.6 as well and so far everything has been running fine for about a week now.

 

I guess that until I see patch notes addressing these issues I will be staying away from 7.7.

[mwilliamson]

Hi all, I am still seeing the memory errors on jobs that run a long time (+24hrs). I have been limping along trying to reduce my execution units to 1 or 2. Still I eventually get the "assert" tmemory error and have to restart the job. I have been shuffling the order of the machines in my source group because the those at the end invariably don't backup when Retrospect crashes. At this point I still think rolling back would be too painful. At least I have some machines backed. I still have had no response back from EMC support. They are aware of the issue.

 

-Matt

[jwright]

Matt:

 

I feel your pain. I think rolling back is not a solution.

 

Last night I had another asset crash -- first since I reported the solution of reinstalling. So, while reinstalling did reduce the frequency of crashing, it was not a solution.

 

Most of my crashes occured when a job ran for 4+ hours, which is why I opted to reinstall.

 

Jeff

[railes]

After experiencing numerous assert crashes plus LTO tapes that were not being fully utilized (filling up on 350 gigs of data instead of the normal 700 gigs), and after finding out from an EMC tech that there was no ETA for a patch, I decided to go back to Multi-Server 7.6. All backups returned to normal - no crashes.

However be forwarned! All your catalogs from 7.7 are unreadable or unusable in 7.6. This includes the backups to file options also. I spent the weekend dealing with that unpleasant surprise.

[robvil]

Then you did not read the release notes. It says 7.7 backups cannot be used by 7.6.

 

Regards

Robert

 

[mwilliamson]

The problem with me rolling back is I don't believe the backups I've done on 7.7 will be readable. I have had some complete so do have some snapshots I want to keep. Were you able to recreate a catalog file in 7.6 using 7.7 backup set files? I can't believe EMC is not going to address this. I am seriously considering another backup product at this point. Retrospect is only more cost effective if it works.

 

-Matt

[Mayoff]

Everyone,

 

We are aware of the memory issues in 7.7 and I believe our engineers have fixed them. We will be releasing a patch for this soon.