Jump to content
jwright

7.7.203 memory leak?

Recommended Posts

It can't be soon enough! I too am experiencing this issue for a customer with a large network, several clients, and scripts that would be complicated to recreate. We just updated over the weekend since things were running fine testing outside of production. We've had NO reliable backups since upgrading from 7.6! Argh!

Share this post


Link to post
Share on other sites

RE: RobertVil

 

The fact that 7.7. alters your backups so that they can no longer be used by 7.6 is not a cause for concern. Why would anyone care if the set works with an older version of restrospect if they have already decided to move to 7.7? This only becomes an issue because this bug causes the application to crash forcing you to roll back to 7.6 to resolve the issue.

 

While I agree that one should always read the release notes, keep in mind that EMC didn't update the release notes with a "late breaking" warning about this major bug. I read them before I upgraded, but there was no mention of the the bug. If you are one of the people experiencing it, it makes your backups unreliable and fixing it is a huge ordeal. A note should have been added to the top of the document.

 

Also, it is worth noting that there is no direct link to the Readme for the Windows 7.7 release. To find it, you have to search the Knowledge Base. When you do, you will see that it was last updated on "December 1, 2009". Other releases have a direct link to the "Read Me" file right on the download page.

 

 

RE: Mayoff

 

Will this be a patch to resolve just this issue, or are you guys rolling other fixes into it? Can you post a reply with the patch number and link when it is ready? Thanks in advance!

 

Jeff

Share this post


Link to post
Share on other sites
We are hoping to have something out next week that fixed about 5 or 6 very critical user reported bugs.

I have two problems in 7.6 that I want to see fixed before upgrading to 7.7.

The first one is getting -2241 while grooming:

http://forums.dantz.com/showtopic.php?tid/28392/post/113602/#113602

The second one is a bug in matching files:

http://forums.dantz.com/showtopic.php?tid/32242/pid/131189/

 

Are any of these (or both) fixed in 7.7? If not, will they be?

Share this post


Link to post
Share on other sites

For those how have to wait it out you may wish to try a feature available in SysInternals VMMAP program. Note: I have not tested this myself and this should be explored in a test environment first... Your mileage may vary.

 

Here is a write up from TechRepublic with links to the software.

"The most beneficial feature of VMMap is probably the Refresh menu option to Empty Working Set. For applications that are known to egregiously waste RAM, this may be a way to counteract the behavior without stopping the application or restarting the system. You should do this with caution and in situations where you can be sure of what the behavior will be on the process in question."

Share this post


Link to post
Share on other sites
For those how have to wait it out you may wish to try a feature available in SysInternals VMMAP program. Note: I have not tested this myself and this should be explored in a test environment first... Your mileage may vary.

 

Here is a write up from TechRepublic with links to the software.

"The most beneficial feature of VMMap is probably the Refresh menu option to Empty Working Set. For applications that are known to egregiously waste RAM, this may be a way to counteract the behavior without stopping the application or restarting the system. You should do this with caution and in situations where you can be sure of what the behavior will be on the process in question."

I have a hard time believing that this suggestion would do anything except cause thrashing. The Working Set is simply the set of VM pages that are actively being used by a program, and so they are kept in RAM rather than being marked free. If pages are in the working set, then the program is actively using them, and will page fault each time they are touched, causing them to be brought back in from the paging store. Emptying the working set (invalidation of the pages in RAM) will simply cause a lot of disk activity from the paging store as pages are brought back in.

 

If a program truly needs a large working set, the only solution is to add more physical RAM. Now, whether Retrospect 7.7 has a poor algorithm for VM implementation (requiring excessive amounts of RAM), or is carelessly locking pages into memory (as is needed while doing I/O to or from them) without releasing them, that's a different issue. But it's not going to be solved (or made any better) by invalidating the working set.

 

Russ

Share this post


Link to post
Share on other sites

For those that complained about the assert issue or the memory leak, please take a second to patch your server and respond with the results.

 

My backups seem to be working pretty smoothly again. Only time will tell I guess.

 

Jeff

Share this post


Link to post
Share on other sites

I have installed the patch. On my first shot the backup ran about 48hour before crashing with the following errors in the operations logs. After restart on my my backup set catalog files was corrupt. I was unable to rebuild as I would continue to get the TMemory HeapAlloc errors. I would up having to delete the entire backup set start over. I restarted the script and it's now been running about 6 hours. Only time will tell if this patch resolved the problem. So far, there is no indication that it has.

 

-Matt

Share this post


Link to post
Share on other sites

Please contact tech support so we can open an escalation based on the crash logs. We went to get that problem fixed for you

Share this post


Link to post
Share on other sites

Unfortunately nothing was appended to the assert log. In fact the jobs running at the time have no log information. The operations log indicates the following error prior to the crash but I'm not sure it's related. I already have an open case...C4232123.

 

TVol::GetInfo: UGetVolumeInformation failed, Y:\, osErr 1398, error -1001

MapError: unknown Windows error 1,398

TVol::GetInfo: UGetVolumeInformation failed, Y:\, osErr 1398, error -1001

MapError: unknown Windows error 1,398

TVol::GetInfo: UGetVolumeInformation failed, Y:\, osErr 1398, error -1001

MapError: unknown Windows error 1,398

TVol::GetInfo: UGetVolumeInformation failed, Y:\, osErr 1398, error -1001

MapError: unknown Windows error 1,398

TVol::GetInfo: UGetVolumeInformation failed, Y:\, osErr 1398, error -1001

MapError: unknown Windows error 1,398

TVol::GetInfo: UGetVolumeInformation failed, Y:\, osErr 1398, error -1001

MapError: unknown Windows error 1,398

TVol::GetInfo: UGetVolumeInformation failed, Y:\, osErr 1398, error -1001

MapError: unknown Windows error 1,398

TVol::GetInfo: UGetVolumeInformation failed, Y:\, osErr 1398, error -1001

MapError: unknown Windows error 1,398

TVol::GetInfo: UGetVolumeInformation failed, Y:\, osErr 1398, error -1001

MapError: unknown Windows error 1,398

TVol::GetInfo: UGetVolumeInformation failed, Y:\, osErr 1398, error -1001

MapError: unknown Windows error 1,398

 

Share this post


Link to post
Share on other sites

Here is the error I am getting when I try to recreate a catalog file...

 

+ Executing Recatalog at 1/27/2010 11:12 AM (Execution unit 3)

To Backup Set Office_Servers...

TMemory::freeSpace: HeapValidate failed

TMemory::freeSpace: HeapValidate failed

TMemory: heap 95 2,524 K

virtual 19 104.5 M

commit 104.5 M

purgeable 0 zero K

Pool:pools, users 4 65

max allowed mem 614.0 M

max block size 8,192 K

total mem blocks 6 48.0 M

used mem blocks 6 48.0 M

file count, size 5 40.0 M

requested 114 33.7 M

purgeable 0 zero K

avail vm size 893,784,064 B

TMemory::mhalloc: VirtualAlloc(268.0 M, MEM_RESERVE) failed, error 8

TMemory::freeSpace: HeapValidate failed

TMemory::freeSpace: HeapValidate failed

TMemory: heap 96 2,525 K

virtual 19 104.5 M

commit 104.5 M

purgeable 0 zero K

Pool:pools, users 4 65

max allowed mem 614.0 M

max block size 8,192 K

total mem blocks 10 80.0 M

used mem blocks 10 80.0 M

file count, size 9 72.0 M

requested 115 33.7 M

purgeable 0 zero K

avail vm size 893,784,064 B

TMemory::mhalloc: VirtualAlloc(268.0 M, MEM_RESERVE) failed, error 8

Backup Set format inconsistency (6 at 987720527)

1/27/2010 1:37:40 PM: Execution incomplete

Completed: 1892990 files, 383.7 GB

Performance: 2769.7 MB/minute

Duration: 02:23:49 (00:01:58 idle/loading/preparing)

 

Share this post


Link to post
Share on other sites

Windows Server 2003 Std SP2

4GB of RAM

 

I just checked this morning and had another crash over the weekend.

 

- 1/31/2010 9:19:41 AM: Copying Local Disk (C:) on ad2

TMemory::freeSpace: HeapValidate failed

TMemory::freeSpace: HeapValidate failed

TMemory: heap 85 4,255 K

virtual 13 183.0 M

commit 183.0 M

purgeable 0 zero K

Pool:pools, users 3 11

max allowed mem 614.0 M

max block size 8,192 K

total mem blocks 2 16.0 M

used mem blocks 2 16.0 M

file count, size 1 8,192 K

requested 98 10.8 M

purgeable 0 zero K

avail vm size 1,498,349,568 B

TMemory::mhalloc: VirtualAlloc(256.0 M, MEM_RESERVE) failed, error 8

TMemory::freeSpace: HeapValidate failed

TMemory::freeSpace: HeapValidate failed

TMemory: heap 86 4,257 K

virtual 13 183.0 M

commit 183.0 M

purgeable 0 zero K

Pool:pools, users 3 11

max allowed mem 614.0 M

max block size 8,192 K

total mem blocks 2 16.0 M

used mem blocks 2 16.0 M

file count, size 1 8,192 K

requested 99 10.8 M

purgeable 0 zero K

avail vm size 1,498,349,568 B

TMemory::mhalloc: VirtualAlloc(256.0 M, MEM_RESERVE) failed, error 8

TMemory::SetSize: priority 2 failed from 128.0 M to 256.0 M

Assert occurred on 1/31/2010 at 9:29:12 AM

 

Share this post


Link to post
Share on other sites

Any response Robin? I have a ticket open and last communicated with K. Dukes....he never responded back after I sent him several sets of error logs. This was a month ago and he said he was escalating it. Thx.

Share this post


Link to post
Share on other sites

Please call back into tech support and we can check the database and confirm if the issue has been escalated. I probably has.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×