Memory Leak in the Linux Client?

[crumrine]

I have been monitoring some problems we are having running the RS linux client on RHEL ES Update 3. It seems to be eating up (virtual) memory like a champ. Very frequently we get out of memory errors (SThreadSpawn: pthread_create() failed with error 12) in the log file.

 

I ran a 'ps avxww | grep retro' and noticed that the retrospect process has a lot of memory (virtual) allotcated to is (could this be why it's exiting with the out of memory error?).

 

Here's the output of command on a couple machines we are having a problem with (after running one backup):

 

machine 1 (570MB allocated):

1419 ? S 0:38 8 152 569055 1696 0.1 /usr/local/dantz/client/retroclient -daemon

 

machine 2 (1.5GB allocated):

2157 ? S 0:06 7 152 1562719 3884 0.1 /usr/local/dantz/client/retroclient -daemon

 

Of course it's not really using that amount of memory, but could that be the reason why it thinks it runs out of memory. Maybe it's not freeing the memory up properly after using it?

 

Upon initial startup of the retrospect client the memory usage is more reasonable:

 

machine 1 (45MB allocated):

26229 ? S 0:00 7 152 45587 892 0.0 /usr/local/dantz/client/retroclient -daemon

 

machine 2 (45MB allocated):

29319 ? S 0:00 6 152 45579 880 0.0 /usr/local/dantz/client/retroclient -daemon

 

Maybe retrospect's client needs to be restarted after every backup??? Does anyone else have this running solidly in a RHEL environment?

 

Thanks,

Brian

[crumrine]

Another interesting (possibly related) symptom of this is that when retroclient chews up all the memory, the service stops responding to the "service" command. "service rcl stop" does not respond - the only way I have found to get the client in an operating state again is to kill it - then start again with "service rcl start".

[crumrine]

This has turning into kind of a daily ritual for us. We get about 1 day of the client running before we have to kill and restart it. Any ideas what might be happening with it? Anything I can try?

[crumrine]

With each backup the client allocates more and more RAM and never releases it - of course this is swapped out, but I think eventually the server says you can't have any more and the client doesn't like that so it locks up.

 

Here's what I did to help the problem (still would like a more permanent solution):

 

I created a job named "Client Restart" and had it backup nothing (using the no files selector). At the end of the backup I have the /etc/retroeventhandler script restarting the client. Here's what I used:

 

if [ "$2" = "\"Client Restart\"" ] && [ "$1" = "EndSource" ]; then

/usr/local/dantz/client/retrocpl -stop >>$LOGFILE

/usr/local/dantz/client/retroclient -daemon >>$LOGFILE

echo "Retrospect Client Restarted" >> $LOGFILE

fi

 

I scheduled this job before each backup - I imagine that we will need to do this after the backup jobs if the problem comes up again. So far we have been not had the client lock up since turning this on.

 

Note: I started this exercise by trying to do this after every source - this didn't work as it treats each volume/subvolume as a separate source and if the client exits while a backup is running it doesn't want to restart while the job is running. Unless you are only backing up one volume/subvolume, I wouldn't recommend this unless you code for a specific volume/subvolume order.

[matuson]

Does anyone know if there has been any progress on this problem?

 

Can Dantz continue to sell the product to people who need the Linux client knowing that there is such a bug in the code?

[natew]

Hi

 

Have you done a full uninstall of the client package, making sure the /usr/local/dantz directory has been removed, the client daemon was stopped, and that all of the retro* files have been removed from /var/log?

 

So far this has always worked fine for me on a RH9 system. With the 6.5.108 client.

 

Thanks

Nate

[matuson]

Nate,

 

No, I haven't tried that yet, but I will do it today and see if the backup runs tonight.

 

Thanks for your reply.

 

Michael

[crumrine]

Nate,

 

I have tried to reinstall the Linux client with no success. The same errors are occuring, where the client allocates more and more memory until it reaches some limit and the OS tells it no more, then doesn't answer requests any more.

 

I have had limited success with the client restarting.

 

Would it be possible to get a version of retrospect client written for inetd/xinetd so that it wouldn't sit around taking up more and more memory? When is the next rev of the Linux client scheduled for?

 

Thanks,

Brian

[crumrine]

I tried the Retrospect 7 client on RHEL 3 and it seemed to be a little better. The Linux client now only locks up after a week or so of usage.

 

Restarting the client is still the safest way for the Linux client to run in our environment.

 

After three weeks of running the 7 client, the server never failed to run out of allocatable memory after 5-6 days.

 

We have been experimenting with the RedHat hugemem kernel for another project. Maybe we can get two weeks out of the client if we run that on these servers. Or we could always upgrade to a 64-bit OS.

 

Brian

[crumrine]

Still leaking memory - retrospect just stops talking after it reaches the end of addressable space (3GB in our case).

 

root 25040 0.0 0.3 3120820 6988 ? S Apr26 4:20 /usr/local/dantz/client/retroclient -daemon

 

Going to turn on my client restarter script again.

 

I will gladly pay for a support incident if that's it takes to get it fixed and get an official bug report filed.

 

Any chance that will help?

[natew]

Hi,

 

I'll see what I can do.

Even if it is a flaw in the client code a tech support incident won't speed up a fix.

 

Thanks

nate