Clients stop responding

[jfgilbert]

Simple network with 3 iMacs, all Intel, all 10.6.1.

Running Retrospect 8.1 build 622 and clients 6.3.027.

I can sometimes backup the clients once, then they become unavailable. I have to fully stop the clients and restart them, sometimes several times, before they start responding again.

When the client does not respond, the scripted backup does not run, of course, and if I try to browse the client, I get an error "Source Unavailable." Sometimes, the client responds when I check it by browsing, then it does not respond when the backup runs later that day. I have perhaps had 2 backups running automatically without problems in a month, and that's with daily checks and restarts.

This has been happening consistently since I installed 8.1. Both clients do the same thing, but not necessarily at the same time - sometimes one works and not the other and I could not discern any pattern.

I am backing up to a NAS RAID, but when I tried to back up to a USB drive attached locally, the same thing happened.

I am only backing up user files, no system files, caches, sparsebundles or anything unusual, so it has nothing to do with SL's compression scheme.

I waited until 10.6 compatibility was claimed before posting my problem, but it seems that something is seriously broken.

Edited October 29, 2009 by Guest

[rhwalker]

What does your network infrastructure look like?

 

How do the iMacs receive their IP addresses? If by DHCP, is it a static map by MAC address so that the clients always get the same IP?

[jfgilbert]

The master and one of the clients are wired to a hub. The other client is wirelessly linked to an Airport Extreme that acts purely as an access point and is also connected to the hub. The hub sits behind a Cisco ASA that runs the DHCP server. All 3 iMacs have manually assigned addresses, DHCP is only used to assign DNS servers to them, and serve other devices.

All the iMacs can be seen from any of them at all times, even when the clients do not respond. The exact same configuration was working fine with 10.5 and the previous version of Retrospect.

Thank you,

jacques

[rhwalker]

That's a real stumper.

 

And how are the clients added to Retrospect?

 

The exact same configuration was working fine with 10.5 and the previous version of Retrospect.

And what was that previous version? 6.1.x or 8.1.x ?

 

Same client versions as before or different?

[jfgilbert]

The previous version was 6.1, and I think it was a different client.

I have removed and added the clients several times, both by scanning or by direct address, it does not seem to matter.

After seeing that there were some reports of problems with corrupt config files, I have uninstalled fully and reinstalled the engine a few times, still with no effect, meaning it works once or twice and then loses the clients.

I have checked the OS/X firewall, it does not matter whether it's off or on with client permission.

There are no error messages on the console, and nothing on the router log.

I am stumped too.

Thank you for your assistance.

[rhwalker]

I have checked the OS/X firewall, it does not matter whether it's off or on with client permission.

Have you tried it with both the client and the Retrospect engine computer's firewall off?

 

How are you installing the clients? Using a local install from the .dmg with the Apple (non-Vise) installer?

 

Russ

[jfgilbert]

Yes, I have turned all firewalls off, it did not help. The last installs I did were with the new client dmg and the proper installer (not Vise).

The strange thing is that it will work sometimes, usually once, so I cannot imagine that it is a network or install issue. One thing I noticed, but could not reproduce reliably is that it may be related to the login status of the client computer. The engine computer may be able to 'see' the client while I am logged on the client, but will no longer see it after I logged off. Still that could only be a coincidence or combined with something else because it is not consistent.

I really appreciate your efforts to help, Russ.

Thanks.

Jacques

Edited October 30, 2009 by Guest
typo

[rhwalker]

And you are sure that there's not a vestige of the old client on the machine? I'm wondering if you might not have two clients competing for port 497.

 

try, in terminal, when it gets into this mode:

 

ps axlwww | fgrep ret

 

netstat -n | fgrep *.497

 

report your results.

 

Russ

Edited October 31, 2009 by Guest

[jfgilbert]

I will try that tomorrow, I am not near the machines now.

Thanks.

Jacques

 

[jfgilbert]

Just ran the commands, this is what I got:

Jacques-Gilberts-iMac:~ jfg$ ps axlwww | fgrep ret

0 10345 1 0 33 0 106004 1940 - Ss ?? 1:15.82 /Applications/Retrospect Client.app/Contents/Resources/retroclient

501 11332 11316 0 31 0 2425696 196 - R+ s000 0:00.00 fgrep ret

Jacques-Gilberts-iMac:~ jfg$ netstat -n | fgrep *.497

udp4 0 0 *.497 *.*

 

The client on which I ran that was not responding, and I logged in to run the commands. After I logged in, it did respond. After running the commands, I logged off, and it stopped responding. Testing a little further, I find that the logged status of the client machine appears to have an effect, more consistently than I had observed in the past.

I checked the console of the machine on which I was running the Retrospect Console, and found these messages that I had not noticed before (don't know how I have missed them, perhaps because I had only looked at the client's console messages):

10/31/09 1:11:36 PM Retrospect[11368] BrowseController::showBrowseView exception: RefVolume::Connect {Desktop} Bad Passowrd or Other Access Failure

10/31/09 1:12:19 PM Retrospect[11368] SourcesController::refreshSource exception: RefStation::CalculateSpeed failed; error -530

10/31/09 1:12:44 PM [0x0-0x4a14a1].com.emc.retrospect.client.cpgui[11373] Usage: retroclient [--help] | [-setpass | -log n | -ip n]

 

Hope that makes some sense,

Jacques

[rhwalker]

Yes that does. It shows that there is only one Retrospect client running, and it probably is the right one.

 

Interesting about the logging in.

 

The Bad Passowrd (sic, yes, it is misspelled in the error dialog) is the clue. Follow that troubleshooting path. Either the client password is bad (for the chat between Retrospect and the client) or else there is a share on that client that can't be authenticated.

 

When you installed on that machine, did you install as an admin and provide the proper credentials?

 

I seem to recall something about a bug whereby Retrospect was unable to mount a share if the share didn't require authentication. Could this be the issue?

 

Russ

[jfgilbert]

Hello Russ, and thank you for staying with this problem for so long.

Yes, I did install as admin, and gave the proper credentials.

I looked at the password issue, but the client behaves normally at times - e.g. when someone is logged in on that machine - so I thought that it could not be an incorrect password. I found a problem reported in another thread where someone was having problems with permissions in the Retrospect Client app, so I checked the resources directory in the client app package. Some of the files were owned by root/admin and others by root/wheel. It did not seem like a problem, since they all had permissions set at rwxrwxr-x, but I changed the group ownership to wheel on all. It did not help.

I am not using any shares in my Retrospect configuration. I have tried to mount the drive of the other machine as a network share on the engine computer, and that seems to work - at least I can browse the drive even when the machine is not logged on. Of course, it completely bypasses the client, so I would expect that the performance would be terrible and it is really not the way it is supposed to work.

I did some more testing and it seems quite consistent now that the client responds when I am logged onto that machine, and stops responding (i.e. Source unavailable) as soon as I log off.

Ran out of ideas.

Jacques

[rhwalker]

I suggest that you give EMC Retrospect support a call.

 

Russ

[jfgilbert]

I suggest that you give EMC Retrospect support a call.

Yes, I suppose I will have to do that, but I don't expect much from them, and it bothers me to have to pay for them to help fix their own problems. I was really disappointed by how this "upgrade" was handled; I had to set up a new file server because they no longer support FTP, then I had some problems and waited for official SL support, then I found that it is still behaving erratically...

You have seen the posts on the forum. I know that people don't post when things work, but it still looks to me like there are way too many unresolved issues. I have been a software developer most of my life and I know a bad project when I see one. This one has so many symptoms, it looks like a caricature: Major upgrade without new functionality, features removed, late to market, pushed out the door without QA, no documentation (!), voodoo fixes (remove and re-install, see if it works better), mysteriously corrupt config, etc. I could tolerate this - for a while - for a standard application, but backup is much too critical. I cannot do test restores everyday, just to check that the backup is readable, but I don't think that I can trust it if I don't, so it is self defeating.

 

Thank you for your help, Russ, you are a real asset to this forum.

Cheers,

Jacques

[rhwalker]

Yes, I suppose I will have to do that, but I don't expect much from them, and it bothers me to have to pay for them to help fix their own problems.

It's my understanding that the support is at no charge for bugs like this during this initial period of instability of Retrospect 8.

 

I have found Retrospect support to be quite good, a cut above what most vendors have. It's not "Bob from Bangalor" who is clueless.

 

The support group is caught in the middle on this, and they didn't write the code. They are doing an admirable job trying to support an undocumented product that was "released" before it was ready with inadequate testing (if any).

 

Russ

[jfgilbert]

You are right, I talked to support yesterday and they are good, helpful, and have an excellent attitude. I do not have a resolution yet, but they are trying. I will post here when I have a solution, in case anybody else sees something similar.

Thanks,

Jacques

[skerkvli]

I too am/was having this same problem, among a few others. I am backing up a local volume, 3 laptops, one desktop and 3 servers. This is an all Mac environment and all on the same subnet. No software firewall settings on any of the machines.

 

In the Sources area of Retrospect Console, the one server is shown but when I try to browse either of the two drives I get the "Source unavailable". In effort to resolve this, I tried to update all remote machines from Agent Version 6.3.019 to 6.3.027. All machines updated except the one server. It never reports that it fails the update, it just never updates. I had trouble for a while where the console would not show the correct version of the Agent on the remote machines unless you did a 'Refresh' on it. If I quit the Retrospect Console and reopened, it still showed the older version of the Agent. This continued for awhile but not know they all show the most recent version 6.3.027 with the exception of the one server not getting backed up.

 

I removed that server from the one backup script and then made repeated attempts to delete the Source (server). Unfortunately , the server kept showing up again in the source list. I could not seem to delete it. It would disappear for awhile but then it showed up again.

 

After many attempts over the course of an hour I got the server to not be a connected source. When I tried to add it I would receive a 'Client is reserved (-505)" error.

 

Reading through the posts, I decided to restart the one server that I could not log into with Retrospect Console. The server had an uptime of 52 days running server 10.5.7 prior to the restart.

 

After a restart, I tried to add the server to the source list and it just spun for several minutes after I typed in the password. Making no headway I launched the Retrospect client on the server in question. Immediately the Retrospect Console added the server to the source list.

 

I was able to add the server's volume to the backup script and the server is now being backed up as I type this. I just hope it stays working so other issues can be worked on.