We have been having having a similar problem trying to run retrospect as a backup server. I believe I have narrowed our problem down somewhat.
Server: new G4 desktop, 512 meg memory, OSX 10.1.5 Retro 5.0.205 with the latest RDU, ASP Quantum DLT 4000 SCSI tape attached to Apple provided SCSI interface.
Clients: All the latest OS9, OSX, and Win
In today's instance:
1. backing up a client failed with a "unable to reserve client - not visible on network" error (-1028)
2. subsequent clients come up as "not responding" when polled.
3. When I quit the server and get into the Client Configuration list the clients are listed as "responding" in the client list, but when you try to do a "get info" on the client you get a "-1028 Client not responding" error.
4. I used the OSX Network utility to try to ping the clients and they respond fine. All other TCP communications (like telnet, web browser, etc. seem to work also).
5. I then tried to quit retrospect, the tape drive light flashed for a while then the server sat there with the spinning ball (I left it in that state for 20 min).
6. I switched to the process viewer and force the LaunchCFMApp associated with retrospect to quit.
7. Then I relaunched retrospect and went to the client config list. The clients were once again responsive to the "get info" command and to wiewing their configurations.
8. I restarted the backup server and it was off and running.
If I don't use a backup server script but a direct scheduled backup script instead, the clients that happen to be powered on and not sleeping back up fine, but the backup server arrangement is a far easier way to get all of our 70+ clients covered.