Interesting. I just finished discovering a specific set of bugs in Retrospect, and challenges in our router(s) and local network apps, that directly lead to the above anomalies in finding and/or connecting to clients. (Yes, all of the following has been submitted as a bug report.)
I'm running a more Windows-centric network, with a little OSx tossed in, so my tools are a bit different.
WireShark: shows packets actually traveling. Most useful is a filter of udp.port==497 || tcp.port==497
tcpdump (command line in linux and/or osx) - monitoring port 497
TCPview (from SysInternals, now owned by Microsoft) - sort on port. Look at what is listening to 497 and on what IP address(es)
(command line) ipconfig and also "route print"
In Retrospect: go into Clients -> choose a client-> access tab -> Interface -> Configure Interfaces ... and see what default interface IP address is.
Things to watch for:
Are UDP broadcast packets being received by clients? (eg 192.168.x.255, port 497)
For multicast, are packets getting to clients? (eg 184.108.40.206/4 -- Retrospect uses 220.127.116.11 UDP port 497)
Are clients responding to those packets (UDP broadcast or multicast) (initially to UDP port 497 on the backup system)
If crossing subnets, is TTL set high enough to reach the client?
What could possibly go wrong? Heh.
Here are anomalies I've seen:
Often, some app will create virtual interfaces. This includes npcap (for Wireshark etc), VMware, TAP-Windows (comes with Windows?), etc... This has led to:
On some of my clients, some virtual interfaces have APIPA addresses (169.254.*) -- which makes it obvious when retrospect chooses the wrong interface to listen on! (Workaround: I uninstalled the TAP-Windows adapter as I don't need it. And I temporarily disabled npcap on the one workstation where that got in the way.)
On my retrospect backup desktop, retrospect chose one of the VMware virtual adapters as the default adapter! This even though the real gig adapter has higher priority etc etc. (Workaround: create another adapter in Retrospect)
The result in either case: I can't see the clients, even though ping works.
I have a network security system. It regularly scans all ports on all subnets. Some (but not all) clients get confused by this, with the retroclient app hung up on an IP connection in CLOSE_WAIT status. The result: the client is never available for backups. Yet it is visible to subnet or multicast.
We switched to a pfSense firewall/router. I just discovered that multicast forwarding is badly broken.(Workaround: manually installed pimd.)
Similarly, UDP broadcast is often blocked by firewalls. Make sure the packets are getting through!
Having fixed and/or worked around ALL of the above, and rebooted everything... I can now reliably use either multicast or subnet broadcast to connect with clients.