Windows clients no longer accessible

[twickland]

We have run into a peculiar issue where most of our Windows 10 client machines can no longer be backed up. They are all connected via direct IP (due to our network's security settings), and have a variety of different client versions from 15.6 to 16.5.1. On each machine, the Retrospect Client app appears to be normal, the IP address has not changed, the Windows firewall is set to enable communication, and the computer is awake, but when access is attempted from the backup server, we get a -519 error. Mac client machines are unaffected.

Even more oddly, a newly-added Windows 10 client was able to be backed up once before going incognito.

I assume this is most likely an issue with our local network, since I would think if the problem were more widespread, there would have been other posts to this forum. Any ideas as to what might be the issue here, so I can intelligently discuss the problem with our network techs, who have no knowledge of any particular requirements that Retrospect might impose?

Does Retrospect still use TCP and UDP port 497?

We're running Retrospect 16.5.1 server on MacOS 10.14.6.

Thanks.

 

 

 

[DavidHertzberg]

twickland,

Here's the Knowledge Base article on the -519 error.  "Error 519 means Retrospect had established a network connection with another computer and was communicating through that connection when something caused the connection to be severed." 

"This error is one of Retrospect’s most challenging errors to troubleshoot because networks involve a large number of variables. Causes of error 519 range from a simple software conflict on an individual workstation to a faulty network component that does not cause trouble during normal (non-backup, less intensive) use. Weak links can exist in the software or hardware level, on the backup computer or the client computer, or in the infrastructure of your network. This document aims to help you narrow down what might at first seem like an unwieldy problem to solve."

When you say "They are all connected via direct IP", do you mean "clients" are all defined with Direct Access Method—known in Retrospect Mac as Add Source Directly?  I've had to do that to avoid -530 errors that seem to be somewhat related to changes in networking hardware.

Yes, Retrospect still uses TCP and UDP port 497; it multicasts 224.1.0.38.  Here's MrPete's post on comprehensive Windows trouble-shooting procedures.

Here are some very cynical WildA**edGuesses:

• Your Professional IT Guys (sexist term,  and naughty acronym when 't' is lower-cased, entirely intentional) installed an anti-virus product that interferes with Retrospect Windows Clients, one not yet fixed in this 2013 post. I'd add Windows Defender Firewall to the list of anti-virus products, except that "a newly-added Windows 10 client was able to be backed up once before going incognito"—which sounds like somebody did something to that machine's software after it arrived in the organization with a presumably-latest version of Windows 10 ( ask PIGs: What was arrival version?) .
• Your Professional IT Guys did something to the networking hardware/software with which your organization's Windows "client" machines are connected.  Your organization's Mac "client" machines are unaffected because Mac users are isolated in one or more "leper colony" departments .🤣
• The Retrospect "Inc." engineers messed up something in the Retrospect Windows 16.5.1 Engine.  They've been doing that in the last two X.5.Y releases, IMHO because management is trying desperately to fully implement major features that were previewed in the corresponding X.0.0 releases.  A known example (which unfortunately probably has nothing to do with your problem) from a recent thread in  this Forum is connected with "Improved NAS support with auto-adding existing NAS share mounts" in 16.5.0.  I happened to mention it in a short phone conversation with the head of North American Sales last week, and he said that is a known problem that will be fixed with a new release "within about a week".  My candidate for the Engine "improvement" that caused your problem is "Networking: Retrospect now honors service priority when choosing a default interface (#7058)"—whatever "service priority " means—in the cumulative Retrospect Windows Release Notes (assuming carryover to Mac Engine).

P.S.: The head of Retrospect Tech Support replied in my Support Case 3 weeks ago "You can try going to Network in the Retrospect Preferences and click Advanced. Change the network timeout from 300 seconds to 9,000 seconds and see if Retrospect is able to complete the backup".  That Support Case was about my getting -559 errors after precisely two hours when running a Recycle backup of my MacBook Pro.  He had also suggested "Inside your energy saver control panel, you may need to set the screen to never sleep (and use a screen saver) and check 'prevent computer from sleeping automatically when display is off'. Or try a combination of those options. In some cases Uncheck Put hard disk to sleep when possible can also help."  Those latter suggestions are obviously for a Mac "client", but possibly Windows 10 has—as Nigel Smith said of macOS in my thread about the -559 problem—"tied computer-sleep and display-sleep together -- the default is that, when your display sleeps your computer does too."  The combination of suggestions fixed my -559 problem.

Edited November 20, 2019 by DavidHertzberg
Add P.S.; clarify first bullet-pointed paragraph; in 4th paragraph, link to MrPete's comprehensive trouble-shooting procedures, add multicast address; add Retrospect Windows terminology to 3rd paragraph

[Nigel Smith]

On 11/15/2019 at 9:38 PM, twickland said:

We have run into a peculiar issue where most of our Windows 10 client machines can no longer be backed up.

FWIW, my Mac RS server has been showing "error -515 (Piton protocol violation)" errors from our Win10 PCs since the beginning of this month -- no successful backups at all.

I don't think it's a network hardware issue, since they *are* being successfully backed up by the Windows RS server!

I was keeping quiet, assuming it was my rather old Mac RS software playing badly with a Windows update, but I'll have a closer look tomorrow.

[DavidHertzberg]

Nigel Smith,

Is it possible that your "client" machines are defined to your Retrospect Windows "backup server" by the Direct Access Method (see pages 296-297 of the Retrospect Windows 16 User's Guide), but defined to your Retrospect Mac "backup server" with Use Multicast?  If so, IMHO that would mean that the latest version of Microsoft Windows Defender has added something that blocks multicast on 224.1.0.38, and that your "rather old Mac RS software" still defines "clients" by a method that used to work.  Alternatively your "clients" are also defined to Retrospect Windows by the default Multicast, but the latest version has some code for working around Windows Defender that isn't in your "rather old" version of Retrospect Mac.  FYI, the -515 error is defined on page 491 of the Retrospect Windows 16 UG. 

BTW, could you tell us why you are backing up the same "clients" with two "backup servers—and what version of Retrospect Mac you are using?

twickland,

I ask again: When you say "They are all connected via direct IP", do you mean "clients" are all defined with Direct Access Method?

[Nigel Smith]

4 hours ago, DavidHertzberg said:

BTW, could you tell us why you are backing up the same "clients" with two "backup servers—and what version of Retrospect Mac you are using?

Simple enough -- we we've been using the Mac version for years, currently on 13.5. With Apple dropping server and enterprise I was looking at moving to Windows (now at 15.6.1), if only for 10GbE, so we've been running both in parallel But the "evaluation" has gone on rather longer than expected -- partly because the goalposts keep moving (eg Apple introducing 10GbE on the Mac Mini) but mainly because I really, really, really don't want to permanently move to Windows 🙂 

I'll have to make a decision soon, at which time version numbers will jump to current. But, at the moment, all clients are being backed up at least once -- and if it ain't broke...

5 hours ago, DavidHertzberg said:

Is it possible that your "client" machines are defined to your Retrospect Windows "backup server" by the Direct Access Method

The clients are "Direct IP" (the terminology RS uses in the Console's Sources summary when you've used an IP address to "Add source directly...") on both servers (since they're statics on a private subnet), so it isn't that. No changes to the Mac server, so it isn't that. Which leaves either network hardware (unlikely, since both servers are on the same switch), the router/IPS security settings (possible, but unlikely since both servers are on the same subnet so are subject to the same policies etc), or Windows. All I need to do is test with another Windows 10 client on the same subnet as the server to start narrowing things down. But, as above -- meh, Windows 😉 

In my experience, there's a commonality between -515 and -519 errors, a brief "ceased to communicate" usually being reported as a -519 but sometimes as a -515 -- especially when, as in my case, the software isn't as up-to-date as it should be. Since the troubleshooting is similar for both we may as well consider them the same.

[twickland]

16 hours ago, DavidHertzberg said:

twickland,

I ask again: When you say "They are all connected via direct IP", do you mean "clients" are all defined with Direct Access Method?

Yes, clicking on "Add source directly..." and entering the client's IP address. We need to do this because our IT people block multicast and the clients are found in multiple subnets. 

Incidentally, for clients added by the direct method, Retrospect typically displays the -519 error message (rather than -530) when the client is simply not visible on the network. It even shows -519 in the add client dropdown if one enters an unassigned IP address.

As for the Windows client that briefly went incognito, it's now back and being backed up successfully. We have a total of 20 Windows clients, only four of which are currently able to be backed up. Three of those are on Windows 10 and one is on Windows 7. Knowing the users, I have a suspicion that those three Windows 10 machines may not have had the latest security patches applied.

Our networking people say there have been no changes to our wired network (we don't use WiFi for any backups). I haven't yet heard from our desktop support folks to get their take on this issue. Something is definitely amiss, as the client machines that are not visible on the network in Retrospect are also unresponsive when port 497 is pinged.

[DavidHertzberg]

Nigel Smith and twickland,

According to the Retrospect Windows 16 User's Guide pages 491-492 (the same information was in the Retrospect Mac 6 UG, but was deleted in the glorious Retrospect Mac 8 upgrade), -515 and -530 errors have to do with connecting via the Piton Protocol (pages 293-294)—while -519 errors have to do IME with loss of an IP address connection during backup (e.g. when I forget my Add-Source-Directly MacBook Pro is being backed up and shut it down). 

The Piton Protocol is Retrospect's 18-year-old method of allowing administrators  to add "clients" via Multicast, which means backup administrators—who are often non-IT key office functionaries with knowledge of what data has to be backed from what machines— don't have to know anything about IP addressing.  It definitely requires use of the 224.1.0.38 multicast address as well as TCP and UDP on port 497; "clients" added with Add Source Directly may not even need port 497—but from what twickland says in his last paragraph directly above I might be wrong about this.

It still sounds to me as if your installations'  PIGs or Microsoft Windows Defender have now installed an anti-intrusion "improvement" that disables either 224.1.0.38 multicast and/or port 497 by default.  I consider this to be a menace to Retrospect's continued viability, and I urge both of you to file Support Cases as soon as you can pin this down for your installations—and please post in this thread what you pinned down.  Be sure to make the point that all the StorCentric-prioritized development (for which I have both official and informal sources) of a version of the "backup server" running on Drobo isn't going to sell many Drobos, if administrators can't back up their Windows "client" machines.  Maybe all that is needed is a prominent mention in the supposedly-forthcoming UG rewrites, plus a Knowledge Base article.

P.S.: What twickland said typically happens didn't apply to me this early this morning.  Waking up after the scheduled time for my No Media Action script that backs up my MacBook Pro every day except Saturday , I stupidly booted the "backup server" in my bedroom before booting the MBP "client" in my study.  According to twickland's experience I should have gotten a -519 error, since my MBP was Added with Add Source Directly.  Instead I got a -530 error.

 

Edited November 21, 2019 by DavidHertzberg
P.S.: Per twickland I should have gotten a -519 error, since my MBP was Added with Add Source Directly; instead I got a -530 error

[twickland]

We have solved the problem. The cause seems most likely to have been Retrospect's new automatic client update feature. In any case, at some point the client software on the affected machines became corrupted or was actually deleted. In the cases we tested, if the client was removed from the sources list and we attempted to re-add it, the process failed due to an incorrect public/private keypair. We decided to just go ahead and reinstall the client software locally on all our "invisible" Windows clients, which solved the problem; we did not need to delete any client configuration files, so these must have been OK.

 

[DavidHertzberg]

twickland,

First, nice problem solving. 😄  Please file a Support Case; maybe the engineers can fix this in the point-release of Retrospect  Mac 16 which I was told would come this week.  Now I know why it was wise for me to stay on Retrospect Mac 16.1. instead of upgrading to 16.5.1. 😌

In related news, I now have a possible explanation for my -530 error—instead of -519—recounted in the P.S. of this post above.  The short version is that my "backup server" seems to have acted as if my MacBook Pro "client" was defined with Multicast, even though it was and is defined with Add Source Directly.

The longer version is that on 19 November I received a fairly-recent model of CalDigit Thunderbolt 3 dock, with which I replaced the not-too-satisfactory two-year-old StarTech USB32DPPRO adapter connecting  my MBP via an ATEN KVM switch to my inherited DisplayPort Apple LED Cinema Display.  Since the CalDigit dock has a variety of outputs, I also used it to replace my Moshi USB3 to Ethernet adapter.  On 22 November, in the course of re-arranging my surge protector plugs to work around a limitation in the CalDigit dock, I tripped and fell (my balance is no longer good after two spinal operations) while working behind one of my study desks.   The fall caused my rear end to come down hard on the HP JetDirect EX Plus adapter that connects my 24-year-old HP LaserJet 5MP's parallel port to my Ethernet LAN. 

The replacement JetDirect EX Plus arrived this afternoon, and I created a replacement static IP 192.168.1.200 entry for it (because the MAC address of the new EX Plus is different) in my Verizon FiOS "gateway" router.  In the process I noticed that my MBP was now showing up in the router at the non-static address 192.168.1.154 instead of at the static 192.168.1.202.  I figured out via System Preferences -> Network that this was because my MBP's Ethernet connection is now via Thunderbolt Ethernet Slot 1 instead of USB 3 (same port, different channel).  I gave the MBP the static 192.168.1.202 address in the router, and before going to dinner ran a "sacrificial" (Rule = No Files) Backup script  to make sure everything would be OK for my weekly Recycle script tomorrow.  It got a -530 error, so I Removed and re-Added the MBP "client" in Sources again using Add Source Direct—after specifying in System Preferences -> Network on my MBP that its Thunderbolt Slot 1 is  to Configure IPv4 Using DHCP with manual address; the "sacrificial" script then ran OK.

The interesting thing is that I had had absolutely no errors—except in the stupid situation linked to in the second paragraph of this post—running Backup scripts for the last week and a half.  IMHO the only reasonable explanation is that the Retrospect Engine on the "backup server" had automatically re-tried finding the MBP via Multicast because the MBP wasn't at its Add Source Direct address—and succeeded.  That would explain why I got a -530 error, rather than a -519, on the one morning when I stupidly let the "backup server" run a Backup script when the MBP "client" really wasn't on the LAN yet.  So what you said typically happens still seems correct.

[Nigel Smith]

On 11/20/2019 at 6:44 AM, DavidHertzberg said:

-515 and -530 errors have to do with connecting via the Piton Protocol

-515 is "data becoming corrupt while being transferred" -- ie the client is still connected (else it would be a -519).
-519 is "network communication failed" -- ie the client was found, then disappeared.
-530 is "client not found" and can be thrown for all sorts of reasons, from the client not being powered on through hubs/switches "invisibly" dropping multicast packets -- including my oft-mentioned "client binding to the wrong interface".

-530 is the easiest to differentiate, since the client was never there 🙂  -515 and -519 are more of a grey area -- IME, a "fluttering" switch port or NIC can give either a -515 (brief, occasional, drops cause data problems without the client dropping for long enough to trigger the "disappeared" time-out) or a -519 (more prolonged failures where RS registers a disconnect though, client-side, everything seems OK because more usual network operations -- file sharing, browsing, etc -- are less sensitive).

FWIW -- my understanding is that all the above involve the piton protocol, regardless of connection type, since that's how RS adds clients, accesses them, and transfers data to/from them.

On 11/22/2019 at 7:18 PM, twickland said:

We have solved the problem.

Hurrah! Though, in our case, it won't have been caused by RS's auto-update -- because we don't use it! Perhaps a Windows update changed a linked library, a security fingerprint, or something -- or, my particular fave, a Defender update did it's usual "reset the firewall" nonsense 🙂 I'll try and get some reinstalls done today and see what happens...