Backups "freeze" part way with no error

[pdb]

Many of my backups have stopped part ways. At the moment there are 5 scripts stopped on "Scanning (C:) on blah-computer", 1 script mid copy (performance and time remaining/elapsed are not moving), and 1 script in "Finding 'another-computer'"

None of these are showing any relevant errors or warnings giving me a clue as to why it has just stopped. I'm able to pause but not stop these are all i can do is force a reboot. This happened on one other occasion and stopped all scripts from executing as well. The interface is still responsive.

Interestingly the logs show the last entries at about 7-730 pm but all I can figure from that is they all stopped "at the same time"

Also, not all of the scripts stopped, some scripts finished after this time period.

Windows 7 Pro x64
Multi Server Premium
Version 15.5.0.179

Edited October 18, 2018 by zz-pdb
more info

[Scillonian]

The version of Retrospect you are running and the OS of the Backup Server and the Clients would be helpful in offering you suggestions.

Also thoroughly check you network hardware. A momentarily failing network switch, which I have had, can wreak havoc with backups.

[johnnymacgo]

Over the weekend I had the same type of freeze\hang with the console stuck at "Finding next-computer-to-backup". I'm running Retrospect for Windows Multi-Server V15.5.0.179 which I upgraded to last week. Before that I was running V15.1.2.100 for a while for no issues. It's on a Windows 2012 R2 server and the only thing that changed was the upgrade of Retrospect so I suspect its and issue with V15.5.0.179. The backup computer is running Windows 10 Pro 1803 64bit agent version V15.5.0.179.

 

 

[JamesOakley]

I've noted this with 15.5.x (Professional) on Windows 10.

The trigger here seems to be if a client is not available for whatever reason. Retrospect hangs looking for that unreachable client. The trouble is, it doesn't timeout like it used to on earlier versions, and move onto the next task. It hangs totally. If you try to close Retrospect, or relaunch from the Dashboard, it never does. The only way to get things back is to kill retrospect.exe in the task manager.

I always notice emails telling me that a backup run had a problem. It takes me longer to notice I haven't had any emails from Retrospect for a while, so I need to find out what's wrong.

[Scillonian]

Never experienced this situation (yet) but there could be Retrospect it trying to show you a dialog that requires your action but has become hidden behind the main Retrospect window*. Under the normal Windows UI tools there is no way to get this hidden dialog to the foreground with the result that the only way to regain control is to kill Retrospect.

This hidden dialog problem is not unique to Retrospect and many Win32 apps experience it on Windows 10 in particular.

There is a way to to bring these hidden dialogs to the foreground using a third party program. (Will detail when I'm back at the Windows PC.)

 

* When a Win32 program wants to put up a dialog for user interaction that dialog should appear above the host program in the 'Z' order. Especially Windows 10 seems to get the 'Z' order wrong and puts the dialog under the host program window with no way in the Windows UI to bring it to the foreground. (I've also had Windows put the dialog at coordinates that are outside the valid display area.)

[JamesOakley]

The dialog issue may be behind this. It's hard to find out, by definition. If that were the case, fixing the z-order wouldn't solve the problem, since Retrospect usually runs without the UI open. If Retrospect is running in the background, the only way to get the UI up to interact with it is to restart it, which cancels any running jobs.

[Scillonian]

Yes, 'Z' order will be irrelevant when running in the background via Launcher Service.

[JamesOakley]

Yes, and in fact it can't be an invisible dialog in my case, because the front-end doesn't stop responding, just the engine. So I cannot "stop" the running job - it just sits there trying to connect to the client. I also cannot close the Retrospect main window, and I cannot relaunch it from the dashboard - if I try, nothing happens. But I can browse around the application, read logs, look at selectors, clients, volumes, etc. - none of which would be possible if the program were paused waiting for me to click "OK" somewhere.

[pdb]

On 10/13/2018 at 11:33 AM, Scillonian said:

The version of Retrospect you are running and the OS of the Backup Server and the Clients would be helpful in offering you suggestions.

Also thoroughly check you network hardware. A momentarily failing network switch, which I have had, can wreak havoc with backups.

whoops, added

Windows 7 Pro x64
Multi Server Premium
Version 15.5.0.179

This hasn't been a problem in the past, even with network issues it would error with "network communication error" or something along those lines. As far as invisible dialogs go, usually Interactive Services Detection from windows pops up with something (image attached)

[JamesOakley]

4 minutes ago, zz-pdb said:

pops up with something (image attached)

Go on, I'm intrigued: when you click "View the message", what is it?

[pdb]

On 10/18/2018 at 2:17 PM, JamesOakley said:

Go on, I'm intrigued: when you click "View the message", what is it?

its just the usual messages of scripts that execute while retrospect console was not open (usually it is). those messages are usually pertaining to scripts that weren't successful because a client was not found or similar (user turned computer off type of stuff).

Don't believe this is related, usually the console is open and so these messages don't come up.Just brought it up since we were talking about hidden messages.

[JamesOakley]

But if it's running in the background, with no UI visible, it shouldn't pop a dialog that you have to click "OK" to before anything continues.

I'm now on 15.6, and it just did it again - no backups ran for 3 days until I noticed that they hadn't. Killed retrospect.exe, and started the program again, and the waiting executions all began to run. The thing is, under 15.x for Windows, even Desktop edition gets multiple execution units, so one stuck job shouldn't freeze the whole setup.

[pdb]

5 minutes ago, JamesOakley said:

But if it's running in the background, with no UI visible, it shouldn't pop a dialog that you have to click "OK" to before anything continues.

I'm now on 15.6, and it just did it again - no backups ran for 3 days until I noticed that they hadn't. Killed retrospect.exe, and started the program again, and the waiting executions all began to run. The thing is, under 15.x for Windows, even Desktop edition gets multiple execution units, so one stuck job shouldn't freeze the whole setup.

the dialog only came up as I tried to start the UI up again.

Same here actually. updated to 15.6 and it's been fine for about a week and today, came to find 5 or so scripts just froze with no error. Had to kill retrospect.

[JamesOakley]

1 minute ago, zz-pdb said:

just froze with no error. Had to kill retrospect.

This is a pretty serious regression since the 12.x versions. Certain errors (specifically including a client being unreachable) no longer fail elegantly with an emailed report, but cause the whole engine to hang so that nothing is backed up. Further more, the hang is serious enough that it can't be restarted from the dashboard - it requires the task manager with elevated permissions.

[DavidHertzberg]

JamesOakley,

I've pointed out here that, on the basis of other threads, Retrospect Windows 15.6 seems to be a "bad release".  The OP of the thread I've linked to in the preceding sentence discusses a rather similar problem.  On the basis of what you're saying, it looks like that may extend back to 15.5. 

Here's why and how to file a Support Request, which IMHO is the only way to get Retrospect Tech Support to notify Engineering that this bug exists.

[JamesOakley]

25 minutes ago, DavidHertzberg said:

Here's why and how to file a Support Request, which IMHO is the only way to get Retrospect Tech Support to notify Engineering that this bug exists.

Before spending time on a Support Request, (from experience) does doing so actually get their attention?

[DavidHertzberg]

On 10/31/2018 at 4:44 PM, JamesOakley said:

Before spending time on a Support Request, (from experience) does doing so actually get their attention?

JamesOakley,

It sure as heck gets more of their attention and memory than simply posting about a bug on this Forum, which nobody at Retrospect Inc. reads anymore (per CEO J.G.Heithcock).  That fact is AFAIK one reason Retrospect Inc. instituted Support Requests; previously the organization had a sad record of often taking 4 years or so to fix bugs.

Besides, how much time do you really need to spend on a Support Request?  After supplying installation details that they definitely need to isolate the bug, all you need to do—as the post linked-to in the preceding paragraph says—is to copy-paste paragraphs from your post(s) here.  You can upload screenshots as part of the Support Request.

If you really want to get someone's attention, you can also phone Werner Walter as indicated in this post, or even send an e-mail to Brian Dunagan as indicated in this post.  However I'm afraid that Brian truly needs to devote his full attention to organize the engineers in fixing this "bad release".  I'm beginning to think that x509 may be right; that, in order to "play catch-up ball" after this situation (third and fourth paragraphs), Retrospect Inc. may have entrusted some enhancements to R. non-V. to developers in China—without ensuring thorough testing.

I think I can safely say that the old-timers at Retrospect Inc. will be sufficiently be scared by knowing that they have put out a "bad release".  If you read between the lines and follow the references in the second and third paragraphs of this article section, you will realize that Retrospect Mac practically "went down the tubes" as a product in 2009-2011 as a result of  the "bad release"—one that was really the fault of EMC management rather than the developers— of Retrospect Mac 8.0.

P.S.: In last sentence of first paragraph added link to Engst 2009 TidBITS overview of Retrospect Mac 8, where "Cracks in Retrospect’s architecture started to show ...." reflects delays in fixing bugs.

Edited December 12, 2018 by DavidHertzberg
In last sentence of first paragraph changed link to point to Retrospect Windows Dashboard bug that was noted in 2013 and "fixed" in 2017

[JamesOakley]

Duly reported

[pdb]

I submitted a support request a while back and they responded pretty quickly with a link to a beta version of Retrospect. This release (15.6.0.302) has been working for about 3 weeks with no issues.

[JamesOakley]

That's more help than I got. I'll wait for the 15.6.x release and upgrade hopefully. Thanks for the good news.

[DavidHertzberg]

10 hours ago, JamesOakley said:

That's more help than I got. I'll wait for the 15.6.x release and upgrade hopefully. Thanks for the good news.

It'll almost certainly be a 15.7 release with other bug fixes.  The fact that it isn't out yet, more than 3 weeks after zz-pdb got a beta version, confirms my guess that 15.5 and15.6 were "bad releases" with a number of bugs.

[DavidHertzberg]

As of 29 November there is a new Retrospect Windows Engine/UI version 15.6.1.104 that probably fixes this bug.  The Release Notes say "Backup: Fixed issue where scripts hung due to Management Console integration (#7753)".  My guess is that it's the official release of what zz-pdb received nearly 4 weeks ago.

It worries me that the release number is still 15.6.x.x instead of 15.7.x.x, and doesn't include any other bug fixes—other than a few related to Remote Backup.

[x509]

I started getting this client timeout issue, same as the OP and others, only after I upgraded from Retrospect 12 to 15.6.  (And I'm regretting that more and more.)

I'm posting this to offer "moral support" to everyone here.  I will try out the new release 15.6.1.104 in the next day or so, and see if it addresses any of my four outstanding issues with this damn 15.6 release.

x509

[JamesOakley]

On 12/2/2018 at 12:41 AM, DavidHertzberg said:

The Release Notes say "Backup: Fixed issue where scripts hung due to Management Console integration (#7753)".

 

Or even: "Remote Backup: Fixed issue where Retrospect would not time out when searching for a remote backup client (#7748)"

Support had me deleting configxx.dat files, so that they were regenerated from the equivalent .xml files. That meant losing all volume information in my installation, which would have meant setting up volume groups from scratch, and then adding source volumes back into each scheduled script before it would run, hoping I remembered correctly how all that was configured.

I didn't last long before I put the backup copies of configxx.dat back, and waited for a proper fix, suspecting that a corrupt config file was probably not my problem.

Edit: They also got me to uninstall InstantScan from the clients. I can probably re-enable that now.

[DavidHertzberg]

23 hours ago, JamesOakley said:

 

Or even: "Remote Backup: Fixed issue where Retrospect would not time out when searching for a remote backup client (#7748)"

....

....

....

JamesOakley,

Do you actually use Remote Backup?  My impression, from this KB article and other Retrospect Inc. marketing material, is that the facility is mostly designed for backing up employee Sally in Shanghai from your constantly-running "backup server" in Britain.  Sally's "client" must be on a Proactive script, but that's OK because you can't keep the time difference straight—and because Sally's "client" is a laptop that she carries around with her when she isn't in your Shanghai office.  If Sally accidentally deletes a file, she can do a User-Initiated Restore to get it back from your "backup server".

Is anyone using Remote Backup yet?  Speak up; inquiring minds want to know.