Backup hangs randomly on client and / or host PC

[spiffey]

I'm really struggling to get Retrospect 8.5 set up and backing up reliably - any help gratefully received!

 

I used to use Retrospect many years ago on both Mac and windows, and found it very reliable. However, after installing it on the PCs at work, it hasn't worked once correctly or reliably - and I don't know why …

 

It is backing up a small Windows-based network (one 'server' PC - but running a desktop OS - and 5 client PCs). All are running Windows 7 Professional SP1 (64bit), using Intel Sandy-Bridge based Quad-Core i5s (apart from the 'Server' PC which is a Quad-Core i7), and are connected to the same Gigabit ethernet hub, running at full-duplex speeds. The 'Server' PC runs the full Retrospect Professional (8.5) software, and all the others are using the Retrospect Client software. All computers back up to a 3TB external hard drive connected via eSATA to the 'Server' PC. I have set up two lots of identical backup scripts, each backing up to a different drive - one for Monday / Wednesday / Friday, and one for Tuesday / Thursday.

 

The server hosts the data files for our Sage accounts and Act! database. All other computers connect to that computer for their data. All computers are running AVG antivirus 2014.

 

Originally, I did a staged introduction of backups, starting with the server. These seemed to go well, and completed without errors.

 

The problems seemed to start (though I can't exactly put my finger on things) when I started adding clients into the mix. I set them all up identically, and yet they sometimes hang (after copying over a nominal amount of data, like 20MB say). Clicking 'Stop …' in either Retrospect itself, or in the Client software, does nothing. Only a restart will fix the issue. I should point out at this stage that the computer itself doesn't hang or crash - only the Retrospect software and backup.

 

I initially thought that this was due to it trying to copy open files; so I started restarting the computers at the end of the day, and leaving them to backup overnight - but this sometimes worked, sometimes not.

 

Also, most days, if there has been a problem with a backup, then the server won't shut down or restart. Once, all the backups completed successfully, but the server wouldn't shut down. I came in the next morning to find the 'Shutting down' screen on. After force restarting the computer, and checking the retrospect log, it seems that it had been trying to shut down for 6 and a half hours.

 

The computers are running Microsoft SQL server; it is used by our database software (Act!) to talk to Sage; however, as the data on both Sage and Act! is backed up by the server, we have not purchased (and feel we don't need to purchase) the SQL backup option.

 

I thought it might be the antivirus; but the issue still occurs despite adding an exception rule for all the retrospect software. I'm starting to suspect that the problem is with the software itself …

 

Please can anyone give me any pointers? Any help gratefully received!

[Scillonian]

... and are connected to the same Gigabit ethernet hub, running at full-duplex speeds. ...

Do you mean switch or are you really using a hub? They work in different ways.

 

... The problems seemed to start (though I can't exactly put my finger on things) when I started adding clients into the mix. I set them all up identically, and yet they sometimes hang (after copying over a nominal amount of data, like 20MB say). Clicking 'Stop …' in either Retrospect itself, or in the Client software, does nothing. Only a restart will fix the issue. ...

Check the NIC drivers are the at the latest compatible version. The symptom of copying a few megabytes then crashing can indicate a NIC driver problem. I've experienced problems like this in the past and updating the drivers solved the problems. For whatever reason driver problems don't always show up when copying files over the network when using Windows Explorer.

 

Also check the Windows Event Logs to see if there are any errors or warnings being recorded around the time when the backup fails.

 

... All computers back up to a 3TB external hard drive connected via eSATA to the 'Server' PC. I have set up two lots of identical backup scripts, each backing up to a different drive - one for Monday / Wednesday / Friday, and one for Tuesday / Thursday. ...

Are all the backups going to one drive or is it a separate drive for each script?

Edited October 25, 2013 by Scillonian

[spiffey]

Thanks very much Scillonian,

 

In answer to your question:

 

Do you mean switch or are you really using a hub? They work in different ways.

It's a switch - SOHO (i.e. unmanaged). D-Link DGS-1008D.

 

Check the NIC drivers are the at the latest compatible version. The symptom of copying a few megabytes then crashing can indicate a NIC driver problem. I've experienced problems like this in the past and updating the drivers solved the problems. For whatever reason driver problems don't always show up when copying files over the network when using Windows Explorer.

 

Also check the Windows Event Logs to see if there are any errors or warnings being recorded around the time when the backup fails.

Thanks for this. I'll give it a try and see what happens.

 

Are all the backups going to one drive or is it a separate drive for each script?

There are two drives; one for the M/W/F scripts and one for the T/T scripts. The thinking behind this was that one drive would be in the fireproof safe while the other drive is backing up. However, I couldn't see any way of doing this on a single script - recycling the backup script happened on a weekly basis rather than a daily one, and didn't seem to be any way of changing it.

[Daniels]

What edition (free or paid) of AVG antivirus are you running? I have found that AVG is processor hog and if it is set up to scan when files are accessed then this cause your hanging problem as AVG is scanning the files while Retrospect tries to back them up thus causing Retrospect to hang because it can not access the file.

[Scillonian]

It's a switch - SOHO (i.e. unmanaged). D-Link DGS-1008D.

Exactly the same switch I had problems with. The idea going around a couple of years ago when I had problems with it was that under sustained heavy load it was overheating and momentarily failing.

 

There are two drives; one for the M/W/F scripts and one for the T/T scripts. The thinking behind this was that one drive would be in the fireproof safe while the other drive is backing up. However, I couldn't see any way of doing this on a single script - recycling the backup script happened on a weekly basis rather than a daily one, and didn't seem to be any way of changing it.

According to the manual you can use multiple backup sets with a Proactive Backup script which will use whichever one of the allocated backup sets is available. Although they work in a different way to regular scripts it is worth investigating for you situation. I have used a Proactive Script with multiple destinations and it worked.

 

For my daily backups I use a Proactive Backup script to a NAS based backup set. For the off site backups I use software on the NAS to replicated the backup set and catalog to one of four hard disks on a rotation.

[spiffey]

What edition (free or paid) of AVG antivirus are you running? I have found that AVG is processor hog and if it is set up to scan when files are accessed then this cause your hanging problem as AVG is scanning the files while Retrospect tries to back them up thus causing Retrospect to hang because it can not access the file.

Thanks Daniels,

 

We're currently using the free AVG 2014 edition. I was wondering if this was the cause of the problem, as it also seemed to be erroring after the 2014 upgrade went through. I know for a fact that it caused a backup on the server itself to fail as it was running a full scan; however, I've set some exclusions on retrospect, so I thought this would rule it out …?

 

I was thinking of trying Kaspersky just to rule out AVG - are there any reported issues between that and retrospect?

[Daniels]

Even with the Retrospect exclusions AVG can still cause problems as it will scan files as they are accessed.  I have not tried Kaspersky however I can confirm that Microsoft Security Essentials as well as Symantec Endpoint Protection do not interfere with Retrospect.

[spiffey]

Exactly the same switch I had problems with. The idea going around a couple of years ago when I had problems with it was that under sustained heavy load it was overheating and momentarily failing.

I'd not considered the switch to be a problem; we've got two of them in the building, and they seemed (to me, at least) to be fairly reliable. There isn't vast amounts of network traffic going through it, and surely that wouldn't account for only about 20MB copying and then hanging - as the network itself doesn't drop out? They are in a well-ventilated area and are warm, but not hot, to the touch. There have been occasions where the backup has worked successfully, and copied over gigabytes of data without issue; the network traffic on those occasions is no more or less than on the occasions where it's not worked.

 

I think the issue is exacerbated by the fact it is happening randomly, without any sort of logic or repeatability. One day it won't work; the next time it will. If I'd been able to find the cause of the hanging, it would make things a lot easier!

 

However, I appreciate your comments, and I've another switched hub (the same model, unfortunately) which I can try to see if it's limited to the one hub. We are talking about getting one switch for the whole building, but until that happens, there's very little I can do about it …

 

According to the manual you can use multiple backup sets with a Proactive Backup script which will use whichever one of the allocated backup sets is available. Although they work in a different way to regular scripts it is worth investigating for you situation. I have used a Proactive Script with multiple destinations and it worked.

 

For my daily backups I use a Proactive Backup script to a NAS based backup set. For the off site backups I use software on the NAS to replicated the backup set and catalog to one of four hard disks on a rotation.

Thanks, I'll have a look at the manual and check this out. The way I've set it up did seem a bit of a clunky way, but it's the only way I could see of doing it at the time.

 

Check the NIC drivers are the at the latest compatible version. The symptom of copying a few megabytes then crashing can indicate a NIC driver problem. I've experienced problems like this in the past and updating the drivers solved the problems. For whatever reason driver problems don't always show up when copying files over the network when using Windows Explorer.

I've had a look at this, and Windows 7 is insisting that the drivers are up to date, but I'm having trouble tracking the exact model of ethernet controller, and Windows only reports it as Broadcom Netlink Gigabit Ethernet. It's integrated into a Dell motherboard. I've looked at the device ID (1691) but broadcom's site doesn't refer to it at all. I don't know how I can find out the exact part number of the controller, to see if there are any newer drivers I can use?

[spiffey]

Also check the Windows Event Logs to see if there are any errors or warnings being recorded around the time when the backup fails.

I've looked in the event logs, and Retrospect reported the following errors, at various times, around the times that the backup failed:

 

"Error -519 (Network communication failed)

 

Are there any common causes for this sort of error?

[Scillonian]

... I can confirm that Microsoft Security Essentials as well as Symantec Endpoint Protection do not interfere with Retrospect.

 

I can confirm that Microsoft Security Essentials (and built in Windows Defender in Windows 8 and 8.1) have not cause me any problems with Retrospect (or any other programs). Also in Microsoft Security Essentials and Windows Defender it is possible to set an exception so it ignores Retrospect's disk activity which I have done.

 

I think the issue is exacerbated by the fact it is happening randomly, without any sort of logic or repeatability. One day it won't work; the next time it will. If I'd been able to find the cause of the hanging, it would make things a lot easier!

 

However, I appreciate your comments, and I've another switched hub (the same model, unfortunately) which I can try to see if it's limited to the one hub. We are talking about getting one switch for the whole building, but until that happens, there's very little I can do about it …

I had the same problem with random backup failures — sometimes several in quick succession then nothing for days. My NAS logging momentary network link failures was what pointed me in the direction of the switch.

 

I've had a look at this, and Windows 7 is insisting that the drivers are up to date, but I'm having trouble tracking the exact model of ethernet controller, and Windows only reports it as Broadcom Netlink Gigabit Ethernet. It's integrated into a Dell motherboard. I've looked at the device ID (1691) but broadcom's site doesn't refer to it at all. I don't know how I can find out the exact part number of the controller, to see if there are any newer drivers I can use?

 

From a post on the Broadcom forums the device ID 0x1691 is the BCM57788 chipset. Current drivers are at: http://www.broadcom.com/support/ethernet_nic/netlink_k57.php

 

Windows will only report that there are newer drivers available if the manufacturer has told Microsoft they are available. Dell can also be very slow, if ever, to update their drivers to the latest available from the part manufacturer.

[spiffey]

Great, thanks Scillonian.

 

I've downloaded the drivers and will try those as the first step. I think I'm also going to remove AVG and try another antivirus option, as some process is both hanging on to external drive (meaning I can't eject it without shutting down) and also the server just chunters away to itself without shutting down. I've looked into this, and it's likely that a process that's running is not quitting. There's not much running on the computer so that's going to be the first thing to try.

[spiffey]

Hmm, it seems that there is more going on here than first meets the eye.

 

Since the issues with the client backups, I've turned all of their schedules off. The thinking behind this is that I'd try to back up the server alone. If this is stable and works OK, I'll then know it's an issue with the backup of clients, then I can investigate that further.

 

Unfortunately, when I checked the server whether it had backed up, it was saying that it was running - but it had just hung. So I first need to track down the issue on the server - there is something that is causing the backup to hang on the server itself, which could be having a knock-on effect on the client backups.

 

I checked the windows event logs; there was a critical error that was related to the fact that, following the hang, I had to force shut down and restart the computer. There were other processes that had critical errors which I've prevented from starting at startup - just to try and eliminate them as causes - I've uninstalled AVG, purged the registry, etc. and installed Kaspersky (as I'd already downloaded a trial version). This has enabled me to exclude all instances of retrospect and other network processes related to it, to hopefully prevent it from scanning while backing items up.

 

However, what has caused some disquiet was some errors have been reported by Retrospect which, following searches online, seem to be quite unique.

 

This is what has been reported:

 

"The description for Event ID 1 from source Retrospect cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

 

If the event originated on another computer, the display information had to be saved with the event.

 

The following information was included with the event:

 

Trouble writing: "1-Backup Set M-W-F" (3574288384), error -106 ( data overwrite attempt)

 

The specified resource type cannot be found in the image file"

 

I've printed off (and will look into) a fix that someone has suggested regarding the 'resource type cannot be found' error which someone came across while attempting to restore from a backup set; but I can't see any occurrences where someone has come across this while backing up. However, I presume that this error is the cause of (or strongly related to) the hang.

 

Can anyone shed some more light on these issues?

[Scillonian]

However, what has caused some disquiet was some errors have been reported by Retrospect which, following searches online, seem to be quite unique. This is what has been reported:

 

The description for Event ID 1 from source Retrospect cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

 

If the event originated on another computer, the display information had to be saved with the event:

 

...

 

The specified resource type cannot be found in the image file

 

This part of the error message can (in the case of Retrospect at least) be ignored. All Windows is saying is that Retrospect doesn't have/provide additional information connected with the error report. I have seen this part associated with other applications too.

 

The following information was included with the event: Trouble writing: "1-Backup Set M-W-F" (3574288384), error -106 ( data overwrite attempt)

 

This is the part you need to be concerned about. This would suggest to me Retrospect is having trouble writing to the disk.

 

Unfortunately, when I checked the server whether it had backed up, it was saying that it was running - but it had just hung. So I first need to track down the issue on the server - there is something that is causing the backup to hang on the server itself, which could be having a knock-on effect on the client backups. I checked the windows event logs; there was a critical error that was related to the fact that, following the hang, I had to force shut down and restart the computer.

 

When you restarted after the forced shutdown did you check the disks for errors? After any forced or uncontrolled shutdown it is good practice to check any disks, both internal and external, connected to the system at the time for errors.

 

Do you have enough space on an internal disk of the server to perform a backup to? This may help to see if there are problems with the external drives.

 

Unlike a USB disk which Windows knows is removable media an eSATA disk will be seen as fixed media the same as if it was an internal SATA disk. If there are communication problems with the eSATA disk this could be causing Retrospect to hang waiting for a response from the disk which may never come. (Although the eSATA port and the eSATA disk may support hot swapping Windows does not support it by default. To disconnect or connect an eSATA disk Windows needs to be shutdown first.)

[spiffey]

This part of the error message can (in the case of Retrospect at least) be ignored. All Windows is saying is that Retrospect doesn't have/provide additional information connected with the error report. I have seen this part associated with other applications too.

Thanks - good to know!

 

This is the part you need to be concerned about. This would suggest to me Retrospect is having trouble writing to the disk.

When you restarted after the forced shutdown did you check the disks for errors? After any forced or uncontrolled shutdown it is good practice to check any disks, both internal and external, connected to the system at the time for errors.

I didn't at the time; I've checked this and the disk is fine. No errors reported.

 

Do you have enough space on an internal disk of the server to perform a backup to? This may help to see if there are problems with the external drives.

It's a 3TB drive, and has 2.43TB available.

 

Unlike a USB disk which Windows knows is removable media an eSATA disk will be seen as fixed media the same as if it was an internal SATA disk. If there are communication problems with the eSATA disk this could be causing Retrospect to hang waiting for a response from the disk which may never come. (Although the eSATA port and the eSATA disk may support hot swapping Windows does not support it by default. To disconnect or connect an eSATA disk Windows needs to be shutdown first.)

That's also good to know.

 

Since I posted the above, I tried the backup again, and it once again hung on backup. However, I was able to successfully shut down retrospect, which means that - I presume - that AVG was partly responsible for the errors on the system. Kaspersky seems to be going OK at the moment.

 

That said, it meant I was able to access the retrospect log to see exactly what the problem is.

 

It had over 13,000 errors, and the following is a sample (they all seemed to be the same):

- 01/11/2013 11:18:17: Copying OS (C:)

Additional error information for Disk Backup Set member "1-Backup Set M-W-F",

Can't write to file E:\Retrospect\Backup Set M-W-F\1-Backup Set M-W-F\AA000523.rdb, error -1023 ( already exists)

Trouble writing: "1-Backup Set M-W-F" (3574288384), error -106 ( data overwrite attempt)

- 01/11/2013 11:18:17: Verifying OS (C:)

 

So it seems that Retrospect has somehow got confused and is trying to overwrite some files which it either shouldn't be trying to, or shouldn't be there.

 

If necessary, I can start again with the backup (uninstall retrospect, forget all the catalog files, reformat the backup drives etc and start again from fresh) as it's never worked reliably from the start - as long as this error is something that can be fixed by starting afresh. However, I don't want to be still sorting this out in six months time, if it is likely to reoccur.

 

Please can someone advise the best course of action?

[Scillonian]

You shouldn't need to uninstall Retrospect or forget the catalog files. A recycle of the backup set should do. This is an option from the 'Actions' button in the backup set's properties. (I can't give exact direction as I don't have access to Retrospect at present.)

 

This should empty the catalog and delete all the .RDB files belonging to the backup set. Just to be extra sure that there are no .RDB files left go to the folder that contained the backup set's .RDB files and check it is empty. (The location is shown in the error message you posted.) If there are any files left there delete them.

 

It is quite possible the hangs and forced shutdowns have corrupted the catalog file such that it has lost track of what .RDB files exist.

[spiffey]

Nice one Scillonian. I'll give this a go now and see what happens!

[Scillonian]

If you still get problems with the external disk try a backup to a fresh backup set on the internal drive and see if that completes without error.

[spiffey]

The Recycle backup worked; the next backup is scheduled for Monday, so I'll let it do a normal, scheduled backup then and see if it works. Assuming it does, I'll add clients one at a time and see how it goes, doing a recycle backup on the first try. (Obviously I'll also ditch AVG, upgrade the network drivers etc. on each client beforehand!)

 

I really appreciate your help with this! :^)

[spiffey]

The backup worked on Monday - but unfortunately I forgot to change the script, so it did another recycle backup rather than a normal one. So I'll have to wait until Wednesday before seeing if it worked.

 

As an aside, after backing up, the server shut itself down - even though the script is set to not shut down or sleep and, in the menu bar, it is set to stay in Retrospect. I can't see anywhere else that you can set the behaviour after the script has run. Can anyone offer any suggestions? When I originally scheduled it to run overnight, it was set to shut down, but I thought I'd changed this.

 

I'm thinking it would be best to delete the script and start again from scratch?

[Scillonian]

Do you have the Retrospect Launcher Service enabled?

 

In the preferences dialog in the Startup section there are options that apply when the Retrospect Launcher Service launches Retrospect to run a script.

 

Also I think the 'what to do when finished' options in scripts only apply to clients, not the server.

[spiffey]

Do you have the Retrospect Launcher Service enabled?

 

In the preferences dialog in the Startup section there are options that apply when the Retrospect Launcher Service launches Retrospect to run a script.

That's done it - I forgot about the overall preferences section. It was set to shut down, so I've changed it to exit, as I'm intending to get the backups done during the day rather than at night.

 

Also I think the 'what to do when finished' options in scripts only apply to clients, not the server.

I thought this might be the case, but I changed it anyway as I couldn't think of anywhere else that I could set it (at the time) …

 

Thanks, once again, for all your help! :^)