Jump to content
derek500

Building Snapshot - sometimes forever

Recommended Posts

After restarting the engine the log of that backup was not captured in the overall Retrospect log (a huge flaw IMO) so I can't confirm how it ended.

Yes, that's really bad.

Share this post


Link to post
Share on other sites

What does the log say?

 

We run the Windows version 7.6 at work. Sometimes when a laptop user "pulls the plug" during "building snapshot", an "error -519" is shown in the log, but the server continues with "building snapshot" until it's killed.

 

I see this at times too...but the job can't be killed. You have to 'kill' the process. :(

Share this post


Link to post
Share on other sites

I see this at times too...but the job can't be killed. You have to 'kill' the process. :(

Right, I wasn't clear. I have to kill the process, too.

Share this post


Link to post
Share on other sites

2013-04-16: See updates in red.

 

Regarding the issue of long "Building Snapshot" when backing up Mac sources, my understanding from the team:

To provide the features and flexibility in Retrospect, the backup process has *roughly* 4 major steps that take varying amount of time:

1. Scan the Source's volume or subvolume (e.g. over 1 million files)

2. Apply Selectors and match against what are already backed up in the Media Set to determine what needs to be backed up from the Source in the current backup session

3. Back up those contents (e.g. may be 300 files instead of over 1 million files)

4. Build Snapshot, including file information (not contents) and folder details (e.g. permissions Mac HFS file system ACLs and extended attributs) for the Source's volume or subvolume (e.g. over 1 million files near 20,000 such folders on certain Macs)

Therefore, time required for step 1 and 4 depend on how many files and folders there are in the Source's volume or subvolume, which can be substantial and time consuming.

  • Like 1

Share this post


Link to post
Share on other sites

Regarding the issue of certain log entries not being in the operations log after a crash, my understanding from the team:

We are evaluating how to address this. Why it occurs is related to Retrospect engine having parallel execution units and other architectural factors, as a result certain log entries are cached in the ConfigXX.dat until the corresponding operation completes and flushes to relevant log entries to the operations log.

Share this post


Link to post
Share on other sites

Thank you for some update information.

 

We just had another case like this.

 

The backup job went thru 100+GB in around 4+ hours..only to be stuck on Building Snapshot for another 4 hours - at which time the client left.

 

What stayed active - the server job.

 

This is a fairly regular occurence and becoming an annoyance - as the process has to be shut down - and in most cases a backup set needs to be rebuilt.

 

With nearly 2TB backup sets, this can take 18-22 hours!

 

Please let us know if anything else has been found.

Share this post


Link to post
Share on other sites

Hi David,

 

Somehow I missed your December posts on this thread. Thank you (and the team) for your attention to these flaws! Our frustration level was getting very high, but the most recent update seems to have smoothed out most of our biggest complaints. The loss of log info definitely still needs to be addressed (maybe a dedicated log cache for each process that gets checked on engine launch and posted if not empty?) but we are definintely pleased to see some reliability updates coming along.

 

Thanks

-Derek

Share this post


Link to post
Share on other sites

stuck on Building Snapshot for another 4 hours

 

You mentioned recently this is for backing up Windows Client using Windows Retrospect Server. Because "Building snapshot..." is sufficiently different between the platforms, I have created a new post on the Windows forum and commented here: http://forums.retrospect.com/index.php?/topic/150440-stuck-on-building-snapshot-for-another-4-hours/

Share this post


Link to post
Share on other sites

I realize you created a related topic under the Windows section, but not this is also a huge problem for backing up Linux clients. This problem has existed since version 8 and I have reported it many times (including in this topic http://forums.retros...post__p__249571).

 

I have cases where I'm backing up Linux clients over a relatively slow network link (to a remote office) and the building snapshot phase can take literally hours even though only a few files representing only KB's of data were actually backed up. The scanning/matching/backup phase takes a couple of minutes at the most.

 

Just wanted to make sure this was on the radar and Linux clients were not forgotten.

Share this post


Link to post
Share on other sites

This issue continues to confound with the Mac version 10.1 running under Snow Leopard trying to back up a brand new windows 7 x64 client with a snappy Core i5 using 8 GB ram and with the latest Windows client install available.  A 5 GB backup for 1415 files copies in ~15 minutes over a wireless n - which is slow but tolerable.  But the snapshot build takes another 40 minutes - and I have seen it take twice as long.  Media verification takes about 60 seconds.  Total time for the 5 GB is 60 minutes (and sometimes as long as 100 minutes).   That is an effective transfer rate of  1.4 MB/sec at best.  That is 83 MB/min, compared to the performance metric Retrospect proudly shows for the job, which is a confabulated 458 MB/min that neglects the time for the snapshot build entirely.  I have a 22 year old Mac IIci that has a 2 MB/sec scsi port that is faster than that.  If Retrospect really is not capable of anything better, perhaps the company might consider relaxing the number of threads that the normal desktop versions can run at once, so that it can concurrently wait on the 5 Windozer laptops I am servicing and the poor Mini does not spend 5 hours+ per day running these backups.  The official concurrent thread limit for the Retrospec program is only one.  On the Retrospect for windows (v7) that I also have, that cannot be changed.  On the Mac you can increase it, but next time Retrospect launches it is back to one thread.  Keeping the Retrospect program running after increasing the number of threads to the number you need to run all your backups concurrently seems to be the only available workaround for the abysmal snapshot build performance I see.   Another possibility is that antivirus software is somehow running interference.  Symantec End Point Protection and the like can slow network transfers from WIndows down to a crawl.  Has anyone else tried any experiments with and without AV enabled?

Share this post


Link to post
Share on other sites

 But the snapshot build takes another 40 minutes - and I have seen it take twice as long.

 

When building the snapshot, Retrospect saves the metadata (such as permissions etc) for EVERY file on the client, not just the backed up files. (If it didn't save all metadata, Retrospect can't restore the client to the state the client had at the time of the backup,)

 

So if the client has millions of files (or has very slow disks), the time is to be expected.

Share this post


Link to post
Share on other sites

Thelander - what you say is absolutely correct - it does have to get metadata from every file.  Total disk space used on the client is 45 GB or so and it has typical number of files - maybe 600K or so.  The disk is an SSD - very fast - fast enough for a full Symantec SEP antivirus scan to run in less than an hour.  The metadata for the Windows machines seems to be much harder for the Mac version of Retrospect to gather than what it it has to do for a much larger number of files on a much slower Powerbook G4 client; the whole backup and snapshot is done in 15 minutes, which is the kind of performance you might hope for.  I do not have a good comparison for the Windows version - it takes a fair amount of time building snapshots for its Windows clients also, although it is not as long.  My Windows setup is on a stronger machine than the dual core Mini.  But  despite the host machine's performance power differing, my observation  appear to indicate something about the nature of Windows metadata when you are trying to save system state etc. - it seems to be harder to do with Windows clients than with Mac clients.  This is also borne out when looking at what it takes to service an older Gen 2 Power Mac client with the Windows v 8.x multi server  host - snapshots happen very quickly - much faster than for Windows 7 clients.

Share this post


Link to post
Share on other sites

You are right, 600k files on an SSD should be fast.

I'm afraid I have no idea what is causing your problem.

 

You should contact Retrospect support to get it resolved. (This is just a user-to-user discussion.)

Share this post


Link to post
Share on other sites

I guess I spoke too soon. After having no problems for several months, I now find myself with another Windows client stuck on Building Snapshot - for 48 hours. I know that the client has shut down and left the building but the server is still stuck here. I have contacted support about this issue in the past and the solution is to stop the engine on the server, which usually means I need to rebuild the catalog. This is where things get frustrating. Is there any way to stop one job on the engine? (10.1.0.221 on Mac OS X 10.6)

 

BTW this thread was always intended (at least by me) to be about never-ending 'Building Snapshot' - not ones that finish in 4 hours.

 

Thanks

-Derek

Share this post


Link to post
Share on other sites

Snap shot building has always been slow for Windows clients.  I am not very convinced that this is an issue that Support has a good answer for; probably the only way to improve performance is to reduce the amount of metadata (e,g, uncheck save system state).  Even Mac version 6 was pokey with WIndows clients - this goes all the way back to G3 and G5 iMacs and continues with the Intel Mini.   The likelyhood of these snapshots getting hung by system sleeps, network glitches, client computers walking off with their owners, and so forth increases greatly when the process takes too long.  The results are unpredictable and Derek's report that things run OK for months and then get hung again for no apparent reason appears to support this.  I too have had ones hang to the point where the job cannot be stopped from the console and either the Retrospect engine has to be stopped or the host machine restarted.  The engine is an all or nothing affair, to answer Derek's question - everything stops  and affected backup sets catalog rebuilds will then be required.  I share his pain (and OS).  Often the client is also left in a state that makes it better to restart it also; occasionally a reinstall of the client and re-pointing the host at what is then a new client also becomes necessary.  So anyway, to respond to Derek's BTW,  perhaps slow snapshots and the never-ending ones are related sufficiently that this thread might benefit from discussing both.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×