Jump to content
peternlewis

Exceptionally long "Updating catalog" phase?

Recommended Posts

I backed up a Macbook air. First backup to a recently recycled backup set.

 

Retrospect took perhaps an hour or two to backup, but then got stuck with the summary saying "1 file remaining", "Updating catalog" (or words to that effect). It then spent about several hours in that state before completing the backup.

 

[/size][/font][font=Helvetica][size=3]
Completed: 748479 files, 28.6 GB[/size][/font][font=Helvetica][size=3]
Performance: 265.1 MB/minute (196.5 copy, 407.4 compare)[/size][/font][font=Helvetica][size=3]
Duration: 07:04:17 (03:23:17 idle/loading/preparing)[/size][/font]

The Macbook Air was plugged in to a power adapter, left open, and set to not sleep.

What's up with that? 7 hours to backup 26GB is a bit pathetic.

Retrospect 10.0.1 (105), server and client running 10.8.2.

Share this post


Link to post
Share on other sites

What type of verification are you using? If you are using thorough verification it can take longer to verify than the actual backup as it has to compare each backed up file to the original file. Try either media verification or no verification and see if the time is reduced.

Share this post


Link to post
Share on other sites

What's the point of a backup without verification?

 

It is now doing a proactive backup to the MacBook air, again, plugged in, set to not sleep, and its bee "Building a snapshot" for three hours. Seriously! It's only got 30GB on the HD, and it's only got 500MB to back up. 3 hours to build a snashot for a laptop is untenable. I can't have the laptop plugged in and awake and open for 3 hours at a time (and more!) just to get an incremental update.

 

I really want to get back to loving Retrospect like I did 15 years ago when it was reliable, but this is ridiculous. Of my four Macs, the only Mac it will reliably back up is the server! The laptops take ages to do incremental backups, so long they would never complete if I don't actively leave them open, and it wont back up my desktop at all, failing with an error. Even though I've purchased Retrospect in the hopes I can make it work, I'm going to have to give up soon.

Share this post


Link to post
Share on other sites

A snapshot saves the permissions and other metadata for ALL the files in the source, regardless of how many files that was actually backed up.

 

So, how many files are there on the source volume?

Share this post


Link to post
Share on other sites

750,000 files is not too bad. A MacBook Air has always been supplied with an SSD, which should be fast. Which Air is it? The first generation? The latest generation?

How much free RAM does the client have during the backup?

Is anything using a lot of CPU (on the client) when the backup runs?

 

 

Try a safe boot, then boot normally before trying again.

http://support.apple.com/kb/HT1564?viewlocale=en_US

http://support.apple.com/kb/ht1455

Share this post


Link to post
Share on other sites

I did a Safe Reboot on the MacBook air - not issues, and no change.

 

 

I tried another backup - 123 files backed up, 96MB, the backup had more or less finished after 10 minutes, and then 3 hours of "Building Snapshot".

 

It is a 13-inch mid 2011 model. It has 4GB of RAM, with (after the backup) 1.5GB free, 1GB wired, 1.3GB active, 0.3GB inactive. Nothing is using appreciable CPU.

 

I removed everything that was taking any CPU, and set off another backup. Again, it "backed up" in about ten minutes, and then went off to "Building Snapshot". It's been going for 40 minutes now, I presume it will take another few hours to complete. Nothing on the MacBook Air (including retrospect) is using more than 1% CPU. Network bandwidth is about 25KB/sec. Disk access on the MacBook Air is Zero.

 

I wouldn't mind so much it is had released the connection to the laptop, but laptops don't stay open for three hours at a time, and closing it while the backup is in process has (at least in previous tests) left retrospect in an unhappy state.

 

Notably, CarbonCopyCloner backed up the entire Mac (to a local harddisk admittedly) in 40 minutes, so I don't see how it can be justified to take 3 hours to build a snapshot of anything.

Share this post


Link to post
Share on other sites

So the school wiped the Mac and re-imaged it, which was, needless to say, a real pain to then restore the files on.

 

But the new backup now does an incremental backup in 11 minutes. So whatever the issue was, it appears to have been resolved, at least for the time being.

 

Unfortunately, this doesn't really result in any useful information to determine what was actually going on and how it could be avoided in the future.

 

Thanks for the assistance.

Share this post


Link to post
Share on other sites

OK, so after the update to 10.1, this is now happening on my other laptop. 5minutes to backup, and then an hour to "update the catalog".

 

I've tried restarting the client and the server, but I don't know what else to do to resolve the issue - a backup from a laptop has to be fast enough that it is finished before the user closes the lid.

Share this post


Link to post
Share on other sites

I've had a support request going for a couple of months now with the same issue - 2 x MacBook Airs (latest 13 inch models), and the building snapshot process taking hours to complete (even if it only backs up a few files). The new version just released hasn't made any difference - in fact, it's actually taking longer to build the snapshots - around 2.5 hours. And this is only backing up a Users Library folder (excluding Cache folder) - not the whole system.

 

Still waiting to hear back from Support after submitting numerous log files.

Share this post


Link to post
Share on other sites

it's actually taking longer to build the snapshots - around 2.5 hours. And this is only backing up a Users Library folder (excluding Cache folder) - not the whole system.

 

Will check the support ticket history tomorrow. In the interim, a sanity check:

Are you using a Favorite as the backup source (instead of a Rule on the entire volume or entire computer as backup source) to backup the Users Library folder? If not, that will significantly shorten the snapshot building time, which depends on how many files and folders exist on the backup source, regardless of how many files are actually backed up.

 

Still waiting to hear back from Support after submitting numerous log files.

 

I am pinging the team for update.

Share this post


Link to post
Share on other sites

Are you using a Favorite as the backup source (instead of a Rule on the entire volume or entire computer as backup source) to backup the Users Library folder? If not, that will significantly shorten the snapshot building time, which depends on how many files and folders exist on the backup source, regardless of how many files are actually backed up.

 

Yes, it's a favourite. We have 2 favourites set per client Mac - Documents folder and Library Folder. Each favourite is tagged as "BU-clients" which is used as the source for the backup script.

 

All other clients are fine (mix of 10.8 and 10.6) but for whatever reason, 2 MacBook Airs we have are taking anywhere from 1 hour to 2.6 hours - below was today's log.

 

Generally all that's being backed up day-to-day are emails from the Library folder.

 

 

Completed: 430 files, 126.1 MB, with 24% compression

Performance: 194 MB/minute

Duration: 01:25:25 (01:24:46 idle/loading/preparing)

Share this post


Link to post
Share on other sites

Have you tried turning off compression as that can slow down the process?

No, I haven't, but the other 25 clients don't have the same issue backing up the exact same scenario, so I doubt compression would be an issue (it's also running on a top spec xServe / SSD / 12GB so I doubt speed would be an issue).

Share this post


Link to post
Share on other sites

In my experience, the script doesn't make any difference - I get the same result whether the Proactive Script (with two laptops) chooses the backup the laptop, or if I backup with a specific script for that laptop. And indeed, the same Proactive Script works fine and completes promptly for one laptop while taking an extra hour for the other (faster!) laptop.

Share this post


Link to post
Share on other sites

I have found that running a problematic client in a separate scheduled script often alleviates the issue of slow backups. Note I only use scheduled scripts and have not tried this with Proactive scripts.

Share this post


Link to post
Share on other sites

Thanks to Disegno Group's and peternlewis' patience in working with us, we have reproduced this issue and is testing a fix. We expect the fix to be included in the next free update.

 

Background:

During the "Building snapshot" phase, (to ensure correct restore) Retrospect is backing up Mac HFS file system's ACLs and extended attributes for every folder in the backup source / favorite, regardless of selectors. Most folders don't have ACLs and extended attributes. Some versions of the Apple developer tools do create the /Users/_UserNameHere_/Library/Developer/Shared/Documentation/DocSets folder which may contain 10,000+ folders with ACLs and/or extended attributes.

 

Cause:

Compared to the Retrospect Windows Engine, the Retrospect Mac Engine has some degree of inefficiency specifically when backing the ACLs and extended attributes for folders from a Retrospect Client (not local HFS volume). This inefficiency is amplified when the Client is connected via WiFi.

 

Preliminary improvement:

After tuning the related code, on some internal test systems including my primary Mac, we are seeing significant improvements in preliminary testing specific to Mac Engine's "Building snapshot..." phase for Mac Client with large number of folders that have ACLs and/or extended attributes.

We are in the process of additional performance and regression testing. The improvement helps reduce overall backup time and targets the "Building snapshot..." phase specifically with folders meeting the mentioned criteria, and doesn't accelerate other phases of the backup process.

 

Applicable cases:

To see if your Mac Client would benefit from the upcoming improvement, run the following command in the Terminal window of the Mac Client computer. The preliminary estimate of time savings would be the result from the command line x 0.2 seconds for Client on WiFi. The savings when processing 10,000 folders meeting the mentioned criteria for Client on WiFi would be over half an hour. There will also be savings but to a smaller degree for Client on ethernet.

sudo find _YourBackupSourcePathHere_ -type d -exec ls -dlF {} \; | grep '^d.........[@+]' | wc -l

Share this post


Link to post
Share on other sites

Excellent news! Thanks very much this will finish off my major issues with Retrospect and let me start advising people to use it again. Fantastic.

 

And finally, after a decade in the wilderness, I am back to having solid, rotating, offsite disk backups again!

Share this post


Link to post
Share on other sites

Great. Will this be fixed for the Windows portion as well ( sorry, I cross posted - my issue is regarding Windows, however it seem to be the same issue). Thanks.

Share this post


Link to post
Share on other sites

Hi David,

 

Thanks for the news and update. I'd just like to mention that my issue with Building Snapshots never ending is when using Windows Clients and a Mac server. I'm not sure if that should get posted under the Windows or Mac subforum. Also, I haven't had this happen yet on the latest release of Mac server (10.1.0(221)) so I'm not sure if that issue was addressed or not.

 

Thanks

-Derek Cunningham

Share this post


Link to post
Share on other sites

Applicable cases:

To see if your Mac Client would benefit from the upcoming improvement, run the following command in the Terminal window of the Mac Client computer. The preliminary estimate of time savings would be the result from the command line x 0.2 seconds for Client on WiFi. The savings when processing 10,000 folders meeting the mentioned criteria for Client on WiFi would be over half an hour. There will also be savings but to a smaller degree for Client on ethernet.

sudo find _YourBackupSourcePathHere_ -type d -exec ls -dlF {} \; | grep '^d.........[@+]' | wc -l

 

 

So I ran this on my computer (which has xCode installed) and I backup the /Users folder as a Favorite Folder

 

 

maser$ sudo find /Users -type d -exec ls -dlF {} \; | grep '^d.........[@+]' | wc -l

3582

 

 

So, you are saying I should see 3582 *.2 = 716 seconds of savings here? Approximately 12 minutes when building the snapshot after each backup? Or not that much in savings as all of my backups are over Ethernet?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×