Jump to content

8.2 still unbearably slow backing up clients


Recommended Posts

I'm giving 8.2 another chance, since shortly I will be forced to upgrade my server to Snow Leopard and will no longer be able to use reliable version 6 any longer. I read that 8.2 was supposed to address the slow backup issue, but for me, it's still unbearably slow when backing up clients.

 

The server is running on a Mac Pro running 10.5.5. I have a single disk media set that is on an eSATA RAID 1 array of 2TB drives. The media set is configured with AES128 encryption and the default Retrospect grooming policy.

 

Backing up the local server performs okay - I would still expect it to be faster, but at least it isn't abysmal:

Completed: 1173355 files, 78.9 GB

Performance: 607.1 MB/minute (571.9 copy, 646.4 compare)

Duration: 05:18:44 (00:54:16 idle/loading/preparing)

 

However, backing up another Mac Pro client, connected over a gigabit network, was another story. This performance was simply dreadful:

Completed: 1223258 files, 117.3 GB

Performance: 109.5 MB/minute

Duration: 1 01:10:34 (06:53:59 idle/loading/preparing)

Note: That's 1 day, 1 hour - 25 hours! The client is running 10.6.4 and the Retrospect client that was included with the Retrospect 8.2 disk image.

 

I monitored network bandwidth, CPU, and disk I/O on both the client and the server. Network utilization on a Gbps circuit occasionally peaked at 60Mbps but predominately was in the 7-20Mbps range, mostly spent at the bottom end (7-9Mbps). CPU and disk I/O on both the client and server were mostly idle. Both machines are quad-core, and both were showing tons of idle time. I reniced to -19 both the Retrospect Engine and retrospect client / retropds processes, to give them the highest possibly priority, but they didn't consume any additional CPU or disk I/O. Both the client and the server weren't doing anything else throughout the day long backup. The backup was performed by a proactive backup script with the default settings. This was the first time the client was backed up. The drive being backed up was an internal SATA 1TB 7200 RPM drive - it has no performance issues locally.

 

I am now running another backup of a Macbook Air client, and it is going just as slowly. It is attempting to backup about 31GB in 800k files, and it has been running for 8 hours so far, having completed 500k files and 21GB. This client is also running 10.6.4 and the newest Retrospect client.

 

Why is it so slow? If it's not CPU, I/O, or network bound, what is slowing it down?

 

Thanks,

-Mike

Edited by Guest
Link to comment
Share on other sites

The speed improvements to 8.2 were *primarily* related to backing up large files. From my testing, 8.2 was not a whole lot faster when backing up small files.

 

Do you have a folder with a bunch of large files in it (like Movies or large database files?)

 

Maybe try making that a "Favorite Folder" and running a backup of just that folder to see how fast it goes?

Link to comment
Share on other sites

Yes, larger files are faster than smaller ones. Are the engineers working on speeding up smaller files? Most clients have a ton of small files that need to be backed up.

 

I backed up a Windows 7 client using the same server, and it was more than twice as fast as the LOCAL Mac server backup:

Completed: 88328 files, 22.6 GB

Performance: 1335.9 MB/minute

Duration: 00:37:44 (00:20:25 idle/loading/preparing)

 

Here is the backup for the same Mac Pro client as before, but backing up a disk with mostly large media files on it - faster, but still slow:

Completed: 35880 files, 64.5 GB

Performance: 401.0 MB/minute

Duration: 02:49:16 (00:04:46 idle/loading/preparing)

 

So, the Mac client had fewer, much larger, files than the Windows client, and the speed was 4 times as slow. Want to hear what's worse, the Windows 7 client above, that's actually running in VMWare Fusion on the SAME Mac Pro client. So, backing up over the same network, to the same physical computer, and the Mac client is way slower (and no, these backups were not running at the same time - I only have the one media set, so only one backup runs at a time.)

 

What can I do to help you guys fix this issue? It isn't an issue with the 6.x server, and the clients are running the same Mac client software regardless of the server version. So, what is broken in the 8.2 server that makes Mac backups perform so incredibly slowly over the network (and not too speedily locally either)?

 

Thanks.

Link to comment
Share on other sites

Retrospect 8 (as was explained to me) is slower because of the need to store the ACLs on the Mac files. That's why the Windows backups will be faster for the same set of files.

 

I was told that for tiny files, the time it takes to make sure the ACL is backed up takes longer than backing up the actual file.

 

Whether there are additional speed improvements to be made along these lines -- only time will tell, I guess.

Link to comment
Share on other sites

The additional call per file required to get the ACLs does not justify the orders of magnitude of slowness we're seeing here. ACLs can be represented with a tiny amount of data and most files/directories don't have any set.

 

The Macbook Air backup finally finished (and it was over a wired LAN.) Just as pathetically slow:

Completed: 797393 files, 31.7 GB

Performance: 46.9 MB/minute

Duration: 18:44:25 (07:15:01 idle/loading/preparing)

 

Most of that 7 hours of "idle/loading/preparing" was spent after the backup had completed, while it was storing the snapshot or whatever that's called. Why did that take 7 hours?

 

Link to comment
Share on other sites

Somebody else posted in another thread that they had a client (running 10.5.8) that took exceptionally long to build the snapshot.

 

I tried replicating it here (backed up an entire 10.5.8 -running G4 MacBook Pro with my default set of applications I deploy here -- about 30G like yours) running 10.5.8 and the snapshot build only took 5-6 minutes -- like it did for your MacPro client.

 

That user had a couple of external hard disks and clean installed 10.5.8 on one of them and backing up only that booted drive took 5-6 minutes for the snapshot, as well.

 

I'm not sure if the poster ever dug down to figure out what was unique about his client hard disk (he only had one client showing this issue...)

 

 

While this doesn't mean you have the same issue, it sounds similar.

 

I might suggest you try what I suggested for him: For that client, make "Favorite Folders" out of the main folders (System, Library, Applications and Users) and see if one of them in particular takes a long time to build the snapshot.

 

if so, you might have to make sub-Favorite Folders to figure out which subfolder within those folders is making this take so long and try to isolate that.

 

I realize this would be a time-consuming issue, but the few of us that tried to replicate the other person's problem could not do it with our setups, so we weren't sure what was unique about his particular client.

 

 

 

Since this is (only) 30G, do you have enough space on your engine computer hard disk to back up to a new disk media set to that computer (bypassing your eSATA RAID) -- just to rule out (or confirm) any issues with the RAID?

 

 

 

 

 

Link to comment
Share on other sites

Somebody else posted in another thread that they had a client (running 10.5.8) that took exceptionally long to build the snapshot.

 

I tried replicating it here (backed up an entire 10.5.8 -running G4 MacBook Pro with my default set of applications I deploy here -- about 30G like yours) running 10.5.8 and the snapshot build only took 5-6 minutes -- like it did for your MacPro client.

 

That user had a couple of external hard disks and clean installed 10.5.8 on one of them and backing up only that booted drive took 5-6 minutes for the snapshot, as well.

 

I'm not sure if the poster ever dug down to figure out what was unique about his client hard disk (he only had one client showing this issue...)

 

 

While this doesn't mean you have the same issue, it sounds similar.

 

I might suggest you try what I suggested for him: For that client, make "Favorite Folders" out of the main folders (System, Library, Applications and Users) and see if one of them in particular takes a long time to build the snapshot.

 

if so, you might have to make sub-Favorite Folders to figure out which subfolder within those folders is making this take so long and try to isolate that.

 

I realize this would be a time-consuming issue, but the few of us that tried to replicate the other person's problem could not do it with our setups, so we weren't sure what was unique about his particular client.

 

 

 

Since this is (only) 30G, do you have enough space on your engine computer hard disk to back up to a new disk media set to that computer (bypassing your eSATA RAID) -- just to rule out (or confirm) any issues with the RAID

 

I don't see how there could be an issue with the RAID when the local backup and the backup of the Windows clients are going to the same media set on the RAID with no issues.

 

The first incremental backup of the Mac Pro ran into the same issue. It spent over 5 hours building a snapshot. I had to stop it - I couldn't keep waiting for it to finish. I tried again and got the same result. And once again, the server, client, and network were idle while this was going on - nothing was happening anywhere.

 

Since none of my Mac clients were getting backed up at that point, I had to go back to 6.1.230 again. Time for me to start looking into BRU or Crashplan or some other alternative, as it's been over a year and Retrospect 8 still isn't usable in a production environment.

Link to comment
Share on other sites

If the problem is with the MacPro *client*, then I would expect the incremental backup to have the same issue building the snapshot (as it has to build a snapshot allowing for the restore of the entire volume.)

 

I'm not going to attempt to dissuade you from looking at other products, but I'd still be interested if you had the time to try to determine why this is happening for you specific client, but not others.

 

And the only way I can think of doing that would be to see if trying to isolate the problem to something specific being backed up from some specific folder on your client.

 

I'd doubt it's a unique *service/process* causing this problem, but maybe your client hard disk as a bunch of unexpected hard links on it that Retrospect has a problem trying to keep track of (or something similar...)

 

I'd bet if -- like the other poster -- you put a hard disk on your MacPro that *only* had the OS on it -- it would build a snapshot in 5 minutes like normal.

Edited by Guest
Link to comment
Share on other sites

If the problem is with the MacPro *client*, then I would expect the incremental backup to have the same issue building the snapshot (as it has to build a snapshot allowing for the restore of the entire volume.)

 

I'm not going to attempt to dissuade you from looking at other products, but I'd still be interested if you had the time to try to determine why this is happening for you specific client, but not others.

 

And the only way I can think of doing that would be to see if trying to isolate the problem to something specific being backed up from some specific folder on your client.

 

I'd doubt it's a unique *service/process* causing this problem, but maybe your client hard disk as a bunch of unexpected hard links on it that Retrospect has a problem trying to keep track of (or something similar...)

 

I'd bet if -- like the other poster -- you put a hard disk on your MacPro that *only* had the OS on it -- it would build a snapshot in 5 minutes like normal.

 

Let me recap the data points here, as it isn't just the one client that is having the issue.

 

Mac Pro Server: Backing up two local disks to the RAID media set. Performs about the same as Retrospect 6.1, at around 600 MB/min. Initial backup took over 5 hours, with about an hour idle/loading/preparing. Subsequent incremental backup took 61 minutes, of which 55 were idle/loading/preparing.

 

Mac Pro Client: Initial backup to the same RAID media set took over 25 hours, of that, 7 hours idle/loading/preparing. Performance was about 100 MB/min, not counting all that idle time. Next incremental backup took over 5.5 hours (gave up before it finished), most of it idle/loading/preparing, as there were only 300 small files to backup.

 

Macbook Air Client: Initial backup to the same RAID media set took over 18 hours, of that, again 7 hours were idle/loading/preparing. Performance was under 50 MB/min.

 

Windows 7 Client running on same Mac Pro as above under VMWare: Initial backup to the same RAID media set took 37 minutes, of which 20 were idle/loading/preparing. Performance was over 1,300 MB/min.

 

The Mac clients have Snow Leopard installed, along with iPhone development tools, Microsoft Office, and Adobe CS5 suite, so they have a ton of files installed. Plus, there's about 3GB of e-mail for Apple's Mail client.

 

The Mac server has Leopard and the developer tools and MS Office, but not Adobe CS5. It runs an IMAP server, so it has even more e-mail stored on it.

 

The Windows client has MS Office installed on it.

 

 

Now, Retrospect 6.1 is no speed demon, but it performs better for the Macs. By comparison, with 6.1, the Mac Pro incremental backups run around 2 hours, with 1.5 spent idle/loading/preparing, at a speed of about 170 MB/min - that's with about 60 days of incremental backups to have to match against.

 

 

Gigabit switched network between client and server.

 

Both Retrospect 6.1 and 8.2 are run (alternately) on the same server. Both are backing up to the same RAID system. Both are backing up the same clients.

 

The Retrospect client running on the clients is the exact same install of the latest client software.

 

The fast Windows 7 backup and the incredibly slow Mac OS X backup are backing up the same client computer, on the same network, to the same backup media set.

 

During the client backups, the network is woefully underutilized, and CPU and I/O on both the client and server is mostly idle.

 

 

So, to recap, it's not happening to a specific client - it is happening to all of my Mac clients. Retrospect 6.1 is way faster at backing up the Mac clients, somewhat slower at everything else. Windows clients backup faster than Mac clients on both 6.1 and 8.2, but on 8.2 the Mac clients backup so slowly as to be unusable.

 

I don't have a lot of time and leeway to experiment much further. I installed 8.0 when it came out and wasted a week with it, and did the same with 8.1, and now with 8.2. I can't leave the clients without backups, so I have to keep going back to 6.1 until I find something else that works.

 

Yes, I know 6.1 can not be used to restore Snow Leopard compressed OS files - I have SuperDuper clones to deal with bare metal OS restores as a workaround. until I can find a backup solution that is reliable and performant.

Link to comment
Share on other sites

From what I recall personally, the sheer number of files has a limited effect on how long it takes to build a snapshot of something -- how long it takes to *scan and match* prior to backup -- yes. But not so much on how long it takes to build the final snapshot.

 

55 minutes for the *server* to be "idle" (which is likely scanning and comparing) is close to what I see for my one 10.6.4 server (about 25 minutes is listed as the "idle" time...)

 

 

However, there *is something* specific to your Mac clients which seems to make the snapshot part of the backup take longer than anybody else (beyond that one other person I saw post about this.)

 

If everybody doing a backup of a full Mac client took 7 hours to build the snapshot, the boards would (rightfully so) be lit up with this problem.

 

Do you have any clients (for comparison purposes) that do *not* have CS5 *or* the development tools on them? (I ask this in the hopes of helping to see if I can replicate this too -- my SL machines have both Office 2004 and 2008 installed on them.)

 

We don't own (yet) a copy of CS5 at my office, so I couldn't try to replicate with that on my end, unfortunately. I do have a machine with the dev tools on it -- at least the current 3.x tools. However, I just back up the "users" folder on this machine.

 

I'd certainly be happy to run a test with whatever version of xCode *you* have installed on a machine in my office to see if it's that. Which version do you have -- 3.x or 4?

 

 

Are all your machines imaged off a central build? If so, that would explain why they all take a long time to build the snapshot, but not *why* they take a long time to build the snapshot...

 

The goals is to find something Q/A could replicate.

 

 

Again -- this is not your problem (well, it specifically is a problem you are seeing that others aren't...), but if it's something you can isolate as to the cause -- then they can probably replicate in-house and fix the issue.

 

 

 

 

 

Link to comment
Share on other sites

Do you have any clients (for comparison purposes) that do *not* have CS5 *or* the development tools on them? (I ask this in the hopes of helping to see if I can replicate this too -- my SL machines have both Office 2004 and 2008 installed on them.)

I do have a Mac Mini that I haven't tried to backup with 8.2 yet. It has a fairly minimal Mac OS X install on it. I will try that when I get a chance, but it may be a while.

 

We don't own (yet) a copy of CS5 at my office, so I couldn't try to replicate with that on my end, unfortunately. I do have a machine with the dev tools on it -- at least the current 3.x tools. However, I just back up the "users" folder on this machine.

I'd certainly be happy to run a test with whatever version of xCode *you* have installed on a machine in my office to see if it's that. Which version do you have -- 3.x or 4?

xcode_3.2.3_and_ios_sdk_4.0.1 is what is installed.

 

Are all your machines imaged off a central build? If so, that would explain why they all take a long time to build the snapshot, but not *why* they take a long time to build the snapshot...

No, they're not imaged - they were installed individually from the source installers.

 

Again -- this is not your problem (well, it specifically is a problem you are seeing that others aren't...), but if it's something you can isolate as to the cause -- then they can probably replicate in-house and fix the issue.

I appreciate you taking the time and effort to look into this further.

Link to comment
Share on other sites

We don't own (yet) a copy of CS5 at my office, so I couldn't try to replicate with that on my end, unfortunately. I do have a machine with the dev tools on it -- at least the current 3.x tools. However, I just back up the "users" folder on this machine.

I'd certainly be happy to run a test with whatever version of xCode *you* have installed on a machine in my office to see if it's that. Which version do you have -- 3.x or 4?

xcode_3.2.3_and_ios_sdk_4.0.1 is what is installed.

 

 

 

So -- there's *nothing* else:

 

Base 10.6.4 and all updates via SU

 

Office (which version?)

 

Adobe CS5

 

xcode 3.2.3 and iOS SDK 4.0.1

 

No anti-virus software? No other 3rd party applications? No extra printer drivers?

 

No other versions of xcode and/or SDK on this *prior* to the versions listed above?

 

I might be able to get my hands on a copy of CS5, but I'd suspect the dev tools. Do you remember the order in which you installed them?

 

I'll see what I can replicate here. I don't have the iOS SDK installed anywhere, but I can easily get a copy of that.

 

 

If you have that mini on your end, I'd suggest doing a backup of it with what's currently on it now -- if *that* takes 7 hours to build the snapshot, I'd be incredibly surprised.

 

Then you might add your apps one-at-a-time and do an incremental backup to see what triggers the snapshot problem.

 

 

Also (just thought of this), you might check the "Volumes" directory on your client to make sure there's nothing weird there.

 

Mine look like this:

 

 

daredevil:volumes root# ls -la

total 8

drwxrwxrwt@ 3 root admin 102 Aug 16 08:43 .

drwxrwxr-t 36 root admin 1292 Jun 23 08:34 ..

lrwxr-xr-x 1 root admin 1 Aug 16 07:59 Macintosh HD -> /

daredevil:volumes root#

 

 

Link to comment
Share on other sites

If the contents of the entire volume(s) are being suspected as a cause of the long times necessary to backup that entire volume, might I suggest that first some simple tests of smaller defined Favorite folders be done to establish some sort of baseline?

 

If your configuration remains unbearably slow when backing up

[color:purple]ClientMac/usr/bin/

[/color]then investigating application installs won't help.

 

But if that (and some other, larger) test backup screams, then considering files (or links) on the volume(s) does make sense

 

 

 

Dave

Edited by Guest
Edited to make clear the issue is being seen on multiple client source machines
Link to comment
Share on other sites

So, trying something while I did other stuff today, I could *not* reproduce this problem doing the following:

 

Started with this client setup:

 

10.6.4 load set that I distribute here. Client is a MacBook Pro 2.1.6 core duo with 2G RAM. It has installed as 3rd party apps:

 

Acrobat 9 -- pro and reader

Citrix ICA client

Daylite 3

Fetch

FileMaker Pro 11

Firefox

Flip4Mac

Fugu

iLife suite

4D client

Office 2004 and 2008

Retrospect client

Sophos Anti-Virus

stuffit expander

 

 

Did a backup of this (compression on/verification off) to a FW hard disk disk media set on a test engine computer (MBP 2.2 with 2G RAM running 10.6.4). No problems:

 

8/16/10 9:54:07 AM: Connected to snapshottest

* Resolved container snapshottest to 1 volumes:

Macintosh HD on snapshottest

- 8/16/10 9:54:07 AM: Copying Macintosh HD on snapshottest

> *File "Macintosh HD/private/var/log/DiagnosticMessages/2010.07.17.asl": can't read, error -1012 ( feature unsupported)

8/16/10 11:33:13 AM: Snapshot stored, 146.7 MB

8/16/10 11:33:26 AM: 1 execution errors

Remaining: 1 files, 1 KB

Completed: 498339 files, 23.2 GB, with 4% compression

Performance: 266.2 MB/minute

Duration: 01:39:19 (00:10:11 idle/loading/preparing)

 

 

Downloaded the demo of the CS5 "design premium" suite and installed that (40+ minutes) on the client machine.

 

Did an incremental backup of the client:

 

8/16/10 12:04:43 PM: Connected to snapshottest

* Resolved container snapshottest to 1 volumes:

Macintosh HD on snapshottest

- 8/16/10 12:04:42 PM: Copying Macintosh HD on snapshottest

8/16/10 12:46:29 PM: Snapshot stored, 167.8 MB

8/16/10 12:46:44 PM: Execution completed successfully

Completed: 71545 files, 8.5 GB, with 9% compression

Performance: 307.2 MB/minute

Duration: 00:42:01 (00:13:42 idle/loading/preparing)

 

 

Longer "idle" time, but that was due to the initial scanning/matching. Building the snapshot took less than 5 minutes.

 

 

Then I downloaded and installed the xCode 3.2.3/iOS SDK 4.0.2 software. Another 45 minutes after that install, I did another incremental backup:

 

 

8/16/10 1:49:22 PM: Connected to snapshottest

* Resolved container snapshottest to 1 volumes:

Macintosh HD on snapshottest

- 8/16/10 1:49:17 PM: Copying Macintosh HD on snapshottest

8/16/10 2:24:00 PM: Snapshot stored, 195.2 MB

8/16/10 2:24:26 PM: Execution completed successfully

Completed: 102482 files, 6.9 GB, with 38% compression

Performance: 352 MB/minute

Duration: 00:35:09 (00:15:03 idle/loading/preparing)

 

 

Idle time was a bit longer again due to the need to match for the additional 100K files, but snapshot build was about the same time (less than 5 minutes.)

 

 

So, while this doesn't help *you*, it means that with the above set of software installed on *my* client -- I can't reproduce your 7-hour snapshot build issue.

 

 

I would recommend you consider trying the "favorite folders" method of backup and see if you can figure out what folder contains what files/links that's causing this on your end.

 

 

If I'm missing some other 3rd party software from what I have listed above that *you* have (admittedly, I have the *current* iOS SDK and I made an assumption on which of the 4 demo CS5 suites you might have...) -- let me know and I can install it on this test machine to see if I can figure out what's up.

 

 

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...