How do I speed up Retrospect backups?

[crisw]

We are a private school. We had been running Retrospect 6 for years. We migrated to 8 this winter. We are running it on the same type of machine we run 6 on- an original, non-Intel 1.33 GHz Xserve with 2 GB of RAM. The network is exactly the same; we didn't change anything.

 

Restrospect 8 runs so slowly that it is unusable. A client with 1 GB of data takes hours to back up. Looking over the console, backup speeds range from to 409 KB to 16.4 MB per minute. In contrast, our still-running Retrospect 6 server averages 300-600 MB/min- with an identical server, running on the same network.

 

Any ideas on why it runs so slowly, and what we can do to speed it up?

[rhwalker]

It will never run fast on a PPC machine, and you've got a very slow PPC xServe at that.

 

The design decision was made to have the backup file format be "little endian" (the byte order for Intel machines) rather than "big endian" (the byte order for PPC machines) so that Retrospect for Windows would be able to have a common backup file format as the Mac version, for exchanging backup sets between the two, and to exchange data over the network in "little endian" format, too.

 

As a result, the PPC version of Retrospect spends all of its time doing byte-swapping of the data structures.

 

To speed it up, you need to get a faster computer running on the Intel architecture.

 

Sorry, but that's just the way it is.

 

Russ

[jwinm2]

Running on high end Intel boxes won't be much faster either I'm afraid. Until there is an update to Retrospect, it won't matter what hardware you use. Network backups are just nowhere near the speed that they were under Retrospect 6. Hopefully they figure out how to fix this and it's not just "the way it is." Unfortunately it's been this slow for quite a long time now and there's been no indication that Retrospect considers this to be an issue. I do hope I'm very wrong about that though.

[crisw]

I'd be very interested in getting more opinions on this. We can move to an Intel box, but if it doesn't give a significant speed upgrade, it won't be worth it for us.

 

Are any users getting decent backup speeds, at least comparable to version 6? If so, what is your setup?

[rhwalker]

I'd be very interested in getting more opinions on this. We can move to an Intel box, but if it doesn't give a significant speed upgrade, it won't be worth it for us.

See the EMC Retrospect 8.1 Read Me about this:

 

Known issues with this release

 

Retrospect 8.1 performs slowly on PowerPC-based Macs. For the best performance, EMC strongly recommends that users upgrade their Retrospect servers to Intel-based Macs. For those who cannot upgrade at this time, EMC software engineers are continuing to improve Retrospect 8’s performance on all processor types, and performance boosts are likely in a future release.

Are any users getting decent backup speeds, at least comparable to version 6? If so, what is your setup?

Ah, that's a very different question. Retrospect 8 on an Intel Mac, when it doesn't crash, can use multiple CPU execution units (cores) to achieve better results, and also runs native Intel code on the Cocoa API (rather than emulated PPC code running under Rosetta using the Carbon API).

 

Retrospect 6 will always outperform Retrospect 8 on a PPC, but Retrospect 6 is unsupported for use with Snow Leopard (10.6.x).

 

And yes, there seems to be a performance issue with Retrospect networked clients under Retrospect 8, regardless of the platform.

 

Russ

[disegno]

No matter what the hardware, Retrospect 8 is painfully slow for networked clients.

 

We have top of the range latest xServe with gigabit network, 12GB RAM, LTO-4 tape and latest Mac Pro clients all running latest software.

 

Backup speeds are about a tenth of Retrospect 6 - so much so R8 is useless for client backups for us, so we're continuing using R6 for this.

 

We're about to upgrade our xServe to 10.6 server, so currently looking at other backup solutions as R8 won't cut it for us anymore with the slow client backups.

 

Christiaan

[Mayoff]

On slower computers you can increase performance by turning off the Use MD5 digest preference

[disegno]

Thanks, I've just tried this, but it made no difference to speed - but as I said, these are the latest xServe / Mac Pro clients. May help on slower ones...?

 

Any idea when this is likely to be fixed? Last lot of dialogue was hinting around October last year for an update to the speed issues...?

 

http://forums.dantz.com/showtopic.php?tid/31016/

 

Christiaan

 

[kevinpoole]

I'm running the Retrospect server on a 2.66 GHz quad core Mac Pro with 6 GB of RAM. In my experience, it's the performance of the Retrospect client software that severely constrains network backup performance, not the Retrospect server engine.

 

As an alternative to using the client software on my Macs, I instead created AFP share points on them to be backed up (which can be done on any Mac OS X machine). Then I made those share points my backup sources on the Retrospect server. Performance went from about 450 MB/min. over a gigabit network to an average of 2.5 GB/min. Turning on deduplication (the "don't add duplicate files to media set" option) reduced the performance to around 2.2 GB/min, but that's acceptable to me. My backups are now significantly faster than they were using an LTO-3 library with BRU Server 1.2.

 

Note that my backup source drives are fiber channel RAID 5 volumes and my destination media sets are made up of individual non-RAID eSATA hard drive volumes. I think the performance bottleneck now is actually the eSATA drives, not Retrospect.

 

I just can't understand why the Intel-native Retrospect client software is so horribly slow.

[CallMeDave]

As an alternative to using the client software on my Macs, I instead created AFP share points on them to be backed up

 

The trade-off for your speed increase is a loss of inode and ACL information preservation.

 

I don't know what permissions files are given when retrospectengine (running as root) mounts a volume (using some specific non-root login/password credentials) and then writes files to the Media Set's Member, but they're not guaranteed to be what they were on the Source volume. Using the client software _does_ (or at least is supposed to; it's another thing I haven't tested) preserve accurate permission information for files.

 

Dave

[rhwalker]

I don't know what permissions files are given when retrospectengine (running as root) mounts a volume (using some specific non-root login/password credentials) and then writes files to the Media Set's Member, but they're not guaranteed to be what they were on the Source volume. Using the client software _does_ (or at least is supposed to; it's another thing I haven't tested) preserve accurate permission information for files.

Backup Bouncer

 

Russ

[kevinpoole]

I willing to live with the lack of ACLs for now, as my backups are actually D2D2D. The first D2D backup is a nightly clone onto a another volume on the server using Data Backup 3.0, which works quite well. The Retrospect backup then pulls data off that clone, not the actual production volume. So, I always have a full copy of my server volume with ACLs intact that's less than 24 hours old. If I have to restore anything from the Retrospect backups, I can rebuild the permissions.

[scsctech]

Running on high end Intel boxes won't be much faster either I'm afraid. Until there is an update to Retrospect, it won't matter what hardware you use.

 

I respectfully disagree. I do have my issues with Retrospect 8, but actual file transfer performance is not one of them. With my iMac clients on 100baseT I am getting performance in the 350-450MB/m range, with clients on gigabit, I am seeing upwards of 1.2-1.4GB/m.

I am using an intel xserve with overland lto-4 autoloader.

 

[Maser]

really? Is this using the client software? Or mounting the clients as shares? Compression or not?

 

(There's a big speed difference in the two -- and if you have software compression turned on...)

 

 

I just looked through my past activities sorted by "performance" -- while these are all incremental backups at this point (and verification is off), *none* of my clients (10.6 or Windows7) go above 273M/min (that's the max).

 

And this is with everybody on Gig Ethernet...

[scsctech]

This is with the client software, hardware compression on, software compression off.

These are my xserves primarily, but I have had similar peaks with some iMac clients on gigabit.

 

Clients on 100baseT, however:

 

Edited January 28, 2010 by Guest

[jwinm2]

I too would love to know how scsc tech is getting those speeds with network backups. This is driving me crazy ( well one of many issues with Retrospect) and is something I need to get fixed. Backing up TB's of data with Retrospect 8 just doesn't work unless I'm OK with it taking an insanely long time. Is there anyone else who is actually getting good performance from network clients?

[Giobbi]

I have exactly the same speed - when backup a share instead of using client software on my xserves.

 

The Retro client software is unusable for the time, to unreliable and waay to slow.

Edited February 21, 2010 by Guest

[scsctech]

Just a few more little 'tricks' to speed up your backup process. In these cases you are sacrificing some things for speed.

 

1. Make verification none. This will remove the step of checking the backup all over again.

2. Turn off MD5 digest in the admin preferences. This MUST be done prior to making media sets. Any existing media sets will need to be re-created. (This info is directly from tech support)

3. Turn off link encryption and WOL.

 

This will pretty much give you a barebones 'backup only' system and this will be about as fast as you can get it.

[Maser]

 

2. Turn off MD5 digest in the admin preferences. This MUST be done prior to making media sets. Any existing media sets will need to be re-created. (This info is directly from tech support)

 

.

 

 

Really.

 

I'll have to test that one out. First I've heard of that one...

 

 

(Turning off software compression makes things go faster, too.)

[roobieroo]

Please do report back what kind of speed increase if any you see after trying those suggestions. I've given up on Retrospect 8 and am moving clients to other backup solutions. If they can speed up network backups and make things more reliable I'lll consider giving Retrospect another look.

[Maser]

I might rerun this because the results seemed odd to me...

 

So I did the following with the following results.

 

This was backing up 65K files/14G of data from my "users" directory as a client over Gigabit Ethernet. Client was not "at rest" -- I was doing normal work stuff while the backups below ran...

 

Made a media set with MD5 on and ran backup with MD5 on (the default): 327M/m

 

Made a media set with MD5 off and ran backup with MD5 off:

355M/m (which would seem to validate the suggestion)

 

Recycled the MD5-on media set and ran with MD5 *off*: 403 M/m (even faster -- but which would imply that tech supports suggestion isn't quite correct as this was faster...)

 

Recycled the MD5-off media set and ran with MD5 *on* -- 350M/min

 

So, having MD5 off (at various points) seems to have a positive effect, but it's only around a 10% difference.

 

[scsctech]

Are you turning off MD5 in the admin console preferences? If so, according to the tech, you cant just toggle it on & off. If its on when you create the media set, that media set will ALWAYS create MD5 checksums even if turn it off afterward. You would have to delete the media set, turn off MD5 (restart the engine just to be safe) then recreate the media set.

 

You may try this procedure if you are in testing mode.

Edited February 25, 2010 by Guest

[Maser]

So for what I did above, then I had two sets -- one created with MD5 on and the other off.

 

So the *fastest* speed I got -- 403M/m -- was with the one with MD5 turned *on* when I created the set.

 

 

Go figure.

 

 

(Admittedly, I did *not* restart the engine between any of these tests. No idea if that would make a difference or not.

[rhwalker]

Go figure.

It might depend on what the true set of preferences was. Retrospect 8 has some problems saving preferences, so it might be a coin toss what the true MD5 setting was unless you created new media sets for each test.

 

There's also the issue of how fragmented your media set drive is, because the media set might be scattered about.

 

Russ

[Maser]

Retrospect 8 has some problems saving preferences, so it might be a coin toss what the true MD5 setting was unless you created new media sets for each test.

 

There's also the issue of how fragmented your media set drive is, because the media set might be scattered about.

 

Russ

 

 

True -- the media set drive had a lot of free space on it, so it shouldn't have been that fragmented.

 

But, yes, a true test would have been to stop the engine and reboot the engine computer between backups, etc. Maybe I'll do that tomorrow if I can sneak this in between other stuff.