Maximizing network performance

[kevinpoole]

Yesterday, I installed the latest version of Retrospect Server 8.1 on a Mac Pro 2.66 with Mac OS X 10.6.2, and the client software on two Xserve 8-cores with Mac OS X Server 10.5.8. The two servers have a combined total of 12 TB of files to back up, and I decided to make a break from tape by setting up a JBOD eSATA enclosure with fifteen 2 TB drives, connected to the Mac Pro via eSATA port multiplier.

 

Being a prior user of Retrospect 5 and 6, and more recently of BRU 1.2, I was anticipating backup speeds over the network that would come close to saturating the real-world bandwidth of our Gigabit Ethernet network, especially since the backup media were relatively speedy eSATA disks.

 

At first, I was horribly disappointed by Retrospect's performance. With disk grooming and deduplication turned off, the fastest speeds I achieved over the network were in the range of 435 MB/min, which is only 58 Mbps -- not even reaching the performance of 100 Mbit Ethernet, let alone Gigabit. After much experimentation, I determined that the Retrospect client was the bottleneck, since running Retrospect Server backups locally on the client machines onto eSATA drives achieved speeds in the range of 3.5 GB/min, and copying files over AFP from the servers was well within the normal Gigabit Ethernet expectations.

 

As an alternative to connecting through the Retrospect client software, I instead set up AFP share points on the Mac servers as backup sources, using Retrospect Server's "Shares" feature. The next time I ran a backup, the performance jumped to an average of 2.4 GB/min, or 328 Mbps. That's less than I was originally hoping for with D2D, but it'll work fine for now, and it's faster than the backups I was performing with BRU to an LTO-3 library.

 

So, my question is: why is the Retrospect client such a bottleneck? If I hadn't been able to find a workaround to its slowness, I would've had to have given up on Retrospect 8, which otherwise has a lot of great features (like the ability to essentially create a homebrew Virtual Tape Library without the massive expense of an enterprise-grade VTL unit).

 

I hope some serious improvements to the client are forthcoming.

 

 

[kevinpoole]

One more suggestion for improving performance: by default, the max. usable capacity of media set members is set to 100%. After running my first backup overnight onto my hard drive members, I discovered that it slowed to a crawl once the first hard drive reached capacity -- down to only 60 MB/min, in fact. I stopped the backup and set the max. capacity on the members to 95%, but things went south when I tried to run the backup again; it kept asking for a new member and wouldn't recognize the existing ones. So, I recycled the whole set and started again with 95% capacity. It's running right now, and I'm hoping for the best.

[baweeks]

If you set the "client" as an AFP share, will Retro 8 back up all the hidden files, etc., on the "client"?

I just don't know.

-baweeks

[kevinpoole]

That's a very good question. I'm not backing up bootable volumes, so all I care about are the visible files. Someone would need to do an experiment to confirm whether invisible items are being included.

[rhwalker]

No, that's not the question. There are some parts of the filesystem (e.g., /private/var/etc, which is the target of a symbolic link for /etc) that aren't accessible from a shared volume, unless the true root is shared, which is not usually easy to do (but can be done).

 

Russ

[kevinpoole]

Ah, yes -- very true. I certainly wouldn't count on being able to create bootable volumes using the AFP method.