Jump to content
ridgedale

How to Achieve Optimum Backup Performance?

Recommended Posts

Hi,
Currently using Retrospect 9 (MacMini running Mavericks) to backup 10Tb Raid6 NAS (will be doubling if not trebling storage capacity in near future) and 5x clients using 2 separate backup cycles (8-week recycle + daily incrementals) to 2 separate external firewire 800 drives on a weekly rotation. NAS and clients connected with 1Gb ethernet.

Recycle backups start at 6:00pm on a Sunday and are not finished when the company users come in on Monday morning. Are there any improvements in the hardware/network configuration that can improve the performance without breaking the bank?

An incremental backup of some 70-80Gb of data has taken upwards of 5 hours to complete earlier this morning at a rate of just 580Mb/s. Are there any settings within Retrospect that can improve the performance?

Any advice on how to achieve optimum network D2D backup performance would be greatly appreciated.

tia

Share this post


Link to post
Share on other sites

A few comments:

The amount of data moved is the biggest factor in backup speed. I am experimenting with grooming to do what I think you are doing.

Rather than doing a recycle backup , I've thought about grooming the media sets down to one backup. That way, you never really do a "full" backup - much faster.

Firewire 800 is not so fast these days. For some operations, disk speed is key. Thunderbolt drives may be the ticket.

For many things on Retrospect, CPU speed of the server is the bottleneck. Make sure you have plenty.

Share this post


Link to post
Share on other sites

ridgedale,

Although what Don Lee says about "the amount of data moved" is correct,  the rather-surprising news is that the primary factor controlling speed of client backups appears to be the speed of the client processor chip.  That is because "the main factor limiting speed of [backups over] networks is the traversal of multiple files in the file system by the client computer."  I reported the results of an unplanned experiment that demonstrated it in this post in another thread, although you may want to read the posts before and after that one to see what my results were based on. 

In particular, because I am backing up onto USB3 portable HDDs—with a much faster connection than Firewire 800—it appears that the speed of the connection from the "backup server" to the Media Set drive is not the controlling factor.  My 21 October post in that thread shows that I get much faster backup speeds for drives that are locally attached to the "backup server" machine, but the P.S. in my 11 October post in that thread seems to show that the speed of the processor chip in the "backup server" machine doesn't make as much difference as you would think, and my 8 January post in that thread shows that the speed of the the "backup server" machine's own drive—which surely must be involved as some kind of participant in a backup run—doesn't seem to make much difference either.

Share this post


Link to post
Share on other sites

Here are some suggestions:

Don't use Recycle Backup except for emergencies

Don's suggestion is the easiest to implement, costs no money and will speed up your backups.  Recycle backups will take a long time.  The main advantage of any backup solution is to backup only changed files.  Instead, set the grooming option.  I've set my grooming option to keep 1 copy for all my backups because I don't have the space for keeping backup changes and it's essentially the same as a recycle backup.  My primary reason for backup is to restore in case of failure and not to store a history of changes.  The only time I use a recycle backup is when there is a problem with Retrospect backing up to a media set and I need to start over from scratch.

Try using Retrospect Instant Scan

How much time on the backup is spent on scanning the source for changes?  Are you using Retrospect Instant Scan on your sources?  Instant Scan will monitor the backup source for file changes.  When it's time to be backed up Retrospect queries the client for changes that need to be backed up.  It's adds some processing overhead on the source computer when the computer is not being backed up but should start a backup sooner than scanning a source disk for changes at the time of backup.

Create separate Media Set for each backup Source

Having a different backup Media Set for each backup source will allow Retrospect to use backup threads.  It will cut down on the "scanning for changes" time because all the backups will be scanning for changes on the clients at the same time instead of: 1st backup scan for changes, backup, 2nd backup scan for changes, backup, etc.

Assuming this is the setup you have now:

You've created 1 media set for the backup destination and set 6 sources to backup to that set.  When it's time to backup, only one source will run at a time.

Instead: on your backup destination create a media set for each source located on the backup destination drive.  When it's time to backup, Retrospect will start all backups at the same time.

Increase Backup Destination Storage Speed

Increasing the backup destination storage speed will increase the backup speed.  It's probably what is slowing you down if you are backing up to a single FireWire drive instead of a Firewire Raid.  FireWire 800 tops out at around a theoretical 800 Mbps.  Your speed at 580 Mbps is about what FireWire can handle, so even having a FireWire raid wouldn't make much of an improvement.  Where are you seeing your backup speed?  Is it 580 Mbps or 580 MB/m.  The Retrospect console shows MB/m.  580 Mbps would be really fast on a single FireWire 800 drive.  You're probably getting 10 Mbps.  

The source read speed will also limit the backup.  If you are backing up computers whose top read speed is 168 Mbps, and your backup destination can handle 800 Mbps, then you aren't utilizing all the bandwidth of the backup source.  This is another reason to create multiple media sets in Retrospect so you can have more than 1 backup happening at the same time to saturate your backup destination bandwidth.

If the Mac Mini has a Thunderbolt connection that would give you the top speed.  10 Gbps.  However, the drive connected to it needs to be able to read and write that fast.  Leading back to getting a raid.  OWC makes a reasonably priced Thunderbolt raid:

https://eshop.macsales.com/shop/Thunderbolt/External-Drive/OWC/ThunderBay-4

If the mini doesn't have Thunderbolt you could try a FW 800 raid:

https://eshop.macsales.com/shop/External-Drive/OWC/Elite-Pro-Dual

https://eshop.macsales.com/shop/hard-drives/RAID/Desktop/

The more drives you have in a raid the faster the read / write speeds.  Thunderbolt is the way to go if you have it.

You can use a disk speed test tool to determine how fast your backup destination disk is.  Blackmagic makes a good one that is free:

https://www.lifewire.com/blackmagic-disk-speed-test-4065592

Even though you are using firewire 800 you are probably not saturating the firewire 800 bandwidth and are limited by your disk speed.

Share this post


Link to post
Share on other sites

abb668b8-e67d-45d2-8fe4-b2a87dc2831e,

Thank you for your suggestions.  I'll respond to them out of your sequence, for reasons that will become obvious.  I'll also refer to the Retrospect Mac 10 User's Guide, both because it's the first complete UG since Retrospect Mac 8 and for another reason that will immediately become obvious. 

Try using Retrospect Instant Scan

I thoroughly endorse this suggestion, so far as it is applicable—which is not at all.  Instant Scan was introduced with Retrospect Mac 10—see page 14 of the UG, and ridgedale says in the OP that he/she is running Retrospect Mac 9.

Increase Backup Destination Storage Speed

As I said in the second paragraph of this post, that won't help much at all. 

I do agree with the immediately-above poster, however, that changing the unit of measurement to megabytes per minute clarifies the problem.  ridgedale may not be aware that lower-case 'b' means bits, and upper-case 'B' means bytes.  IME people who use Mb/s are usually taking raw speed from a data communications utility, which includes TCP/IP overhead that runs more than 10%.  I prefer to convert such figures to data-transferred MB/s by assuming 10 bits per byte, which exaggerates the overhead slightly to 20% but makes for easier calculation—while converting to 1/60th of the MB/minute speed unit Retrospect uses in the log and Console.

If we change ridgedale's unit of measurement per the preceding paragraph, 580Mb/s becomes 3480MB/min.  That comes out in the general range of what I get backing up drives that are locally-attached to my "backup server", so that might be correct if ridgedale is talking about the rate for a NAS that is SMB-mounted—so that the Mac Mini is doing the traversal of multiple files on the NAS.  If OTOH the figure is actually 580MB/min, that actually agrees quite closely with what I get backing up my fastest client (where the client machine is doing the traversal of multiple files).

Create separate Media Set for each backup Source

First, this is really applicable only if ridgedale is running a Server Edition of Retrospect 9, not the Desktop Edition.  Page 179 of the UG describes the Preferences option Allow n Activity Threads.  That pop-up defaults to 4 in a Server Edition, but can be permanently set higher.   If ridgedale is backing up 5 clients and a NAS at one time, the pop-up should be set to 6.  However if ridgedale is running the Desktop Edition, setting that pop-up will only be effective for the current execution of the "backup server"—after which it will default back to 1; that is EMC's 9-year-old policy for making the Desktop Edition run as if it were still the single-threaded Retrospect Mac 6 (to the extent possible without having two separate code bases).

Second, ridgedale should be careful to have each of his/her separate Media Sets on a different physical disk.  Otherwise—although I can't remember at the moment where the warning is documented—there will be "write-head-thrashing" going on between the simultaneously-running backups, which will slow them down significantly.

Don't use Recycle Backup except for emergencies

That really depends on what kind of data ridgedale's installation needs to back up.  The Grooming options are described on pages 215-217 of the UG, but ridgedale should also read this Knowledge Base article (while ignoring item 1, which refers to a choice not introduced until Retrospect Mac 13).  EMC designed Grooming to allow installations to comply with regulatory requirements (as stated in the second sentence of "Automated data grooming" here, except that the specified-number-of-months refers to an option not available until Retrospect Mac 12).  If ridgedale's installation needs to backup locally-created audios/videos or advertising-related documents, in which each version must be preserved, then Grooming may not be appropriate.

Edited by DavidHertzberg
Under "Create separate Media Set ...", EMC owned Retrospect in 2009, changed "drop-down" to "pop-up"; under "Increase Backup Destination ...", split into 2 prgfs. and added a third one

Share this post


Link to post
Share on other sites

Hi Don, David and abb668b8-e67d-45d2-8fe4-b2a87dc2831e,

Thanks you for all your feedback. It is very much appreciated.

Firstly to clarify/correct my reference to the speed of the backup: 580Mb/s should have been 580MB/s as David correctly surmised. The version of Retrospect being used is the Desktop Backup 9.02 version.

To expand on the reason behind the decision to opt for two 8-week cycles of recycle and incremental backup scripts, some years ago I was responsible remotely for a Windows network that also a similar NAS as a file server. In that instance a PC was dedicated to handling the backups (it did nothing else). The backups had been configured as an initial full backup followed by years' worth of recycled backups. One day the customer informed me that the NAS had failed catastrophically overnight. I can't remember the exact details (Restropsect 6 Professional (Windows) and NAS RAID 5 configuration, if I recall correctly) but ultimately multiple drives in the NAS had failed rendering it dead. The only option available was the daily backups that had been running for years. A new set of drives were ordered and installed, and the RAID reconfigured without any problems. Then came the task of restoring the data. This is where everything became a massive problem. It took over week to restore the user data to the newly configured NAS server from the Retorspect backups. The restore process was glacial and it was evident that Retrospect was checking and comparing every historical incremental backup to find the data to be restored! :(

Following that experience I decided to change to all implemented Retrospect backup configurations so that they run on 8-week overlapping cycles (effectively a 16-week cycle to two separate backup drives) to reduce the retore overhead in the event of catastrophic server failure and still provide the option to recover files going back 4 weeks at a minimum, in essence as follows:

Backup1_Week1_Recycle (Sunday) - to Backup1 External Drive
Backup1_Week1_Incremental (Monday-Saturday) - to Backup1 External Drive
Backup2_Week2_Incremental (Sunday-Saturday) - to Backup2 External Drive
Backup1_Week3_Incremental (Sunday-Saturday) - to Backup1 External Drive
Backup2_Week4_Incremental (Sunday-Saturday) - to Backup2 External Drive
Backup1_Week5_Incremental (Sunday-Saturday) - to Backup1 External Drive
Backup2_Week6_Incremental (Sunday-Saturday) - to Backup2 External Drive
Backup1_Week7_Incremental (Sunday-Saturday) - to Backup1 External Drive
Backup2_Week8_Incremental (Sunday-Saturday) - to Backup2 External Drive
Backup1_Week9_Incremental (Sunday-Saturday) - to Backup1 External Drive
Backup2_Week10_Recycle (Sunday) - to Backup2 External Drive
Backup2_Week10_Incremental (Monday-Saturday) - to Backup2 External Drive
Backup1_Week11_Incremental (Sunday-Saturday) - to Backup1 External Drive
Backup2_Week12_Incremental (Sunday-Saturday) - to Backup2 External Drive
Backup1_Week13_Incremental (Sunday-Saturday) - to Backup1 External Drive
Backup2_Week14_Incremental (Sunday-Saturday) - to Backup2 External Drive
Backup1_Week15_Incremental (Sunday-Saturday) - to Backup1 External Drive
Backup2_Week16_Incremental (Sunday-Saturday) - to Backup2 External Drive

I tried some time ago purchasing and testing file transfers using an eSATA adapter connected to a NAS but found that the integrity of the fiel transfers were being compromised. I just felt I couldn't trust the eSATA technology at that time to transfer large volumes of data. The mac Mini unfortunately does not incorporate a Thunderbolt port otherwise that the deicision would have been to use extrnal Thunderbolt enabled drives for the backup storage.
It would appear from all the feedback provided that the primary key to Retrospect performance is the specification of the computer dedicated to the backup task with processor speed and RAM high on the list of requirements. In that case the solution would be to either upgrade or replace the macMini as well as upgrade Retrospect. It might be helpful if I provide some log stats to give you all a better handle on the situation as I not convinced the backup of the clients is an issue. I'll do that overnight after the next backup has completed.

Thanks again for all your input.

 

Share this post


Link to post
Share on other sites
11 hours ago, ridgedale said:

Hi Don, David and abb668b8-e67d-45d2-8fe4-b2a87dc2831e,

Thanks you for all your feedback. It is very much appreciated.

Firstly to clarify/correct my reference to the speed of the backup: 580Mb/s should have been 580MB/s as David correctly surmised. The version of Retrospect being used is the Desktop Backup 9.02 version.

....

....

....

....It would appear from all the feedback provided that the primary key to Retrospect performance is the specification of the computer dedicated to the backup task with processor speed and RAM high on the list of requirements. In that case the solution would be to either upgrade or replace the macMini as well as upgrade Retrospect. It might be helpful if I provide some log stats to give you all a better handle on the situation as I not convinced the backup of the clients is an issue. I'll do that overnight after the next backup has completed.

Thanks again for all your input.

 

ridgedale:

First, I don't think you read my preceding post in this thread very carefully. I did not say 580MB/sec, I said 580MB/min.  I learned in third grade, IIRC, that there are 60 seconds in a minute; I also learned in 1964 that there are 8 bits in a byte.  580 MBytes/sec would be 8 * 580 Mbits/sec = 4640Mbits/sec.  You say in your OP "NAS and clients connected with 1Gb ethernet", so 4640Mbits/sec would be impossible unless they are actually connected with 5Gbit Ethernet—which very few installations have yet.  I hate to be heavy-handed, but I strongly suggest that you spell out "bit" and "byte" and "sec" and "min" for the rest of your posts in this thread.  Just in case your posting errors result from your reading rapidly-changing figures from the Console, I also suggest that for those posts you use non-changing figures from the Retrospect Log, which is discussed on page 206 of the Retrospect Mac 10 User's Guide.

Once you start doing that, we can actually discuss the speed you are getting backing up your installation's NAS, versus the speed you are getting backing up your installation's client machines.  On the one hand,  "the specification of the computer dedicated to the backup task" would affect backing up your installation's NAS—if it is set up as a shared drive in that dedicated  "backup server". OTOH, my tests—as recounted in the second paragraph of this previous post in the thread—show that it is the specifications of a Retrospect client machine—and primarily its processor speed—that is the primary key to Retrospect performance in backing up that client.  When you quote from the Log figures for Performance of a particular Source, please quote the figures before the word "copy" in parentheses, not the before the word "compare".  I'll get into the "compare" operation in another post, because I think that de-coupling it from the "copy" operation might help to speed up your backups.

While you are at it, it would be nice if you clicked on your "handle" to the left of a post you have made, and clicked Edit Profile at the upper right corner.  You could then put in a Gender, so I could stop referring to you as "he/she" and "him/her".  You could also put a general Location; I'm guessing from your times of posting that you are probably a Brit.  Lastly you could briefly describe your prior experience in Interests; whether you come from an IT background, and maybe whether you are a consultant, would help us in how we respond to your posts.

Edited by DavidHertzberg
Added 2nd and 3rd sentences to second paragraph, saying that whether a particular machine's specifications affect backup speed depends on whether you are backing up a NAS or a client machine

Share this post


Link to post
Share on other sites

ridgedale:

I promised to discuss de-coupling the "compare" operations in Backups from the "copy" operations, because this would in fact speed up your Sunday Backups.  EMC introduced the separate Verify script in Retrospect Mac 8 just for that purpose.  To use this feature, you would have to change the the pop-up in the Backup options of your scripts to No Verification—as described from the bottom of page 112 through page 113 in the Retrospect Mac 10 User's Guide.  You would then schedule separate Verification scripts to run during the work week, as described on pages 158-159 of the UG.  It would be preferable to schedule the Verification scripts to run as early as possible on Mondays, to avoid having to investigate more than the minimum number of  "warnings" that were really caused by someone changing a file after it was backed up.

In addition, what I wrote in the "Create separate Media Set for each backup Source" section of this post doesn't mean that you couldn't use the facility because you are running the Desktop Edition.  It just means that you would have to ensure that the Preferences option is changed after your Mac Mini "backup server" is booted for your Sunday Backup runs.

 

Share this post


Link to post
Share on other sites

ridgedale:

It looks as if you might be able to double your Backup performance for as little as US$119.  I just went through the cumulative Release Notes for Retrospect Mac, searching for "Improved".  Both versions 14.0, 13.0, and 12.0 improved performance; in 12.0 there are "Performance increases for backup and restore, up to 100% faster" (I transitioned from Retrospect Mac 6.1 directly to version 12, so I wouldn't know).  Of course that price is for upgrading the Desktop Edition; upgrading to the Single Server 20 Workstation Clients edition in order to "Create separate Media Set for each backup Source" and non-underhandedly backup to those Media Sets in parallel would cost US$499; both prices are without Annual Support and Maintenance.

BTW it's surprising you can run Retrospect Mac 9 under Mavericks; according to the Release Notes for version 10.5  "OS X Mavericks ready — OS X Mavericks 10.9 is fully supported in this version (pending final release)."

Edited by DavidHertzberg
Quoted performance improvement for version 12.0; specified price for Single Server 20 Wrkstn.; added prgf. saying Mavericks officially not supported until version 10.5

Share this post


Link to post
Share on other sites

Just to give you a simplified recapitulation, ridgedale:

If your main time-consumer in Recycle backups are your installation's  client machines, then—short of replacing them with ones with significantly-faster processors (which I assume would constitute "breaking the bank")—then IMHO you are limited to the software solutions in my two posts directly above this.  Here is a newly-written post  containing my hypothesis as to why backups using the Retrospect Client software are inherently slow.

That same newly-written post explains why, if OTOH your main time-consumer in Recycle backups is your installation's NAS—and that NAS is setup in the Retrospect "backup server" as a shared Source, then there may be an additional hardware solution.  That solution would be to replace your installation's Mac Mini with a relatively-low-cost fast new Mac.  Unless you want to wait several months for the fabled reinvention of the Mac Pro (which is rumored to include a lower-priced model that would replace the Mac Mini), that fast new Mac would be the 21.5-inch iMac with Retina 4K display.  It would cost your installation US$1300, plus the cost of a FireWire800-to-USB3 adapter.

The question of which is the main time-consumer in your installation's Recycle backups is why I have laid so much stress on your accurately reporting MByte/min rates for both the client and NAS Sources backed up .  Along with, of course, the MByte/GByte totals backed up for each Source.  Without those, we can't give you accurate advice.  And, BTW, what the dickens does "5x clients" in your OP mean; is it N clients, where 50 ≦ N ≦ 59?

Edited by DavidHertzberg
Expanded 3rd major paragraph beyond a single sentence; parenthetical mention in 2nd prgf. of possible future low-cost Mac Pro; meaning of "5x" clients

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×