Is a Recycle required now and then ?

[hevanw]

I've been using Retrospect 7.5 for almost a decade or so, in combination with CrashPlan Home. Retrospect was sort of the 'backup' (no pun intended) to CrashPlan where I only ran Retrospect once week for all Windows clients at home from a 24/7 Windows home server (running Windows 10 Pro).

 

With CrashPlan Home being sunset, I'm now making Retrospect my main backup solution for all clients and therefore am upping the frequency to every 2 hours with a pro-active schedule. I'm evaluating the latest version of Retrospect for this (where especially the much improved speed looks to be very helpful). I then also store/backup my Retrospect backup files to the cloud every night as an off-site backup. This is currently working pretty well as the incremental backups cause only small changes to the rdb-files which then cause limited uploads to the cloud.

 

Now, this will work well as long as I keep doing incremental backups. However, in my old backup strategy, I regularly recycled backups. This is now less ideal since a Recycle will basically generate a completely new backup set which then again needs to be completely uploaded to the cloud.

 

Question: will never doing a recycle eventually cause problems with the backup sets, particularly in terms of performance and reliability ? Also, I realize that grooming a backup set (which will happen when I hit the max. allocated size for a backup set) may cause the rdb files to get modified so much that their cloud upload may require a full upload again.

[DavidHertzberg]

What hevanw wants to do is covered in this Retrospect Mac thread from 1.5 years ago.  He/she should look at question C] in post #1 in that thread, then jump to post #7 in that thread.  He/she should then implement step C6 in that post, except that—since hevanw is using Retrospect Windows—he/she should use Transfer Backup Sets as the scripted operation.  That is described on pages 214-219 of the Retrospect Windows 14 User's Guide.

 

The idea is that hevanw should never completely upload a new Backup Set to the cloud.  He/she should only Transfer the latest updates to his/her on-site backup to his/her Cloud Backup Set.  hevanw should separately Groom his/her Cloud Backup Set, which of course must be done using the "performance-optimized grooming" option, which is described in the "What's New" chapter of the Retrospect Windows 11 UG but—due to the brilliance (insert appropriate smiley here) of the Retrospect Inc. UG-writing committee—has been "un-described" from the Retrospect Windows 12 UG.

 

One complication is that hevanw is going to be using Proactive script(s) with a frequency of every two hours for on-site backup.  That means that he/she will have to ensure that his/her cloud backup script runs at least that often.  Otherwise hevanw may have to use Transfer Snapshots instead, and try to mess with the options on page 226 of the Retrospect Windows 12 UG to get the equivalent result.

 

In this post is a translation table that will enable hevanw to understand the thread linked to in the first paragraph in Retrospect Windows terms.

Edited October 19, 2017 by DavidHertzberg
Retrospect Windows-to-Mac translation table now in Forums post

[hevanw]

 

The idea is that hevanw should never completely upload a new Backup Set to the cloud.  He/she should only Transfer the latest updates to his/her on-site backup to his/her Cloud Backup Set.  hevanw should separately Groom his/her Cloud Backup Set, which of course must be done using the "performance-optimized grooming" option, which is described in the "What's New" chapter of the Retrospect Windows 11 UG but—due to the brilliance (insert appropriate smiley here) of the Retrospect Inc. UG-writing committee—has been "un-described" from the Retrospect Windows 12 UG.

 

One complication is that hevanw is going to be using Proactive script(s) with a frequency of every two hours for on-site backup.  That means that he/she will have to ensure that his/her cloud backup script runs at least that often.  Otherwise hevanw may have to use Transfer Snapshots instead, and try to mess with the options on page 226 of the Retrospect Windows 12 UG to get the equivalent result.

 

 

 

Thanks for the elaborate response !

 

Note that I'm currently not yet using Retrospect's own Cloud support for my uploads to the cloud. CrashPlan just considers the rdb-files as regular files and uses it's own algorithms for incremental backups to the cloud. As mentioned, this is working pretty well. Even though my entire Retrospect folder (i.e. where all the rdb-files of all clients are stored) is now about 1.4 TB and growing (as long as I don't groom), CrashPlan only sends a few GB per night to my CrashPlan cloud storage to complete the backup of the entire Retrospect folder.

 

Anyway, for now I will not worry about doing Recycles. From what I see from the online information (still need to go through the actual User Guide) I should have no need to recycle my backups in the foreseeable future.

[DavidHertzberg]

IMHO hevanw must go read the Proactive section on pages 248 through 264 in the Retrospect Windows 12 User's Guide, consider what his/her installation really requires, and "de-CrashPlan-ify" his/her mind.

 

First of all, hevanw can forget about "upping the frequency to every 2 hours with a pro-active schedule" unless he/she wants to schedule a separate Proactive script for each two hours in the day; see "Customizing the Schedule" on pages 261-263.  Retrospect has two types of backup scripts for two types of computer users.  For Sam StayInTheOffice,  there is a Backup script that can be scheduled to run as often as once an hour to backup his always-connected-to-the LAN computer.  For Suzie SalesPerson, there is a Proactive script that is scheduled to run for the better part of the working day in order to backup her laptop computer whenever she arrives and connects it to the LAN.

 

I set up Retrospect in 1995 and ran it until 2003 primarily to benefit my (now ex-) wife.  She was (and still is) the epitome of Sam StayInTheOffice, and was doing extensive writing and art work on her desktop Mac.  Nevertheless she was fine with a once-a-day Backup script, since she is smart enough to save new versions of her files under different names.  I might have been a prototype of Suzie SalesPerson, except that I did my out-of-the-home work on a company-owned terminal or desktop computer that I was not responsible for backing up—so I only needed a daily Backup of any files I created on my desktop Mac at home.  We simply didn't need to run a Proactive script, nor did we need to run a Backup script more than once a day (hard disk drives usually gave advance warning before they failed).

 

CrashPlan was designed to do incremental backups every 15 minutes primarily in order to spread out the load on CrashPlan Central; the fact that CashPlan for Home LAN also (while it's still available) does this is just an incidental byproduct that spreads out the load on a LAN disk backup destination used by several computers.  Because Retrospect was designed when expensive "backup server" tape drives were the only feasible destination for backing up multiple LAN computers, it uses an alternate approach—which is to Backup LAN computers in succession at a time which is scheduled (preferably when that LAN computer is idle) by the administrator.  Proactive scripts were designed as an alternative for backing up Suzie SalesPerson, because you can never be sure when her laptop will be on and connected to the LAN.  Does hevanw really have a Suzie SalesPerson in his/her installation, and do his/her Sam StayInTheOffice people really need to be backed up more than once a day?

 

If the answer to those two questions is yes, then hevanw can still do with Transfer Backup Sets what I suggested in the first paragraph of post #2 in this thread—provided that Suzie SalesPerson doesn't connect to the LAN more frequently than Sam StayInTheOffice needs to be backed up.  If on the other hand hevanw's installation is really a home one with multiple users, then he/she should consider doing what I and my wife did—which is to turn on our computers while having breakfast and run a Normal Backup script scheduled to run before (if the "backup server" is already on) or during breakfast.

[hevanw]

(wonder why in all your posts you refer to the other forum members in 3rd person...? Looks rather goofy...)

 

Anyway. I used to do for a decade exactly what you describe in your last paragraph. Every Friday morning I would turn on all client PCs and take backups. This was the only feasible way in the beginning as I was still backing up to DVD-RWs then (using Retrospect 7). I had multiple backup sets such that I could rotate and recycle them like twice a year, at which time I would have to tell my wife not turn off her laptop as a recycle is a full backup and thus would take several hours.

 

However, with a NAS as target, there is really no reason to limit yourself to such infrequent backups. I'm now running proactive backups since 2 weeks or so (evaluating latest Retrospect) and realize I should have done this much earlier (basically as soon as I abandoned the DVDs). No longer the hassle of having to make sure that clients are on when running the backups (in fact, I ran the scripts manually then), and at the same time having much more frequent backups. And yes, even though it's a home setting, the schedule more corresponds with your Suzy SalesPerson model where the times the laptop is on and available for backup is rather arbitrary.

 

The 2h frequency may be overkill, but it's again a trade-off between backup overhead and backup recency. The longer you wait with your backups the more potential data can be lost when a disaster strikes. At the moment I don't have the impression that the overhead of Retrospect is that high that a 2h frequency would cause a problem. The only reason my wife now knows her laptop is being backed up so frequently was because of the W10 notifications that popped up every 2 hours, but that was easy to turn off.

 

Now as for cloud storage/backup, that's of course a different matter, since then other variables come in play like limited network speed (esp. upstream), cost of upload, cost of online storage, etc...

 

I also read that specific section in the UG, and the only thing that really concerns me is that Proactive Backup is only available with the proper license. So I need to double check whether it's actually available in the Desktop version..  EDIT: Proactive just seems to be part of the Windows Desktop (and I then assume any) version : https://www.retrospect.com/en/products/win/desktop .

[DavidHertzberg]

All I'm saying in post #4 in this thread is that, if hevanw wants to use the Transfer Backup Sets (in Retrospect Windows terms) to cloud approach in step C6 of post #7 in the linked-to thread in post #2 of this thread, he/she is going to have to make sure that no computer in the household gets itself into the Proactive script queue more than once in the intervals between step-C6-to-the cloud-script runs.  

 

That approach was corrected and validated 1.5 years ago by Mayoff, when he was still posting to these forums.  I'm not Mayoff, but merely an unpaid volunteer trying to fill in—along with other volunteers—for his absence.  That's why I refer to other Forums members in the third person; I feel it's less confrontational.  If hevanw wants my posts to appear less goofy, he/she should fill in a gender in his/her Forums profile so I won't have to use all these non-gender-assuming pronouns.

 

As far as the item about a license requirement for Proactive scripts on page 248 of the Retrospect Windows 12 User's Guide, I'm pretty sure that's a typo.  This morning, while my "backup server" was doing a Normal backup of my most-active Mac, I started to create a Proactive script to see if I could.  Retrospect didn't have any problem, and I have the Retrospect Mac 14 Desktop edition.

[DavidHertzberg]

....

....

 

Now, this will work well as long as I keep doing incremental backups. However, in my old backup strategy, I regularly recycled backups. This is now less ideal since a Recycle will basically generate a completely new backup set which then again needs to be completely uploaded to the cloud.

 

Question: will never doing a recycle eventually cause problems with the backup sets, particularly in terms of performance and reliability ? Also, I realize that grooming a backup set (which will happen when I hit the max. allocated size for a backup set) may cause the rdb files to get modified so much that their cloud upload may require a full upload again.

 

 

Why does hevanw need to Recycle his/her local Backup Sets?  Is this a holdover from the days of backing up to DVD-RWs?  If hevanw Recycles a local Backup Set, what happens if the computer disk drive he/she is using as a Source fails in the midst of the Recycle backup?

 

I happen to Recycle one of my three portable USB3 backup drives each week.  But that's because I do 6 Normal backups onto that drive after I've Recycled it and then take it off-site to my bank safe deposit box, not because I think the backup drive has that limited a lifetime.  In the one case where I had to Restore a laptop drive from the Backup Set 6 months ago under "combat conditions", the Restore worked fine.  

 

Welcome to the wonderful early 21st Century,  hevanw.  If you want to keep the size of your local Backup Sets below some limit, try Grooming them separately from any Grooming you may do to your Cloud Backup Set.

[hevanw]

Thank you. This is exactly what I wanted to know, and yeah I guess the recycling was still a remnant of the days where I backed up to DVDs.

[DavidHertzberg]

If hevanw is worried about portions of his/her local Backup Sets going bad on disk, he/she should consider running an occasional Verification script.  That is described starting on page 230 of the Retrospect Windows 12 User's Guide.  Note that the fourth paragraph on page 230 says "In certain circumstances, Retrospect does not have access to MD5 digests generated during backup. This is true for all backups created using versions of Retrospect prior to Retrospect 12.0, as well as backups that took place when Retrospect’s 'Generate MD5 digests during backup operations' preference was disabled."

 

dzeleznik has been doing this regularly, and about 2 months ago reported  errors in this thread.  As reported in post #8 of that thread, some of dzeleznik's Verification errors turned out to be real.  However iCompute, whose Retrospect Mac thread is linked to in post #2 of that thread, reported that his Verification errors turned out to be false.  There is some discussion in dzeleznik's thread of a fix to bug #6668, and whether it broke some things in the process of fixing others.

 

If one of hevanw's Verification runs reports a problem with a backed-up file, he/she should be able to Restore the file from his cloud Backup Set.  Retrospect Inc. has informed us that backups to cloud Members of Cloud Media Sets are always error-free, and therefore we should turn off the Verify option on such scripts to save time and processing—for which the cloud provider will likely charge.  There was a complaint by jhg, for which I have now found my last post in that thread, that his/her Verification run was reporting errors in messages that did not include the file name—which meant he/she didn't know which file(s) to Restore from an off-site backup.  There was a bug fix in Retrospect Windows 12.1 related to this.

 

P.S.: It is of course possible for one of  hevanw's Catalog Files to go bad, since these are almost certainly on a local hard disk drive connected to his/her "backup server"—even the Catalog File for his/her Cloud Backup Set.  Instructions for doing a Rebuild start on page 480 of the Retrospect Windows 12 UG.  I personally have never done a Rebuild of a Catalog File "in anger", probably because I do a Recycle of each of my Backup Sets every 3 weeks.  I have done a Rebuild as a test; the important thing is to be able to lay your hands on all the disks in the Backup Set in the proper sequence.  I have no idea exactly how to do a Rebuild of a Cloud Backup Set whose Member is in the cloud.

 

P.P.S.: Using Advanced Search on posts with my name, found the post—mentioned in the third paragraph of this thread—in a thread starting with the complaint that jhg's Verification run was reporting errors in messages that did not include the file name—which meant he/she didn't know which file(s) to Restore from an off-site backup.  There was a bug fix in Retrospect Windows 12.1;  the error message in that thread was "Backup Set format inconsistency".