Jump to content

Rebuild catalog - Problem with one of the tapes


Recommended Posts

Nigel,

SAN is for live data for the most part. We tend to keep projects that are up to 18 months old because clients in the music industry have a tendency to need the material to make something new from it; long story. So, I this is what I'm thinking would be good for me.

1. Archive the entire SAN to LTO; happens nightly and is rotated with three media sets throughout the week. (Archive is taken care of)

2. With a NAS or internal drives in the MacPro, 12 T of disk storage to keep 18 months of backup, with the use of a filter in RS, and writing the data to Disk sets. (backup taken care of)

3. Groom disk sets so that any project or file that's older than 18 months gets groomed off the disk media sets.

 

Am I understanding this correctly? The great thing for me about RS, is that it does it's thing and I ONLY have to make sure the correct LTO tape is in the drive. It's pretty much set it and forget it. I'm hopefully what I described above can also be set and forget for the most part. 

Link to comment
Share on other sites

oslomike and Nigel Smith,

Let's get our terminology in sync with Retrospect's terminology, which is mostly 30 years old and wasn't adopted by the rest of the IT industry.  I'll refer to the Retrospect Mac 13 User's Guide, which is—fortunately for this thread—the first Mac version where the august Documentation Committee stopped copying the contents of the previous "What's New" chapter into other chapters of the UG (later converting the "What's New" chapter into marketing fluff):

Page 144 says about "archive"

Quote

Archiving lets you copy files from a volume to a Media Set for off-line storage. Archiving allows you to remove seldom-used files from a hard disk while maintaining a copy of those files on your storage media. With archive scripts, you can choose to move—rather than just copy—files from the source to the destination. For example, you might want to move the files for a particular project off your main hard disk after the project is completed, but still have those files be easily findable if you ever need to refer to them.

Note: An archive script has one major difference from a backup script. Archiving has the matching options disabled by default so that all files from the source are copied, even if they have previously been copied to the same Media Set. This is done for two reasons. By  placing all the files belonging to an archived project together on the backup media,Retrospect ensures the fastest restore of the archived files. Additionally, when the “Delete source files after copying and verifying” option is selected, only files archived and verified during that session will be deleted from the source.

That concurs with your use of "archive", except that Retrospect has a particular type of script named Archive—described from the rest of page 144 to the top of page 146.   An Archive script does what the quoted paragraphs say, but as a version of a Backup script with some enhancements and some limitations.  One of the enhancements is an option to delete from the Source (Nigel Smith "move") the data that has been backed up to the Destination.

Page 15 shows the latest version of the "Grooming" dialog for Retrospect Mac.  Pages 221-223 describe Grooming of a Disk Media Set, but they have not been updated to include the Months to keep specification shown (above v13's "Performance/Storage optimized" popup) for Groom to Retrospect defined policy in the dialog on page 15 —an enhancement that was added in Retrospect Mac 12.  Pages 163–164 tell how to create a Groom script.

What Nigel Smith calls a "filter" is called a "custom Rule" in Retrospect Mac; it used to be called a "custom Selector" before Retrospect Mac 8, and is still called one in Retrospect Windows—despite its having the capability of excluding files (which is why Retrospect Mac 8 renamed it a Rule).  Creating one is described under "Adding or Editing Rules" on pages 195–199; using them—as well as "built-in Rules"—is described on pages 193–195.

 

 

 

 

Edited by DavidHertzberg
Add Note paragraph to quote from page 144 of UG. Add in parentheses to "Grooming" paragraph that v.12 Months to Keep spec is above v.13 "optimized" popup. Clarify new v.13 Documentation Committee policy.
Link to comment
Share on other sites

On 1/22/2021 at 5:41 AM, DavidHertzberg said:

Let's get our terminology in sync with Retrospect's terminology...

I was trying -- and failing! -- to avoid RS terminology to get away from the software specifics and get oslomike to look at it more theoretically. That's difficult to do when RS uses the same terms as you'd naturally use: backups and archives 🙂

You can archive an (RS) backup. You can restore from an (RS) archive. So in practice they're shades of grey rather than black-and-white different, but it can help in creating a strategy to think of them as different.

Backups -- what you need for disaster recovery -- disk failure, accidental deletion, building fire, etc.
Archives -- safe storage of things not needed now but may be needed later, or must be kept for compliance etc.

In an ideal world you'd have all the data you ever used/created on super fast, instantly accessible, storage which you'd then back up (multiple copies, obviously!) in case of disaster. In the real world we usually can't afford to do that, so we keep the data we need, or will likely need, on the super fast storage and anything that is unlikely to be needed (or isn't needed instantly-on-demand) can be archived to cheaper slower, or even off-line, storage.

So oslomike's plan, above, is a good backup plan (assuming the tape sets are rotated so one is always off-site!) with "accidental archiving" -- old files are somewhere on the backup tapes, and still retrievable, though there's no provision for removal of those old files from the SAN. I'd add in an actual "deliberate archiving step", which doesn't need to be anything fancy -- eg once a year, copy all projects that finished more than 18 months ago to 2 (or more!) dedicated "archive" sets, verify those by retrieving data from them, then remove that data from the SAN. The more business critical that archival data, the more care and copies you should take -- you might want to archive to local (slow) disk storage, to a cloud service with multi-region replication, and to tapes kept in off-site secure storage, with bonus points if you use different methods for each so you aren't tied to Retrospect for it all (for example, you could use RS or some other method to push actual files to the cloud service rather than using RS's storage format).

Whether that's worth it to you is another matter, but it might give you some ideas.

Link to comment
Share on other sites

Thanks Nigel. I appreciate the lesson on the different between backup and archive. 🙂 I admit I've never used the term "archive", I've always used "backup." For me they've been one in the same because I've always done my backups to the tapes and I archive them in different safes in different locations. I have a rotating set of three, with one complete set always in a safe close to the backup machine for when I need to restore something, which is fairly often. BUT NOW, after al the help and insight I've received here, I'm going to make a proper backup to local disks. AND I think I need a little help with the script. Here's what I'd like to do:

I would like to make a Disk Media backup script that will backup any file or folder on a RAID disk that has been created or modified within then last 24 months. Anything that is older than 24 months that hasn't been  touched, gets groomed off the Disk Media Backup set. Is this possible? I have looked at the Rules possibilities in RS, but I haven't figured out which rule(s) are the ones that are best. It seems a couple will do the job, but I suspect there are nuances in how they behave that will affect the outcome. Any tips?

Thanks.

Link to comment
Share on other sites

6 hours ago, oslomike said:

I would like to make a Disk Media backup script that will backup any file or folder on a RAID disk that has been created or modified within then last 24 months.

I wouldn't, for a few reasons:

  1. I like to disaster recover to the last, best, complete state. If you select as above there could be files that were on the SAN but won't be restored -- if they aren't worth backing up, why are they still on the SAN? If they should be on the SAN, as part of an on-going project, shouldn't they be backed up even if they haven't been modified?
  2. You could get round the above by restoring to the last time-point then overlaying any missing older files from previous backups -- but that's a lot of work and error-prone compared to simply restoring everything from a single snapshot
  3. You should also include files that haven't been modified in the last 24 months but have been accessed -- obvious examples are templates that you open then "Save As...", or images that you link (rather than embed) in documents. Perhaps in your case a drum loop that you import into a project? You might need that original loop, even though it's never itself modified. Not all systems set an access flag, some are way too keen and set it for things you wouldn't consider an access for your backup purposes, so you should test it very carefully

Given the above, I'd back up everything on the SAN and rely on my archiving process to manage the amount of data on there.

I also wouldn't groom the disk set, but that's because I find it much easier to keep things straight in my head if a set has everything backed up between its start and end -- I just archive off to tape and start a new set when it gets unmanageable. YMMV, so remember that there are two ways to groom in RS:

  1. As a property of the Disk Media Set, where you only set the set to retain the last n backups of a source
  2. By running a separate Groom script, where you use RS's Selectors to decide what to keep/remove

If you still want to go by file properties, a Groom script will be the necessary. But given that you are considering backing up to both disk and tape I strongly recommend you look at the "Staged Backup Strategies" section (pp183 of the RS17 Mac UG). Not, in your case, for speed of getting data to your tape drive but because it reduces load on your SAN and gets it back to "production speed" more quickly -- if your users work odd hours and across your backup window, they'll thank you (Hah! When does a user ever thank a sysadmin?).

So I think I'd do:

  1. Nightly backups of all the SAN's files to the Disk Media Set
  2. Daily transfers from that to tape (means only 1 of your 3 tape sets is on site/travelling at a time, reducing risk of loss)
  3. Have a grooming policy that balanced how far back you usually go to restore a previous version of a file with the capacity of your Disk Set media

That last is particular to your setup, remembering that you can still go back further than n backups by going to your tapes -- it'll just take longer than a Disk Media restore.

So many ways to do things! Pick your poison, try it out, refine as you understand your needs more.

Link to comment
Share on other sites

Hi Nigel,

Lot's of info here. Thank you for all the good pointers and detail. I know it takes time to write these posts, so yes, thanks for being so generous. You mentioned that I should just backup everything on the SAN. I was hoping/thinking that making a disk media set I would do that and make it so I could do quick restores any project in any point in time that was worked on within that past two years without having to get out the LTO tapes. The life saver scenario for me is for the occasion where I  accidentally saved over a version of a song which has been heavily modified, only to find out the artist preferred the version I was working on two days ago, which I destroyed. My backups/archives to LTO tapes happen daily in the middle of the night, I was planning on having the Disk Media Set backup scripts also run nightly. And I was hoping I could set it and mostly forget it. In my mind, grooming sounded like a good idea. Set it up and always have 24 months of backups online and ready to be restored. As for grooming scripts, option 2, using selectors was what I was thinking, but maybe it's not possible. I don't know if this is possible, but it's the goal. I'll keep reading and trying to learn how it all works and what is actually possible. Thanks again.

36 minutes ago, Nigel Smith said:

Given the above, I'd back up everything on the SAN and rely on my archiving process to manage the amount of data on there.

I also wouldn't groom the disk set, but that's because I find it much easier to keep things straight in my head if a set has everything backed up between its start and end -- I just archive off to tape and start a new set when it gets unmanageable. YMMV, so remember that there are two ways to groom in RS:

  1. As a property of the Disk Media Set, where you only set the set to retain the last n backups of a source
  2. By running a separate Groom script, where you use RS's Selectors to decide what to keep/remove

If you still want to go by file properties, a Groom script will be the necessary. But given that you are considering backing up to both disk and tape I strongly recommend you look at the "Staged Backup Strategies" section (pp183 of the RS17 Mac UG). Not, in your case, for speed of getting data to your tape drive but because it reduces load on your SAN and gets it back to "production speed" more quickly -- if your users work odd hours and across your backup window, they'll thank you (Hah! When does a user ever thank a sysadmin?).

So I think I'd do:

  1. Nightly backups of all the SAN's files to the Disk Media Set
  2. Daily transfers from that to tape (means only 1 of your 3 tape sets is on site/travelling at a time, reducing risk of loss)
  3. Have a grooming policy that balanced how far back you usually go to restore a previous version of a file with the capacity of your Disk Set media

That last is particular to your setup, remembering that you can still go back further than n backups by going to your tapes -- it'll just take longer than a Disk Media restore.

So many ways to do things! Pick your poison, try it out, refine as you understand your needs more.

Link to comment
Share on other sites

oslomike,

On second thought, I agree with Nigel Smith that the relevant definition of small-'a' "archive he describes in this up-thread post, rather than the big-'A' "Archive" definition described in this earlier up-thread post, is the one relevant to your needs.  In that respect I think you should consider using a "seekrit" capability of Grooming scripts that was introduced in Retrospect 15.1.

This "Grooming Selectors" capability—although the Retrospect engineers evidently "busted a gut" creating it in time to meet the deadline for the GDPR's "right of erasure"—is documented only in this Knowledge Base article, because it was created too late to be described in the "What's New" chapter of the User's Guides based on 15.0.  (The "What's New" chapter of the next edition of the User's Guides based on 16.0 is primarily about the glories of the Management Console—created as a key part of Product Management's ultimately-unsuccessful "go big or go home" strategy.  The policy of the august Documentation Committee described in the last sentence of the first paragraph of this even-earlier up-thread post meant that the capability could not be documented in any other chapter of the UGs for versions later than 15.)  The "seekrit" is that you can create one or more conditions that can specify any directory or filename or other criterion that would identify a particular project, to be used in the Exclude section of a Rule (the post-Retrospect-Mac-6 term for Selector) in a storage-optimized Groom script's Rules—per "Adding or Editing Rules" on pages 166–170 of the Retrospect Mac 16 User's Guide.

You can then adapt these same conditions as a companion Include-section Rule for use in a Copy Media Set script's Rules tab.  The two scripts, when the Copy Media Set script is run—with results checked—before the Groom script, "archive the project" onto the destination of the Copy Media Set script while removing it from the source of the Copy Media Set script.  The article implies the Options tab for the Media Set being Groomed need only specify "Performance-optimized grooming" in the popup; if that's wrong, also specify "Groom to Retrospect-defined policy" with 999 in the "Months to keep" box.

Of course you'd have to upgrade to at least Retrospect Mac 15.1 to use this capability, though I recommend Retrospect Mac 16.6.  I suggest you phone European Retrospect Support and speak to a salesperson; you probably can negotiate a reduced price for an upgrade to a not-the-latest major version.

Link to comment
Share on other sites

David, this is very interesting. In my case, for my Disk Media Backup, can I select an entire disk as the directory and set the script to include (Backup) everything Created within the past 24 months, and exclude (remove) everything created before 24 months ago? I didn't quite find out how to do the last part. I do have a license for v17 so these "seekrit"  options should be available. I'll look into it some more.

Link to comment
Share on other sites

11 hours ago, oslomike said:

You mentioned that I should just backup everything on the SAN.

...and...

11 hours ago, oslomike said:

As for grooming scripts, option 2, using selectors was what I was thinking,

...makes complete restores difficult unless you never have anything older than 24 months on your SAN. That's where no backup rules and an "in set" Grooming policy win out -- you can always restore your SAN to how it was up to n snapshots ago with a single operation. Using date-based Rules will mean you can only restore all files that match that criteria, and may then have to go through other sets/tapes to find older items (which can be a real pain, takes time, and is prone to error).

So you need to ask yourself "If my SAN fails and I have to replace it, do I want to restore it as it was at a certain time or will restore only files modified in the last... be good enough?".

Link to comment
Share on other sites

Realistically, I only really need a backup of projects on the SAN that have been worked on up to a year back in time. In the case of a SAN failure, it's not as important to restore the entire SAN to the state it was in before it failed; anything missing is on LTO tapes. Two years is a good buffer, and if a script could keep me current with projects that are up to two years old, I would be very happy with that. If I wanted to do that, is there a script that I can set up that will do exactly that?

Link to comment
Share on other sites

2 hours ago, oslomike said:

If I wanted to do that, is there a script that I can set up that will do exactly that?

Not until you can define that exactly 😉

Rules are smart, but also very literal. I've already explained about "last access" vs "last modified", but also consider that you will be in a situation where only half a project can be restored, because some of the files in it were last used 729 days ago and others at 731 days.

If you work in projects, IMO it makes more sense to manage your backups by removing them from the SAN (to archive, obv) 2 years after they've finished -- they won't get backed up any more because they aren't there(!), and your Grooming policy will remove them from the Disk Media Set after the appropriate number of backups. No need for a rule in your backup script (and remember that, if time is important, every file on the volume has to be scanned before it can be excluded by the rule), no need to run a separate Grooming script.

If you still want to use a date-based rule, your first job is to test all the software that you use to access your SAN-stored files and find out whether "last access" will be honoured. Without that you won't be able to backup files that are read but not modified within your date window.

Link to comment
Share on other sites

oslomike,

I can see now that I didn't read this up-thread post by you and its immediate successor carefully enough.😔  What you are calling your "archive" is an LTO Tape Media Set containing—ideally—every version of every file your installation has ever created.  But that "archive" is too clumsy for "returning customer" restores, so you also want a "For Fast Restores" Disk Media Set containing only what you consider to be "active projects".  The problem's that you want Retrospect to automatically delete from "For Fast Restores"  all backed-up files—and only those files—that no longer belong to "active projects".

As Nigel Smith has made very clear, you're just not going to be able to get that result automatically using Retrospect's set-it-and forget-it Grooming based on any kind of dates associated with a file.  IMHO the best you'll be able to do is to get that result with a minimum amount of manual effort, based on using the "seekrit" v15 Grooming-with-Rules capability described in the second paragraph of my last up-thread post.  And you'll only be able to get that result if you keep each project's files in one or more separate folders, which I suspect you do—or you wouldn't have stayed in business long.🧐

Here's a minimum-manual-effort approach using Apple's macOS 10.13 High Sierra Notes.app, which I've never used but is described in this iMore.com "Mastering High Sierra" article:  Start Notes and create a new Note, then add a table to which you add a column (maybe you won't have to do this; the article's sections are inconsistent on the default number of columns) so that it will contain a 3-column row for each "active project".  The first column in each row will be your name for the project, likely including client name and "album" name.  The second column in each row will be the year-month-day last-activated date for the project.  The third column in each row will be the path fragment to the project's folder on your SAN.  If there can be multiple folders for a single project, in separate paths, I suggest adding additional columns rather than trying to cram multiple path fragments into the third column.  Add a row for any project after the first one, for which 2 empty rows will have automatically been created—I suggest putting column titles into the first of those 2 rows.  Then, each time a project is re-activated, change the last-activated date in the second column for its row and drag that row to the bottom of the table.  The result will be a table whose rows are sorted least-recent-to-most-recent on last-activated date.  (Incidentally, if you can do this with a spreadsheet application instead of Notes.app, feel free to do so—using automatic year-month-day date-sorting on the second column.)

Then, every few days look at the Note—noticing any rows whose second column is now more than 24 months old.  For each such "inactive" project, create per the last sentence of the second paragraph of my last up-thread post a Rule with the contents of the third column of the row replacing the KB article's path fragment /Users/Bob/ (don't forget to include a trailing forward-slash) in the last Mac screenshot in the Knowledge Base article.  If you have more than 3 non-empty columns in the row, click the '+' button and add extra lines to the rule with all fields the same as the first line except using the path fragments from the additional columns, and make sure Any is in the top-left popup.  Give the new Rule the name "Remove" followed by the project name from the first column in the Note row, and click the Add button.  Then click the radio-button at left of the Rule you have just created in the Rules tab of a Groom script created per page 140 of the Retrospect Mac 16 User's Guide—after check-marking the "For Fast Restores" Media Set in the Media Sets tab, and click the Run button to run that Groom script immediately—unless you want to use the Schedule tab to run that Groom script later.

After the Groom script has run successfully, you'll want to remove the row for its now-Groomed-out project from the Note.  This section of the iMore.com article at least tells how to delete the contents of all the row's cells.  The "Manage rows and columns" section of this support.apple.com article—which appears to be more comprehensive than the iMore article—tells you how to delete the row.  You may also want to delete the project-specific Rule—if you don't want to save it for later duplication and modification into another Rule; see the P.S. below.

This approach eliminates the use of the Copy Media Set script discussed in the third paragraph of my last up-thread post.  You won't need that—or its Include Rules corresponding to the Exclude Rules, because you're already running a backup to your "archive" Tape Media Set every night.

P.S.: In case you've not understood "Working with Rules" on pages 164–165 in the Retrospect Mac 16 User's Guide, the GUI for Retrospect Mac 8 was designed assuming that a Rule could be used in multiple scripts—so it's defined by clicking the Rules tab in Preferences.  A Rule is specified to be used in a particular script by clicking the Rules tab for that script—which shows all the Rules that have been defined, and clicking the radio button to the left of any Rule you want to use.  This GUI was not modified when the use of Rules in Groom scripts was added to Retrospect 15.1, but it's not too clunky for the purpose described in the fourth paragraph of this post—provided you use the Preferences facilities for "Duplicating Existing Rules" and "Deleting Rules" described on page 170 of the UG.

 

Edited by DavidHertzberg
P.S.: Clarify the GUI for defining a Rule, using it in a particular script, duplicating it to be modified into another Rule, and deleting it.
Link to comment
Share on other sites

Nigel and David,

Thanks! You've super helpful. Unfortunately though, I think I was asking too much of Retrospect, and maybe even of myself. I've been trying keep things under control work-wise,  balancing my real job and at the same time be the IT manager, which I'm not. I'll have to dial it back a little until I get more time. I've found a solution that will work for for the daily disk backup using a different product, while I'll continue to do all my archives with Retrospect and the LTO tapes. I've learned a lot from you both. Thanks again.

mike

Link to comment
Share on other sites

On 2/3/2021 at 9:08 AM, oslomike said:

Nigel and David,

Thanks! You've super helpful. Unfortunately though, I think I was asking too much of Retrospect, and maybe even of myself. I've been trying keep things under control work-wise,  balancing my real job and at the same time be the IT manager, which I'm not. I'll have to dial it back a little until I get more time. I've found a solution that will work for for the daily disk backup using a different product, while I'll continue to do all my archives with Retrospect and the LTO tapes. I've learned a lot from you both. Thanks again.

mike

oslomike,

I don't understand how your different  product for daily disk backup could achieve the result you said you were looking for in this up-thread post. Nigel Smith's  post further down the thread basically said you would have significant difficulties achieving that automatically with date-based rules, which why I suggested in the next post that you achieve it with minimum manual effort using project-based rules.

I don't believe there is any other backup application that has the equivalent of date-based rules that are more flexible than Retrospect's.  I also don't believe that any backup application could have the equivalent of project-based rules that are more flexible than Retrospect's.  I might be wrong in my second belief, because the advent of the GDPR's "right of erasure" has probably unleashed a storm of creativity among the developers of backup applications.  However, given the wide variety of ways in which various commerce applications could store customer-identifying data—which is what the "right of erasure" is all about, I suspect that the GDPR solution for any backup application would have to have the same kind of flexibility that Retrospect's use of Rules in Grooming provides.  And that kind of flexibility would require the same kind of minimal manual effort as an adaption for project-based Grooming.

Please Private Message me with the name of the different product you plan to use for daily backup.  I'd ask that you not post it on these Forums, because past experience has shown that the long-time head of Retrospect Technical Support "gets his knickers in a twist" when a Forums post mentions the name of a competing backup product.  However I think he'd make an exception for backup products mentioned in this Knowledge Base article, which consist of one competitor that is much more expensive than Retrospect plus the free-but-limited Apple Time Machine.  I just checked the manual for the much-more-expensive backup product, and it doesn't seem to have any capability that Retrospect Mac Rules don't have.  As for Time Machine, we know its date-based filtering is more limited than Retrospect and its project-based filtering is essentially equivalent to Retrospect's.

  • Like 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

×
×
  • Create New...