Recommendations D2D2T backup setup

[Nigel Smith]

On 5/6/2020 at 2:21 PM, Joriz said:

What kind of hardware are you actually using? Like a Mac Pro or a Mac Mini (with thunderbolt to SAS ?)

Currently a 2014 Mac Mini in a Sonnet enclosure with dual interface 10GbE card. Attached to an old (but brilliantly reliable) ADIC Scalar 24 with a single LTO-2 drive via a Promise SANLink2 Thunderbolt to FC adapter (flakey, leads to a lot of prematurely "full" tapes, but the data is always good). Disk-wise we use a Thunderbolt-chained pair of LaCie 8Bigs for 84TB of space, but can overflow onto networked Synologys if required. The server itself, including RS catalogs, is backed up with Time Machine to an external USB drive as well as with RS.

Next iteration will probably be a new Mac Mini with built-in 10GbE in a Thunderbolt 3 enclosure, still using the 8Bigs but permanently adding one or more Synology NASs for even more disk capacity ("private" network via the card while built-in handles the clients), and adding a SAS card (H680?) to connect to a Quantum Superloader 3 with single LTO-8 drive.

As you can tell, there's a lot of using what we've got while upgrading what we can/must. That's partly funding issues, partly because I hate spending money (even other people's!) if I don't have to, but mainly because I like to stick with what I know works rather than deal with the inevitable issues with new kit.

The above is with all due respect to our hosts. I've had a Drobo on my desk since soon after they came out in the UK, have been very happy with it, but relatively low storage density/lack of dual PSU[1]/rack mounting means chained 8Ds or similar aren't an option. And I've dreamt of having a BEAST in the room for years, but they are too deep for our racks (shallow because they have built-in chilled water cooling), and since we're on the third floor we have a loading limit -- the BEASTs have great storage density, but we'd have to leave the rack half empty because of the weight! If/when we relocate the room or replace the racks, BEASTs will be on my shopping list...

I'll skip over the Windows PC running RS that came after the Mac Mini, when we were considering jumping ship after rumours of Apple letting the Mini die, because <shudder>Windows!>/shudder>. The RS side isn't too bad -- oh look, it's Retrospect 6 with some new features! -- but, as a Mac guy, Windows always seems clunky to me. It's still running, still doing its job (in parallel with the Mac Mini), but it never feels right...

As always, different requirements and situations lead to different solutions -- and I'd never hold up what we've done as a good example, just what works for us 😉

[1] Yeah, I know -- "Why do you want dual PSUs when the Mini only has one?" Because, in my experience, it's a lot easier to recover/rebuild the server than a big storage volume after a hard power-off. And we've got a good UPS, so the most likely reason for a power-off is clumsiness in the rack (yanking the wrong lead!) or a PSU failure which we've never had with a Mini but have seen a few of in other server room devices -- expensive Apple components, for the win!

[DavidHertzberg]

On 5/12/2020 at 4:03 AM, Nigel Smith said:

etc...

And this is my confusion: p120

...which implies that you can select whether to have all, some, or only the most recent while Copy Media is always all, and p121

...where "backups" is plural, even for a single source.

While I realise no manual is big enough to explain every single someone might come across, "Copy the most recent backup for each source" wouldn't take up much more virtual ink if, indeed, that is what happens.

Nigel Smith,

I finally figured out the explanation for the confusing Copy Backup popup option explanation in the Retrospect Mac 17 User's Guide (you're citing that version, though Joriz runs 16.6) page 121.  We must detour to page 177 of the Retrospect Windows 17 UG, where it is written of Transfer Snapshot—the equivalent operation with the same options described in a slightly-different sequence:

Quote

6.Select the Snapshot(s) to transfer.

  Once you select a source Backup Set, there are a number of ways to select which Snapshots to transfer.

7.The most recent Snapshot for each source transfers the most recent Snapshot and associated files for each volume (or subvolume) in the active Snapshots list.Retrospect transfers whatever the most recent Snapshots are at the time the script runs.

8.The most recent Snapshot for each source selected transfers the most recent Snapshot and associated files for each volume (or subvolume) you select in the active Snapshots list.  Retrospect transfers whatever the most recent Snapshots are for the selected sources at the time the script runs.

9.All active Snapshots for each source transfers all active Snapshots and associated files. To see the list of active Snapshots and their sources, temporarily choose the “Selected Snapshots”option. Make sure to choose “All active Snapshots for each source” again before clicking OK.To make an older Snapshot active, click Add Snapshot.

10.Selected Snapshots transfers only those Snapshots (and associated files) that you select from the active Snapshots list. Control-click or Shift-click to select multiple Snapshots. To make an older Snapshot active, click Add Snapshot.

  “The most recent Snapshot...” options are very useful for Transfer Snapshots scripts since the list of a  Backup Set’s active Snapshots changes each time you back up.

11.Transfer Snapshots scripts only copy active Snapshots. To copy all Snapshots, use a TransferBackup Sets script. See Scripted Backup Set Transfer for more information.

So why did they butcher this explanation for Retrospect Mac 8 and thereafter?  The reason is that the term "snapshot" was abolished in the Retrospect Mac GUI because by 2008 the term had acquired a standard CS meaning, eventually even at Apple.  Starting in 1990 the Mac UG (p. 264) had defined it:

Quote

Snapshot– In pervious [sic] versions of Retrospect, a Snapshot refers to the point-in-time file and folder listing that is captured during a backup operation to depict a volume’s state (that is, all its files and their paths). Makes it easy to restore a hard disk to its exact state as of a given backup. Retrospect [Mac] now uses the term backup to include both session and Snapshot data. Also see backup.

The term "active Snapshot" is not defined even in  the Windows UG; it means a Snapshot that the "backup server" has given the status "active" by keeping it in a source Media Set's Catalog.  As we see from Eppinizer's  first paragraph quoted here up-thread,  it is the single latest Snapshot if the source Media Set has grooming disabled—but is the "Groom to keep this number of backups" number of latest-going-backward-chronologically Snapshots otherwise.

So that's why the choices in the Copy Backup popup have the word "backups" as plural.  I'll create a documentation Support Case asking that the august Documentation Committee put Eppinizer's definition of "active" backups/Snapshots into the UGs.  But "a backup that is kept in the Catalog" sounds silly.

[Nigel Smith]

Thanks for that, David -- a very clear explanation. And you're right -- the thing that's missing is a definition of "active backup", which is unfortunate given that it is fundamental to how "Copy Backups" works.

Indeed, that's the only place the term "active backup" appears in the User Guide! Which is disappointing, since one of the things that should really, really, be clear in any instructions for a backup program is what will be backed up.

But from what you say we'll get similar results using either "Copy Media" or "Copy Backup" with grooming retention set to at least the number of backups in a "rotation period" -- in Joriz's case that would be 15 (Mon-Fri = 5 backups a week, 3 tape sets rotated weekly -- 5x3=15).

I'm starting to think that if you want everything, rather than a subset, on every tape set then "Copy Media" is more appropriate. But, again, I'd hesitate to say without proper testing.

[DavidHertzberg]

Thank you for the compliment, Nigel Smith; I did a number of revisions to my preceding post to make it clear without being too lengthy.

I just submitted a Support Case (note to self: #73687), covering both the relevant User's Guide deficiencies—both Mac and Windows variants—and the apparent Cumulative Release Notes error I spotted per this up-thread post.

As far as the Mac User's Guide deficiency is concerned, it may help to be aware that according to DovidBenAvraham Retrospect Mac 8—including a totally new GUI and terminology changes as well as the enhancements of Retrospect Windows 7.5—was hurriedly released in 2009 after EMC first end-of-lifed Retrospect Mac and then allowed the re-hiring of some of the employees who had worked on it.  According to the head of North America Sales, an old-timer with whom I had a longer-than-expected phone conversation the other day, one of those employees was both brilliant and rigid—and he was the one who rewrote the Mac UG.  Then he moved to VMWare, so he wasn't around to deal with Retrospect Mac terminology and documentation problems.

One of those problems is that the Mac UG "now uses the term backup to include both session and Snapshot data".  Of course the Retrospect Mac Engine, which since late 2009 has been common code with the non-GUI part of the Retrospect Windows Engine (not the Windows-only GUI ), still uses Snapshots in the Retrospect sense; you can see that merely by watching the Console text summary of a running Backup script.  And of course the session data isn't stored in the Catalog for an active backup; it's only the Snapshot.  So I said in the Support Case that, besides copying "Eppinizer"'s explanation of "active backup" onto page 120, they're going to have to enhance it to include an explanation of "Snapshot" referring to the Glossary definition on pages 264-265.

P.S. (because I needed some sleep): As I interpret Joriz's posts on the first page of this thread, he has three interlocking problems in his campaign to have a disaster-recovery backup—preferably off-site—that is up-to-date:  (1) His boss won't pay for more than the 3 x 2 = 6 LTO-7 tapes (costing US$65 apiece, according to my Googling) currently owned by the company. (2) Any disk-to-tape script he runs on weekdays must fit into the portion of 24 hours that remains after his server-disk-to-backup-disk incremental Backup scripts have run.  (3) Veeam is churning bigger Retrospect incremental server disk backups than it ought to, because Retrospect is seeing VMWare files Veeam already backed up as having been changed in some way. 

To cope with these problems, I have made the following two suggestions: (a) Run a disk-to-tape script nightly, but make it as short as possible by having it add to tape only what the Snapshot of the latest disk-to-disk Backup shows has just been backed up.  (b)  Do something to prevent Veeam seemingly re-making backups of files it has already backed up.  Suggestion (a) can be implemented by either using Copy Backup scripts whose source is disk Media Sets with grooming retention of only the latest backup—with a Groom script run only once a week, or by using Copy Media Set scripts that rely on the “Don’t add duplicates to the Media Set” option to screen out everything except what has been just backed up.  The first alternative would IMHO run at least a little faster, but would require a US$69 update to Retrospect Mac 17; the second alternative would allow more frequent grooming of the disk Media Sets, which might be desirable if Joriz is running out of space on them.  Suggestion (b) can be implemented either by finding out how to prevent VMWare-Veeam from churning, or by switching from Veeam to R. V. (and it's only my guess, which I'll check with Sales,  that R. V. would churn less)—which would undoubtedly be very scary to both Joriz and his boss.

P.P.S: I added to my Support Case a suggestion that the Retrospect term "Snapshot" be changed to "Manifest".  It's the same number of characters, and younger administrators are probably more familiar with a cargo or customs manifest than they are with a Kodak snapshot.  The point is that the term "manifest" hasn't been co-opted by the Computer Science community to mean something else  What do you think?

[Nigel Smith]

On 5/14/2020 at 11:28 AM, DavidHertzberg said:

To cope with these problems, I have made the following two suggestions: (a) Run a disk-to-tape script nightly, but make it as short as possible by having it add to tape only what the Snapshot of the latest disk-to-disk Backup shows has just been backed up

From my earlier back-of-an-envelope calculations, both D2D and D2T should fit in overnight. More importantly, because he isn't backing up during the day, the "to tape" part can happen during the day as well (my guess is that he was assuming nightlies would take as long as the weekend "initial" copy, rather than being incremental), so he should have bags of time.

On 5/14/2020 at 11:28 AM, DavidHertzberg said:

(b)  Do something to prevent Veeam seemingly re-making backups of files it has already backed up.

I know nothing about Veeam's file format, only that it's proprietary (rather than eg making a folder full of copies of files). It may be making, or updating, single files or disk images -- block level incrementals may be the answer. Or it may be that Veeam is actually set to do a full backup every time...

 

On 5/14/2020 at 11:28 AM, DavidHertzberg said:

P.P.S: I added to my Support Case a suggestion that the Retrospect term "Snapshot" be changed to "Manifest". 

It is a snapshot, in both computerese and "normal" English -- a record of state at a point in time. I don't think the fact that it is different to a file system snapshot, operating system snapshot, or ice hockey snap shot 😉 requires a different term -- the context makes it clear enough what's meant, IMO.

[DavidHertzberg]

On 5/15/2020 at 8:55 AM, Nigel Smith said:

From my earlier back-of-an-envelope calculations, both D2D and D2T should fit in overnight. More importantly, because he isn't backing up during the day, the "to tape" part can happen during the day as well (my guess is that he was assuming nightlies would take as long as the weekend "initial" copy, rather than being incremental), so he should have bags of time.

I know nothing about Veeam's file format, only that it's proprietary (rather than eg making a folder full of copies of files). It may be making, or updating, single files or disk images -- block level incrementals may be the answer. Or it may be that Veeam is actually set to do a full backup every time...

 

It is a snapshot, in both computerese and "normal" English -- a record of state at a point in time. I don't think the fact that it is different to a file system snapshot, operating system snapshot, or ice hockey snap shot 😉 requires a different term -- the context makes it clear enough what's meant, IMO.

Joriz,

Nigel Smith is absolutely correct in his first paragraph; I should have written "run a disk-to-tape script every day" instead of "run a disk-to-tape script nightly".  Don't forget to specify that the disk-to-tape script must run in the same activity thread as the disk-to-disk script, as I said in the third paragraph of this up-thread post.

Nigel Smith is even more correct in his second paragraph; 😁 it never occurred to me that Veeam is doing block level incremental backups, because I don't use that feature myself (I've only got one set of small databases that get updated maybe twice a month).  A fast Google search brought up this Veeam Community Forums page; I'm sure you can find the appropriate pages in the Veeam manual, of which this may be one.  Read pages 198-202 in the Retrospect Mac 16 User's Guide to find out how to use the corresponding block level backup feature in Retrospect.

Nigel Smith,

I think "snapshot" was a wonderful term when Dantz came up with it in 1990 or before.  Unfortunately first Unix and then Windows NTFS and now Apple APFS began using it to mean something else beginning by the early 2000s.  If the Retrospect User's Guides add the explanation for "active Snapshot" that I suggested in this post up-thread and in my Support Case, it's therefore going to have to include something like "this is Retrospect's definition of the term 'Snapshot', not everybody's filesystem's definition".   Maybe that clarification could quote the well-known Humpty Dumpty statement from Through the Looking-Glass.🤣

The word "snapshot" started out as a military or hunting term, and was appropriated by the Eastman Kodak company around 1890 to mean something different—in the context of someone carrying a camera instead of a rifle.  Considering that every backup  administrator will be familiar with the use of a filesystem "snapshot" for a backup—I predict even Retrospect will be doing it by next year, IMHO it would be asking a lot of an administrator to switch context just in that one place.  That's why I suggest that Retrospect "Inc." should avoid any confusion and convert to "manifest"; I noticed that "backups" confused you.