Another reason not to use Storage Groups

[twickland]

Numerous people, notably David Hertzberg, have questioned whether using Storage Groups is a good idea. We have been using a Disk Storage Group media set for over a year with no particular issues until last week, when we started getting strange errors for two clients: 

Can't access Backup Set [Username], error -2262 (Catalog File in use by another machine)

Neither quitting and relaunching Retrospect Engine nor rebooting the backup computer made any difference, and when I attempted to rebuild the catalog, I was unable to do so because Retrospect failed to accept the media set's password. (And that even though the media set was configured to remember the password for any access.)

This issue cropped up shortly after upgrading to v18.0.0, which may or may not be relevant.

[Mayoff]

We have released version 18.1, which contains a lot of bug fixes. Please give that version a try. If you still have trouble, please contact support directly.

Configuring Retrospect to remember the password for an encrypted backup set does not apply to a catalog rebuild.  If the password is not accepted during a catalog rebuild attempt, it typically means something is wrong with the password being entered. You might see the same error during a restore attempt.  The catalog rebuild process reads the header of the .rdb file to identify if it is encrypted and when the password is not accepted, it means the decryption attempt failed. 

[DavidHertzberg]

twickland,

First, the Retrospect Mac cumulative Release Notes do not list a fix for that bug in version 18.1.  I'd tell you to submit a Support Case—which I'm sure you know how to do, except that my experience with the -530 bug several years ago shows that Tech Support is normally not allowed to submit a bug to the engineers unless Tech Support can reproduce it on their own system.  Judging from your OP, it doesn't sound as if you can help them do that because the bug only affects two "clients".  FWIW the Release Notes say version 18.0 had "Storage Groups: Fixed rebuild issue with 3+ members (#9292)" and "Rebuild: Fixed issue where process can fail to finish (#9256)", but those may not be relevant unless the fixes in 18.0 were botched.

Second, you should be aware that there is no single Catalog for a Storage Group. A Storage Group is merely a "hack" (in the non-pejorative sense I picked up from MIT students in the 1960s) that provides a container for multiple Media Sets—one Media Set for each machine-volume Source combination ever backed up to that Storage Group.  This Knowledge Base article says in several places "For xxxxxx [operation name]  in Retrospect for Mac, a Storage Group can be treated like a media set. The user interface is the same." In some of those places the KB article says "For xxxxxx [operation name] in Retrospect for Windows, a Storage Group is presented as a folder of backup sets. You can select a backup set.  [Sometimes that second sentence gives you a choice, including selecting a Backup Set or a Snapshot for Transfer.]"  What that difference reflects is that the GUI for a Storage Group in Retrospect Windows allows you to access it deeper than the container level, but the GUI for a Storage Group in Retrospect Mac never has been enhanced to allow you that deeper access.

That's why I have out-and-out said that using Storage Groups for Retrospect Mac is not a good idea.  The KB article says "For rebuild in Retrospect for Mac and Retrospect for Windows, a Storage Group can be treated like a media set. The user interface is the same."  That means when you run a Rebuild for a Storage Group, the "backup server" Engine actually tries to run a Rebuild for each component Media Set—evidently by following on its own—for the single member allowed for the component Media Set on the Storage Set's drive—the procedure that includes step 5 on page 188 of the Retrospect Mac 16 User's Guide (which I'm referring to because the StorCentric Slasher deleted in the Retrospect Mac 17 UG what would have been on the equivalent of pages 186–188).  When the head of Retrospect Tech Support writes in the preceding post "something is wrong with the password being entered", AFAICT he must mean that a password required to be entered by you or stored for the Storage Set as a whole (I'm not sure which of these two alternatives the head of RTS means) doesn't match the password—or lack of a password—stored in the .rdb file header for that particular Media Set.

If Retrospect isn't requiring you to enter a password for each component Media Set in the Rebuild I have only some suggestions—truly "Hail Mary" ones requiring doing Finder-deletes inside the Storage Group's folder in Library->Application Support->Retrospect->Catalogs:  [1] Is it possible you changed or added a password for the Storage Group after these two Media Sets were created?  If so, maybe you could delete those Media Sets and recreate them via a Recycle Backup script that only uses the Sources for those two Media Sets.  [2] Is it possible you were originally simultaneously running another script that was backing up the Sources for those only two Media Sets to the Storage Group?  If so, again maybe you could delete those Media Sets and recreate them via a Recycle Backup script that only uses the Sources for those two Media Sets.  [3] Is it possible the Storage Group was originally created for those only two Media Sets using 17.0.1 or earlier?  The Release Notes say for 17.0.2 "Storage Groups: Fixed issue where rebuilding a storage group can fail to delete previous catalog (#8672)".  If so, again maybe you could delete those Media Sets and recreate them via a Recycle Backup script that only uses the Sources for those two Media Sets.

However you might first try for the two component Media Sets the first suggestion—Remove and re-Locate—in this 2012 post by you yourself.

Please post whether Retrospect Mac 18.1 fixed your problem.

[Mayoff]

>except that my experience with the -530 bug several years ago shows that Tech Support is normally not allowed to submit a bug to the engineers unless Tech Support can reproduce it on their own system. 

This is 100% false. If support believes the issue is a bug, it will be submitted as a bug. In no way does tech support have to reproduce a bug themselves. 

>that's why I have out-and-out said that using Storage Groups for Retrospect Mac is not a good idea

That is a choice you can make for yourself, but we have 1000's of customers successfully using Storage Groups on local disk backup sets and cloud backup sets. 

[DavidHertzberg]

On 6/18/2021 at 9:11 AM, Mayoff said:

>except that my experience with the -530 bug several years ago shows that Tech Support is normally not allowed to submit a bug to the engineers unless Tech Support can reproduce it on their own system. 

This is 100% false. If support believes the issue is a bug, it will be submitted as a bug. In no way does tech support have to reproduce a bug themselves. 

>that's why I have out-and-out said that using Storage Groups for Retrospect Mac is not a good idea

That is a choice you can make for yourself, but we have 1000's of customers successfully using Storage Groups on local disk backup sets and cloud backup sets. 

"100% false" would surely be news to your subordinate named Jeff, who made the later Additional Notes responding to mine in Support Case #53523—my long-running attempt to get my -530 errors fixed. 🧐 Here are the results of a browser search on "troubleshoot" within  that Support Case, searching earliest to latest :

The reply in the Additional Note for February 22, 2017 18:11 PM essentially consists of

"Please send me all of your retroclient.log files from the client machine, found in /var/log. There should be 10 of them. 

Lets focus solely on what you are calling Bug 1 for now, as it is much more tangible, and we have a reproducible error that we can troubleshoot [my emphasis]. 

Please install the .317 client and reproduce the issue. Since most of the client internals come from the engine anyway (we send a downloadable service to the client when we establish a good connection), I suspect that you will still notice the -530s continue. If they don't, then future client logs should give us an indication of why, and may even clue us into what happened to cause what you call bug 2. 

In order to get engineering eyes on the problem [my emphasis], I need to be able to tell them these issues exist with everything updated to the latest build. We can always revert the client if we need additional testing later, and the client logs you send may give us an idea of what went bad. 

Both of your "clues" have been mentioned in this ticket previously, and that information has already been added to the escalation." 

The reply in the Additional Note for March 08, 2017 23:51 PM, signed by Jeff,  contains the two paragraphs

"So I agree with you that "-530 bug 1" exists, and once engineering has the time available, I hope to get a testbuild from them [my emphasis] that addresses it."

and

"We see inconsistencies like this often when troubleshooting, its one of the reasons why predicting the results of a test isn’t the same as performing the test itself. [my emphasis]"

The reply in the Additional Note for May 15, 2018 20:08 PM , signed by Jeff,  contains the paragraph

"Once you have reproduced the various issues [my emphasis], please gather your /library/application support/Retrospect/operations_log.utx file from the server machine, and the 10 or so retroclient.log files found in /var/log on the client machine."

Your "we have 1000's of customers successfully using Storage Groups" ignores a key question: what variant of Retrospect are these customers using?  In case you didn't notice, the second paragraph of mine you quote has the words "for Retrospect Mac" in italics.  What I said earlier in the post you quote is "What that difference reflects is that the GUI for a Storage Group in Retrospect Windows allows you to access it deeper than the container level, but the GUI for a Storage Group in Retrospect Mac never has been enhanced to allow you that deeper access."  I filed Support Cases #76584 and #78624 for enhancing that Retrospect Mac GUI, but they've never been acted on—probably because StorCentric has refocused the Retrospect engineers' tasks since 25 June 2019.  In the Retrospect Mac Forums you'll find threads started by several administrators complaining that their use of Storage Groups has been anything but successful.  Those, supplemented by my reading the applicable Knowledge Base article, are the basis for my giving  the advice you dislike.

P.S.: Your "on local disk backup sets and cloud backup sets" glides over another key question: on what type of media are the majority of Storage Groups being successfully created?  My guess is that—at least for Retrospect Mac—it's on the cloud.  Even there—per the Knowledge Base article—Retrospect Mac has no Storage Group GUI facility for limiting a Restore or any variety of Copy (referred to as Transfer in the KB article) to a particular component Media Set except using a Rule (using a Rule worked when I tried it for a test Storage Group a couple of years ago).  Moreover, Retrospect Mac has no Storage Group GUI facility for adding a second Member on another local disk to a component Media Set—which would be necessary to avoid space overflow on the component Media Set.  Would any Retrospect Mac administrator successfully using a Storage Group care to post a comment?😃

Edited June 19, 2021 by DavidHertzberg
P.S.: For Retrospect Mac administrators—other than twickland prior to 18.0, are they _successfully_ using a Storage Group on any type of media except the cloud?

[DavidHertzberg]

twickland,

It belatedly occurred to me that you might be using the Desktop Edition, which as of Release 18 cannot use Storage Groups—which also applies to the Solo Edition.  And that's not "inaccurate and unsubstantiated information", because it comes from the latest revision of "Retrospect Backup Compare Editions"—which is a new Retrospect.com website document directly linked-to from the new Knowledge Base article "Retrospect Backup 18 Licensing Changes".  But I did a quick check of your past posts, and you're probably still using a Single Server Edition.

However, having been an applications programmer who once or twice introduced bugs into (my) previously-working code (I can admit it now that I'm retired), it occurred to me that maybe the component Catalogs which had -2262 errors on Rebuild are for Sources residing on NAS shares. That same  "Retrospect Backup Compare Editions" excludes—originally for Solo and Desktop Editions (exclusion removed—from the website document as well as the code—for Desktop Edition in 18.1 after my plea to Sales)—NAS Source Protection.  Heaven forfend that I should "post derogatory information" about any Retrospect engineer—even an unspecified one, but maybe the 18.0 coding for that exclusion was sloppy.

[twickland]

Since the storage group disk media set was failing only with new backups, I decided simply to use it only for restores going forward and to create a new regular disk media set for new backups.

As for the media set password,  I use the same password for all our media sets. That would indicate either that I mistyped the password when creating the media set or that something became corrupted after the creation.

[DavidHertzberg]

twickland,

My aging brain has now come up with a new hypothesis, based on my outsider's understanding of the way Storage Groups work.  If you create a Storage Group, Retrospect will have to create within it a new component Media Set each time you back up a machine-volume combination that hasn't already been backed up to that Storage Group—I've tested this.  If the Storage Group was created as password-protected or encrypted (page 41 of the Retrospect Mac 18 User's Guide), then Retrospect will have to create the new component Media Set as appropriately password-protected or encrypted.  This implies there should be a file at the Storage Group level that saves the password and level of encryption, so Retrospect can later apply those to a newly-created component Media Set.  If the 2 "clients" that had the problem were the last ones added to a Backup or Proactive script that uses the Storage Group, it sounds to me as if the file (if any) at the Storage Group level became corrupted before the 2 clients were added to the script.  If so, the component Media Sets for those 2 "client"'s drives would have been created with a different—or no—password versus the one originally specified for the Storage Group.

I strongly recommend you create a Support Case, but also speak to Retrospect European Tech Support.  If you upgraded to Retrospect 18 within the last 30 days, you're definitely entitled to personalized support for this issue; however—assuming my hypothesis is correct—you've exposed a vulnerability in Storage Group processing that should get the attention of Tech Support in any case.  Tech Support may be able to tell you how to read the proper file at the Storage Group level, and then how to Finder-move or Copy-Media-Set-copy (using a Rule) the component Media Sets for those 2 "clients" to a new Storage Group created with the password and level of encryption the file at the Storage Group level now contains.   Obviously—if you can do this—you'd then have to remove those 2 "clients" from their existing Backup or Proactive scripts, and add them to equivalent scripts specially created for them.

Edited July 11, 2021 by DavidHertzberg
In 2nd paragraph, change "Finder-copy" to "Copy-Media-Set-copy"; the Rule is needed to restrict that Storage Group operation to the desired Media Sets. In first paragraph, maybe—due to programmer oversight—the hypothesized Group-level file doesn't exist.

[macpro]

The main reason for me to stop using Storage Groups is that it takes too long to start the Retrospect console.

When syncing with the server, it counts up to 149 media sets and when it's done, it actually shows 35 media sets.
That's probably because Storage Groups contain "sub media sets".

[DavidHertzberg]

On 7/12/2021 at 4:04 AM, macpro said:

The main reason for me to stop using Storage Groups is that it takes too long to start the Retrospect console.

When syncing with the server, it counts up to 149 media sets and when it's done, it actually shows 35 media sets.
That's probably because Storage Groups contain "sub media sets".

Why are you using Storage Groups in the first place, macpro?  The justifications for doing so are summarized in the third paragraph of this 2019 post, supplemented by the first non-disclaimer paragraph in this 2020 post.  If your use-case doesn't fit those justifications, IMHO you shouldn't be using Storage Groups.😎

But let's assume your use case does fit those justifications.  I don't think you really understand what the Media Sets list in the Retrospect Mac Console shows when you use Storage Groups.  Because of deficiencies (IMHO) in that GUI as opposed to the Retrospect Windows GUI, the Mac Console doesn't show the component Media Sets within a  Storage Group.  Those component Media Sets nevertheless exist as far as the "backup server" is concerned , though.  You almost certainly have 149 Media Sets, including those components.  The Mac Console GUI just shows a single line for each Storage Group, no matter how many component Media Sets it contains, which is why your Console shows only 35 "media set" lines.  Look at the top screenshot on page 42 of the Retrospect Mac 18 User's Guide, and contrast that with the top screenshot on page 48 of the Retrospect Windows 18 User's Guide—where the more-complete GUI shows a line for each of the component Backup Sets (the Retrospect Windows term for Media Sets) within a Storage Group.

149 Media Sets includes one for each combination of "client" and drive—or Favorite Folder—you've ever backed up to your Storage Groups.  Having that many component Media Sets allows you to do a Proactive backup simultaneously of up to 14 (assuming your "backup server" machine has speedy hardware and at least 20MB RAM) combinations of "client" and drive. One use case for that would be your having many "client" machines that cannot be connected to the "backup server" at different times-of-day, such as a "virtual"  office using a VPN or—using Remote Backup—doing Work From Home.

BTW, you don't say what version of Retrospect Mac you are running, or what Edition.  Retrospect 18 bans Storage Groups for Desktop/Solo Editions.

[Nigel Smith]

On 7/13/2021 at 2:22 AM, DavidHertzberg said:

Because of deficiencies (IMHO) in that GUI as opposed to the Retrospect Windows GUI, the Mac Console doesn't show the component Media Sets within a  Storage Group.

Which just goes to show how tricky UI design can be because, IMHO, a Storage Group should be treated like any other media set -- those don't show internal client/volume pairs, so why should a Storage Group? Mac consistency, FTW!

On 7/13/2021 at 2:22 AM, DavidHertzberg said:

One use case for that would be your having many "client" machines that cannot be connected to the "backup server" at different times-of-day, such as a "virtual"  office using a VPN or—using Remote Backup—doing Work From Home.

I think the generic use-case for parallelising backup clients is that "the server processes data faster than a single client supplies it". You can parallelise with multiple scripts, each with its own destination set, or with one or more scripts writing to one (or, less often, more) Storage Groups.

My personal experience suggests that for "local" clients (those your RS Server can direct-IP or multi/subnet broadcast to wherever they are in the world), multiple scripts/sets is best. The benefits to me are organisational, de-dup within sets, faster catalog rebuilds when things go wrong (because you only have a subset of clients per media set), etc.

But if you have "remote" or hybrid-working clients then Storage Groups are the way to go, mainly because (as at my last set of tests, anyway) there is no way to back up different remote clients to different media sets -- it's "one big bucket", hence the slower catalog rebuilds, although there's no de-dup (even across volumes of the same client). The only way to parallelise write operations to a single set is by using a Storage Group. And having multiple remote clients, which often have restricted upload speeds, are when parallelising on the server can really help you get all your backups done.

As always, YMMV.

[DavidHertzberg]

On 7/14/2021 at 8:27 AM, Nigel Smith said:

Which just goes to show how tricky UI design can be because, IMHO, a Storage Group should be treated like any other media set -- those don't show internal client/volume pairs, so why should a Storage Group? Mac consistency, FTW!

....

(Disclaimer: Anything I may say about the intentions of Retrospect "Inc." in this or any other post is merely the result of "reading the tea leaves", the "tea leaves" being documentation and public announcements supplemented by an occasional morsel from Retrospect Sales.  I have never been paid a cent by Retrospect "Inc." or its predecessors, and I pay for my upgrades. Any judgements expressed are—obviously—mine alone. The same is true of Retrospect's history, especially with references to here.  Any apparent aspersions I cast are meant to apply to the widest applicable group of Retrospect "Inc." employees, not to an individual employee.)

IMHO your opinion, Nigel Smith, is wrong in this case—a very rare situation.😮  That's because a Storage Group is really a hack or kludge (latter term spelled as if  derived from the Scottish word for toilet, although it probably actually refers to a U.S. company that invented an automatic feeder for printing presses) for grouping together multiple Media Sets so that they can be referred to as if they were one Media Set.  As I explained most recently in the first paragraph of this up-thread post, each time you back up a machine-volume combination that hasn't already been backed up to a particular Storage Group you automatically create within it a new component Media Set.  As the second paragraph of this later up-thread post says, redesigners of the Retrospect Windows GUI chose to show those components—redesigners of the Retrospect Mac GUI chose not to show them. I suspect that wasn't for "Mac consistency", but instead was caused by coding difficulty—traceable to the Retrospect Mac Console designer's having retired subsequent to 2008.

The problem described by twickland in this thread evidently has to do with new Media Set components automatically added to a Storage Group that was originally defined as password-protected (although it would also be applicable to a Storage Group that was originally defined as encrypted).  I originally hypothesized in the first paragraph of this up-thread post that the Retrospect "backup server" maintains a file at the Storage Group level containing the password/encryption status for all components, and that that file had become corrupted by the time twickland backed up a new machine-volume combination.  However it later occurred to me that the engineers might have failed to code for the case of a Storage Group defined as password-protected/encrypted; if that "conceptual bug" occurred—meaning there is no such file at the Storage Group level, component Media Sets later automatically created  for the Storage Group would not be defined as password-protected/encrypted.  Rebuild for a Storage Group in either variant can only be done for the Storage Group as a whole, explaining why  twickland reported "the storage group disk media set was failing only with new backups".  I may know by tomorrow which version of my hypothesis is correct, per Support Case #80364 (the strangest I've ever filed; I refused to do any testing).

On 7/14/2021 at 8:27 AM, Nigel Smith said:

....

I think the generic use-case for parallelising backup clients is that "the server processes data faster than a single client supplies it". You can parallelise with multiple scripts, each with its own destination set, or with one or more scripts writing to one (or, less often, more) Storage Groups.

My personal experience suggests that for "local" clients (those your RS Server can direct-IP or multi/subnet broadcast to wherever they are in the world), multiple scripts/sets is best. The benefits to me are organisational, de-dup within sets, faster catalog rebuilds when things go wrong (because you only have a subset of clients per media set), etc.

But if you have "remote" or hybrid-working clients then Storage Groups are the way to go, mainly because (as at my last set of tests, anyway) there is no way to back up different remote clients to different media sets -- it's "one big bucket", hence the slower catalog rebuilds, although there's no de-dup (even across volumes of the same client). The only way to parallelise write operations to a single set is by using a Storage Group. And having multiple remote clients, which often have restricted upload speeds, are when parallelising on the server can really help you get all your backups done.

As always, YMMV.

Two other client-server backup applications have tried to parallelize (sorry, U.S. spelling) the single-destination-set use-case "the server processes data faster than a single client supplies it", but they can only do it for a tape destination.  As you've said, Nigel Smith, Retrospect has simplified parallelization via multiple scripts by hiding scripts' multiple Media Set destinations—which must be on a single HDD or cloud "bucket"—within a single Storage Group.

You're also correct; there's no way to back up different "clients" tagged "Remote Backup Clients" to different explicitly stand-alone Media Sets.  As I urged Alanna to do in this 2020 post, I filed Support Case #73054 suggesting that "Remote Backup Clients" be treated as a prefix to a Tag for which a suffix would be allowed.  The "Remote Backup Clients" prefix would still tell the "backup server" to not reach out for such "clients", but to wait until an individual "client" reached out to a running Proactive script using public/private key-pairs.  However, in contrast to the present limitation, a running Proactive script wouldn't back up—using Storage Group parallelization—a "client" reaching out unless the full Tag—including the suffix—matched a script-defined  Source.

[DavidHertzberg]

(Disclaimer: Anything I may say about the intentions of Retrospect "Inc." in this or any other post is merely the result of "reading the tea leaves", the "tea leaves" being documentation and public announcements supplemented by an occasional morsel from Retrospect Sales.  I have never been paid a cent by Retrospect "Inc." or its predecessors, and I pay for my upgrades. Any judgements expressed are—obviously—mine alone. The same is true of Retrospect's history, especially with references to here.  Any apparent aspersions I cast are meant to apply to the widest applicable group of Retrospect "Inc." employees, not to an individual employee.)

Results of my testing (Retrospect Mac 16.6 under macOS 10.12 Sierra on "backup server"): Once you Rebuild a Storage Group that has been defined as password-protected, "Has password" disappears from Options in the Overview pane in the panel produced by selecting the Storage Group in the Console's Media Sets list and clicking the Summary tab.  This is for a Storage Group for which a single local HDD had been the source in the first run of a Backup script (resulting in a single component Media Set), and then for which a second local HDD had been added as a source in the second run of the same Backup script (resulting in two component Media Sets).  After the second script had been run—but before the Rebuild of the Storage Group, the Options had still shown "Password-protected" in front of "No automatic grooming".

My new conclusion: At least as far back as Retrospect Mac 16.6, there is a bug—resulting in the removal of password protection from a Storage Group after a Rebuild—in what I said should happen in the first paragraph of this up-thread post.  However Retrospect Mac 18.0 introduced a another bug—resulting in twickland's  -2262 errors—whose uneven occurrence in Retrospect Mac 18.0 may be be causing the refusal to accept a password twickland also reported in his OP.  You can't use password protection (or probably encryption) for Storage Groups; I'll add this to Support Case #80364.

Interesting revelations of my testing:  [1] When I created a "StorageGroupTwickland" as password-protected, Retrospect Mac 16 immediately created a "StorageGroupTwickland" folder within the "Retrospect" folder on the destination local volume—with an empty "1-StorageGroupTwickland" folder within it.  When I then made  the Backup script runs, Retrospect also created  "Macintosh HD" and "Macintosh HD New" folders within the "Retrospect" folder on the destination local volume—with "1-..." and "2-..." folders within those folders.   [2] Because I had mistakenly defined the mandated Member for "StorageGroupTwickland" as slightly too small, Retrospect asked for allocation of an additional Member when the first Backup exceeded the defined space for the Member "1-Macintosh HD".  I defined it on the same destination volume, but there's no reason why I couldn't have defined that additional Member on another volume—permitting component Members on multiple volumes (which I previously thought couldn't be done for a Storage Group).  Each defined Member size for a particular destination volume appears to define the size of an allocation for each component Media Set on that volume.

[Nigel Smith]

On 7/15/2021 at 4:18 AM, DavidHertzberg said:

IMHO your opinion, Nigel Smith, is wrong in this case—a very rare situation.😮

Trust me, it isn't!

On 7/15/2021 at 4:18 AM, DavidHertzberg said:

That's because a Storage Group is really a hack or kludge

Agreed. But whether Mac or PC handling of them is correct (c'mon -- Mac wins every time!) is a matter of perspective. Looking top-down, a media set only has one catalog. A Storage Group only has one catalog. It looks like any other media set in the RS UI, and the manual even states it should be treated the same as any other media set. So it *shouldn't* show the component directories, since no other media sets do.

When it comes to UX and abstraction, consistency trumps absolute accuracy every time. And why display something that shows no usable purpose?

I don't know, and I don't really care, why Mac and Win do this differently. I know which I prefer, so do you, but those are only opinions and neither of us is "right".

On 7/17/2021 at 10:57 PM, DavidHertzberg said:

I defined it on the same destination volume, but there's no reason why I couldn't have defined that additional Member on another volume—permitting component Members on multiple volumes (which I previously thought couldn't be done for a Storage Group).

While I would hope that you are right about this, I'd recommend that anyone who wants to use this in production thoroughly tests it first. There are limits on destination volume size -- my "use as much space as you want" Disk Media set splits every 8TB, for example -- and I'd want to be sure that the above allowed you to separate across "proper" logical/physical volumes rather create more RS "media set" volumes in the same place.

Nice work on the password protection checking!

[DavidHertzberg]

On 7/20/2021 at 5:10 AM, Nigel Smith said:

....

Agreed [ that a Storage Group is really a hack or kludge]. But whether Mac or PC handling of them is correct (c'mon -- Mac wins every time!) is a matter of perspective. Looking top-down, a media set only has one catalog. A Storage Group only has one catalog. It looks like any other media set in the RS UI, and the manual even states it should be treated the same as any other media set. So it *shouldn't* show the component directories, since no other media sets do.

When it comes to UX and abstraction, consistency trumps absolute accuracy every time. And why display something that shows no usable purpose?

I don't know, and I don't really care, why Mac and Win do this differently. I know which I prefer, so do you, but those are only opinions and neither of us is "right".

....

Sorry, Nigel Smith, but a Storage Group has one Catalog for itself plus one Catalog for each component Media Set.  On my "backup server", within Macintosh HD SSD->Library->Application Support->Retrospect->Catalogs, there's a "StorageGroupTwickland" folder (my naming imagination is limited on Friday nights) with a StorageGroupTwickland.rbc Catalog File directly following it.  Within the "StorageGroupTwickland" folder are Catalog Files "Macintosh HD.rbc" and "Macintosh HD New.rbc", created when I successively ran scripts backing up first one—then both—other local drives to StorageGroupTwickland. As a final proof, last night just for kicks I ran another Rebuild of StorageGroupTwickland; the Console Activities panel showed Rebuild activities simultaneously running (different Activity Threads) for StorageGroupTwickland.rbc and the two component Media Sets.

As for "no usable purpose", if twickland had been running Retrospect Windows 18 he might have been able to individually Rebuild his component Backup Sets (Retrospect Windows term for Media Sets)—and thus recover his existing Storage Group backup.  Of course he would then have had to individually Transfer Backup Sets (Retrospect Windows term for Copy Media Set) for those component Backup Sets to the corresponding component Backup Sets of a new Storage Group that still had the proper consistent password protection, but the Retrospect Windows UI would have enabled him to do that too.

IMHO the inadequate explanation of underlying functionality—which I've had to supplement in this post as well as up-thread— in the "How to Use Storage Groups" Knowledge Base article, which has finally simply been copied into the Retrospect 18 User's Guides, is a sterling example of the erroneous policy—begun IIRC in  2016—of having the engineers explain new features in KB articles because it was too much trouble/expense to revise the User's Guides.  I'm not casting aspersions on individual engineers—they're not trained as technical writers—but on Retrospect Inc. management as a whole.
 

Quote

 

While I would hope that you are right about this [permitting component Members on multiple volumes], I'd recommend that anyone who wants to use this in production thoroughly tests it first. There are limits on destination volume size -- my "use as much space as you want" Disk Media set splits every 8TB, for example -- and I'd want to be sure that the above allowed you to separate across "proper" logical/physical volumes rather create more RS "media set" volumes in the same place.

Nice work on the password protection checking!

 

I'll see if I can test this over the next few days.  As for the "nice work", it was mostly recognizing the unexpected when it appeared under my nose.😎

[Nigel Smith]

16 hours ago, DavidHertzberg said:

Sorry, Nigel Smith, but a Storage Group has one Catalog for itself plus one Catalog for each component Media Set.

I know what RS does at the filesystem level. But in the UI, which is where I suspect most of us interact with RS most of the time, it is presented as "just another media set" on the Mac and as something different in Windows. Again, I prefer the Mac software's consistent approach -- you obviously don't. And I can't disagree with your preference since Storage Groups aren't "just another media set" (eg lacking cross-client dedup within the set).

I'd prefer it even more if there was a consistent approach, one way or the other, across platforms -- but that's probably asking too much.

16 hours ago, DavidHertzberg said:

As for "no usable purpose", if twickland had been running Retrospect Windows 18 he might have been able to individually Rebuild his component Backup Sets (Retrospect Windows term for Media Sets)—and thus recover his existing Storage Group backup

It may have changed in v18, but in v17 you had to rebuild the entire Storage Group on both platforms, even if only a single "component catalog" was damaged. A pain, but at least the rebuild is also parallelised across multiple threads. See this old post for my workarounds, either of which could possibly be adapted to twickland's particular problem.

[DavidHertzberg]

13 hours ago, Nigel Smith said:

I know what RS does at the filesystem level. But in the UI, which is where I suspect most of us interact with RS most of the time, it is presented as "just another media set" on the Mac and as something different in Windows. Again, I prefer the Mac software's consistent approach -- you obviously don't. And I can't disagree with your preference since Storage Groups aren't "just another media set" (eg lacking cross-client dedup within the set).

I'd prefer it even more if there was a consistent approach, one way or the other, across platforms -- but that's probably asking too much.

It may have changed in v18, but in v17 you had to rebuild the entire Storage Group on both platforms, even if only a single "component catalog" was damaged. A pain, but at least the rebuild is also parallelised across multiple threads. See this old post for my workarounds, either of which could possibly be adapted to twickland's particular problem.

I much prefer the Retrospect Windows approach, as shown in the "How to Use Storage Groups" KB article.  That article shows that, in Retrospect Windows, you can restrict Restore and Transfer (Retrospect Windows term for non-Finder-level Copy) and Verify to a particular Backup Set (Retrospect Windows term for Media Set) without using a Rule (termed a Selector in Retrospect Windows).  If that restriction were also extended to Rebuild—which I suspect would be easy as far as the "backup server" is concerned, the OP's original error messages in both this thread and the"Rebuilding only a portion of Storage Group Set?" thread you cite up-thread would have been easily remedied via Rebuilds of the component Media Sets.

My guess—as an outsider—is that the "hide the existence of a Storage Group's component Media Sets" approach of the Retrospect Mac Console has as much to do with the relative lack of engineer competence in Console modifications as it does with a difference in philosophical approach.  That's not casting an aspersion on any current engineer, only acknowledging the reported departure of the original Console designer some years ago.
 

[Nigel Smith]

7 hours ago, DavidHertzberg said:

That article shows that, in Retrospect Windows, you can restrict Restore and Transfer (Retrospect Windows term for non-Finder-level Copy) and Verify to a particular Backup Set (Retrospect Windows term for Media Set) without using a Rule

Agree about the Transfer and Verify (although Rules are hardly irksome), but I don't understand about Restore -- sessions are listed by Machine + Volume + Media Set, so it's trivial to do a point-in-time (or Browse and select particular files) for any client/volume pairing in any Storage Group. In Windows it's actually more clicking to get to previous snapshots/sessions.

I've not tried it, but in Windows do you have to restore from a particular source? Can you easily search for every occurrence of "Important.docx" across a Storage Group, or would you have to search (and restore from) each "component set" individually?

As you rightly say, it's potential in improving Rebuilds is massive, if it can be done.

I'm not sure how similar Mac and Win RS actually are! There may be more to it than just how their respective consoles allow you to view what's under the hood. Creating Storage Groups by splitting a single over-arching catalog into a "meta catalog" plus a bunch of client/volume "component catalogs" was a very pragmatic solution to a pressing problem -- whether that could be done the same way on both platforms is something way outside my understanding.

What we really need is a preference so we can switch both Mac and Win consoles to our preferred method of display 😉

[DavidHertzberg]

On 7/22/2021 at 12:23 PM, Nigel Smith said:

Agree about the Transfer and Verify (although Rules are hardly irksome), but I don't understand about Restore -- sessions are listed by Machine + Volume + Media Set, so it's trivial to do a point-in-time (or Browse and select particular files) for any client/volume pairing in any Storage Group. In Windows it's actually more clicking to get to previous snapshots/sessions.

I've not tried it, but in Windows do you have to restore from a particular source? Can you easily search for every occurrence of "Important.docx" across a Storage Group, or would you have to search (and restore from) each "component set" individually?

As you rightly say, it's potential in improving Rebuilds is massive, if it can be done.

I'm not sure how similar Mac and Win RS actually are! There may be more to it than just how their respective consoles allow you to view what's under the hood. Creating Storage Groups by splitting a single over-arching catalog into a "meta catalog" plus a bunch of client/volume "component catalogs" was a very pragmatic solution to a pressing problem -- whether that could be done the same way on both platforms is something way outside my understanding.

What we really need is a preference so we can switch both Mac and Win consoles to our preferred method of display 😉

You're correct about Restore of a client-volume pairing, Nigel Smith; I didn't look closely at the lower part of the Mac screenshot in this section of the KB document.🙁  But that section doesn't make it clear whether doing a Restore from such a pairing will give you the Restore Assistant options as described on page 106 of the Retrospect Mac 18 User's Guide.  However I later tried clicking the Restore button, and it gave the options—though I didn't actually do a Restore.

I don't have a Windows machine, but I was fairly sure you can't "easily search for every occurrence of 'Important.docx' across a Storage Group".  IMHO you're over-estimating the current capabilities of the "meta catalog" for a Storage Group; I looked at one via TextEdit on my Mac "backup server", and it wasn't comprehensible but was quite small.  However I later tried clicking the Restore button and using the option to do a search for filenames containing "Retrospect", and it found files in each of the component Media Sets—though I didn't actually do a Restore.

The "backup server" source code was made common across the Windows and Mac variants in 2009.  I'm pretty sure the only special Retrospect Windows provisions are its still-built-in GUI and its different terminology—which sometimes sneaks into Retrospect Mac messages as "backup set".

I'd love "a preference so we can switch both Mac and Win consoles to our preferred method of display".  But IMHO as an outsider that doesn't have any possibility of happening—given evident redirection since June 2019 by StorCentric, except as a feature of the projected new LAN Console connecting a Mac or Windows machine to a Drobo NAS variant of the "backup server" via a Web server.   That new LAN Console isn't expected any earlier than Retrospect 19.

Edited July 23, 2021 by DavidHertzberg
Edit: In first paragraph, clicking the Restore button gives the options in the Mac User's Guide. In second paragraph, search for filenames found them in each component Media Set.

[DavidHertzberg]

On 7/20/2021 at 5:10 AM, Nigel Smith said:

(Never mind, this is an un-deletable posting error by DavidHertzberg)

[DavidHertzberg]

twickland and Nigel Smith,

Last night I repeated the tests reported in this up-thread post, because I was pretty sure I'd remembered the Options phrase incorrectly.  After creating "StorageGroupTwickland" as password-protected, with a Catalog on "Macintosh HD SSD" and a large-enough Member on "Macintosh HD Sierra" (an extra partition on the HDD that I originally inherited with my 2010 Mac Pro), I first ran a No Media Action script to backup "Macintosh HD" and then reran the same Backup script adding "Macintosh HD New".  At the end of the second run, the Summary Options for "StorageGroupTwickland" still included "Has password".

I then did a Rebuild of "StorageGroupTwickland", and discovered that Rebuild elimination of "Has password" really results from kludgey Storage Group implementation.  The Rebuild GUI asked me to Add a Member, and the only appropriate Member is "1-StorageGroupTwickland" in the Retrospect folder and "StorageGroupTwickland" sub-folder on "Macintosh HD Sierra".  Note that "1-StorageGroupTwickland" is an empty folder; unlike the "1-Macintosh HD"  and "1-Macintosh HD New" sub-sub-sub-folders contained within their respective sub-sub-folders in the "StorageGroupTwickland" sub-folder,  it contains no .rdb files.  Thus, despite what the head of Retrospect Support says in this up-thread post, the Catalog Rebuild process cannot read "the header of the .rdb file to identify if it is encrypted".  Maybe Retrospect 18 attempted to fix this, with results per the OP.   Are .rdb headers of password-protected Media Sets also so marked, or are they marked only in the .rbc Catalog? Was Rebuild of a password-protected Storage Group alpha-tested?

[Nigel Smith]

On 7/22/2021 at 6:52 PM, DavidHertzberg said:

IMHO you're over-estimating the current capabilities of the "meta catalog" for a Storage Group;

I think the "meta catalog" just stores the "higher level" info -- client/volume pairs in the SG, when each was last backed up -- and the "component catalogs" store the session info data. In Mac RS the search "chains" -- in the GUI you search the meta cat, but RS searches each of the component cats then aggregates the results for presentation. I'd hope that Win RS does the same and that the manual's "To restore from a specific source, you need to select the specific source within the Storage Group" is an "and you can restore from a specific source..." rather than "you can only...", but I can't test that at the moment.

On 7/22/2021 at 6:52 PM, DavidHertzberg said:

The "backup server" source code was made common across the Windows and Mac variants in 2009.

Assuming that's the "universal Retrospect engine", that still leaves an awful lot of wiggle room! Same design and pseudo-code? Same actual code but compiled per platform? Even if it's the latter ( as I suspect, comparing version fixes across platforms) that still leave space for things like "if macOS then mac_db_code else win_db_code".

Not saying they have, but we can't discount the possibility!

[DavidHertzberg]

At 9:13 a.m. EST—2 hours before Retrospect "Inc." normal CA opening time on July 26th, I received the following e-mail—which is a copy of an Additional Note in Support Case #80364:

Quote

Agent Response: 

Your emails [presumably Problem Statement and Additional Notes in Support Case #80364] have been made available to engineering for evaluation.  To be clear, Password Only is not actually an encryption method. It is a simple way to keep unauthorized users from having quick access to the backup set, but it contains zero encryption. All of the backup data is stored the same way as any non-encrypted backup set.  If you really need encryption, select an actual encryption type. 

Please let us know if you have any additional questions,

Robin Mayoff
Director, WW Support
Retrospect, Inc

At last, after nearly two weeks, some action!  I guess Tech Support believes this issue is a bug.  😁

Nigel Smith,

I took a look tonight at the contents of StorageGroupTwickland.rbc and "Macintosh HD.rbc" using TextEdit.  I can't figure out any of the contents (maybe the engineers are obfuscating it for security), but the contents of the first .rbc file is much less than the contents of the second .rbc file.

The log for my daily Backup script run this morning said "Finished scanning backup set data files to Backup Set Media Set White".  I think that's adequate evidence for "Same actual code but compiled per platform", with "if macOS then mac_db_code else win_db_code" not being applied to all log messages.  As per up-thread, it's obviously applied to the Retrospect Windows "backup-server"-only GUI (ancient, but more modifiable than Retrospect Mac Console).

Edited July 28, 2021 by DavidHertzberg
In final paragraph, add extra sentence about ancient-but-modifiable Retrospect Windows "backup-server"-only GUI only being compiled for _that_ variant.

[Nigel Smith]

11 hours ago, DavidHertzberg said:

I think that's adequate evidence for "Same actual code but compiled per platform", with "if macOS then mac_db_code else win_db_code" not being applied to all log messages.

Apologies, I was less than clear (rushed, after my first attempt got eaten).

I can see a lot of space in "same code but compiled for different platforms" for quite a lot of variation between the final products. The example was supposed to suggest they might have used different DB engines -- maybe for performance, licensing, or compatibility reasons. There could be lots of these variations, or very few -- I simply don't know.

Your catalogs are as I'd expect, the meta holding "summary data" for the set and each client/volume (including links to their component catalogs) and the component containing the meat of that client/volume's backup history (metadata for every file backed up in every session, etc).

[DavidHertzberg]

Nigel Smith,

DB engines?  I'm glad it's a couple of hours since dinner; otherwise I'd get sick laughing.🤣  In this 2010 post by the very-knowledgeable user rhwalker, he said

Quote

All of the backup data (files, metadata, etc.) is in the .rdb files which, taken together, comprise Retrospect's database. If the rdb files are missing, there's no hope.

 

The "catalog" file (.rbc) is simply an index into the database, telling where the different files are so that Retrospect can retrieve them. The catalog can be recreated or rebuilt. However, if the data is missing, well, you are SOL.

He may still be talking about the Retrospect 6 Catalog File; that format was changed in Retrospect Mac 8, to the extent that Retrospect Mac 6 Catalogs were initially non-convertible.  However I'm still running Retrospect Mac 6.1 on an old Mac to back it up; I just looked at one of that Mac's Catalog Files with TextEdit, and it's incomprehensible.

twickland,

I see you're looking at the Forums again.😄  Can you please post a Support Case now, supplementing my Support Case #80364, to help engineering figure out the -2262 and Rebuild errors in your OP in this thread?  If you do that, maybe we can get an 18.x bug-fix release that'll let you use a Storage Group again.

Can you please furthermore clarify why you are using Password Only protection for your Media Sets?  Despite its inadequate documentation in the User's Guides, I already had figured out what the head of Tech Support said in his e-mail reproduced here up-thread.  Let's pretend I moved to the UK and went to work for you in IT.  Assuming you had to keep your "backup server" running 24/7, you might simply want to password-protect your Media Sets to prevent me stealthily doing a Restore to my own machine of confidential files you had backed up from management's computers.  OTOH, if your installation processes information you want to keep secret from the Chinese, you might want to encrypt your Media Sets in case I were an undercover spy for the Ministry of State Security or the People's Liberation Army.  However Retrospect encryption would slow down your backups because it's software-only, so you might be willing to be satisfied with with Password Only because of my (let's further pretend) having been vetted by MI5.  Is this analysis correct?