Another reason not to use Storage Groups

[Nigel Smith]

17 hours ago, DavidHertzberg said:

DB engines?  I'm glad it's a couple of hours since dinner; otherwise I'd get sick laughing.🤣

Don't know why -- since a database engine is what CRUDs the data in a database, Retrospect has a database, and records in that database can be created, read, etc, there's some kind of DB engine in there. I'm not saying it's MySQL or similar, but there's something.

I'd hope that, with the push to a "common engine", it's the same on all platforms. But that's hope, not certainty. And anyway, it's only an example of the possible differences between nominally common code compiled to different platforms.

I'm sure the meat of the RS engine is more-or-less the same, but it's the gristle round the edges that always gets stuck in your teeth...

[DavidHertzberg]

Nigel Smith,

Sorry to get you agitated, but I think you're missing the point that—per the quote here up-thread—a Media Set's "data base" is contained in the .rdb ("Retrospect data base", geddit?) files crammed into its Members together with the .session files.  The format of these is sufficiently unchanged that, per pages 15–16 of the Retrospect Mac 10 User's Guide, you can generate a Retrospect Mac 10 Media Set Catalog from the Members of a Retrospect Mac 6 Backup Set Catalog.  (You recommended this in a post in a 2019 thread.) I've searched the Forums to find an example of someone creating a Retrospect Mac Media Set Catalog from the Members of a Retrospect Windows Backup Set or vice-versa.  I can't find one and don't have a Windows machine to test, but I'll bet it can be done—because I'll bet the format of Member files in either variant is the same.  Probably twickland should ask Tech Support about it when he files his delayed Support Case.  (FWIW, here's a Knowledge Base article that's totally useless.)

Let me give you a personal example of the wide meaning of the term "data base", derived from the death of my ex-wife  in January 2021.  When I finally got access to her non-password-protected iMac several months later, I found a  Microsoft Word file she'd created in 2020.  The file consisted of 155 comma-separated e-mail addresses, most preceded by a person's name. (If you're curious why I didn't simply use her list from Gmail, I didn't know the password for the Macbook she used for a "don't contact me, I'll contact you" e-mail to 33 people after the effects of lymphoma and its treatment caused her to go into isolation.)  I copied each name and its e-mail address into a Contact in a newly-created Group in my Apple Contacts application.  That included, for about a dozen e-mail addresses, substituting a '?' for a first name or last name I couldn't guess.  I then exported that Group in Virtual Contact File (.vcf) format, and attached the .vcf to an e-mail to my ex-wife's executor—who has used it to generate successive e-mails to those 155 people.  The  point of this example is that Apple Contacts can qualify as a useful Data Base Management System, but comma-separated lists can't—for the same "data base".

 

[Nigel Smith]

On 7/29/2021 at 6:26 AM, DavidHertzberg said:

Sorry to get you agitated, but I think you're missing the point...

No, you are. Yes, the .rdb files are the database and yes, the catalog file is the index. But a component of the Retrospect Engine is a database engine because that's what manipulates the database. Is it the same on Mac and Windows versions of RS? We don't know, and even if the inputs and outputs are the same we still don't know -- it's a black box.

To (rather sadly) quote myself:

On 7/28/2021 at 8:19 PM, Nigel Smith said:

I'd hope that, with the push to a "common engine", it's the same on all platforms. But that's hope, not certainty. And anyway, it's only an example of the possible differences between nominally common code compiled to different platforms.

I'm sure the meat of the RS engine is more-or-less the same, but it's the gristle round the edges that always gets stuck in your teeth...

You can see evidence of that last line for yourself by comparing Mac and Win release notes -- most fixes are in both, but a few are specific to one or the other.

[DavidHertzberg]

(Disclaimer: Anything I may say about the intentions of Retrospect "Inc." in this or any other post is merely the result of "reading the tea leaves", the "tea leaves" being documentation and public announcements supplemented by an occasional morsel from Retrospect Sales.  I have never been paid a cent by Retrospect "Inc." or its predecessors, and I pay for my upgrades. Any judgements expressed are—obviously—mine alone. The same is true of Retrospect's history, especially with references to here.  Any apparent aspersions I cast are meant to apply to the widest applicable group of Retrospect "Inc." employees, not to an individual employee.)

I'm feeling in a slightly combative mood this morning,  Nigel Smith; please excuse me.  I don't believe a .rdb file qualifies as a database, because there are no "backup server" facilities for doing an Update or Delete of one per CRUD—only a Create or Read.  That's because Retrospect 1.0 was released for Mac in 1989 (bottom of the official History article, just above "Patents").  In 1989 large-capacity HDDs were still way too expensive for Mac owners to afford them, so all backup had to be to magnetic tape—a sequential-memory medium.  And, by Gadfrey, the first line under "Patents" in that same article is "5150473-Data storage format for addressable or sequential memory media – January 16, 1990, our backup set format".  My guess is that the .rdb format had been perfected from Dantz Development's point of view by the time "Retrospect 5 for Windows released. (First Windows release)" in 1999, so they didn't change it then and haven't since.  If you want to Delete a .rdb file—even one on HDD or in cloud storage—as opposed to Member removal from a Retrospect Catalog, you must use a "Finder" facility; there's no way you can Update one unless you're a Retrospect "Inc." engineer.

The practical consequence of this is that IMHO the source code for doing a .rdb Create or Read must be essentially the same—and be written by Dantz and its successors—for the Retrospect Mac and Windows variants of the "backup server".  No doubt there are some variant-dependent fillips required because of differences in the C++ compilers used for the variants, but these would handled by "if #Mac" and "if #Windows" macro expansions in the source code.  Because of the application's requirement for reading .rdb files created many years previously, the engineers wouldn't want the format to change.  Besides, I'm sure in their minds the .rdb format long ago achieved perfection.

[DavidHertzberg]

twickland,

Neither in your OP nor in your later post did you say what error message you were getting for "when I attempted to rebuild the catalog, I was unable to do so because Retrospect failed to accept the media set's password".  iodigitale has since reported in the OP of another thread that "when I try to rebuild a catalog and it asks for the password, I receive the following error 'RefBackupsetContainer...TCipher is Invalid'".

Is that the error message you were getting?  iodigitale didn't say whether he was doing a Rebuild of a Storage Group or a stand-alone Media Set, but—judging from the number of Media Sets in his screenshot—I'd guess it was a stand-alone Media Set.  He said he's using "the latest Retrospect 18.1.1 (120) on the Mac".

[Nigel Smith]

On 8/4/2021 at 12:00 PM, DavidHertzberg said:

I don't believe a .rdb file qualifies as a database

Trying not to quote Ralph Waldo Emerson, but you can't have it both ways -- either rhwalker is right and "All of the backup data (files, metadata, etc.) is in the .rdb files which, taken together, comprise Retrospect's database", or you are and they don't. I'll let you decide for yourself, and leave well alone until twickland reports back.

[DavidHertzberg]

Nigel Smith,

I should have written "I don't believe a .rdb file qualifies as a database requiring a database engine".  My guide is the lead section of this Wikipedia article, whose first two paragraphs distinguish between a "database" and a "database management system (DBMS)".  I totally agree with rhwalker—quoted here up-thread—that "All of the backup data (files, metadata, etc.) is in the .rdb files which, taken together, comprise Retrospect's database."  I'd merely add the .session files, which AFAIK add information on which .rdb file(s) contain a backup of a particular Source file/folder.  All I'm saying is that the "backup server"contains no code for doing an Update or Delete of an .rdb or .session file, so it doesn't need to use a packaged DBMS (see the first paragraph in this section of the same WP article)—which you call a "database engine" and which might differ in the Mac vs. Windows variants.

Here's an illustrative example.  Back in the mid-1960s the standard test of whether someone had graduated from Programmer Trainee to full-fledged Programmer was whether he/she could write pseudo-code for a "three-tape update".  The three "tapes" are sequential files, each one of which contains records sorted by "account number".  The first "tape" is an "old master file" with one record for each "account".  The second "tape" is a "transaction file" containing one or more transactions—including Creates and Updates and Deletes (pseudo-code for Reads was usually considered too application-specific to be included in the test) for some—but not all—of the "accounts".  The third "tape" is a "new master file" incorporating the applied transactions.  (in the late 1970s, when I worked for the company that later expanded into Verizon, there were three telephone-engineers-retrained-as-"programmers" in my Disbursements Accounting development "district" who couldn't pass the test when faced with a real-world assignment.  IIRC I had to give them some helpful hints.) 

The point of this example is that the "three-tape update" test requires creating your own application-specific—i.e. non-packaged—DBMS.  And, by Gadfrey, an .rdb file is actually a portion of a sequential "new master file" whose "account numbers" consist of Source file/folder names concatenated with their dates-times and the backup date-time.  According to this 2016 post  by madison347 followed by a reply by the head of Retrospect Tech Support, .session files were added to Members in Retrospect Mac 13 (and Retrospect Windows 11) to allow faster Catalog File rebuilds; however .session files are AFAICT really segments of a disk-saved index to the .rdb files in a Member. 

My overall conclusion is that, since the strictly-sequential creation of .rdb and .session files within a Member is merely a fancy fleshing-out into application-specificity of the Create logic in a classic "three-tape update", it probably doesn't require a DBMS written by an outside software provider—which you call a "database engine" and which might differ between Retrospect Mac and Retrospect Windows.  And the same would probably be true for the reading of .rdb files; even if speeded by the use of .session files, it would just be a fancy fleshing-out into application-specificity of the Read logic usually omitted from the "three tape update" test—so it also probably doesn't require a DBMS written by an outside software provider.

Edited August 16, 2021 by DavidHertzberg
The first sentence failed to express the point of the post; replaced it.