Jump to content

Rebuilding only a portion of Storage Group set?


Recommended Posts

So, we now have these storage groups that function like a collection of mini-sets and mini-catalogs, with each client or source getting a mini-set/catalog. When I have to repair or rebuild a set on any of my consulting clients' Retrospect installations (WAY more often than I should if I had a truly reliable backup program IMHO), I see the individual "mini-sets" getting scanned and rebuilt. Is there a way to trigger a repair or rebuild for an individual source in a storage group?

Right now I've got a client that isn't backing up due to a "chunk checksum (-641)" error. I'm so tired of rebuilding multi-terabyte backup sets that I really don't want to have to put everything on hold yet again while this 5TB disk is scanned and the catalog rebuilt. Since each source appears to have its own, self-contained catalog and folder of band files on the backup disk, couldn't Retrospect just rebuild the individual source whose catalog has failed?

Thx,

Fred

Link to comment
Share on other sites

fredturner,

Not, as far as I can determine, directly in Retrospect Mac 16.6 (which I'm still using) or 17.0.   Here's why and how to create a Support Case for a feature request.  Make sure you mention that you're a "Partner", meaning a consultant; evidently Retrospect Product Management really loves "Partners".

You could indirectly fix a particular problem via a four-step process, somewhat per the fourth substantive paragraph in this post.  First,  do a Copy Media Set of the the Storage Group to a new Media Set—using a Rule that Includes only the particular Source Host and Volume whose backup had the -641 error.  Second, do a Groom of the Storage Group—using a Rule that Excludes the particular Source Host and Volume whose backup had the -641 error.  Third,  do a Copy Media Set of the new Media Set back to the Storage Group—using a Rule that Excludes the date of the backup when the -641 error occurred.  Fourth, do a No Media Action backup of the Source Host and Volume whose backup had the -641 error to the Storage Set—which IMHO should back up the files that were excluded in the third step.  Note that I have not tested this process.  Also note that the four-step-process may take longer than doing a Rebuild of the Storage Group; it certainly involves more manual effort.

Link to comment
Share on other sites

fredturner,

This 2016 post says the way that OP got rid of a -641 error was to do a Recycle backup of a particular "client", even though other tests of the "client" SSD indicated no errors..  The thread is in the "Retrospect 9 or higher for Macintosh" Forum, but "Allowed catalog rebuild to continue after skipping invalid backup data that causes error -641" is a bug fix in the cumulative Release Notes for both Retrospect Windows 15.0 and Retrospect Mac 15.0—so whatever generates a -641 error must be in the underlying code for both variants.

I don't know whether doing a Recycle backup would be acceptable for your situation.  Sorry I didn't do a Forums search for "-641" before.

Link to comment
Share on other sites

On 4/2/2020 at 1:36 AM, fredturner said:

When I have to repair or rebuild a set on any of my consulting clients' Retrospect installations (WAY more often than I should if I had a truly reliable backup program IMHO), I see the individual "mini-sets" getting scanned and rebuilt. Is there a way to trigger a repair or rebuild for an individual source in a storage group?

I won't comment on storage groups, since I don't use them (nor, tbh, do I see the point of them on most situations...).

If you aren't already, make sure the catalogs are getting backed up daily. That way, when trouble strikes, you can restore the last known good catalog and rebuild from that -- a lot faster than rebuilding the catalog from scratch.

You can then use the time saved to find out why you are having to rebuild catalogs so often, because that doesn't match my experience (previously, yes, because hardware was less reliable -- but not now).

Link to comment
Share on other sites

On 4/15/2020 at 4:48 AM, Nigel Smith said:

I won't comment on storage groups, since I don't use them (nor, tbh, do I see the point of them on most situations...).

....

....

Nigel Smith and others,

I felt the same way as you do, and wrote a couple of posts in the Ars Technica thread on Retrospect saying "the basic motivation for the development of Storage Groups was to provide the administrator with the benefits of having a different destination for each source machine-drive combination—without imposing onerous manual record-keeping requirements."  But you and I may have both missed—or not believed—a statement in the first paragraph of the Knowledge Base article that has been copied into the version 17 User's Guides.  It says

Quote

With Storage Groups, you can run parallel backups to the same disk destination with a single ProactiveAI script. Scheduled scripts support Storage Groups as destinations, but the backups run on a single execution and not in parallel.

I made a couple of double-check phonecalls to Retrospect "Inc." on the afternoon of 17 April, and a junior Tech Support person set up a Support Case in which he replied "You can backup multiple clients/sources to the same storage group set at the same time with both ProactiveAI and standard scripts."   Previously the head of North American Sales made the same statement for Proactive scripts only, but didn't say he'd personally tested a Proactive run with multiple "clients" and a Storage Group destination.  So I asked, in an Additional Note to my Support Case, for a definite statement that a Proactive script makes itself multi-threaded if its destination is a Storage Set.  In that same Additional Note, I also said

Quote

I have suggested in other Support Cases that Retrospect Inc. hire a tech writer, instead of allowing engineers to write the User's Guides and Knowledge Base articles. This IMHO is an business school textbook example of how their not having done so is hurting sales. Somebody should convince Mihir Shah to shell out the money to do so.

Now that Tech Support has confirmed (see P.S.) that a Proactive script can make itself multi-threaded if its destination is a Storage Group, it justifies the kludge Alanna complained about in the OP of her thread about using Remote Backup.  It also at least partially eliminates the need for the suggestion I made in the first sentence of the second paragraph of this post in that thread, and invalidates my un-qualified last sentence in the fourth paragraph of that post.  But, per the remainder of the that same second paragraph, the kludge may not solve all problems when Remote Backup is applied to Working from Home—unless there is sufficient RAM for the necessary parallel threads (1MB per thread, 14 threads maximum) or there is more than one "backup server".

The foregoing doesn't mean that the engineers shouldn't fix the deficiencies in the Retrospect Mac GUI for Storage Groups, described in the OP of this thread and in the third and fourth paragraphs of this post in Alanna's thread.

P.S.: I got proof from T. S.  late on 19 April of multi-threading on a Proactive script, via a screenshot of a Retrospect Windows Activity Report.

Edited by DavidHertzberg
P.S.: I got proof from T. S. late on 19 April of multi-threading on a Proactive script. In 3rd paragraph, 14 threads maximum—not 16.
Link to comment
Share on other sites

On 4/18/2020 at 4:04 AM, DavidHertzberg said:

But you and I may have both missed—or not believed—a statement in the first paragraph of the Knowledge Base article that has been copied into the version 17 User's Guides.

Didn't miss it -- but I may have missed the point of it!

As I understand it, I can do similar by running multiple Proactive/Scheduled scripts, each targeting their own backup set, all stored on the same destination resource. You end up with more catalogs and sets to manage, but the benefit is that each is smaller. You lose file-level dedup in both scenarios.

I can see why Storage groups might be useful, but I'm already running multiple sets and so parallelise operations across the whole estate rather than within a single set.

As always, YMMV as you'll have different requirements and resources (where "you" is everyone reading this -- not specifically you, David!).

Link to comment
Share on other sites

Nigel Smith,

You've obviously not been "blessed"—as Alanna evidently has—with a directive from management, or a demand from other employees, that nearly everyone in the centralized office Work From Home because of the COVID-19 pandemic.  An easy way to handle that, which avoids quickly setting up a VPN and then making all those employees bring their home routers—with router-brand-specific GUIs you've no experience with—into the office so you can (if the VPN permits it) open ports 497 and 22024 for TCP and UDP (requiring thorough hand-washing by you and the other employee before and after), is to use the Remote Backup feature introduced with Retrospect 15.6 in the fall of 2018.  The 2018 customer use case for Remote Backup was for—e.g.—a company with its main offices in Britain that has a few salespeople such as Sally in Shanghai and Albert in Adelaide, each of whom has files on his/her laptop that the company's backup administrator wants to be sure are backed up on a "backup server" running 24/7 in Britain. The "backup server" can't "reach out" either if Sally and Albert's machines don't have predictable Internet addresses, either because Sally and Albert don't work from fixed local offices, or because—in the case of Sally—that office cannot be contacted from England via Retrospect multicast because of a Great Firewall.

Therefore Remote Backup piggy-backs on Retrospect's public-key cryptography facility. The Retrospect Client installer for each particular Remote machine designates the Internet address of the "backup server" and includes a customized public key. The "backup server" maintains a table of Remote-machine-specific public keys and their calculated corresponding private keys. When a Remote "client" machine contacts the "backup server" using the address stored on its Client with a backup message specifying its own calculation of the private key—and encrypted with the customized public key, the "backup server" uses the calculated customized private key from the table to verify that the message is from the authorized Remote "client" and the corresponding public key to decrypt it. (Please excuse sloppiness in the preceding sentence; I know nothing about cryptography, and I've no inside info.)

Unlike Using Multicast and Using Subnet and Add Source Directly, that approach does not reveal to the "backup server" the Name of any Remote Backup machine—which would be meaningless for an administrator since it isn't resolvable to a fixed reachable-by-the-"backup-server" IP address. Thus such a Remote "client" cannot be specified by Name as a Source for any backup script. To solve this problem, in 2018 the Retrospect Inc. engineers devised a kludge. Retrospect has a Tag facility for grouping multiple machines; thus only the Tag for those machines need be specified as a Source in a script. The Remote Backup facility interprets the exact Tag "Remote Backup Clients" as a synonym for all "client" machines that can "reach out" to the "backup server" with a public key that is in the "backup server"'s Remote Clients table. Remote Client backup only applies to Proactive scripts, so unscheduled "reaching out" can occur any time such a script is running—in a sequence depending on when the Remote "client" happens to connect to the Internet.

That's the limitation that can make the "Remote Backup Clients" Tag a true kludge for Work From Home. As Alanna discovered in March, any Proactive script that specifies the exact Tag "Remote Backup Clients" as a Source will backup all "clients" appearing in the Remote Clients table that "reach out" to the "backup server" while it is running that Proactive script. Since an individual Retrospect script is not multi-threaded (because it is executed within a particular Activity Thread) if its destination isn't a Storage Group, all those "Remote Backup Clients" would be backed up one-by-one. This wasn't a problem in 2018; if salespeople Sally in Shanghai and Albert in Adelaide are in different time-zones from each other, they're presumed not to have their laptops connected to the Internet at the same time, and—if the company has only a few such remotely-located salespeople—it's OK to back them up via a single Proactive script running 24/7. However that kludge creates problems when a massive number of employees in the same time-zone start working from home because of COVID-19. Let's assume that each such "client" takes 0.5 hours to back up, and you want to back up each such "client" daily. That assumption puts a limit of 48 "clients" that can be backed up with a single Remote Backup script, even if all such "clients" are connected to the Internet 24 hours per day. (That this assumption is not unreasonable is borne out by the fact that my 2016 MacBook Pro takes about 0.5 hours for an incremental Retrospect non-Proactive backup, even though its speedy connection to my Mac Pro "backup server" is over an in-the-apartment MoCA cable connection.) But the kludge is eliminated if the destination is a Storage Group; if so a single Proactive script is multi-threaded—the maximum 14 threads can back up 672 0.5-hour "clients" daily on a fast-processor "backup server" with 16MB available RAM.

So if you're forced to use Remote Backup , you can't "do similar by running multiple Proactive/Scheduled scripts, each targeting their own backup set, all stored on the same destination resource."

P.S.: Insert second paragraph showing how Remote Backup uses Retrospect public-key cryptography to allow a "client" to "reach out" to a "backup server"

P.P.S.: Revise second paragraph, because I think I understand public-key cryptography a little better now—but the last un-parenthesized sentence is still sloppy

 

Edited by DavidHertzberg
In 4th paragraph, maximum 14—not 16—threads can back up 672 0.5-hour clients daily.
Link to comment
Share on other sites

On 4/24/2020 at 4:17 PM, DavidHertzberg said:

You've obviously not been "blessed"—as Alanna evidently has—with a directive from management, or a demand from other employees, that nearly everyone in the centralized office Work From Home because of the COVID-19 pandemic.

Wrong -- we're now into Week 6 of working from home.

On 4/24/2020 at 4:17 PM, DavidHertzberg said:

An easy way to handle that, which avoids quickly setting up a VPN

...was to issue everyone an external hard drive to use with Time Machine or Windows backup.

Rest of my attempted reply just got eaten. Suffice to say:

  1. Have tried Remote Backups before -- fail for us because you can't segregate clients into different sets
  2. Keep meaning to try Remote combined with Rules -- can you run multiple Proactives using Remote tag, each using a different set, each with different Rules to filter by client name? Previously felt client list was too long for this to work, but RS 17's faster Proactive scanning may make it feasible
  3. Tags need an "AND" combiner and not just an "OR". That may not be sensible/possible -- include ability to use Tags in Rules and you'd make Rules way more powerful
  • Thanks 1
Link to comment
Share on other sites

Nigel Smith,

Based on my experience with a long-time close acquaintance (we've never been in each other's apartments), I'm rather skeptical about your Work From Home approach of issuing "everyone an external hard drive to use with Time Machine or Windows backup"—unless your fellow employees are one and all fairly familiar with computer hardware.  My acquaintance is quite knowledgeable in her specialized field—which isn't related to computers or engineering, and is a part-time teacher of university courses that require her to grade student exams and papers etc..  I happened to ask her in January 2018 how she was backing up her Mac, and she said she had bought some kind of wireless external HDD to use with Time Machine.  I no longer recall the details, but the HDD was USB-powered and had a battery that was good for 10 hours.  It turned out she wasn't routinely plugging in the USB cable to her Mac—her only USB electrical power source, apparently because she was afraid of tripping over a USB cable in her small apartment.  So Time Machine was not in fact backing up her Mac; thank goodness she hadn't had a disk crash.  How do you know some of your fellow employees aren't as ignorant as she was?

OTOH I think your idea "to try Remote combined with Rules" is brilliant for Work From Home.😀    Backup of my MacBook Pro with a No Files Rule—which is essentially what the idea institutes for Rule-excluded "clients"—takes about 3 minutes. If "multiple Proactives using Remote tag, each using a different [presumably Media] set, each with different Rules to filter by client name" actually works with Retrospect 17's faster Proactive scanning,  it would get around the problem described in the second and third paragraphs of this up-thread post.  But it would only get around the problem for Work From Home, where at-home machines can be presumed to have previously been on the same Retrospect-backed-up LAN—and therefore are guaranteed to have known and non-conflicting client names (otherwise they couldn't have been used as Sources in pre-Work-From-Home backup scripts).  It wouldn't get around the problem for the general use case for which Remote Backup was designed in 2018, in which the company might have distinct salespeople David in Davos and David in New York (who wanted to work in Denver, but management said he couldn't)—each of whose machines might be named "David's MacBook Pro".  I suppose in that case the Rule could also specify Source Host Login Name (page 178 of the Retrospect Mac 17 User's Guide), which might be needed if the company had two home office employees named David in different Subnets—e.g. because they worked in different home-office buildings.

I'd like to add your idea as an Additional Note to my Support Case (note to self, #73054), crediting you and linking to your post directly above.  In the first Additional Note to that Support Case (split off because of the approximately-2000-character limit on a Support Case Problem Statement), I proposed—as per the sixth paragraph of this post in another thread—allowing "Remote Backup Clients" to be treated as a prefix in a manually-defined tag such as "Remote Backup Clients A" or "Remote Backup Clients B" that could be used as a Source that backs up only those "client" machines defined with that particular full Tag.  If a Retrospect Tech Support test shows that your idea works, the "backup server" Engine modifications that my proposal requires would be unnecessary—at least in the short term.  Both your idea and my proposal would, by making it unnecessary to use a Storage Group as a destination in order to get effective multi-threading among multiple simultaneously-running Proactive scripts, also temporarily make unnecessary the enhancements to the half-baked Retrospect Mac GUI for Storage Groups (described by the phrase "For ... in Retrospect for Mac, a Storage Group can be treated like a media set" in the Restore and Transfer and Rebuild and Verify sections of this Knowledge Base article—see fredturner's OP in this thread) .

Link to comment
Share on other sites

6 hours ago, DavidHertzberg said:

How do you know some of your fellow employees aren't as ignorant as she was?

I don't -- in fact I guarantee that right now, despite the majority of my users being post-grads and higher and so certainly not stupid, there will be a handful who haven't bothered to plug in their external drives for a couple of weeks.

And I don't care 🙂.

Our PhDs/PostDocs want (and have) a high degree of IT autonomy, and with that comes responsibility for their data. We can't make them plug in their HDs, any more than we can make them store their scientific data on the central servers -- but they know it's their future they are risking so they are generally pretty good about these things. They were all using external HDs and Time Machine/Windows Backup anyway -- the disks issued were so they run duplicates, backup up any home machines they might be using for work, etc.

The same is true for the Admin side, plus Biz-Op data is held on our servers (accessed over VPN etc) and is still being backed up as usual.

We use RS in addition to external HDs. I'd love to have RS working remotely the way I need it to, but until it does I can't rely on RS even in more "normal" times, so external HDs are required. Being useful when home-working was an unexpected benefit of such a "portable" backup solution.

RS's Remote Backup is, frankly, half-baked. As you say, it needs either Multiple "Remote" tags; making the "Remote Backup Clients" tag a special case which "AND"s with other tags (rather than "OR"s); or making tags usable in Rules. Then we can back up different clients to different sets, still use file-level de-dup, etc -- basically, treat a Remote Client the same as a local one.

Feel free to use any of my ramblings in any way you see fit. If I get time I'll be spinning up an evaluation of RS17 (put on hold since the lockdown) -- I'll try and include Remote tests in that since my previous experience is probably out of date.

Link to comment
Share on other sites

  • 2 months later...
On 4/27/2020 at 1:09 PM, Nigel Smith said:
  • Keep meaning to try Remote combined with Rules -- can you run multiple Proactives using Remote tag, each using a different set, each with different Rules to filter by client name? Previously felt client list was too long for this to work, but RS 17's faster Proactive scanning may make it feasible

Finally got around to having a play with this. While RS17 still treats tags as "OR" when choosing which clients to back up in script and you can't use tags within a rule, you can use "Source Host" as part of a rule to determine whether or not a client's data is backed up by a particular Remote-enabled script. It involves more management, since you'd have to build and update the "Source Host" rules for each script, but there's a bigger problem:

Omitting by Rule is not the same as not backing up the client.

That's worth shouting -- the client is still scanned, every directory and file on the client's volume(s) or Favourite Folder(s) will be matched, a snapshot will be stored, and the event will be recorded as a successful backup. It's just that no data will be copied from client to server. (TBH that's the behaviour I should have expected from my own answers in other threads about how path/directory matching is applied in Rules.)

So if you have 6 Proactive scripts, each backing up 1 of 6 groups of clients daily to 1 of 6 backup sets, every client will be "backed up" 6 times with just 1 resulting in data being copied. That's a lot of overhead, and may not be worth it for the resulting reduced (individual) catalog size.

Also note: a failed media set or script will not be as obvious since it won't result in clients going into the "No backup in 7 days" report, since the "no data" backups from the other scripts are considered to be successful.

For me, at least, Remote Backups is functionality that promises much but delivers little. Which is a shame -- if Remote Backup was a script or client option rather than a tag/group attribute, or if tag/group attributes could be evaluated with AND as well as OR logic, I think it would work really well.

  • Thanks 1
Link to comment
Share on other sites

Excellent investigation, Nigel Smith.  However, IMHO as a retired applications programmer,  there's about zero chance of the engineers making any substantial change to the way Rules/Selectors are implemented—either moving them forward into the scanning phase or adding AND logic.

In regard to the scanning phase, this post ending another thread (read its preceding posts if you enjoy inter-administrator arguments 🤣 ) shows the engineers sped up the scanning phase by nearly 40% in Retrospect Mac 16.0—which enabled them to abandon any attempt to make Instant Scan work with APFS-formatted volumes.  (We peasants now know, which we didn't in 2018–Spring 2019, that they had to exert substantial effort in any case because Apple was about to eliminate 32-bit APIs in macOS.)    IMHO the engineers are unlikely to slow down the scanning phase for all backups in order to fix an edge case for "no data" backups.  I never pay attention to the Dashboard, which has had substantial bugs ever since I first encountered it in Retrospect Mac 12;  that's because I know which of the 7 drives has been backed up in my diminutive home installation.

In regard to adding AND logic to Rules/Selectors, the engineers haven't yet implemented the GUI enhancement to Retrospect Mac that would eliminate the problem fredturner complained about in the OP of this thread.  After Mihir Shah of StorCentric finally lets them do that, IMHO they'll solve the [your re-phrase]  "all Remote Backup clients are included in all Proactive/Remote scripts" problem by treating a manually-defined Tag that has the prefix "Remote Backup Clients" as indicating it is eligible for Remote Backup with a script that specifies a Tag with that prefix—as I proposed in the sixth paragraph of this post in another thread.  That solution will be a lot less work for administrators to use, and it'll be a lot less programming than what you've proposed.

Edited by DavidHertzberg
In second sentence of third paragraph, re-state problem using Nigel Smith's phrase—because my _phrasing_ was inexact as he points out directly below
Link to comment
Share on other sites

5 hours ago, DavidHertzberg said:

However, IMHO as a retired applications programmer,  there's about zero chance of the engineers making any substantial change to the way Rules/Selectors are implemented—either moving them forward into the scanning phase or adding AND logic.

Agreed -- and, IMHO, that's the wrong place to do it anyway.

Proactive scripts, the only ones we're concerned with, generate then periodically refresh a list of clients to watch out for. That's where the logic should be implemented -- list generation.

6 hours ago, DavidHertzberg said:

IMHO they'll solve the "only one Proactive script for all Remote Backup clients" problem

Wrong problem. You can have multiple Proactive scripts for Remote Backup clients -- the issue is that all Remote Backup clients are included in all Proactive/Remote scripts.

I do like your solution of multiple, user-definable, Remote Backup tags. Obvious to the Admin, in keeping with what is already there and presumably easiest to implement (always easy to say that when on the outside!).

Another option would be to keep the single tag but to add AND logic to client list generation from tags. They could even go further and replicate Rules within source-selection-by-tag -- think how powerful that would be, with the advantage of using the same structure as Rules. A lot more work for the programmers though, and probably of little use to the majority of users.

A third would be to divorce Remote Backups from tags altogether. It can instead be considered as both a script attribute (this script accepts Remote clients as sources) and a client attribute (this client advertises its IP to the server) and so be moved to being "Options" for both. That makes sense in that Remote Backups are a functional attribute whereas tags are organisational but, again, would be a lot more work for the programmers and could also throw up client compatibility issues.

If I was in charge of road-mapping I'd put the first solution as a priority, the third as something to work towards, and dismiss the second as nice-but-unnecessary. Meanwhile I'll carry on looking for a way to work with what we've got, whether that means single Remote Backup set with a stupidly large catalog or taking the hit of the Rules-based solution in an attempt to keep catalogs a manageable size.

Link to comment
Share on other sites

Nigel Smith,

It's the same problem; I just phrased it very badly—so badly that I've now replaced my phrasing with yours in my post directly above your post.  Thanks. 😀

However I think your second paragraph directly above indicates you need to re-read the last sentence in the first paragraph and the second and third paragraphs in this up-thread post.  I'm not sure you've grasped the idea that a script backing up Remote Clients can't have any pre-existing list of those "clients", because it can't know in advance—directly or by inference from unique "client" names—what their Internet addresses are going to be.  Whenever it comes online, each such "client" must send a message to the "backup server" saying "back up my current IP address with any Remote Backup script you've got running that specifies the tag 'Remote Backup Clients', because I know your public key for doing that". 

My multiple, user-definable, Remote Backup tags solution just expands the message to say "back up my current IP address with any Remote Backup script you've got running that specifies a tag prefixed with 'Remote Backup Clients' and suffixed with a character string I'm now sending you , because I know your public key for doing that".  The suffix string would be stored on the "client" in the same public_key folder containing the pubkey.dat file by the Retrospect Client installer, per pages 228–230 in the Retrospect Mac 17 User's Guide.

An illustrative but tricky part of my solution would be changing the suffix string stored on a "client", as shown in the following example (which assumes single-character suffix strings):  Your London-based company suddenly discovers the unexpected possibility of getting a big product order from a government agency in Beijing, and decides to rush the salesperson Albert in Adelaide there—along with his existing laptop.  (For plausibility, let's further assume Albert is a Chinese-Australian with Mandarin-speaking parents, who sent him to "Sunday school" to learn to read and write Standard Chinese.)  You must arrange for Albert to change the Remote Backup suffix on his laptop from 'A' (for Australia) to 'C' (for China), so that it will be backed up by the Proactive script intended to back up any of your company's Remote Backup "clients" located in China.  There would have to be a Client-installer-supplementing Retrospect utility program for making that change, unless you trust Albert to find the proper file and use a text editor to change 'A' to 'C'.

Link to comment
Share on other sites

38 minutes ago, DavidHertzberg said:

I'm not sure you've grasped the idea that a script backing up Remote Clients can't have any pre-existing list of those "clients", because it can't know in advance—directly or by inference from unique "client" names—what their Internet addresses are going to be.

I believe you're wrong. I can't be absolutely sure of how it works, I'm not an RS engineer, but here's a couple of screenshots immediately following a full restart of the server:

 image.thumb.png.d92397e338fb2af78bf5968f0662722e.png

image.thumb.png.ab26fe8c3504f4b9e45b925fbff949ed.png

Apologies for the fuzziness -- remote desktopping in a remote desktop session can play havoc with screenshots 🙂

On the same network as the server, "Luggage" is a Direct IP client and "RS_16_Test" is subnet broadcast. At home, ie Remote clients, "admin2's MacBook" finished a backup to "SG_Test" one minute before the server restart and is still awake/available, "Admin's MacBook Air" has been asleep since its last backup more than four hours before the restart.

As you can see, they are all in the Proactive list, even though two can't be contacted. RS might, of course, be including anything from the source list with a "known" IP address, as shown in the second screenie, but that also shows that the "known" IP address persists even across restarts -- so a client will always have a "last known" IP address even if that isn't its current one.

(Note that a Remote Backup isn't just "here's my IP, come back to me when you're ready". I deliberately turned off all port forwarding on my router for this test -- the backup session must be initiated client-side, hence the private IP address.)

So each Proactive script does appear to have a list of clients, generated from the script's Source list.

This isn't points-scoring on my part -- it helps refine our ideas on how to move forward because:

3 hours ago, DavidHertzberg said:

My multiple, user-definable, Remote Backup tags solution just expands the message to say "back me up with any Remote Backup script you've got running that specifies a tag prefixed with 'Remote Backup Clients' and suffixed with a character string I'm now sending you , because I know your public key for doing that".

...is more complicated than needed. If RS interpreted any tag name starting "Remote Backup Clients" as it does now, but also allowed you to create eg "Remote Backup Clients - Group 1" and "Remote Backup Clients - Group 2" so you could add different "remote groups" to different scripts, everything would work just as with other tags -- no messing around with the client and no problem with your China example, just update the tag selection on the server and you're done. If that's a bit clunky you could have a special class of tags, "Remote Backup tags" that could have any name and yet still be understood as Remote tags.

Thanks David -- this is really helpful in trying to work out a way forward for our setup. Even if nothing comes of it, it's had me poking around and trying different ways of using what's already available. Including -- gulp -- Storage Groups!

Link to comment
Share on other sites

I believe I'm right, Nigel Smith.  The reason is at the bottom right of your second screenshot, of  "Admin's MacBook Air",  where it says  "Tags: Automatically Added Clients".  As to what that means, here's a quote from page 72 of the Retrospect Mac 17 User's Guide, under the sub-section "Using Public/Private Key Authentication with Retrospect Clients":

Quote

3. If you want Retrospect to automatically log in clients with the proper public key, check “Automatically add clients”. This is recommended. The Retrospect server will then periodically check the network for new clients with the matching public key and automatically add them to Retrospect’s Sources list. Clients so added will be tagged with the “Automatically Added Clients” tag [my bolding], providing both a place to look in Retrospect for automatically added clients and also a way to create a script that will use the tag to automatically back up such clients. (For more information on tags, see the section on Tags in Chapter 3.)

That identical paragraph—under the identical sub-section—is on page 63 of the Retrospect Mac 14 User's Guide, which predates the introduction of Remote Backup.  So "Admin's MacBook Air"—and no doubt "admin2's MacBook"—is on what your "backup server" considers to be its network, which probably includes VPN-connected Retrospect-defined Subnets.  You haven't said whether the "backup server" is at your work or home location.  (However I'm sure Malcolm McLeary,  our friendly Forums security guru from Australia, would thoroughly approve of your use of public/private key authentication for these "clients". )  But why didn't "Automatically add clients" just work for Remote Backup; why did the engineers need to do additional programming?

Yes indeed, the Remote Backup feature introduced in Retrospect 15 is an extension of public/private key authentication.  But the difference is that IMHO (I'm not a networking expert) a Retrospect Engine can't "check the network" in the same way for a Remote Client when that "network" is the entire Internet (and I'm sure the Public Security Bureau of the PRC would object to Retrospect's doing that for a "client" machine—such as the one belonging to Sally in Shanghai—that is behind the Great Firewall).  That's why Retrospect Inc. added Remote Backup, in which—as it says on page 228 of the Retrospect Mac 17 UG—"When a remote client contacts the Retrospect engine for a backup [my bolding],  the running ProactiveAI script [my bolding] will accept the connection and begin a backup, assuming the destination is available."  This is reinforced on the same UG page, in the statement "Scheduled scripts and Immediate executions (Windows-only) are not supported because of their serialized scheduling process. ProactiveAI scripts can schedule backups for any client that is available, including remote clients."  In fact the cumulative Release Notes for Mac 17.0.1 have "Remote Backup: Fixed issue with not logging out clients after ProactiveAI backup (#7967)".  So "Automatically add clients" is Engine-initiated, but Remote Backup is Proactive-script-initiated.

The second sentence of your next-to-last paragraph directly above is simply a restatement of my prefix solution with multi-character suffixes, such as "Group 1",  allowed (unless you're proposing numeric-only suffixes with " - Group " automatically inserted, which I think would be less administrator-friendly than even single-character suffixes).  OTOH the separate class of tags proposed in the last sentence of that paragraph would save administrators the trouble of typing "Remote Backup Clients" as the start of each such tag.

Edited by DavidHertzberg
Add question to end of my second paragraph (the one below the quote), provide additional UG evidence in my third paragraph for the answer in my third paragraph
Link to comment
Share on other sites

3 hours ago, DavidHertzberg said:

I believe I'm right, Nigel Smith.  The reason is at the bottom right of your second screenshot, of  "Admin's MacBook Air",  where it says  "Tags: Automatically Added Clients".

Sorry, but that's irrelevant. "Automatically add clients"  is for onboarding (note "check network for new clients" [my emphasis] in your quote) so Joe Bloggs can do his own client install and the server will automatically register the new client -- rather than the usual install client, go to server, manually discover client on network, add to server. The tag is preserved so you can use it scripts later on.

It's neat, but not so useful to Admin's like me who always define Favourites on clients -- since the client must be online to do that, you may as well add them manually. I can see it would be good for anyone who always backs up whole clients (you can also apply Rules, obviously), since it removes the requirement for both client and Admin to be available when onboarding.

Anyways...

Take another look at my first screenshot. That's the list of clients the SG_Test Proactive script is waiting to back up, straight after a server restart. The MacBook Air is asleep, and has been for the previous 3 hours -- it isn't online, it can't have sent any message to the server since the restart, yet the server knows it is to be backed up to SG_Test. So the list must be generated regardless of any client's availability.

Note: To be even clearer -- the server, "Luggage", and "RS_16_Test" are at work, and all on the same network. "admin2's MacBook" and "Admin's MacBook Air" are at home, behind a NATing router with no port forwarding, not using a VPN -- they cannot be contacted by the server, and any backup network session must be initiated by them. And that's why, as you point out, it says what it says on p228 of the UG.

Server generates the Proactive script's client list from the script's source list, without reference to availability, the script then iterates through the list in order of "need". (I don't know what it does for "unavailable" clients, whether it skips them or tries a multicast/subnet broadcast before moving on. That "last seen at home" Remote client may now be back on the work ethernet -- does the now-local client report its presence to the server or does the server discover it?)

Seriously -- try it for yourself. Add a new client to the server, but don't tag it. Turn the client computer off. Make a new Proactive script but don't run it, create a tag for the client, add the tag to the script's Sources, then start the script and look at what it is waiting to back up -- you'll see your turned-off client in the list.

I don't have enough test machines to find out what happens when a Remote client that is due for backup advertises its availability but the script has other, more "in need" clients available. Perhaps another good reason to use Storage Groups, to avoid that very problem...

3 hours ago, DavidHertzberg said:

The second sentence of your next-to-last paragraph directly above is simply a restatement of my prefix solution

No, it's a different approach to the same end -- you talk of "storing a suffix string on the client", I'm doing it wholly with tags on the server.

Analogy time! (Yes, I'm twiddling my thumbs while a web server rebuilds -- apologies...)

Acme Explosives are a careful company -- as you'd expect, given the business they're in. Every day each manager (Proactive script) phones each of their members of staff (RS client) to ask for a report on what had happened (backup) since they last spoke.

Then Covid-19 hit. Acme furloughed most of their staff, set a few to work from home (Remote clients), and kept one manager in the office. The manager didn't know anyone's home number so, instead, the home-workers phoned in their daily reports to that manager, and everything was good.

As the pandemic dragged on, more and more workers were taken off furlough and set to work from home. The single manager couldn't handle all the reports so Acme had to put more managers on site. Although they each had a phone, they all shared the same number, so there was no control over who collected a worker's report -- each worker phoned in again and again until they'd all recorded it (roughly the current situation in RS with multiple Remote-enabled Proactive scripts) or each one listened to the report and either recorded it or said "Sorry, not my responsibility, please call back and try again" (my work-round using Rules).

Being smart businessmen, Acme called in the consultants. One proposed that each manager had an extension number and that each home-workers' phone be programmed to speed-dial the appropriate extension. Easy for the both worker and Acme once set up, but if the home-worker's manager changed a technician would have to go to their house and reprogram the speed-dial. The other also suggested extensions, but that all calls in were to a central number and then routed by an operator to the appropriate manager, as shown by tags on the switchboard. A bit more work on every call -- but easily changed if required, without the operator leaving their desk.

History has yet to record which solution they eventually went with. But it must have been successful because, since then, business at Acme Explosives is booming...

Sorry -- just couldn't resist 🙂 

Link to comment
Share on other sites

On 7/28/2020 at 3:06 PM, Nigel Smith said:

I believe you're wrong. I can't be absolutely sure of how it works, I'm not an RS engineer, but here's a couple of screenshots immediately following a full restart of the server:

 image.thumb.png.d92397e338fb2af78bf5968f0662722e.png

image.thumb.png.ab26fe8c3504f4b9e45b925fbff949ed.png

Apologies for the fuzziness -- remote desktopping in a remote desktop session can play havoc with screenshots 🙂

On the same network as the server, "Luggage" is a Direct IP client and "RS_16_Test" is subnet broadcast. At home, ie Remote clients, "admin2's MacBook" finished a backup to "SG_Test" one minute before the server restart and is still awake/available, "Admin's MacBook Air" has been asleep since its last backup more than four hours before the restart.

As you can see, they are all in the Proactive list, even though two can't be contacted. RS might, of course, be including anything from the source list with a "known" IP address, as shown in the second screenie, but that also shows that the "known" IP address persists even across restarts -- so a client will always have a "last known" IP address even if that isn't its current one.

(Note that a Remote Backup isn't just "here's my IP, come back to me when you're ready". I deliberately turned off all port forwarding on my router for this test -- the backup session must be initiated client-side, hence the private IP address.)

So each Proactive script does appear to have a list of clients, generated from the script's Source list.

This isn't points-scoring on my part -- it helps refine our ideas on how to move forward because:

...is more complicated than needed. If RS interpreted any tag name starting "Remote Backup Clients" as it does now, but also allowed you to create eg "Remote Backup Clients - Group 1" and "Remote Backup Clients - Group 2" so you could add different "remote groups" to different scripts, everything would work just as with other tags -- no messing around with the client and no problem with your China example, just update the tag selection on the server and you're done. If that's a bit clunky you could have a special class of tags, "Remote Backup tags" that could have any name and yet still be understood as Remote tags.

Thanks David -- this is really helpful in trying to work out a way forward for our setup. Even if nothing comes of it, it's had me poking around and trying different ways of using what's already available. Including -- gulp -- Storage Groups!

Nigel Smith,

You've glossed over the fact that your Proactive script "SG_Test" was evidently created before you "turned off all port forwarding on my router for this test".  (Evidence is the script name, where SG stands for Storage Groups.) That IMHO is how you managed to designate Favorite Folders on "Admin's MacBook Air" and "admin2's MacBook" as sources for the script.  If you can get one of your PhDs/PostDocs to temporarily install a Retrospect Client on his/her from-the-get-go Remote machine per pages 228–230 of the Retrospect Mac 17 User's Guide, and then create a Proactive script using "Remote Backup Clients" as a source, I predict you won't be able to then designate a Favorite Folder on that machine as a source in that script.  All you've shown is that a Proactive script does indeed—or did as of 16.6—generate a list of clients it already has defined, and remembers that list—including possibly-obsolete IP addresses—across "backup server" restarts.  Per the cumulative Release Notes for Mac 17.0.1  "Remote Backup: Fixed issue with not logging out clients after ProactiveAI backup (#7967)", your 16.6 script may not be modifying the list for Remote "clients" as it should.  I can't test this for you, because I don't have any candidate Remote "clients" that I could associate with my diminutive home installation.😞

You've also glossed over the original concept of Tags: they are merely groups of drives or Favorite Folders that can be referred to in the "backup server" GUI by the same Tag name.  The "Remote Backup Clients" Tag is a kludgey perversion of this concept, which needs to be enhanced by something that can be contained in a particular Retrospect Client installation—i.e. a suffix name.  Just defining a "Remote Backup Clients - Group 1" to the "backup server" isn't enough; how would you define which truly-Remote "clients" that Tag refers to without particular "clients"  identifying themselves as members of "Group 1", using some string associated with their installed Retrospect Client? 😕

 

Edited by DavidHertzberg
Change italicized phrases to bolded phrases, because italics may not catch Nigel Smith's eye; rephrase next-to-last sentence if first paragraph
Link to comment
Share on other sites

My testing setup was as close as I could get to our proposed "production" use -- no point in doing otherwise in this case. So...

1 hour ago, DavidHertzberg said:

You've glossed over the fact that your Proactive script "SG_Test" was evidently created before you "turned off all port forwarding on my router for this test".

No -- I took the laptops into work, wiped them, got hem on the ethernet, installed RS client, registered them with RS Server and created Favourites. I then took them home.

From home I VPNed from my desktop into the work RS Server and set up the test scripts and tags. I then turned on the laptops, got them connected to my home wireless network, and let things happen -- the laptops were at no time connected via VPN.

Sidenote: If it was created using port forwarding, the client record "Interface" would necessarily be "Direct IP" and it would be my router's public address that was shown. That's because there's no way (at least that I can find, correct me if I'm wrong) to change from "Direct IP" to "Multicast" or "Subnet Broadcast" if you can't actually contact then select the client using the changed method.

1 hour ago, DavidHertzberg said:

All you've shown is that a Proactive script does indeed—or did as of 16.6—generate a list of clients it already has defined, and remembers that list—including possibly-obsolete IP addresses—across "backup server" restarts.

Sidenote 2: Apologies for any confusion, but the RS Server I'm testing with is the latest update to v17. "RS_16_Test" is the previous v16 test server which now has the engine turned off so is client-only. I was short on time and so grabbed an available machine -- I really should have thought ahead and renamed it!

Again no -- I've shown that the list of clients that Proactive is waiting to access is generated on the fly using the script's Source list, without reference to the clients' locations or availability. And that the Source list is updated, without need to start/stop the script, either periodically or in response to tag changes. You can try this yourself by setting up a new Proactive script using a new tag that hasn't been applied to any clients, start it running, then applying that tag to a client -- after a few moments the client will appear in the ProactiveAI list.

And if you can't do that yourself -- here's a video:

https://youtu.be/LiZ8b6KZ3OY

And inB4 "but Remote Backup Client tags are different":

https://youtu.be/4Ok_z2Lrn9A

1 hour ago, DavidHertzberg said:

including possibly-obsolete IP addresses

And again no -- though I'm much less certain on this one 😉 -- the ProactiveAI list doesn't contain IP addresses, they're held in the Sources record for the client. I'd guess that Proactive signals the Engine which client is to be backed up and Engine refers to the Sources record for the client to determine connection type etc. I don't know how that works with Remote clients. But it isn't really important to the discussion.

1 hour ago, DavidHertzberg said:

The "Remote Backup Clients" Tag is a kludgey perversion of this concept

Agreed. But...

The current kludgey perversion binds all so-tagged scripts to all so-tagged clients. I want to limit that binding so that a client can be bound to one, some, or all scripts as I desire. It can't be beyond the wit of man to have multiple Remote Backup Clients tags, each with an unique identifying attribute, so that matching-tagged scripts bind to matching-tagged clients -- after all, that's exactly how all the other tags work, and there's no messing around with the client installation required for those.

Link to comment
Share on other sites

Thank you for your YouTube videos, Nigel Smith; they clarify your "production" use.  They show that you first cabled the laptops to the same LAN as your "backup server", and basically followed the procedure from the bottom of page 228 to the top of page 230 in the Retrospect Mac 17 User's Guide.  However they show you manually created the Tag "Remote Backup Clients" and used it to identify specific Favorite Folders on your laptops, which explains why those Favorite Folder names appear in your (topmost) screenshot of the Proactive Activity list up-thread.  That's what had confused me.

What you're forgetting is that Tags are only defined on a "backup server"; they don't appear on the "client" machines that they group together.  As I've said up-thread, a truly remote "client" sends a message to the "backup server" whose IP address is stored in server.txt—displayed as the Public Address of its Client.  Apparently the message currently says "back me up with any Proactive script; legitimized because I have this public key stored in my pubkey.dat." 

Page 228 of the UG doesn't explain how the "backup server" knows which backup to on-demand restore for a particular remote "client".  Maybe it does so by the IP address stored by a Proactive script in Sources for a particular Remote Client backup, which would explain "Remote Backup: Fixed issue where remote backup failed if client added by name (#7705)" in the cumulative Release Notes for Retrospect Mac 15.6.1.  But a remote "client"'s current IP address could have changed.  In any case what's needed is a "unique identifying attribute"—as you say, but one stored in association with the Remote Client that can be combined with the Tag "Remote Backup Clients" to designate which Proactive script is to backup that particular "client" machine. 

I propose a suffix to the Tag, stored in association with the machine's Remote Client.  There should also be a simple Retrospect utility program, which would be run by the possessor of a Remote  "client" machine to either establish the suffix or to change it.  If you repeated your test after the Retrospect engineers instituted use of the suffix for Remote Backup, you'd have to run the utility program to establish a suffix on each of your two laptops.  If the salesperson Albert were moved from Adelaide to Beijing with his existing laptop, as described in the last paragraph of this up-thread post, he'd be told to run the utility program to change the suffix  The utility would also be used if the possessor needed to acquire or replace his/her remote "client" machine without returning to headquarters; replacement examples—ripped from the headlines 🤣—would be the Shanghai PSB arresting Sally for subversion and then releasing her without her laptop, or Albert taking his laptop on a trip from Adelaide to Kangaroo Island and having it destroyed in the 2020 bushfires.

Whether the suffix should be a group number or a multi-character alphanumeric string is a GUI question, although I think a multi-character alphanumeric string (may be empty) is the most administrator-friendly approach.  A Proactive script would have to store the suffixes for its Remote Backup sources.

Link to comment
Share on other sites

13 hours ago, DavidHertzberg said:

They show that you first cabled the laptops to the same LAN as your "backup server"

The videos don't show that (although that's how the clients were installed, registered, and Favourites defined) as you can see, the tag changes were done to a machine that couldn't be contacted by the server (because it was at home and turned off!).

The videos show that the list of machines ProactiveAI is going to try to back up is based solely on server-set tags, without reference to or need of contact with the client itself. So it's pretty obvious that I'm not forgetting that tags don't appear on client machines.

14 hours ago, DavidHertzberg said:

Apparently the message currently says "back me up with any Proactive script; legitimized because I have this public key stored in my pubkey.dat." 

Yes, but it isn't a huge leap to have multiple Remote tags that can be assigned server-side to both script and client and that message interpreted by the RS engine as "back me up with any matching Proactive script; legitimised because I have this public key stored in my pubkey.dat."

It's almost the same solution as you propose but with three small and one big difference. The small ones are "no faffing with the client when changes need to be made", things are obvious to the Admin (no digging for secret strings), and the fact that adding Remote clients to scripts is exactly the same as adding any other tagged client (consistency is good). The big one is that you could apply multiple Remote tags to the client record so they can be backed up with multiple scripts (doable with multiple secret strings but nowhere near as easy or obvious to the Admin).

14 hours ago, DavidHertzberg said:

Page 228 of the UG doesn't explain how the "backup server" knows which backup to on-demand restore for a particular remote "client". 

I think they've assumed you'd have be a real idiot to want more than one remote-enabled script -- they've got me down to a T 🙂 

From my playing, it alternates between the two available scripts -- but that's probably because there's no contention due to low client numbers and use of Storage Groups on both. I would hope it follows the "usual rules" -- if more than one destination is available for backup, pick the least-recently (yuck! Pardon my English, but I can't think of a better way of putting it) used.

Link to comment
Share on other sites

On 7/31/2020 at 10:15 AM, Nigel Smith said:

The videos don't show that (although that's how the clients were installed, registered, and Favourites defined) as you can see, the tag changes were done to a machine that couldn't be contacted by the server (because it was at home and turned off!).

The videos show that the list of machines ProactiveAI is going to try to back up is based solely on server-set tags, without reference to or need of contact with the client itself. So it's pretty obvious that I'm not forgetting that tags don't appear on client machines.

Yes, but it isn't a huge leap to have multiple Remote tags that can be assigned server-side to both script and client and that message interpreted by the RS engine as "back me up with any matching Proactive script; legitimised because I have this public key stored in my pubkey.dat."

It's almost the same solution as you propose but with three small and one big difference. The small ones are "no faffing with the client when changes need to be made", things are obvious to the Admin (no digging for secret strings), and the fact that adding Remote clients to scripts is exactly the same as adding any other tagged client (consistency is good). The big one is that you could apply multiple Remote tags to the client record so they can be backed up with multiple scripts (doable with multiple secret strings but nowhere near as easy or obvious to the Admin).

I think they've assumed you'd have be a real idiot to want more than one remote-enabled script -- they've got me down to a T 🙂 

From my playing, it alternates between the two available scripts -- but that's probably because there's no contention due to low client numbers and use of Storage Groups on both. I would hope it follows the "usual rules" -- if more than one destination is available for backup, pick the least-recently (yuck! Pardon my English, but I can't think of a better way of putting it) used.

I don't see how your scheme could apply different "Remote Backup Clients"-derived tags to different "client" machines, and later have a Proactive script recognize which tag should be associated with a particular "client" sending a "back me up" message to the "backup server", without initially having had that "client" machine on the same LAN as the "backup server".   You'd want this to happen so that a particular Proactive script could say "my sources include some Remote Backup Clients, but not this one, so I won't even scan it".  You pointed out up-thread that a Rule won't avoid scan + snapshot time.

The reason your scheme would need to initially have each potentially-Remote "client" machine on the "backup server"'s LAN is that Retrospect provides no other way of defining a machine as a source.  That Sources->Add would at a minimum have to capture the machine's Computer Name, because the machine's later "back me up" message would have to include the sender's Computer Name to allow a Proactive script to determine whether its list of "Remote Backup Clients"-derived tags applied to the sender.  The "client" machine's name would have to be unique on that LAN, although it might only have to be unique to a subnet of the LAN if the machine's Retrospect Client stored—and later transmitted with the Computer Name—the subnet name.

It's also not possible to define a Favorite Folder—or an added drive—on a machine that isn't connected to the LAN.  (I've checked this out by shutting down the "client" on which I've drafted this post.)  However you can add or remove a tag for a defined source.  You were able to define Favorite Folders—on their existing drives—for your own laptops because you'd brought them in and connected them to your "backup server"'s  LAN.  However I'd advise you against having PhDs/PostDocs bring in their machines for temporary LAN attachment, because that's now a good way 🙄 for you—or them—to catch COVID-19.

Taking a broader view, there are two different use cases for Remote Backup.  As designed by Retrospect Product Management in 2018, the use case was for AlwaysRemote (my term) machines, those that would never be put on the same LAN as the "backup server".  Product Management assumed that an installation would only have a few such machines, so they probably didn't worry too much that their phantom-Tag "Remote Backup Clients" would oblige a Proactive script whose sources included that phantom-Tag to back up all machines sending the "back me up" message.  However Alanna's OP in another thread was the first on these Forums to discuss the RecentlyRemote (my term) use case, in which she evidently planned to back up—as split up among different Proactive scripts—many machines that had been recently moved for Work From Home in the same time zone.  As I've said in the preceding three paragraphs, that won't work with your scheme unless the RecentlyRemote machines are first brought or sent to the "backup server"'s LAN location—and unless Proactive scripts and the Retrospect Client Installer are both enhanced to store Computer Names to be included in the "back me up" messages.

I think my suffix scheme would be preferable, because it handles both the AlwaysRemote and RecentlyRemote use cases.  It wouldn't allow you to define Favorite Folders for AlwaysRemote machines, but neither would your scheme per two paragraphs up.  Most of the code for my proposed suffix-changing utility already exists, because it's included in the version of the Retrospect Client Installer described for Management Console Automatic Onboarding.

 

Link to comment
Share on other sites

Briefly stepping away from the "David & Nigel Show" and back to OP's original question -- Fred, I hope you're still with us!

On 4/2/2020 at 1:36 AM, fredturner said:

Since each source appears to have its own, self-contained catalog and folder of band files on the backup disk, couldn't Retrospect just rebuild the individual source whose catalog has failed?

At the moment, no. But rebuilding is parallelised, and once any client is complete it can be backed up again even while others are rebuilding. But I don't fancy doing multi-terabyte rebuilds for just one corrupt catalog either, and I can't (yet!) find a way of using backed up catalogs and the Repair function to shorten the process.

I suspect that the "gold standard" work-around would be to:

  1. Remove "broken" client from original set's Source list (either machine entry or tags)
  2. Create a new Storage Group media set
  3. Move the folder of the backup files of the "failed" client to the new Storage Group, putting it in the same place in the SG hierarchy as it was
  4. Rebuild the new Storage Group media set
  5. Check you can restore from the rebuilt set
  6. Delete backup files and client catalog from original set
  7. Do a Transfer of the now-rebuilt backups from the new set to the original
  8. Check that original set's version of client is now working again
  9. Re-add client to original set's Source list
  10. Delete new Storage group data and catalogs

If haven't tested this, but it seems like it should work. Try yourself on test data before using in anger!

This shortcut, however, I have tried:

  1. Remove "broken" client from original set's Source list (either machine entry or tags)
  2. Create a new Storage Group media set
  3. Copy the folder of the backup files of the "failed" client to the new Storage Group, putting it in the same place in the SG hierarchy as it was
  4. Rebuild the new Storage Group media set
  5. Check you can restore from the rebuilt set
  6. Replace original catalog with the rebuilt one
  7. Check that original set's version of client is now working again
  8. Re-add client to original set's Source list
  9. Delete new Storage group data and catalogs

And it works! But I'd urge you to give it a thorough thrashing in test mode before entrusting any of your precious data to this method...

Both methods will do want we want -- rebuild just one client catalog out of many. The first requires an extra Transfer operation, the second the extra disk space for the data copy. neither comes with any official stamp of approval 🙂 Try them both and see what you think.

Link to comment
Share on other sites

On 8/1/2020 at 11:06 PM, DavidHertzberg said:

I don't see how your scheme could apply different "Remote Backup Clients"-derived tags to different "client" machines, and later have a Proactive script recognize which tag should be associated with a particular "client" sending a "back me up" message to the "backup server", without initially having had that "client" machine on the same LAN as the "backup server".

The trite answer is "Easily!".

Note that tags are not applied to "client machines" -- they never have been, and the "Remote Backups Clients" tag is no different. They are applied to the "client record" on the server. They effectively say "allow incoming calls from this client" (when in the client record) and "allow incoming calls to this script" (when used in the script's Source).

So I'm proposing a "class" of Remote tags, each instance having a user-defined attribute -- they all say "allow incoming" but finesse it with "and direct them to scripts with the matching attribute".

Perhaps it is easier to think of them the other way round -- they would behave exactly the same as all other tags and, additionally, allow remote access.

On 8/1/2020 at 11:06 PM, DavidHertzberg said:

The reason your scheme would need to initially have each potentially-Remote "client" machine on the "backup server"'s LAN is that Retrospect provides no other way of defining a machine as a source.

Are you sure? 😉   You've actually raised the next issue I want to test -- does automatic onboarding work with Remote clients? I haven't bothered with that yet, because our use of Favourites mandates local addition, but now I've time to scrub a machine and start a client from scratch again.

On 8/1/2020 at 11:06 PM, DavidHertzberg said:

As designed by Retrospect Product Management in 2018, the use case was for AlwaysRemote (my term) machines, those that would never be put on the same LAN as the "backup server".

Sorry David, but that's just plain wrong. From the KB article: "Clients can seamlessly switch between network backup on a local network to remote backup over the internet. You do not need to set up the remote backup initially. You can transition to it and back again" -- true as far back as Nov 2018.

Link to comment
Share on other sites

14 hours ago, Nigel Smith said:

Are you sure? 😉   You've actually raised the next issue I want to test -- does automatic onboarding work with Remote clients?

And the bad news is -- it does...

"But Nige," I hear you say, "surely that's a good thing, allowing us to onboard Remote clients without making them come to the office?" I don't think so, because Remote Clients are automatically added:

  1. ...without the "Automatically Added Clients" tag, so there's no easy way to find them
  2. ...with the "Remote Backup Clients" tag, which means they will be automatically added to all Remote-enabled Proactive scripts
  3. ...with the client's backup "Options" defaulting to "All Volumes", so external drives etc will also be included

I let things run and, without any intervention on my part, the new client started backing up via the first available Remotely-enabled script.

Edit to add:

I didn't notice this last night, but it appears that the client was added with the interface "Default/Direct IP", in contrast to the clients automatically added from the server's network which were "Default/Subnet". I don't know what this means if my home router's IP changes or I take the laptop to a different location (will the server consider it to now be on the "wrong" IP and refuse to back it up?) or if I know take it into work and plug in to the network (will the server not subnet-scan for the client, since it is "DirectIP"?).

End edit

Given the above I'd suggest everyone think really carefully before enabling both "Automatically add.." and "Remote Client Backups"  unless they retain control of client installation (eg over a remote-control Zoom session) -- especially since I've yet to find out what happens if you have a duplicate client name (the next test, which I'm struggling to work out how to do).

Edited by Nigel Smith
Extra data, not spotted at time of original post
  • Thanks 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...