Rules - lack of wildcards and chaining exclusion rules

[AlexB]

I am "new" to Retrospect, having not used it for many, many years. Burned recently by Time Machine. I'm using Retrospect Desktop 17.5.2 on macOS Catalina. Some things are like riding a bike, some are not.

One place I'm struggling is getting rules working for me. I would like to not have Retrospect back up some downloaded items like /Users/*/Library/Calendars and /Users/*/Library/Mail. I initially attempted the wildcard (supported on Windows?!) but does not appear to work on Mac. Then I tried creating a rule that had the All conditions with a sub (took me a while to figure out the option key magically let me do this - a note for improved documentation) - so Mac Path contains /Users and Mac Path contains /Library/Mail. Those work in a rule fine. But if I attempt to define them in a separate rule and then include them in a subsequent rule ("Saved Rule includes"), I can't get that to work. I can include the exclude Caches rule that comes with Retrospect, but if I attempt to include one of my own rules, it does not work. I am copying-pasting the name to be sure of no typos.

First, am I correct that you can't use * or ? in Mac rules?

Second, if I want to define an exclusion, and then include it in another rule, what's the way to do that? 

For now, it's working, with a big rule with all my exclusions, but I'd to know how to solve this. 

I know other people have complained about this but I remember Retrospect had a way you could preview rules - missing that. For now, I'm doing little test backups and seeing what ends up in the backup. And I see what others were complaining about the confusion that excluded folders may still get backed up, but they are empty.

Thanks - Alex

[DavidHertzberg]

AlexB,

First, this 2012 post for Retrospect Mac 9+ by the very-knowledgeable Don Lee says "There is no 'wildcard' or regular expression capability in the rules ...".  Pages 440–441 of the Retrospect Windows 16 User's Guide says Retrospect Windows has a "wildcard" capability for Selectors.

Second, both the first screenshot on page 167 of the Retrospect Mac 16 User's Guide and the last screenshot in this Knowledge Base article show a trailing slash at the end of a "contains" path specification.  A Mac Path should end with a forward slash; a Windows Path should end with a backslash.

[Nigel Smith]

18 hours ago, AlexB said:

But if I attempt to define them in a separate rule and then include them in a subsequent rule ("Saved Rule includes"), I can't get that to work.

Can you screenshot your "failing" rule setup?

Also, be careful how you "embed" rules -- a saved exclusion rule goes in the "Includes" section when you embed it (as you've done above with the built-in) and IIRC multiple exclusion rules should be "Any"ed.

As David says, trailing slashes on your Mac paths -- implied by the documentation and even if not strictly necessary prevents a false match with eg "/Users_Important_Files" -- and no, there's no documented wildcarding. There is an "Is like" match option, but I don't think anyone knows how -- or even if -- it works!

21 hours ago, AlexB said:

Then I tried creating a rule that had the All conditions with a sub (took me a while to figure out the option key magically let me do this - a note for improved documentation)

pp177 of the User Guide -- as much as I like to complain about the documentation, this is something they did include (albeit in a "blink and you'll miss it" way). It should certainly be more obvious, eg a tooltip in the UI.

[DavidHertzberg]

AlexB,

Guided by Nigel Smith's mention of "is like" in his post immediately above, I did a Forums search and struck gold in this 2010 post by kswisher.  So "is like" provides a wildcard capability, but there apparently is no regular expression capability.  Page 432 of the Retrospect Windows 16 User's Guide says—for Find (in code borrowed for Rules?)—that the character '*' can be used to match multiple characters, and the character '?' to match a single character. 

Observe that, in the 2010 post, ", works" after a condition is a note meaning it worked for kswisher's test and ", not work" after a condition is a note meaning it didn't work for kswisher's test.  It probably would also be worthwhile to read preceding posts in that thread.

"is like" is mentioned on page 168 of the Retrospect Mac 16 User's Guide, which I prefer to use because it had not yet been subject to the tender ministrations of the StorCentric Slasher (see the P.S. of this October 2020 post).

[Nigel Smith]

On 2/2/2021 at 9:54 PM, DavidHertzberg said:

Guided by Nigel Smith's mention of "is like" in his post immediately above, I did a Forums search and struck gold in this 2010 post by kswisher

I didn't want to mention kswisher's work without some checks of my own -- there's even more chance of undocumented behaviours being broken by "updates" than the documented ones!

Some quick tests suggest this is still valid, so "Folder Mac Path is like */Users/*/Documents/" will select the Documents folder of every user account of a machine.

Note that "*" is "greedy", so "*/Users/*/Xcode/" will match /Users/nigel/Documents/Programming/SuperSecretStuff/Personal/ButNotHidden/Xcode/.

Given the lack of official documentation you should test, test, and test again.

On 2/1/2021 at 8:49 PM, AlexB said:

I know other people have complained about this but I remember Retrospect had a way you could preview rules - missing that.

While there's no preview, I do this by running the backup then using the "Past Backups" pane to "Browse" with "Only show files copied during this backup" checked. But you should still be able to do it the way I used to -- create a new media set, back up an entire client to that set, then do a "Restore" using "Search for files...", select your rule(s) and the backup set, then the destination. The "Select Backups" step will allow you to preview the results. When you are doing a lot of fiddling to get a rule right, this can be a lot quicker than repeated backup attempts (and there's a lot less impact on the client!).

Also note that Rules don't reduce scan time -- every file on a (RS-defined) volume is scanned/tested, there are no "don't even look in this folder" shortcuts. The only way to do that is via the RS Client's "Privacy" settings.

[AlexB]

Thanks all for the pointers! I will go back to my rules and add / to the end of my paths - point taken. Very interesting on the "is like". I would like to see Retrospect improve the documentation to explain this rule and that it is the only way you can use a wildcard.

On 2/2/2021 at 12:55 PM, Nigel Smith said:

Can you screenshot your "failing" rule setup?

Also, be careful how you "embed" rules -- a saved exclusion rule goes in the "Includes" section when you embed it (as you've done above with the built-in) and IIRC multiple exclusion rules should be "Any"ed.

I tried a few things here. One, I said I could not get it to use my own rule as the saved rule. I have tested again, and that is now working. Not sure what I did before, but I can include as a single rule and it works in my test backup (see first image).

I still am not getting multiples to work - when I do this (see second image), the Downloads folder now gets backed up, telling me the 2nd rule is not being applied (see third image for 2nd rule).

As I said, I'm running ok right now, just would like to know how this should work.

Thanks,
Alex

 

[DavidHertzberg]

(Disclaimer: Anything I may say about the intentions of Retrospect "Inc." in this or any other post is merely the result of "reading the tea leaves", the "tea leaves" being documentation and public announcements supplemented by an occasional morsel from Retrospect Sales.  I have never been paid a cent by Retrospect "Inc." or its predecessors, and I pay for my upgrades. Any judgements expressed are—obviously—mine alone. The same is true of Retrospect's history, especially with references to here.)

AlexB,

WRT "embedded" Rules, it appears this 2014 post by jotrago is the best analysis in the Forums. 

Note—however—that the post appeared in the "Server, SBS and Multiserver" Forum, and that it uses the Retrospect Windows term Selectors instead of the Retrospect Mac term Rules.  IMHO that doesn't make any difference to the question you raised, because—although the GUIs differ—I doubt that the Retrospect engineers went to the trouble of developing two different sets of underlying implementation code.  If you question my doubt about their going to that trouble, look at pages 366–376 of the Retrospect Windows 17 User's Guide.

[Nigel Smith]

18 hours ago, AlexB said:

I still am not getting multiples to work - when I do this (see second image), the Downloads folder now gets backed up, telling me the 2nd rule is not being applied (see third image for 2nd rule).

If you can't merge multiple saved rules (I'll test later), try chaining them:

Rule A --  Include "Saved rule includes All Files Except Cache Files", Exclude "Folder name is Temp"
Rule B -- Include "Saved rule includes Rule A", Exclude "Folder name is Downloads"
Rule C -- Include "Saved rule includes Rule B", Exclude "Folder name is .dropbox"
etc.

Obviously each Exclude could contain as many clauses are you like.

So a run using Rule C, above, would exclude ".dropbox", "Downloads", "Temp", and all cache files.

[AlexB]

OK - I'll just conclude that you can only chain from a single rule at a time. Thanks for the replies.

- Alex