Retrospect can't find disk backup members on NAS

[prl]

Since installing 16.1.2, I haven't been able to do backups to disk media sets on a NAS. Now, when the backup wants to start writing to the media set, it goes to Needs media, and when I use Choose Media..., I can navigate down into the top levels of the NAS filesystem, but when I get to the directory that contains the top-level directory of the disk that contains the media set members, I can open and display some directories, but not the Retrospect directory that contains the backup members.

I get similar results when I try to use Verify on the backup set, or to create a new backup set on the NAS.

I can access the directories where the Retrospect stored the media sets in Finder (as myself) and by using U*ix commandline access to list the directories both as myself and when su'd to root (which RetrospectEngine runs as).

This all worked for several years before I upgraded to 16.1.

Anyone have any ideas?

Retrospect for Mac 16.1.2 (102)
MacOS 10.11.6 (Please don't suggest that I upgrade - my Mac is a MacBook Pro 13-inch, Mid 2009, and this is the most recent MacOS version that runs on it)
NAS: SMB shares served from a Fritz!Box 7390 router (yes, it's slow, but it worked until the Retrospect upgrade)

[Nigel Smith]

On 7/13/2019 at 4:10 AM, prl said:

Anyone have any ideas?

What happens if you let Retrospect make a new set in a new, separate, directory on the NAS -- i.e., instead of your current NAS/folder1/folder2/Retrospect structure you do NAS/folder1/folder3? Can it create and, afterwards, verify the test set OK?

If so, check for any permissions differences between the two using the NAS's management interface. Sometimes a NAS, which is usually running a SAMBA variant, can present itself in different ways to different OS X programs.

And if the first step works but you can't see any differences, try copying one of your "old" catalogs into the "new" Retrospect directory. Can you access that now? In which case you might be able to get round the problem by moving all your catalogs to the new folder.

[Monafly]

I had the same problem and after talking to RS Support found that there was a bug in an early version of 16.1.2 that was fixed in build 102. This solved my problem, but you state that this is the build you are using so maybe there is something else amiss. 

I'd check that you are for sure using 16.1.2 (102) and then check with RS support if it persists. 

[prl]

Thanks for the suggestion, but from Retrospect>About: "Version 16.1.2 (102)".

So no luck there.

Nigel, thanks for your suggestion. I haven't had time to try it out yet, but I will. I may also try moving the existing backup members to another directory on the NAS, too.

The NAS doesn't have much in the way of permission control. Only registering users with passwords, specifying the name of the directory subtrees that are exported to them and whether they have read or write permission.

The NAS feature of the router seems to have been a bit of an "oh, look, there's room for Samba, so let's put it in" effort. It was removed a while back for space reasons, but there were enough user grumbles that they found space for it again.

[Nigel Smith]

On 7/24/2019 at 3:41 AM, prl said:

The NAS doesn't have much in the way of permission control.

Doesn't need much. NASs usually use Windows ACLs for permission control, which don't directly translate to POSIX/OS X permissions. So it's always a "best approximation", can be tighter or looser than expected/intended, and can be interpreted in different ways by different programs (if they aren't using OS X's APIs).

I'm not expecting my workround to work, but it's worth trying before you contact Support -- more data points will help them help you.

[prl]

OK, I tried making a new backup into a directory that Retrospect had newly created on the NAS, and it all worked just fine.

I then renamed that directory to the name of the original backup top level directory and moved the backup set directories into it, but still no good. I think I'll abandon the old backup sets and create new ones.

All a bit annoying.

[Nigel Smith]

You might still be carrying some cruft over from the "old" directory structure.

Instead of what you did, copy (not move) the members from old to new directory, creating any required sub-directories by hand as you go. Set all permissions to the same as the newly-created top level Retrospect directory. Get Retrospect to "Rebuild" the media set, adding members as required, but make sure to save the new catalog in a different location so you don't overwrite the old one.

That's quite a lot of work. But it could get you out of a hole if you need to keep the old sets available for restores -- you never said in the OP if Retrospect still had the read-access that restores require. If it does then I wouldn't bother, just move onto the new sets.

[prl]

I tried doing a partial copy of one of the disk media sets, and it seemed to be accessible from Retrospect. I'm currently doing a full copy of both disk media sets. It's s.l.o.w. I'll let you know how it goes.

Thanks for the suggestion.

It's still no clearer why two long-functioning media sets decided to "disappear" from retrospect.

[prl]

OK. The copy finished and I got around to testing the copy. I can navigate into the copied directory for a Verify, but when I select the data member, it goes straight back to wanting me to Choose Media again, without putting anything useful into the log.

Also, very oddly, when the top level directory of the copy is called "NewRetro", I can navigate in the directories inside it, but if I rename it to "Retrospect" (the original name), I can no longer navigate inside it. Also, When the old copy is renamed from its original name "Retrospect", to "OldRetro", I can navigate into its directories, but a Verify of its members fails in the same way as the Verify of the copy.

Looks like I'll need to set up new backups.

[Nigel Smith]

Could you go back to whatever version you were using before you "upgraded" to 16.1?

[MikeHutch]

I get this problem also, while using Retrospect Desktop 16.6.0 (114), using a WD MyCloud 4TB NAS, and MacOS Mojave 10.14.6.

Usually I could rectify it by rebuilding the catalogue, but not this time, can't see the folders in the Retrospect folder. However, one curious detail this time, is that I can restore from the Media Set, but not write to the affected member.

I'll try the above suggestions, with the folder rename, to see if that works.

 

[DavidHertzberg]

I don't have a NAS, but—after 15 years experience over the last 20 years—I've been running Retrospect Mac 16.6 for nearly a year.  It's my observation that earlier 16.x releases were a mess, and that 16.6 was a late-in-the-cycle release to clean up earlier mistakes.  My destinations rotate on a weekly basis between 3 portable HDDs, one of which is always in my bank safe deposit box.  Another HDD is always cabled to my "backup server", and the third—having been brought back from the bank—normally sits right inside my apartment door in case I have a flood when the bank isn't open.  From time to time I use the spare space on the inside-the-door HDD as a destination for tests of Retrospect features, and I've  had some "interesting occurrences".🤣

The squeaky-clean approach is to have a "Retrospect" folder at the top level of each destination "drive", and to have a zero-byte "marker" Unix executable file named "Backup Media" directly inside "Retrospect".  Also directly inside "Retrospect" should be a folder for each Media Set whose destination can be that "drive"; the folder has the name of the Media Set, e. g. "Whatever".  Inside that folder should be at least one Member folder whose name is that of the Media Set preceded by a number and a dash, e. g. "1-Whatever" or "2-Whatever"; these leading-numbered folders are where the corresponding .rdb and .session files for the Member are stored.   Adding a Member to Media Set "Whatever" creates folder "Whatever".

This approach was the multi-HDD-volume-destination Disk Media Set upgrade from Retrospect's multi-tape-per-Backup-Set destinations, where the tapes are labeled e. g.  "1-Whatever" and "2-Whatever".  Over the years since Retrospect Mac 8,  Retrospect Inc. engineers seem to have gotten creative for the benefit of administrators wishing to use the spare space on their destination HDDs for other purposes.  IME everything works beautifully so long as there is one—and only one—folder on a destination drive with a zero-byte Unix executable file named "Backup Media" directly inside it.  It seems you can get away with naming the folder enclosing that "marker" file something other than "Retrospect", and even with having more than one folder named "Retrospect" on the HDD so long as only one folder contains the "marker" file.   However you can get into trouble by having multiple folders each containing the  "marker" file, especially if they are one inside another.  I suspect all you complaining administrators have some variant of that problem, encouraged by the possibility of having more than one destination "drive" on the same NAS.

Now that I've told you this secret, I believe I'm obligated to kill everyone who reads it.  Please aid that by PM'ing me your names and addresses.🤣  If you can't see your way clear to doing that, please put at least your home country—and home state for U.S. residents—in the Location field of your Profile.  It's very helpful to those of us who answer questions on these Forums, because it gives us some idea of your English proficiency and at which hours you'll be posting. Gender too.  And no pseudo-Latinate cutesy-poo; leave that to the weird posters on the Ars Technica forums.TIA. 😃

P.S.:  I did test; what's written in the third paragraph about uniqueness of the "marker" file is wrong—see this post below.

Edited January 13, 2021 by DavidHertzberg
P.S.: I did test; what's written in the third paragraph about _uniqueness_ of the "marker" file is _wrong_.

[prl]

Thanks, Dave. I'm afraid I've long since given up on Retrospect and I now use Apple's TimeMachine for my backups.

Now that we're running Catalina on our Macs, the NAS I was using (which can only do SMB v1 - don't ask) is no longer accessible from the Macs anyway.

Posting from Australia (Terra Australis Incognita, IIRC, in real late Latin ).

[Nigel Smith]

On 11/15/2020 at 12:51 AM, prl said:

Now that we're running Catalina on our Macs, the NAS I was using (which can only do SMB v1 - don't ask) is no longer accessible from the Macs anyway.

NetBIOS is disabled in Catalina, which is probably what's breaking things. See here for how to re-enable. You may also have to use "cifs://serverAddress" rather than "smb://serverAddress" to force an old-style connection.

Or upgrade the NAS, of course 😉

 

[prl]

Thanks.

Doing that (I'd in fact already tried it when I first installed Catalina) allows me guest access to the NAS, but not authenticated (Registered user...) access. Whenever I try, , the Connect to Server popup shakes and does nothing, not even an error popup. It doesn't seem to make any difference whether I use the smb: or cifs: protocol name.

[Nigel Smith]

So it's connecting, but not authenticating...

Did you also set

minauth=none

in /etc/nsmb.conf?

[prl]

Thanks, Nigel. I hadn't, but adding it didn't help either.

By using mount -t smbfs, I can see unequivocally that it's an authentication issue (using a guest mount of the same server/share specs works):

mount_smbfs: server rejected the connection: Authentication error

My /etc/nsmb.conf now contains:

[default]
port445=both
signing_required=no
minauth=none

According to the Mac nsmb.conf man page, port445=both and signing_required=no are defaults anyway.

BTW, I rebooted to make sure that /etc/nsmb.conf was re-read. Is that necessary?

[Nigel Smith]

11 hours ago, prl said:

Thanks, Nigel. I hadn't, but adding it didn't help either.

Given that AVM themselves say you'll not get things to work and that unit is EOL and so not getting firmware/OS updates, I'd give this up as a bad job. Not all Samba implementations are equal, and it seems that AVM used one that you can't force to work with Catalina.

Assuming you don't want to get a new router or replace your USB drive with a "proper", standalone, NAS, I'd go for a fileserver with the drive connected. Got an old Mac or PC sitting around? Use that. Otherwise, nice little project for you -- a Raspberry Pi running Pi OS or Ubuntu Server, which'll not only be more secure but probably faster too! A Pi 4 Model B, case and PSU would only cost ~£50

[prl]

Thanks, Nigel. I've been thinking about getting a NAS, but so far can't justify myself getting one. The AVM router's NAS was convenient, but quite slow (max throughput ~5MB/s). My current backup arrangement using TimeMachine doesn't require a NAS.

[MikeHutch]

Good news. I've just discovered something surprising about the NAS members problem, relating to editing, rebuilding or repairing the set!

I have a Media Set that lost it's connection to Member Set 1, on my NAS. So I decided to repair it. I then decided to repair it... got to the 'Add Member' dialogue window showing your available storage devices, and selected the Retrospect folder on my NAS drive. I thought, OK lets see what happens, so I clicked on Retrospect folder, and pressed 'Next...'. Low and behold, the program found the members itself, after a few seconds. Clicked on the correct backup set, and pressed Next.. again, and it repaired the Media Set. It seemed to have worked!

I always thought that you have to find the member folders your self (I'm using Retrospect 16.6 (114)) manually - or at least, that's what I had to do in older versions. Of course the user guide wasn't helpful in giving any further instructions on exactly how you are suppose to find the members in a media set. 😞

[DavidHertzberg]

Mike Hutch,

I think that Step 3 on page 185 of the Retrospect Mac 16 User's Guide—which has been the same since at least the Retrospect Mac 10 UG—glides over a complication when it says "Retrospect displays a dialog that allows you to navigate to the first member of the Media Set".  The screenshot on that page shows a top-level folder named "Retrospect", and directly within that folders named for various Media Sets.  That's what I called "the squeaky-clean approach" in the second paragraph of this up-thread post.  It works fine in my simple home installation, and now evidently works fine for you.

The complication is that evidently many Retrospect installations don't follow "the squeaky-clean approach" for their destination Member disk drives.  They prefer to use subsidiary folders inside the "Retrospect" folder to classify Members by the period of Media Set use—such as "2019 Backups" and "2020 Backups"—or by installation sub-organization—such as "Engineering Backups" or "Accounting Backups".  You can apply your organizational experience to come up with more possibilities.  Some installations use other backup applications in addition to Retrospect, and therefore prefer a top-level folder on each destination Member disk drive to be named something like "Marketing Backups" instead of "Retrospect"—implying multiple subsidiary "Retrospect" folders.  These must be navigated to in the Add Member dialog.

To handle such installations, the Retrospect Inc. engineers at least as far back as 2015 devised the zero-byte "marker" Unix executable file named "Backup Media" approach I mentioned in the second and third paragraphs of  this same up-thread post.  As I understand it, you have to have such a "marker" file in the  folder into which you directly create a folder for any Media Set whose destination can be that "drive".  In my example in that up-thread post, the created folder has the Media Set name "Whatever", and will contain subsidiary folders named "n-Whatever" where n is an unsigned integer.  Adding a Media Set or manually adding a new Member is described on pages 87-89 of the Retrospect Mac 16 UG.  The top of page 89 says 

Quote

Retrospect adds a Retrospect folder on the member disk you have defined, containing another folder with the name of the Media Set, which in turn contains another folder with the Media Set member number.

but I don't think you have to keep the added "Retrospect" folder once the folder with the name of the Media Set has been added.  However, as Scillonian says in the last paragraph of this 2018 post in the Professional forum, "The Retrospect folder can exist as a top level folder on any HDD connect [sic] directly to the Backup Server either internally or externally. On an SMB share (e.g. Samba on a NAS) the Retrospect folder must reside in a share folder ....".  AFAIK that means there has to be some folder containing the "marker" file directly above the folder named for the Media Set.

IMHO , having to read in the UG what I have just written would scare the pants off most new administrators reading it.  That's why Retrospect Inc. put only a description of "the squeaky-clean approach" into the UG, and why I humorously wrote in the last paragraph of that up-thread post "Now that I've told you this secret, I believe I'm obligated to kill everyone who reads it."  What Retrospect Engineering needs to do is to create a Knowledge Base article containing some version of what I've written in the second and third paragraphs of my up-thread post, supplemented by some version of what I've written in the second and third paragraphs of this post, and link to it from the pages I've listed in the third paragraph of this post.  You're the most recent administrator to have had this problem, Mike Hutch, so here's why and how to file a documentation Support Case; copy from my posts.

P.S.: I'll test "but I don't think you have to keep the added 'Retrospect' folder ...." in the next-to-last paragraph (directly after the embedded quote) on Friday evening, when I can free up a portable HDD.

P.P.S.: I did test; what's written in the third paragraph implying uniqueness of the "marker" file is wrong—see this post below.

 

 

Edited January 13, 2021 by DavidHertzberg
P. S.: I'll test "but I don't think you have to keep the added 'Retrospect' folder ...." in the next-to-last paragraph on Friday evening. P.P.S.: I did test; what's written in the third paragraph implying _uniqueness_ of the "marker" file is _wrong_.

[Nigel Smith]

I'll admit, I get very confused about what to select during various Media Set operations... It's usually "the directory that contains the 'retrospect' directory that contains your media sets", sometimes "the 'retrospect' directory that contains your media sets", and occasionally the media set itself. So I just work through them, in that order, until it works -- and then forget for the next time!

[MikeHutch]

Thanks for the info David - very in-depth. By the way, my [Retrospect] folder structure is in the standard configuration you've described in your second paragraph.

Anyway, I'm good now, so no problems.

[DavidHertzberg]

MikeHutch, Nigel Smith,  prl, and everybody else,

I've just finished the last of the experiments I promised in the P.S. of this up-thread post, and you can forget anything I said about uniqueness of the "marker" file in that post or in this preceding up-thread post.  In May 2019 using Google I found—but couldn't find a few days ago—this 2012 post that said the "Backup Media" zero-byte "marker" file was devised to exclude destination Members from a Backup or ordinary Copy operation (termed a Duplicate operation in Retrospect Windows); they should be copied with "Transfer" operations such as Copy Backup or Copy Media Set.  My experiments last night show the presence or absence of a "marker" file doesn't  affect the creation of Media Set Members.  (I'm running Retrospect Mac 16.6, on a Mac Pro "backup server" that boots macOS 10.12.)

Instead my experiments show that the "backup server" is a very intelligent program.  The first thing I did last night was to rename the top-level "Retrospect" folder on my G-DRIVE White portable HDD to "Spectrowet" (I'm not very imaginative), and to delete all folders—but not the "marker" file—beneath it.  I then Removed the single "1-Media Set White" Member from "Media Set White"; I had to hit the Recycle button for the Media Set before the minus-sign-button would activate.  After that I Added  a new "Media Set White" Member to the "Spectrowet" folder, which hierarchically generated a "Retrospect" folder and a "Media Set White" folder and a "1-Media Set White" folder within the the "Spectrowet" folder.  It also generated another "marker" file inside the "Retrospect" folder, meaning there is now one "Backup Media" zero-byte "marker" file within the "Spectrowet" folder—at the same level as the "Retrospect" folder—and another "Backup Media" zero-byte "marker" file within the "Retrospect" folder.  As the last step in the experiments I ran a No Media Action script to do a Backup of my MacBook Pro over the LAN to "Media Set White"—which of course automatically became a 3.25-hour Recycle backup because I had previously hit the Recycle button for "Media Set White".  That was the end of the experiments.

After the backup and Catalog-updating phases of the script had completed, I Stopped the compare phase (which I 'd intended to omit, but forgot to uncheck it in the script's Options) and re-established a "1-Media Set White" Member on G-DRIVE White hierarchically under a renamed-and-emptied top-level "Retrospect" folder.  The experiments above show that the "backup server" program figures out what folders and files need to be created underneath the folder designated by Add Member, and only creates them if necessary.  If you've designated an existing "Retrospect" folder, it'll just create the subsidiary folders (and "marker" file if it doesn't already exist) needed for the Member.  If OTOH you've designated an existing folder with another name, it will create the complete hierarchy underneath that folder—including a "Retrospect" folder directly beneath the designated folder.  Manually subverting its intelligence—as prl reported doing up-thread, or having "Retrospect" be the name of a NAS "shared folder", creates problems.

[DavidHertzberg]

It was late Friday night again, so I was able to do non-"squeaky-clean" (2nd paragraph) experiments on the G-DRIVE Blue I'm going to Recycle at 3 a.m. Saturday.  They continue to show that the "backup server" is a very intelligent program per the 3rd paragraph here, though it can be subverted.

My first experiment was to replicate the second approach in that 2nd paragraph linked-to in this post's first sentence.  I hit the Recycle button for "Media Set Blue", removed its Members, and erased the portable HDD.  I then created folders named "One" and "Two" (I'm still not very imaginative🤣) at the HDD's top level, and Added a Member under "One" and another Member under "Two".  This created a "Retrospect" folder—containing a zero-byte "Backup Media" file as well as the subsidiary folders—under each one of "One" and "Two".  I limited the size of "1-Media Set Blue" to 40MB, in order to guarantee spillover onto "2-Media Set Blue" for a 72MB Recycle Backup of my MacBook Pro.  Finally I ran that No Media Action Backup, which defaulted to Recycle because I'd Recycled "Media Set Blue", to its 4-hour completion—including the compare phase to make sure I could access both Members.  This whole experiment was uneventful, showing you can have "Retrospect" folders under multiple "departmental" folders.

My second experiment was to replicate the first approach in that 2nd paragraph.  I hit the Recycle button for "Media Set Blue", Removed its Members—which proved impossible for "1-Media Set Blue" until I Removed and Added "Media Set Blue" and then added it back to the four scheduled scripts that use it, and erased the portable HDD.  I then created folders named "One" and "Two" under a  top-level "Retrospect" folder level, and Added a Member under "One" .  This created two additional nested "Retrospect" folders—each containing a zero-byte "Backup Media" file as well as a subsidiary "Media Set Blue" folder (the lowest-level folder containing a "1-Media Set Blue" folder)—under folder "One".  Finally I re-ran that same No Media Action Backup, which defaulted to Recycle because I'd again Recycled "Media Set Blue", far enough to see the copying phase was working—not to completion because that would have delayed my 3 a.m scheduled scripts using "Media Set Blue".  This whole experiment was not so uneventful, due to my difficulty in Removing "1-Media Set Blue" and my inadvertent creation of two additional nested "Retrospect" folders.  However it showed that,  if you Add carefully, you can have "Retrospect" folders under multiple folders that classify Members either by period of Media Set use—such as "2019 Backups" and "2020 Backups"—or by installation sub-organization—such as "Engineering Backups" or "Accounting Backups".