Jump to content

Nigel Smith

  • Posts

  • Joined

  • Last visited

  • Days Won


Posts posted by Nigel Smith

  1. 19 hours ago, phalmos said:

    Hard to say how much material changes on our master drive.

    You can guesstimate it if you've logged the used capacity of your source volume over the last few weeks -- that'll give you the amount of new data. Then the sum of all incremental backups over that time minus the amount of new will give an idea of your churn.

    But if you've got 40TB on the source, even only 0.5% churn a day plus some new stuff will fill your 80TB target -- especially since, as Lennart says, performance optimised grooming can leave a lot ungroomed.

    General recommendation for target volume capacity is 2-3 times the source capacity, assuming "regular" usage -- obviously you can get away with less if your data is mainly archival in nature and so doesn't change much. It might be a good idea to go through what was backed up each time (see Lennart's screenshots above) to see what's happening -- keep an eye out for things like database files (sounds like you have a lot of images, perhaps you are cataloguing them somehow?) where a small change will mean a multi-GB whole file backup.

  2. How big is your backup set? Retrospect has a maximum size for the "virtual media" used when doing a disk backup -- around 8.5TB IIRC. Once your backup set reaches that limit you'll be asked to "Choose Media" and RS will create a new folder for the next piece of "virtual media".

    The tricky bit is choosing the right place in the "Choose Media" dialog. On the volume you are storing your backups there will be a directory called "Retrospect" containing a directory for each of your disk backup sets. You need to choose the volume (or directory) that contains the "Retrospect" directory to add the next piece of "virtual media".

    I've a mounted volume called "Backups" containing the folder "Retrospect" which contains "2017" (the disk backup set) which, in turn, contains "1-2017", "2-2017", "3-2017"... So in the "Choose Media" dialog I need to select "Backups" -- RS does the rest.

    Another problem might be permissions. You say "local backup" -- is this directly attached storage or are one or more of the volumes concerned remote (eg mounted NAS volumes)?

  3. On 8/4/2021 at 12:00 PM, DavidHertzberg said:

    I don't believe a .rdb file qualifies as a database

    Trying not to quote Ralph Waldo Emerson, but you can't have it both ways -- either rhwalker is right and "All of the backup data (files, metadata, etc.) is in the .rdb files which, taken together, comprise Retrospect's database", or you are and they don't. I'll let you decide for yourself, and leave well alone until twickland reports back.

  4. On 7/29/2021 at 6:26 AM, DavidHertzberg said:

    Sorry to get you agitated, but I think you're missing the point...

    No, you are. Yes, the .rdb files are the database and yes, the catalog file is the index. But a component of the Retrospect Engine is a database engine because that's what manipulates the database. Is it the same on Mac and Windows versions of RS? We don't know, and even if the inputs and outputs are the same we still don't know -- it's a black box.

    To (rather sadly) quote myself:

    On 7/28/2021 at 8:19 PM, Nigel Smith said:

    I'd hope that, with the push to a "common engine", it's the same on all platforms. But that's hope, not certainty. And anyway, it's only an example of the possible differences between nominally common code compiled to different platforms.

    I'm sure the meat of the RS engine is more-or-less the same, but it's the gristle round the edges that always gets stuck in your teeth...

    You can see evidence of that last line for yourself by comparing Mac and Win release notes -- most fixes are in both, but a few are specific to one or the other.

  5. 17 hours ago, DavidHertzberg said:

    DB engines?  I'm glad it's a couple of hours since dinner; otherwise I'd get sick laughing.🤣

    Don't know why -- since a database engine is what CRUDs the data in a database, Retrospect has a database, and records in that database can be created, read, etc, there's some kind of DB engine in there. I'm not saying it's MySQL or similar, but there's something.

    I'd hope that, with the push to a "common engine", it's the same on all platforms. But that's hope, not certainty. And anyway, it's only an example of the possible differences between nominally common code compiled to different platforms.

    I'm sure the meat of the RS engine is more-or-less the same, but it's the gristle round the edges that always gets stuck in your teeth...

  6. 11 hours ago, DavidHertzberg said:

    I think that's adequate evidence for "Same actual code but compiled per platform", with "if macOS then mac_db_code else win_db_code" not being applied to all log messages.

    Apologies, I was less than clear (rushed, after my first attempt got eaten).

    I can see a lot of space in "same code but compiled for different platforms" for quite a lot of variation between the final products. The example was supposed to suggest they might have used different DB engines -- maybe for performance, licensing, or compatibility reasons. There could be lots of these variations, or very few -- I simply don't know.

    Your catalogs are as I'd expect, the meta holding "summary data" for the set and each client/volume (including links to their component catalogs) and the component containing the meat of that client/volume's backup history (metadata for every file backed up in every session, etc).

  7. On 7/22/2021 at 6:52 PM, DavidHertzberg said:

    IMHO you're over-estimating the current capabilities of the "meta catalog" for a Storage Group;

    think the "meta catalog" just stores the "higher level" info -- client/volume pairs in the SG, when each was last backed up -- and the "component catalogs" store the session info data. In Mac RS the search "chains" -- in the GUI you search the meta cat, but RS searches each of the component cats then aggregates the results for presentation. I'd hope that Win RS does the same and that the manual's "To restore from a specific source, you need to select the specific source within the Storage Group" is an "and you can restore from a specific source..." rather than "you can only...", but I can't test that at the moment.

    On 7/22/2021 at 6:52 PM, DavidHertzberg said:

    Assuming that's the "universal Retrospect engine", that still leaves an awful lot of wiggle room! Same design and pseudo-code? Same actual code but compiled per platform? Even if it's the latter ( as I suspect, comparing version fixes across platforms) that still leave space for things like "if macOS then mac_db_code else win_db_code".

    Not saying they have, but we can't discount the possibility!

  8. 7 hours ago, DavidHertzberg said:

    That article shows that, in Retrospect Windows, you can restrict Restore and Transfer (Retrospect Windows term for non-Finder-level Copy) and Verify to a particular Backup Set (Retrospect Windows term for Media Set) without using a Rule

    Agree about the Transfer and Verify (although Rules are hardly irksome), but I don't understand about Restore -- sessions are listed by Machine + Volume + Media Set, so it's trivial to do a point-in-time (or Browse and select particular files) for any client/volume pairing in any Storage Group. In Windows it's actually more clicking to get to previous snapshots/sessions.

    I've not tried it, but in Windows do you have to restore from a particular source? Can you easily search for every occurrence of "Important.docx" across a Storage Group, or would you have to search (and restore from) each "component set" individually?

    As you rightly say, it's potential in improving Rebuilds is massiveif it can be done.

    I'm not sure how similar Mac and Win RS actually are! There may be more to it than just how their respective consoles allow you to view what's under the hood. Creating Storage Groups by splitting a single over-arching catalog into a "meta catalog" plus a bunch of client/volume "component catalogs" was a very pragmatic solution to a pressing problem -- whether that could be done the same way on both platforms is something way outside my understanding.

    What we really need is a preference so we can switch both Mac and Win consoles to our preferred method of display 😉

  9. 16 hours ago, DavidHertzberg said:

    Sorry, Nigel Smith, but a Storage Group has one Catalog for itself plus one Catalog for each component Media Set.

    I know what RS does at the filesystem level. But in the UI, which is where I suspect most of us interact with RS most of the time, it is presented as "just another media set" on the Mac and as something different in Windows. Again, I prefer the Mac software's consistent approach -- you obviously don't. And I can't disagree with your preference since Storage Groups aren't "just another media set" (eg lacking cross-client dedup within the set).

    I'd prefer it even more if there was a consistent approach, one way or the other, across platforms -- but that's probably asking too much.

    16 hours ago, DavidHertzberg said:

    As for "no usable purpose", if twickland had been running Retrospect Windows 18 he might have been able to individually Rebuild his component Backup Sets (Retrospect Windows term for Media Sets)—and thus recover his existing Storage Group backup

    It may have changed in v18, but in v17 you had to rebuild the entire Storage Group on both platforms, even if only a single "component catalog" was damaged. A pain, but at least the rebuild is also parallelised across multiple threads. See this old post for my workarounds, either of which could possibly be adapted to twickland's particular problem.

  10. On 7/15/2021 at 4:18 AM, DavidHertzberg said:

    IMHO your opinion, Nigel Smith, is wrong in this case—a very rare situation.😮

    Trust me, it isn't!

    On 7/15/2021 at 4:18 AM, DavidHertzberg said:

    That's because a Storage Group is really a hack or kludge

    Agreed. But whether Mac or PC handling of them is correct (c'mon -- Mac wins every time!) is a matter of perspective. Looking top-down, a media set only has one catalog. A Storage Group only has one catalog. It looks like any other media set in the RS UI, and the manual even states it should be treated the same as any other media set. So it *shouldn't* show the component directories, since no other media sets do.

    When it comes to UX and abstraction, consistency trumps absolute accuracy every time. And why display something that shows no usable purpose?

    I don't know, and I don't really care, why Mac and Win do this differently. I know which I prefer, so do you, but those are only opinions and neither of us is "right".

    On 7/17/2021 at 10:57 PM, DavidHertzberg said:

    I defined it on the same destination volume, but there's no reason why I couldn't have defined that additional Member on another volume—permitting component Members on multiple volumes (which I previously thought couldn't be done for a Storage Group).

    While I would hope that you are right about this, I'd recommend that anyone who wants to use this in production thoroughly tests it first. There are limits on destination volume size -- my "use as much space as you want" Disk Media set splits every 8TB, for example -- and I'd want to be sure that the above allowed you to separate across "proper" logical/physical volumes rather create more RS "media set" volumes in the same place.

    Nice work on the password protection checking!

  11. On 7/13/2021 at 2:22 AM, DavidHertzberg said:

    Because of deficiencies (IMHO) in that GUI as opposed to the Retrospect Windows GUI, the Mac Console doesn't show the component Media Sets within a  Storage Group.

    Which just goes to show how tricky UI design can be because, IMHO, a Storage Group should be treated like any other media set -- those don't show internal client/volume pairs, so why should a Storage Group? Mac consistency, FTW!

    On 7/13/2021 at 2:22 AM, DavidHertzberg said:

    One use case for that would be your having many "client" machines that cannot be connected to the "backup server" at different times-of-day, such as a "virtual"  office using a VPN or—using Remote Backup—doing Work From Home.

    I think the generic use-case for parallelising backup clients is that "the server processes data faster than a single client supplies it". You can parallelise with multiple scripts, each with its own destination set, or with one or more scripts writing to one (or, less often, more) Storage Groups.

    My personal experience suggests that for "local" clients (those your RS Server can direct-IP or multi/subnet broadcast to wherever they are in the world), multiple scripts/sets is best. The benefits to me are organisational, de-dup within sets, faster catalog rebuilds when things go wrong (because you only have a subset of clients per media set), etc.

    But if you have "remote" or hybrid-working clients then Storage Groups are the way to go, mainly because (as at my last set of tests, anyway) there is no way to back up different remote clients to different media sets -- it's "one big bucket", hence the slower catalog rebuilds, although there's no de-dup (even across volumes of the same client). The only way to parallelise write operations to a single set is by using a Storage Group. And having multiple remote clients, which often have restricted upload speeds, are when parallelising on the server can really help you get all your backups done.

    As always, YMMV.

  12. Try the newest version of Retrospect. If you can, try v17 as well, on that same machine. The more relevant information you can send to Mayoff & team the better (your star sign probably isn't important at this point 🙂 )! You could also try a simple Finder Copy between the two images, just to check that works as expected.

    Certainly a Copy script should maintain the copied files' metadata, and it does on my tests with macOS 10.15.7, RSv17, and mounted encrypted disk images. But there's always a chance that something's been changed on the OS side, and it's just that you're seeing the effects in RS...

  13. Sounds like the Retrospect Engine is spontaneously quitting/restarting for some reason. Try stopping it, and turning off "Launch on System Startup", in the Retrospect pane of System Preferences. Restart the computer so you are working off a clean sheet.

    With Activity Monitor open, start the Retrospect Engine. Note the PID in Activity Monitor, go away for 10 minutes, come back and check again. If the PID is the same then the Engine hasn't quit, so try launching RS Console and seeing what happens.

    You may even find that switching to a manual start cures the problem -- IIRC there was someone last year who was seeing similar, and it appeared to be a timing issue where Engine was active before an external device and got into an endless restart loop when the device appeared.

    Failing that, keeping an eye out for the PID change (indicating a restart) which will give you a time period before that to check Console logs for problems.

    Try all the above in "standalone" mode -- disconnect any external devices bar keyboard/mouse/display, unplug the network and turn off wireless. It may be something totally unrelated, so keep it as simple as possible.

    It also looks like you didn't completely remove all preferences -- there are scheduled activities showing. A complete uninstall/re-install might help, if only for troubleshooting.  

    • Thanks 1
  14. 9 hours ago, DavidHertzberg said:

    If a user puts a "client" to sleep while a Scheduled script is backing it up, do you know if Wake-on-Lan will reawaken it if enabled for that "client" Source?

    Interesting question! I think not, because I'm guessing that WoL is part of the discovery phase only. Unless that's how they did the fix you mentioned way back when?

    And sleep is only for security if you've got "require password on wake" set. IMO it's more about energy saving, extending component life (not just disks) etc -- especially when moving laptops around (I get so annoyed when I see people walking up and down stairs at work with laptop open and running -- and also get a desperate urge to trip them up, just to see what happens!).

  15. 18 hours ago, Gintzler said:

    Have you guys ever gotten that a source is busy and nothing will kick it out of Busy mode?

    Client? Often-- eg when a backup is in progress and the user sleeps their machine, pulls the network plug, etc. Client remains "in use" so the server can't back it up. Either restart the client machine or (if your RS client permissions allow) turn the client off and on again(Option-click on the client "Off" button to fully kill the client process).

  16. On 2/10/2021 at 8:50 PM, DavidHertzberg said:

    I'm surprised at you! 🙄  Don't you realize that the whole idea of modern  (since low-cost large-capacity HDDs were introduced in the 2000s; previously it was that tape drives were expensive) client-server backup is based on the principle of not trusting ordinary users to be responsible for backing up their own machines? 



    Prepare to be picked up by the fearsome Retrospect Support Police and be taken to a backup administrator re-education camp.🤣

    In my defence, your honour...

    My personal preference is to lock things down because I don't trust ordinary users. But there are some who cause so much aggravation that, for the sake of my own sanity, they get the "This is why you shouldn't do that..." speech during which it is made clear that if they do do that then it is completely on their own head when they do it wrong. And I've got the email trail to prove it...

    Plus, in this case it's jethro who wants to choose his backup times -- and I'm sure he can be trusted to do it right! And if he doesn't and it all goes pants, I've got the forum screenshots to prove that it wasn't my fault, your worship 😉

  17. 5 hours ago, DavidHertzberg said:

    That enhancement isn't in Retrospect Mac 17, which means the only way you can currently do Remote Backups using a Mac "backup server" is with Proactive scripts

    Totally untested, but you might be able to spoof what jethro describes by having a 24hr/day Proactive script with a large "days between backups" setting -- RS wouldn't back up the client whenever that client contacted it, only when the user used "Backup Now". That setting would, of course, apply to all clients in that script, which would be all those tagged for Remote Backup.

    I'd question why you'd want to do that though! Far simpler to just set things up as normal for Remote Backups, and if you only wanted those to happen at certain user-decided times (perhaps they always want to wait until they've finished both their work and their evening's Netflix viewing before clogging their network connection with RS traffic) allow them to turn their client on and off as it suits them.

  18. 18 hours ago, AlexB said:

    I still am not getting multiples to work - when I do this (see second image), the Downloads folder now gets backed up, telling me the 2nd rule is not being applied (see third image for 2nd rule).

    If you can't merge multiple saved rules (I'll test later), try chaining them:

    Rule A --  Include "Saved rule includes All Files Except Cache Files", Exclude "Folder name is Temp"
    Rule B -- Include "Saved rule includes Rule A", Exclude "Folder name is Downloads"
    Rule C -- Include "Saved rule includes Rule B", Exclude "Folder name is .dropbox"

    Obviously each Exclude could contain as many clauses are you like.

    So a run using Rule C, above, would exclude ".dropbox", "Downloads", "Temp", and all cache files.

  19. Check by using the RS Console to "browse" the mounted volume. If you can, and given that your backups are working, you can consider it a spurious alert message.

    (Full Disk Access -- SystemPolicyAllFiles -- includes SystemPolicyRemovableVolumes so Engine and Instant should be OK.)

    My favourite quote about FDA/F&F is "Currently, this system is so complex that it appears unpredictable." I guess that extends to application's error messages, too 😉 

  20. Everything David says. And I'd add that:

    1. Most VPN servers don't allow "network discovery", either Bonjour (like you'd use to list available printers etc) or Retrospect's version, between subnets.
    2. Remote Backup is a lot more flexible in that the client wouldn't need to be on the VPN to be backed up. That also reduces the load on your VPN server, helping the people that need to use it.
    3. If the use of VPN is a requirement, eg compliance issues, you can actually use Remote Backup through, and only through, your VPN. Otherwise you'll have to open the appropriate ports on the server to the Internet (probably including port-forwarding on the WatchGuard).
    4. Most home connections are slow. Get initial backups done while the machines are directly connected to the "work" network, then use incrementals when the machine is out of the office. In your situation you could try a transfer from your current backups to a new, Storage Groups based, set (I've not tried this myself, so don't know if you can). RS will do this switch to incrementals seamlessly, just as it would usually.
    5. There's no deduplication across different volumes in Storage Groups, so you may have to allow for more space on your target media.
    6. Deffo upgrade to v17!
  21. 1 hour ago, boomcha said:

    I'm given RS a call to see what they say, and they are recommending upgrading to 17.

    That isn't exactly a surprise 🙂 But there's a few more things you can try first (I'd still recommend upgrading to v17 eventually, for better compatibility).

    1 hour ago, boomcha said:

    This also happens on external HFS+ drives on another client as well consistently as well. That one is running Sierra but same version

    So that is completely different -- an external drive rather than a system disk, HFS+ rather than APFS, Sierra rather than Mojave? Sounds like the only constants are the server and the client version, yet it isn't happening with every client...

    What are the commonalities/differences between the clients for which this happens and the ones that don't? Don't just look at software versions, but also what's installed and what is or isn't enabled eg FileVault, sleep options.

    Give the v15.5 client a go, even if you haven't a spare test machine. If it doesn't work you can simply uninstall it, re-install the v14 version, and re-register the machine with your server.

    And ultimately -- if it is always "first attempt fails, second attempt works" as you describe... Simply schedule a "primer" script to hit the troublesome systems before your real script 😉 You could even do it with a rule that excluded everything -- all files would be scanned, priming the target for the real script, but little or no space would be needed on your backup target.

  • Create New...