More news after some more testing and waiting for instant scan to catch up.
The CPU usage of RetrospectInstantScan seems to be in the low single digits percentage every time I look at them. That's amazing. It used to be above a 100% of a core when it didn't actually work. I have a backup running right now on this machine (iMac 2014 i7) and while copying files, RetrospectClient uses less than 10% CPU in Activity monitor (up to 30% in iStat Menus, though). I've seen it peak to around 60-70% before, but maybe that was before the index was properly built (even though I suspect the index has nothing to do with speed during upload, but who knows). If, in fact, there is an index (see below).
The scanning period has not gotten faster than in the tests I did earlier, so I'm unsure if there's actually an index from instant scan. I suspect there isn't one. When looking in /Library/Application Support/Retrospect/RtrISAExec.dir, there's nothing there and I think that's where the index data is supposed to be. There is, however, a retroISA_log.utx file modified at the latest two days ago, with a size of 9.6 MB. Inside, the last entries are from March 5, so I think it stopped updating when I installed the 16.0 client. This leads me to think instant scan is no more, but that full scan has been made so much more effective that it doesn't matter much. Or, maybe the instant scan is saving data elsewhere. Why there's still an instant scan daemon mystifies me, though. Unless it's the old daemon they forgot to stop and delete.
Another great bonus: there's a new feature in "sources", namely "automatic update" of the client, which seems to work, at least it updated one of the machines for me. That will be a huge relief, since it means we won't have to run around updating machines, or fiddle with the remote update thing, which I always found obnoxious. It seems to just do it on its own.
Latest logs for the two machines, "after8" is a 2013 Macbook Air over wifi (with around 150 Mb/s) and "razr" is the 2014 i7 iMac on wired Gb. All backups to a Synology 1517+. Backup server is a 2019 Mac Mini i7.
When looking at the log for the Macbook Air, you'll see it slower than the iMac, but a bit faster than it was before. It's not as fast as I'd like it to be, but it's within the range of acceptable. Arq runs for 13-15 minutes on the same machine, but that's without validation. Arq is also set to only backup the home folder, while Retrospect is set to backup the entire disk (with some exceptions).
Now, interestingly, on the iMac, Arq is really slow at times, taking more than two hours to do a backup of the home folder in some cases and 20 minutes in other cases. I really don't know why. So right now, Retrospect rules on the fast machine, while Arq rules on the slower laptop.
In conclusion: I think, for the first time, I can set MacOS machines to hourly proactive backups without completely hogging the machines (maybe 2 hour intervals would be more reasonable). The performance now is on par with Arq, and probably close to what can be done on the platform. Arq is still somewhat quicker, but that is maybe at least in part because it defers validation to a separate and more infrequent session, while Retrospect does it every time (if set to verify). But Arq also has slowdowns that I don't know the reason for.
Remains the question why "Instant Scan" is still a setting in the client. Weird.
Another side note: it looks as if backups have started working for sleeping machines, albeit very slowly. But I haven't looked into that systematically.
+ Normal backup using Razr at 2019-03-09, 12:52:07 (Activity Thread 1)
2019-03-09 12:52:08: Finished scanning backup set data files
To Backup Set Razr...
- 2019-03-09 12:52:07: Copying Macintosh HD on razr
2019-03-09 13:04:02: Found: 3797143 files, 864033 folders, 596.3 GB
2019-03-09 13:04:52: Finished matching
2019-03-09 13:08:23: Selector "Martin's machines" was used to select 3 681 955 files out of 3 797 143.
2019-03-09 13:08:46: Copying: 2406 files (2.5 GB) and 0 hard links
2019-03-09 13:11:20: Building Snapshot...
2019-03-09 13:11:20: Checking 864 033 folders for ACLs or extended attributes
2019-03-09 13:14:22: Finished copying 84 113 folders with ACLs or extended attributes
2019-03-09 13:14:35: Copying Snapshot: 2 files (1.2 GB)
2019-03-09 13:15:14: Snapshot stored, 1.2 GB
2019-03-09 13:15:14: Comparing Macintosh HD on razr
*File "Macintosh HD/usr/local/var/mongodb/diagnostic.data/metrics.2019-03-05T19-37-16Z-00000": different data size (set: 991 232, vol: 995 328)
*File "Macintosh HD/usr/local/var/mongodb/diagnostic.data/metrics.interim": different data size (set: 2 948, vol: 3 345)
2019-03-09 13:17:14: Execution completed successfully
Completed: 2 406 files, 2.5 GB
Performance: 1 108.6 MB/minute (943.9 copy, 1 354.4 compare)
Duration: 00:25:06 (00:20:24 idle/loading/preparing)
+ Normal backup using After8 at 2019-03-09, 14:04:31 (Activity Thread 1)
2019-03-09 14:04:31: Finished scanning backup set data files
To Backup Set After8 Martin...
- 2019-03-09 14:04:31: Copying Macintosh HD on after8 (2)
2019-03-09 14:20:40: Found: 2412938 files, 532691 folders, 340.7 GB
2019-03-09 14:20:59: Finished matching
2019-03-09 14:22:12: Selector "Martin's machines" was used to select 2 344 605 files out of 2 412 938.
2019-03-09 14:25:48: Copying: 322 files (715.7 MB) and 0 hard links
2019-03-09 14:27:49: Building Snapshot...
2019-03-09 14:27:49: Checking 532 691 folders for ACLs or extended attributes
2019-03-09 15:01:46: Finished copying 358 898 folders with ACLs or extended attributes
2019-03-09 15:01:54: Copying Snapshot: 2 files (255.3 MB)
2019-03-09 15:02:04: Snapshot stored, 255.3 MB
2019-03-09 15:02:04: Comparing Macintosh HD on after8 (2)
*File "Macintosh HD/Applications/Junos Pulse.app/Contents/Plugins/JUNS/accessRecovery.ini": didn't compare
*File "Macintosh HD/usr/local/var/postgres/server.log": different data size (set: 123 352 400, vol: 123 367 500)
2019-03-09 15:02:34: Execution completed successfully
Completed: 320 files, 712.1 MB, with 52% compression
Performance: 662.4 MB/minute (431.5 copy, 1 473.2 compare)
Duration: 00:58:03 (00:55:54 idle/loading/preparing)