Jump to content

insont

Members
  • Content count

    26
  • Joined

  • Last visited

  • Days Won

    5

insont last won the day on March 9

insont had the most liked content!

Community Reputation

8 Neutral

About insont

  • Rank
    Member

Contact Methods

  • Website URL
    ursecta.com

Profile Information

  • Gender
    Male
  • Location
    Uppsala, Sweden

Recent Profile Visitors

253 profile views
  1. insont

    Why out of space?

    Good one. I'll remember that.
  2. insont

    Why out of space?

    Problem solved. It was all a matter of PBKAC. I kept raising the folder quota limit on the Synology, but completely missed that there is also a quota set on the user, and yes, that one was still set to 20 TB for the Retrospect server user. Now when I go to media set member and change the "use at most", Retrospect picks up the new volume size and it works. One thing remains, though, and that is that I'm still seeing two shares where there should only be one. But I'm going to ignore that...
  3. insont

    Why out of space?

    It's a small hassle, but my idea was that if I retire a machine, I can also move the corresponding media set offline. The important work documents are "copy backuped" to Wasabi (that's the cloud media set). You can see that in the two screenshots I included in my second post above. Interestingly, the first "Retrospect" share entry shows 5% of 28 TB, (1429 GB) which reflects the total space available on the Synology, while the second "Retrospect" share shows 5% of 20 TB (1024 GB) which is what Retrospect seems to limit itself to. That was, until a week ago or so, the actual limit on the Retrospect volume, which I then bumped up to 22 TB. So none of the two variants of the share Retrospect displays is correct. One reflects the full drive, not the actual share, the other reflects the size of the share a week or more ago. First of all: there is only one "Retrospect" share, but Retrospect sees two, and sometimes even three, shares with the same name but different sizes. That can't be right. Just to make sure it's just the one real share, I added a folder "test" to one of them and "test2" to the other. As you can see, both folders show up in both shares, so they must be one and the same. (I manually set them to 100% to show the total capacities Retrospect sees.) When I set up the media members, I did set those limits, so that's correct. I do think the basic problem must be that Retrospect is seeing the wrong total size of the volume, though. Or will Retrospect always stop 10% shy of the available space? Even if so, why two different shares and sizes?
  4. insont

    Why out of space?

    Yes, all these media set members are on the same volume. It's on a Synology where I restricted the Retrospect volume to 22 TB total. Currently, the media set members take 20 TB, and there's 2 TB free. Any reason I shouldn't have them on the same volume? So the question is, why doesn't Retrospect use the last 2 TB? And why don't all the other media sets show as out of space, only those six? (Those are the ones most recently backed up, so that could explain it.) In effect, Retrospect has stopped working 2 TB short of a full volume.
  5. insont

    Why out of space?

    I should add another oddity. When editing members, I get two instances of the Retrospect share, even though there is only one (verified in Finder and by checking /Volumes). I also get different suggested "use most" for them. Before restarting (as I mentioned above), I had three instances of the Retrospect share. When accessing members through the "add member" dialog box in "Repair", I can drill down and see that I actually see the same files with the same dates in all three "shares" (that was before I restarted and still saw three of them). BTW, this is all Retrospect 16.0.0, MacOS 10.14 on the server. /Martin
  6. I'm getting "waiting for media" errors and clearly a bunch of media sets are out of space. But it doesn't make sense to me, since if you look at the capacity, used, and free, it doesn't add up. All the media members are on the same Synology volume, which has 2 TB free space. All the media sets have a single media member each, all go to disk. One exception is the Cloud set. I tried repairing the top media set, succeeded but made no difference. I've restarted the Retrospect server software, and the Synology (followed by restart of Retrospect server software). I don't understand what's going on here. Anyone? /Martin
  7. Yes, APFS on both (it's well-neigh impossible to not have APFS on 10.14). About Instant Scan in settings: I don't agree. It's easy as can be to not show that label and button if it's not relevant. So your theory of how they'd be so exhausted after all these months not to be able to add like ten lines of code (at most, probably more like three) for that seems a bit fanciful. I can't believe it's not on purpose. Your whole idea that it took almost a year because it was so difficult is based on absolutely nothing. They might as well have been doing something else entirely. We don't know their priorities. OTOH, they had the same problem, i.e. a pointless label and button, in 15.6 when instant scan didn't do anything. Except then it actually consumed a lot of CPU doing nothing. So something's not right. No matter why, it seems to work now. And even though I think it took them way too long, I'm willing to forgive and forget since the result seems pretty good. So let's cut the whole discussion of why they did what they did, since we can't know. Unless you have some real insider information outside of guesswork.
  8. More news after some more testing and waiting for instant scan to catch up. The CPU usage of RetrospectInstantScan seems to be in the low single digits percentage every time I look at them. That's amazing. It used to be above a 100% of a core when it didn't actually work. I have a backup running right now on this machine (iMac 2014 i7) and while copying files, RetrospectClient uses less than 10% CPU in Activity monitor (up to 30% in iStat Menus, though). I've seen it peak to around 60-70% before, but maybe that was before the index was properly built (even though I suspect the index has nothing to do with speed during upload, but who knows). If, in fact, there is an index (see below). The scanning period has not gotten faster than in the tests I did earlier, so I'm unsure if there's actually an index from instant scan. I suspect there isn't one. When looking in /Library/Application Support/Retrospect/RtrISAExec.dir, there's nothing there and I think that's where the index data is supposed to be. There is, however, a retroISA_log.utx file modified at the latest two days ago, with a size of 9.6 MB. Inside, the last entries are from March 5, so I think it stopped updating when I installed the 16.0 client. This leads me to think instant scan is no more, but that full scan has been made so much more effective that it doesn't matter much. Or, maybe the instant scan is saving data elsewhere. Why there's still an instant scan daemon mystifies me, though. Unless it's the old daemon they forgot to stop and delete. Another great bonus: there's a new feature in "sources", namely "automatic update" of the client, which seems to work, at least it updated one of the machines for me. That will be a huge relief, since it means we won't have to run around updating machines, or fiddle with the remote update thing, which I always found obnoxious. It seems to just do it on its own. Latest logs for the two machines, "after8" is a 2013 Macbook Air over wifi (with around 150 Mb/s) and "razr" is the 2014 i7 iMac on wired Gb. All backups to a Synology 1517+. Backup server is a 2019 Mac Mini i7. When looking at the log for the Macbook Air, you'll see it slower than the iMac, but a bit faster than it was before. It's not as fast as I'd like it to be, but it's within the range of acceptable. Arq runs for 13-15 minutes on the same machine, but that's without validation. Arq is also set to only backup the home folder, while Retrospect is set to backup the entire disk (with some exceptions). Now, interestingly, on the iMac, Arq is really slow at times, taking more than two hours to do a backup of the home folder in some cases and 20 minutes in other cases. I really don't know why. So right now, Retrospect rules on the fast machine, while Arq rules on the slower laptop. In conclusion: I think, for the first time, I can set MacOS machines to hourly proactive backups without completely hogging the machines (maybe 2 hour intervals would be more reasonable). The performance now is on par with Arq, and probably close to what can be done on the platform. Arq is still somewhat quicker, but that is maybe at least in part because it defers validation to a separate and more infrequent session, while Retrospect does it every time (if set to verify). But Arq also has slowdowns that I don't know the reason for. Remains the question why "Instant Scan" is still a setting in the client. Weird. Another side note: it looks as if backups have started working for sleeping machines, albeit very slowly. But I haven't looked into that systematically. + Normal backup using Razr at 2019-03-09, 12:52:07 (Activity Thread 1) 2019-03-09 12:52:08: Finished scanning backup set data files To Backup Set Razr... - 2019-03-09 12:52:07: Copying Macintosh HD on razr 2019-03-09 13:04:02: Found: 3797143 files, 864033 folders, 596.3 GB 2019-03-09 13:04:52: Finished matching 2019-03-09 13:08:23: Selector "Martin's machines" was used to select 3 681 955 files out of 3 797 143. 2019-03-09 13:08:46: Copying: 2406 files (2.5 GB) and 0 hard links 2019-03-09 13:11:20: Building Snapshot... 2019-03-09 13:11:20: Checking 864 033 folders for ACLs or extended attributes 2019-03-09 13:14:22: Finished copying 84 113 folders with ACLs or extended attributes 2019-03-09 13:14:35: Copying Snapshot: 2 files (1.2 GB) 2019-03-09 13:15:14: Snapshot stored, 1.2 GB 2019-03-09 13:15:14: Comparing Macintosh HD on razr *File "Macintosh HD/usr/local/var/mongodb/diagnostic.data/metrics.2019-03-05T19-37-16Z-00000": different data size (set: 991 232, vol: 995 328) *File "Macintosh HD/usr/local/var/mongodb/diagnostic.data/metrics.interim": different data size (set: 2 948, vol: 3 345) 2019-03-09 13:17:14: Execution completed successfully Completed: 2 406 files, 2.5 GB Performance: 1 108.6 MB/minute (943.9 copy, 1 354.4 compare) Duration: 00:25:06 (00:20:24 idle/loading/preparing) + Normal backup using After8 at 2019-03-09, 14:04:31 (Activity Thread 1) 2019-03-09 14:04:31: Finished scanning backup set data files To Backup Set After8 Martin... - 2019-03-09 14:04:31: Copying Macintosh HD on after8 (2) 2019-03-09 14:20:40: Found: 2412938 files, 532691 folders, 340.7 GB 2019-03-09 14:20:59: Finished matching 2019-03-09 14:22:12: Selector "Martin's machines" was used to select 2 344 605 files out of 2 412 938. 2019-03-09 14:25:48: Copying: 322 files (715.7 MB) and 0 hard links 2019-03-09 14:27:49: Building Snapshot... 2019-03-09 14:27:49: Checking 532 691 folders for ACLs or extended attributes 2019-03-09 15:01:46: Finished copying 358 898 folders with ACLs or extended attributes 2019-03-09 15:01:54: Copying Snapshot: 2 files (255.3 MB) 2019-03-09 15:02:04: Snapshot stored, 255.3 MB 2019-03-09 15:02:04: Comparing Macintosh HD on after8 (2) *File "Macintosh HD/Applications/Junos Pulse.app/Contents/Plugins/JUNS/accessRecovery.ini": didn't compare *File "Macintosh HD/usr/local/var/postgres/server.log": different data size (set: 123 352 400, vol: 123 367 500) 2019-03-09 15:02:34: Execution completed successfully Completed: 320 files, 712.1 MB, with 52% compression Performance: 662.4 MB/minute (431.5 copy, 1 473.2 compare) Duration: 00:58:03 (00:55:54 idle/loading/preparing)
  9. Just ran a backup on a 2014 iMac with wired Gb network (the Macbook Air is on wifi), and things are decidedly looking more useable than before Retrospect 16. The scanning took just 2 minutes 11 minutes or so and the entire process 20 minutes. Retrospect is back in the race now. I still can't make out if it's using instant scan or not, but with this speed it must be. /Martin + Normal backup using Razr at 2019-03-06, 18:19:20 (Activity Thread 1) 2019-03-06 18:19:20: Finished scanning backup set data files To Backup Set Razr... - 2019-03-06 18:19:20: Copying Macintosh HD on razr 2019-03-06 18:30:18: Found: 3796060 files, 863974 folders, 596.4 GB 2019-03-06 18:30:50: Finished matching 2019-03-06 18:32:18: Selector "Martin's machines" was used to select 3 681 740 files out of 3 796 060. 2019-03-06 18:32:30: Copying: 2330 files (3.9 GB) and 0 hard links 2019-03-06 18:35:25: Building Snapshot... 2019-03-06 18:35:25: Checking 863 974 folders for ACLs or extended attributes 2019-03-06 18:36:42: Finished copying 83 900 folders with ACLs or extended attributes 2019-03-06 18:36:55: Copying Snapshot: 2 files (1.2 GB) 2019-03-06 18:37:28: Snapshot stored, 1.2 GB 2019-03-06 18:37:28: Comparing Macintosh HD on razr *File "Macintosh HD/Users/mw/Pictures/Photos Library.photoslibrary/private/com.apple.mediaanalysisd/MediaAnalysis/mediaanalysis.db-shm": didn't compare *File "Macintosh HD/Users/mw/Pictures/Photos Library.photoslibrary/private/com.apple.mediaanalysisd/MediaAnalysis/mediaanalysis.db-wal": different data size (set: 32 992, vol: 45 352) *File "Macintosh HD/usr/local/var/mongodb/diagnostic.data/metrics.2019-03-05T19-37-16Z-00000": different data size (set: 360 448, vol: 368 640) *File "Macintosh HD/usr/local/var/mongodb/diagnostic.data/metrics.interim": different data size (set: 3 457, vol: 2 913) 2019-03-06 18:39:34: Execution completed successfully Completed: 2 330 files, 3.9 GB Performance: 1 595 MB/minute (1 332.2 copy, 2 003.9 compare) Duration: 00:20:13 (00:15:14 idle/loading/preparing)
  10. I installed 16.0 yesterday and am test running it now on my 2013 MacBook Air with MacOS 10.14.3, APFS. There is a process "Retrospectinstantscan" running so that's good news. Retrospect did a full scan of almost 2,200,000 files, over 300 GB, in 20 minutes. No indication that it was using any index from instant scan, but I don't think you can expect that so early after installing it. Will have to see over the next few days if it starts using an index. The preference pane for the Retrospect client does say that Instant Scan is enabled. (But so it did for the 15.6 as well, even when it wasn't.) After 23 minutes, it started copying files. The total running time, including copying of almost 8 GB of files, was an hour and 17 minutes, as compared to over three hours in my September post (see above). The scanning phase went down from about 2.5 hours to 20 minutes. Yay! The client clearly uses less CPU now. It also backs off if something else uses significant CPU, so the machine is running much cooler, without the usual jet engine sounds. Nice. I'll be back in a couple of days to see if Retrospect actually catches up to the performance of competing-product-that-shall-not-be-named-here. It still has a way to go, but it's entirely possible that a few days of indexing can fix that. /Martin + Normal backup using After8 at 2019-03-06, 11:52:18 (Activity Thread 2) 2019-03-06 11:52:19 : Finished scanning backup set data files To Backup Set After8 Martin... - 2019-03-06 11:52:18 : Copying Macintosh HD on after8 (2) 2019-03-06 12:10:57 : Found: 2414406 files, 532724 folders, 337,8 GB 2019-03-06 12:11:14 : Finished matching 2019-03-06 12:12:16 : Selector "Martin's machines" was used to select 2 344 893 files out of 2 414 406. 2019-03-06 12:15:24 : Copying: 4216 files (7,9 GB) and 2 hard links 2019-03-06 12:32:42 : Building Snapshot... 2019-03-06 12:32:44 : Checking 532 724 folders for ACLs or extended attributes 2019-03-06 13:04:33 : Finished copying 359 066 folders with ACLs or extended attributes 2019-03-06 13:04:38 : Copying Snapshot: 2 files (255,4 MB) 2019-03-06 13:04:47 : Snapshot stored, 255,4 MB 2019-03-06 13:04:47 : Comparing Macintosh HD on after8 (2) *File "Macintosh HD/Applications/Junos Pulse.app/Contents/Plugins/JUNS/accessRecovery.ini": didn't compare *File "Macintosh HD/usr/local/var/postgres/server.log": different data size (set: 121 455 000, vol: 121 471 000) 2019-03-06 13:09:21 : Execution completed successfully Completed: 4 217 files, 7,9 GB, with 25% compression Performance: 762,5 MB/minute (483,3 copy, 1 811,4 compare) Duration: 01:17:03 (00:55:45 idle/loading/preparing)
  11. David, That was truly horrible. /Martin
  12. David, Yes, I know that Retrospect doesn't implement instant scan anymore, but they sure do scan. Not so instant, though. And yes, Arq doesn't do fsevent monitoring, but where the full scan on Retrospect is comparable to the full scan in Arq, there's a nine-fold difference in scanning speed. Nine. At least. And yes, Retrospect has said they're going to fix it, but after seeing no sign of that for eight months now, can I not be excused for getting really irritated? Especially since Arq (and other products) show that this problem is clearly solvable? And considering that Retrospect cost me around ten times the price of Arq, the price-performance ratio between the two is 1:100. I promise to stop complaining once Retrospect shows any sign of at least trying to do something about this. And so far, nothing. /Martin PS: my name really is "Martin". "Insont" is only a handle. I thought that was obvious by now.
  13. Meanwhile, Retrospect finished on the same machine. "Preparing to backup" took 2 hours and 54 minutes, and the actual backup of 5.5 GB took another 6 minutes for a grand total of exactly three hours. Arq did the same job in 21 minutes. The reason Retrospect found 5.5 GB to backup while Arq only backed up 890 MB is because Arq runs successfully every hour, while it can take days for my machine to be on long enough for Retrospect to get past the "Preparing to backup" to actually do a backup. But, as I already said, the huge difference is in the scanning, and both products scan about the same volume of data.
  14. I did a cursory comparison between Retrospect and Arq on my iMac just now. Retrospect is set to backup the whole disk, with some exceptions, and usually is in the scanning phase ("Preparing for backup") for hours, then doing the actual backup for a couple of minutes or longer, depending. This while running a CPU at above 100% (right now, for some reason, it's down at 10% during scanning). If I sleep the machine, the scanning is restarted from the beginning when I wake it up again. So if I don't use the machine for more than an hour at a time, Retrospect in effect never gets anything done. Except heating the room, that is. Note that this is a 1 TB Apple SSD with APFS, 4-core i7, a GBit net with copper and fiber, and the destination is a Synology 1517+ on the local net. Both Retrospect and Arq, in this comparison, back up to the same Synology. MacOS 10.14.3. Retrospect 15.6.1. Arq 5.15.2. (I also run several laptops, which I've mentioned before, but this time I'm on this iMac.) Arq, on the other hand, doesn't have a separate scanning phase; it scans and uploads at the same time. So it always gets something done right away. I have it set to only backup the home directory, but that is about 85% of the total size of data on the drive. While it was running, it took about 35% of a CPU, with no noticeable warming up. The whole scan and upload took 21 minutes, scanning 586 GB and uploading 890 MB. And, as I said, totally unobtrusive. By default it backs up hourly. Now, that's what Retrospect should do, as well. The separate scanning phase seems to be the fundamentally bad idea here. And it's excruciatingly slow to boot. Something else they should copy: Arq can use SFTP to the backup store. So even if you have a local NAS, you can backup without mapping the drive, which makes it way less likely that crypto malware destroys your backups as well. Yes, I know you can unmap network drives in Finder once you've set the credentials in Retrospect, but I've had that fail a couple of times. I don't like it. I figure it will be a lot less likely malware will see and be able to connect to an SFTP destination. Arq also has S3, so I can use both Retrospect and Arq with Wasabi. In summary: Arq is painless where Retrospect is painful. And Arq is safer, having SFTP. (I may have missed something in Retrospect, though.) Do note, however, that I'm currently anti-Retrospect biased, so I'm probably blind to the advantages of Retrospect over Arq right now. But that's humanity for you. /Martin
  15. Backing up my MacOS machines is still taking 3-5 hours every time, always with a heavy CPU load. And any interruption makes it start all over again, so in practice I have to leave the machines on all day just for Retrospect. I've reduced the backups to one day a week simply because I can't stand feeling (and hearing) all that hot air. Meanwhile I tried ArqBackup and it has none of these problems on the same machines. It blows through the backups in no time. It runs every hour with no noticeable CPU load. If a one man product like Arq can do this on APFS, why can't Retrospect? The performance difference is completely surreal. Arq seems to be very similar to Crashplan in this respect. Or rather, similar to practically anything except Retrospect. I'm starting to wonder if all that money I spent on three years of support on Retrospect wasn't wasted. It sure looks like they don't really care. No solution of any kind has appeared yet and this has been going on for quite a while now. If it wasn't for that investment, I'd have thrown out Retrospect by now. Sorry about that. I needed to vent. /Martin
×