Jump to content
insont

No more instant scan on MacOS?

Recommended Posts

I'm backing up a couple of Macs and Retrospect seems to place an inordinate load on them. My Macbook Pro 17", for instance, is hyperventilating several hours per day even though I hardly use it. I had Crashplan until they cancelled the service, and even though they backed up every 15 minutes, it was totally inconspicious and effective. Retrospect is a steamroller compared to that. Admittedly, Retrospect backs up more files, but still.

Every now and then backups went smoother, and that was when Instant Scan did its work (as verified in the logs), but most of the time Instant Scan simply bogs down the machines but doesn't actually work.

I had a week-long exchange with Retrospect support about this, trying to figure out why Instant Scan is totally ineffective, but once they escalated, I was told Instant Scan is no more. Not that it's having problems, no, that it has been abandoned. Does that mean my Mac backups will continue to be so grossly overloading the machines? I'll quote verbatim the last message I did get from Retrospect below. To me it sounds like Retrospect on MacOS will forever remain a dog.

(I should mention that Instant Scan already didn't work before I upgraded to 10.13 and APFS. I also assume Retrospect support meant to say "APFS" and not "AFP", which makes no sense in relation to 10.13. And makes no sense in any case on the local machine where Instant Scan works. Or maybe they actually did mean AFP and then I understand even less.)

"Dear Martin, 
Retrospect Engineering Reply can be found below. 
Instant scan doesn't work on AFP, and most likely customer is using AFP if they are on OSX 10.13.
We recommend disabling instant scan in ALL cases, we are disabling instant scan by default in future versions of Retrospect.
Thank you for using Retrospect,
The Retrospect Support Team "

Share this post


Link to post
Share on other sites

insont,

The only 2018 information I could find about this is this thread on the Apple Developer Forums.

Forgive me for asking this, but I notice your profile contains no information about you,  that you joined the Forums on 30 June, and that your profile says "The recent visitors block is disabled and is not being shown to other users."  Is what you are saying you heard from Retrospect Engineering true, or is it "fake news".

Share this post


Link to post
Share on other sites

David,

I copied verbatim what Retrospect support sent me. How can I prove that to you?

And yes, I created this login specifically to post about this, since the reply from Retrospect upset me. I didn't expect, however, to be accused of lying in the process. About "recent visitors block is disabled" I know nothing. I haven't touched my profile from the default. I'm sorry if I broke some kind of rule by not doing that.

BTW, since you doubt my veracity, I'd suggest you ask Retrospect support if this is true. Either they confirm what I just said, making me somewhat happy, or they backtrack and say they are going to support Instant Scan in the future or that I misunderstood, and I'm even happier. To me their reply seems not quite thought through and not good for the product, so you asking could make them think again.

Oh, and the case is nb 00061949 that contains this exchange, including the message I quoted. You're welcome to reference that.

Martin

Edited by insont
wrote "forum" instead of "support"

Share this post


Link to post
Share on other sites
6 hours ago, insont said:

David,

....

....

BTW, since you doubt my veracity, I'd suggest you ask Retrospect support if this is true. Either they confirm what I just said, making me somewhat happy, or they backtrack and say they are going to support Instant Scan in the future or that I misunderstood, and I'm even happier. To me their reply seems not quite thought through and not good for the product, so you asking could make them think again.

Oh, and the case is nb 00061949 that contains this exchange, including the message I quoted. You're welcome to reference that.

Martin

insont/martin,

I doubt your veracity because:  [1] No other threads, other than the ones you have posted in, have reported a problem with APFS and Instant Scan for Retrospect Mac except for this one back in October 2017.  [2] Retrospect Engineering has lately given fair warning about planned feature changes that would affect some administrators adversely; they put yellow-flagged Notes about two such changes (once of which has already been expanded into a Knowledge Base article) in the cumulative Release Notes for Retrospect Mac 15.1 (and one of those was already in there for Retrospect Mac 15.0)—so I find it a bit difficult to believe that Retrospect Engineering (which IME communicates with customers only through a Support Engineer) would announce such an adverse change directly to you.

[2] is not impossible to believe; the next-to-last substantive paragraph in this post suggests that IMHO the "real" engineers are being given unprecedented freedom in documentation.  Also, the Apple Developer Forums thread linked to in my preceding  post indicates there is a problem with the APFS change in "normalization" of file names.  Finally, this would be such a massively adverse change that Retrospect Product Management may have "chickened out" on announcing it, which would explain why you "had a week-long exchange with Retrospect support" and why this Knowledge Base article has not  been updated.

As a mere customer, I am not allowed to view Support Cases other than my own.  However I will file a Support Case, referencing this thread and Support Case 61949—which Retrospect Support has the capability of viewing.

 

Share this post


Link to post
Share on other sites

David,

Retrospect Engineering didn't reply directly to me; it was forwarded by the support engineer. I thought that was obvious from the quote where they mention me in third person. I just picked up the whole support thread from Retrospect Support and copy/paste it below. It's completely unedited, I left all references in. Start reading from the bottom.

BTW, I have the problem on several machines, but I concentrated on one for the support case.

 

Quote

June 28, 2018 09:24


Dear Martin, 

Retrospect Engineering Reply can be found below. 

Instant scan doesn't work on AFP, and most likely customer is using AFP if they are on OSX 10.13.

We recommend disabling instant scan in ALL cases, we are disabling instant scan by default in future versions of Retrospect.


Thank you for using Retrospect,
The Retrospect Support Team 

---------------------------

Click the link below to update your support case. Note, the case is automatically closed if no response is received within 5 days). 

* Case # 00061949 - http://retrospect.com/support/case/5000P00000iaXqm
ref:_00DU0Kyj8._5000PiaXqm:ref
Attachment

June 25, 2018 19:41


Attached 608 KB file: RetroISA.zip
Comment

June 25, 2018 19:41


Including the requested files.
Cube

June 25, 2018 07:26


Dear Martin, 

Retrospect Agent Reply can be found below. 


Agent Response: 

Please send us your  retroISA_log.utx

This files can be found in: Library/Application Support/Retrospect/ and provide the Mac System Report – https://support.apple.com/en-us/HT203001

Thank you for using Retrospect,
The Retrospect Support Team 

---------------------------

Click the link below to update your support case. Note, the case is automatically closed if no response is received within 5 days). 

* Case # 00061949 - http://retrospect.com/support/case/5000P00000iaXqm
ref:_00DU0Kyj8._5000PiaXqm:ref
Comment

June 22, 2018 15:58


Hi,

I did as you suggested, removed the ISA Scan files, updated server and client and restarted Instant Scan. I verified it runs by using launchctl load. Now nothing happens. I waited a couple of days but the database isn't recreated. Yesterday I removed even the folder RetroISAScans, but even the folder does not recreate. The Instant Scan process consumes very little CPU. I'm used to it consuming more than one core, but now it's down under 1%. It looks as if it is doing hardly anything.
Backups work, but scan the whole file system, taking forever (three hours or more). I would have thought the Instant scan database would recreate in a day or two, but it doesn't look like it.

Any more tips?

Regards,

Martin
Cube

June 19, 2018 08:32


Dear Martin, 

Retrospect Agent Reply can be found below. 


Agent Response: 

stops Instant Scan, it might be an issue with Instant Scan’s internal data files.

Locate Instant Scan’s folder in /Library/Application Support/Retrospect/RetroISAScans and remove the files starting with RetroISAScan.
 
Start Instant Scan and wait for the first initial scan to finish.

Update Retrospect Server to 15.1.2.101 https://s3.amazonaws.com/download.retrospect.com/software/mac/v1512/Retrospect_15_1_2_101.dmg and all your clients to 15.1.0.x

Thank you for using Retrospect,
The Retrospect Support Team 

---------------------------

Click the link below to update your support case. Note, the case is automatically closed if no response is received within 5 days). 

* Case # 00061949 - http://retrospect.com/support/case/5000P00000iaXqm
ref:_00DU0Kyj8._5000PiaXqm:ref
Attachment

June 18, 2018 16:11


Attached 1.03 MB file: Archive.zip
Comment

June 18, 2018 16:11


Hi,

I'm uploading the files. The launchctl command told me the daemon was already running.
Cube

June 18, 2018 09:51


Dear Martin, 

Thank you for contacting Retrospect support.  Retrospect Agent Reply can be found below.

We understand you have the following question:

Instant Scan seems not work on one client

Agent Response: 

Please send me your Operations_log.utx file  located in  Library/Application Support/Retrospect/

On the Client computer send me the files retroclient.log and retropds.log

and Library/Application Support/Retrospect/retroisa.utx

Also on the client check Instant Scan is well started: sudo launchctl load /Library/LaunchDaemons/com.retrospect.retroisa.plist

Thank you for using Retrospect.

The Retrospect Support Team 

Additional Resources: 
Have you tried our online self-service tools? 
* Knowledgebase - http://retrospect.com/kb
* Forum - http://forums.retrospect.com
* Training Videos - http://www.youtube.com/user/RetrospectInc/featured?hl=en


-----------------------------------------
Click the link below to update your support case. (Note, the case is automatically closed if no response is received within 5 days). 

* Case # 00061949 - http://retrospect.com/support/case/5000P00000iaXqm 

------------------------------------------

Your Problem Description.: 
Instand scan doesn't seem to work. On one machine, a MacBook Pro 17", the entire proactive backup takes hours, I've had days it's been six or nine hours, to complete, even though this machine sees very light use, mostly emailing. As far as I can see, Retrospect always scans the entire drive, never using Instant Scan results. Instant Scan is on in preferences.
I've had this problem even before v15, but had hoped v15 would have cleared it up, but it didn't. Client is also updated.
I have no idea where to start troubleshooting this.
ref:_00DU0Kyj8._5000PiaXqm:ref
Comment

June 17, 2018 09:57


Instand scan doesn't seem to work. On one machine, a MacBook Pro 17", the entire proactive backup takes hours, I've had days it's been six or nine hours, to complete, even though this machine sees very light use, mostly emailing. As far as I can see, Retrospect always scans the entire drive, never using Instant Scan results. Instant Scan is on in preferences.
I've had this problem even before v15, but had hoped v15 would have cleared it up, but it didn't. Client is also updated.
I have no idea where to start troubleshooting this.

 

Share this post


Link to post
Share on other sites

The OP in the 21 May Apple Developer Forums thread I linked to in this previous post used the "handle" "suyashdruva", so I just did a little Googling for "suyash druva".  It appears that his name is Suyash Singh, and he works for a company named Druva.  And a look at an overview video on Druva.com confirms they have product(s) that do AWS cloud backups including for  "endpoints".  Reading some of the references in this section of the Wikipedia "Backup" article has made me aware that "endpoint" is industry slang for end user devices such as PCs, laptops and mobile phones.  And that would include Macs, so the APFS problem insont/martin reports sounds as if it's Mac-backup-industry-wide and not just limited to Retrospect.

So "Walnut Creek, we have a problem"; sorry for my initial skepticism, Martin.

BTW, even the newest of my 3 Macs is on macOS 10.12.6, and they use good old HFS+.  So I'll let Instant Scan continue to shorten my daily incremental backups of one Mac by 8-10 minutes.

Share this post


Link to post
Share on other sites

David,

Nice detective work. Yes, what you found seems very likely to be the explanation. As long as that API problem persists, Retrospect can't do all that much about it. (Except if it's only a problem of identifying file deletions I would expect it not to be that disastrous. If the agent tries to back up a deleted file that is still in the scan index, it could just skip it. At least that's what I would assume. The inserts and updates seem to work right, so... but there could be more to it than that, of course.)

Let's hope Apple gets its act together, though, and get this fixed.

Share this post


Link to post
Share on other sites

suyashdruva says in the last 21 May 2018 post in that Apple Developer Forums thread "I observe case and normalization are preserved for operation other than delete. So why behavior is different for file/folder delete event? It seems while accumulating events, paths are getting normalized except this case."

CrashPlan had (and evidently still has) a File System Scan that was scheduled to run by default at 3 a.m., because it "Requires more resources".  CrashPlan used to use Spotlight instead of FSEvents—which Retrospect uses, so that section of the page used to say "Spotlight does not report deleted files in real-time, so CrashPlan only detects deleted files during the scheduled scan".  Maybe Retrospect will have to adopt the same approach.

P.S.: insont got in his immediately-preceding post while I was researching the second paragraph for this one.  I think what I've just said still is applicable.

P.P.S.: For the masochists and/or frustrated systems programmers among you, here's an applicable blog post—with comments—on APFS filename normalization.

Share this post


Link to post
Share on other sites

The latest Retrospect doesn't seem to implement Instant Scan either. I can't even enable it anymore. I have to leave laptops on for hours just to get anything backed up at all. See this log from today, from a Macbook Air where the scanning and snapshot took three hours, then the actual copy around two minutes. Tomorrow, it's going to scan for three hours again... unless I forget and close it, which will make the whole thing repeat. It's not reasonable to consume three hours just for file scanning for every backup on a relatively small computer. (And to think Crashplan used to run backups every 15 minutes, each one taking seconds. Without significantly loading the CPU.)

 

 +  Normal backup using After8 at 2018-09-20, 17:36:11  (Activity Thread 2)
    2018-09-20 17:36:12 : Finished scanning backup set data files
    To Backup Set After8 Martin...
    -  2018-09-20 17:36:11 : Copying Macintosh HD on after8 (2)
    2018-09-20 19:59:26 : Found: 2268745 files, 497287 folders, 316,4 GB
    2018-09-20 20:00:04 : Finished matching
    2018-09-20 20:01:50 : Selector "Martin's machines" was used to select 2 225 329 files out of 2 268 745.
    2018-09-20 20:07:04 : Copying: 935 files (709,7 MB) and 0 hard links
    2018-09-20 20:09:20 : Building Snapshot...
    2018-09-20 20:09:21 : Checking 497 287 folders for ACLs or extended attributes
    2018-09-20 20:40:57 : Finished copying 336 384 folders with ACLs or extended attributes
    2018-09-20 20:41:11 : Copying Snapshot: 2 files (239,6 MB)
    2018-09-20 20:41:26 : Snapshot stored, 239,6 MB
    2018-09-20 20:41:26 : Comparing Macintosh HD on after8 (2)
...
    2018-09-20 20:42:07 : Execution completed successfully
    Completed: 935 files, 705,2 MB, with 52% compression
    Performance: 549,5 MB/minute (371,1 copy, 1 057,8 compare)
    Duration: 03:05:55 (03:03:21 idle/loading/preparing)

 

/Martin

Share this post


Link to post
Share on other sites

Well, I did write support to ask if they had plans to fix Instant Scan in the future or if they'd given up on MacOS, and got some encouraging news:

Quote

In our next full release of Retrospect the client scan APIs will be completely overhauled. For local backups you can help speed things up by changing your current api settings by following the instructions in this article: https://www.retrospect.com/en/support/kb/macos_api_request_impossible This is only for LOCAL BACKUPS for the time being. Client API changes are being worked on and preliminary testing shows them being upwards of 10 times faster than they are currently. Thank you for using Retrospect.

/Martin

Share this post


Link to post
Share on other sites
On 10/6/2018 at 6:03 AM, insont said:

Well, I did write support to ask if they had plans to fix Instant Scan in the future or if they'd given up on MacOS, and got some encouraging news:

/Martin

insont,

That's good, because I just found a 15 May 2018 Knowledge Base article change confirming what you've been saying.  The  paragraph

"Mac Customers: Please note that Instant Scan is not supported with APFS."

has been added below the first paragraph in the KB article "Instant Scan Frequently Asked Questions". which is under the catch-all heading "Resources" in the KB .  That first paragraph  begins with the sentence "Retrospect 10 for Macintosh and Retrospect 8 for Windows introduced a new feature called Instant Scan."  This section of the permalinked old version of the Wikipedia article says those versions were introduced in 2012, so the original article almost certainly dates from sometime shortly after that year up through April 2015—when the companion article "Instant Scan Advanced Options" was published.

Sorry to have previously expressed doubt, Martin.  Pretty sneaky announcement there, Retrospect Inc. :rolleyes:; you didn't even point Martin to it in your final reply to his Support Case.

Share this post


Link to post
Share on other sites

I got that feedback from support recently as well and reported it in another post here.

Frankly, when using SSD's in clients, I don't see a huge advantage to ISA. Whether in incremental or full backup. It is a CPU hog and space hog. Maybe also duplicates processes already in place (in the case of APFS). I will compare again in 15.6 with ISA disabled to confirm this. Once turned off, intend to leave it that way. Locally, on HFS+ formatted (HDD) drives, maybe some advantage, but since the server does its thing quicker locally and is essentially unattended, I don't care too much. 

My 2¢...

 

Henry

Share this post


Link to post
Share on other sites

Henry,

Well, I'm certainly seeing a difference. Going from 2 minutes on a Windows machine to 2 hours or more on a Mac with a built-in SSD, with comparable backup sizes. That's two hours spent in "preparing for backup". So let's hope that is due to (the lack of) Instant Scan, else it's beyond useless, if it never gets fixed.

Possibly this only happens to some people, of course, but it's happening on all my Macs on APFS (four of them), and not on virtual and real Windows machines (three of them). Am I really the only one seeing this?

/Martin

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×