Backup script no longer backs up - how can i 'debug' without actually running the script?

[x509]

I have a backup script "Media" that backs up a one folder on Drive D (an SSD) and my entire Drive E (a regular HDD) which is for bulk storage of photo files.  The last date that this script actually backed up Drive E was 10-18, but I have been adding files every day since, including today. 10-21.  Nothing on my system has changed, except I upgraded from Retrospect 12 to Retrospect 15 on 10-16.

I can't seem to find a way to sort of 'interactively debug' this script, by seeing which new files it would back up.  This script uses a selector, but when I 'verify' the selector, the result shows all the files on the entire drive that could be backed up, not just the files that need to be backed up now.  I was hoping that if I could 'debug' interactively, I might be able to pinpoint the problem in this script.  Trying to debug this script by actually running it is very inconvenient, so I was hoping there is a better, easier way.

My system is Win 10 Pro 64, 32 GB RAM, fully patched up.  If it matters, I used Symantec/Norton Internet Security, provided by my ISP Comcast.

 

x509

[Lennart_T]

In the log it says how many files there are and how many that were selected by your selector. Is there any (significant) difference in the number of files?

The files on E that were not backed up, are they completely new files? Or are they files that were copied from D to E after being backed up from D?

You could try a "Restore Files and Folders" and see if you could restore one of those files.

[x509]

Lennart,

I physically could not work on this issue until today, because of the problems I have had with the Dashboard (separate post ...)

So to try to show everyone the actual problem with this script not working, I'm going to post a bunch of screenshots.  This screenshot shows the last time the backup actually worked, on 18 October.  This folder tree is my digital photo collection.  I use Adobe Lightroom, which moves photos from the camera's memory card into folders on Drive E: based on the date of photos.  This screenshot shows the folders that Retrospect has backed up and could restore.

 

This screenshot shows the actual E:\Photos folder structure.  Note that I have added photos on 19, 20, and 21 October.

Now I'm going to show portions of the execution log for my MEDIA backup script.  This script is from 18 October.  65 new files were backed up.  This execution worked as expected.

Now for 19 October, no files were backed up, even though the 2018/10/19 folder contains 134 files.  Note also above that Retrospect doesn't have these files backed up, so it can't offer to restore them for me.

The 2018/10/20 folder contains 95 files, but none were backed up in the 2018-10-20 backup operation.

And the 2018/10/21 folder contains 418 files, which also were not backed up.

Also, I configure Lightroom to create XMP files as an extra form of backup.  If I apply keywords, adjust the GPS data, crop an image, or do other editing, that information is always written out to the XMP file.  So that means that even if I don't add new photos to the E:\Photos directory and subdirectoryies if I do any work in Lightroom, some files need to be backed up.

I can't figure out why this script stopped working, because I haven't made any changes to the script, including the various Selectors I use for each backup operation within the script.  I hope this post is clear.  If it is not, just tell me and I'll respond to questions as soon as possible.  Now that I (think I) have reconfigured Retrospect to bypass the Dashboard, I don't have to wait for days before being able to respond.

x509

 

[Lennart_T]

It is interesting to note that on the 18th, Retrospect found 55150 files.

On the next day, Retrospect found only 55104 files.

This leads me to believe there are problems with the disk's directory (the E: disk). Try repairing the disk.

https://windowsinstructed.com/run-chkdsk-windows/

[x509]

Lennart,

I did run a chkdsk operation (from the CMD window, I'm old fashioned that way.)  No problems.

However, there is actually a very good explanation.  When I tried to do a test restore for the folder 2018/10/18, Retrospect showed 65 files available for restore.  However, I just checked that folder on Drive E: and there are currently only 18 photo files + 18 XMP files created by Outlook.  There is actually a simple explanation for that change.  As part of the editing process in Adobe Lightroom, I delete unwanted photos.  So some of that difference can be explained by edits I did to the folder 2018/10/18 on the next day.  Of course, I might have been editing photos taken on earlier days.

I should explain that the way Outlook works, it does not import the actual photos into its internal database but just catalogs them, so it can record edits, metadata, etc.

It just occurred to me that this problem appeared just around the time that I upgraded to 15.6.  Should I submit a trouble ticket with Retro support?  Or are there tools in Retro that I just don't know about that might help me to debug this problem myself?  Right now, I'm doing "emergency" backups of Drive E: to a spare external hard disk, but that's not a viable solution for me long-term.

 

x509

[Lennart_T]

1 hour ago, x509 said:

Lennart,

I did run a chkdsk operation (from the CMD window, I'm old fashioned that way.)  No problems.

However, there is actually a very good explanation.  When I tried to do a test restore for the folder 2018/10/18, Retrospect showed 65 files available for restore.  However, I just checked that folder on Drive E: and there are currently only 18 photo files + 18 XMP files created by Outlook.  There is actually a simple explanation for that change.  As part of the editing process in Adobe Lightroom, I delete unwanted photos.  So some of that difference can be explained by edits I did to the folder 2018/10/18 on the next day.  Of course, I might have been editing photos taken on earlier days.

I should explain that the way Outlook works, it does not import the actual photos into its internal database but just catalogs them, so it can record edits, metadata, etc.

It just occurred to me that this problem appeared just around the time that I upgraded to 15.6.  Should I submit a trouble ticket with Retro support?  Or are there tools in Retro that I just don't know about that might help me to debug this problem myself?  Right now, I'm doing "emergency" backups of Drive E: to a spare external hard disk, but that's not a viable solution for me long-term.

 

x509

I'm out of ideas, so contacting Retrospect support would probably be a good idea.  

[x509]

Lennart,

Re-reading my post above I wrote “Outlook” twice when I meant “Lightroom.”   Sorry for any confusion. 

Checking my software install records this problem started once I installed Retrospect 15.6. Next stop for me is the support page. 

Edited October 28, 2018 by x509
spelling fix

[DavidHertzberg]

5 hours ago, x509 said:

Lennart,

Re-reading my post above I wrote “Outlook” twice when I meant “Lightroom.”   Sorry for any confusion. 

Checking my software install records this problem started once I installed Retrospect 15.6. Next stop for me is the support page. 

x509,

It sounds to me as if this problem, too, is a result of Retrospect Windows 15.6 being a bad release.  (And no, telling it "bad, bad release" won't get it to behave—as it might if 15.6 were a canine instead of a piece of software. )  After you post a Support Request, I'd suggest e-mailing Brian Dunagan again.

[x509]

1 hour ago, DavidHertzberg said:

x509,

It sounds to me as if this problem, too, is a result of Retrospect Windows 15.6 being a bad release.  (And no, telling it "bad, bad release" won't get it to behave—as it might if 15.6 were a canine instead of a piece of software. )  After you post a Support Request, I'd suggest e-mailing Brian Dunagan again.

I certainly will.  I could also point out that 30 days (!) of free support for a complex piece of software is not enough.  I'm guessing that most companies "pay the man" for support.  For me, that would be ridiculous. 

 

x509

[Scillonian]

x509

Do you have InstantScan running on the drives in question?

I have encountered situations before where, although InstantScan is active and reports no detectable errors, the volume databases stops updating and so as far as Retrospect is concerned no new files have been added files have been added, changed, or deleted on the volume. When a file is added, changed or deleted the volume database should be updated fairly soon afterwards so if the file time stamp of the volume database is not updating then InstantScan is not updating.

The InstantScan volume databases are in the C:\ProgramData\RetroISA\RetroISAScans folder. To reset stop the InstantScan services, delete the volume databases, the restart the InstantScan services.

Edited October 29, 2018 by Scillonian

[x509]

Guys,

Sorry for going silent on this thread. 

I do have Instant Scan on. 

I'm working on some suggestions from Retrospect support.  As soon as I have something definite, I will report it here.  So far, if I run an immediate backup, the backup works properly.

x509