Why is my duplication script trying to erase all my files just because I have moved the destination?

[haggis999]

I was forced to rebuild my primary Synology NAS after a recent disk crash and had to copy all the data from a backup Synology NAS. For various reasons, I then chose to reorganise the top-level shared folder structure on the primary NAS, with the result that my digital photos are now stored in a new location.

When I tried to run my previous Retrospect 12 script to duplicate the Pictures folder on my Windows PC to the NAS (after modifying the script to use the new location), it started to delete a large number of photos from the NAS. I stopped this after it it had deleted 153GB of the images, out of a total of 470GB (the deleted photos are still in the Synology recycle bin).

Given that the PC and NAS versions of my Pictures folder were almost, if not totally identical, such behaviour was very puzzling. Why might Retrospect be wanting to replace all the images?

For reasons I don't really understand, there is a small difference in the time stamps for all the deleted files I have just checked, but I doubt if this has anything to do with the recent reorganisation on my NAS. For example, one of the deleted images has a time stamp of  1 Sep 2014, 15:31 on the PC and 1 Sep 2014, 15:37 on the NAS. I suspect that this has always been the case, and if so, it never previously interfered with the Retrospect duplication process.

[haggis999]

I have just noticed a recent thread called 'Duplicate always erases all files first', which has a reply from Scillonian that I thought might also apply in my case.

When I rebuilt my NAS, I created a special Synology version of RAID called SHR and took the option to use the Btrfs file system. Given Scillonian's comments in the earlier thread, I tried turning off all duplication of file and folder security settings. However, when I tried to re-run my duplication script it still initiated a deletion operation. 

[Lennart_T]

Your "new" NAS might have a different software version, making the file's metadata look different. For that reason, Retrospect does not see your files on the "new" NAS as identical (in all respects) with the files on your "old" NAS.

At least that's my theory about what happened.

[haggis999]

5 minutes ago, Lennart_T said:

Your "new" NAS might have a different software version, making the file's metadata look different. For that reason, Retrospect does not see your files on the "new" NAS as identical (in all respects) with the files on your "old" NAS.

At least that's my theory about what happened.

I have always updated the DSM operating system on my Synology NAS on a regular basis, but this has never interfered with past duplications. My recent change to use the Btrfs file system on the NAS still seems the most likely reason for my current problem. Perhaps the turning off all duplication of file and folder security settings that worked for Scillonian in a similar situation is not enough to overcome the problem for Btrfs.

[Scillonian]

In the past when you updated DSM the UUID (Universal Unique IDentifier) of the volume on the NAS would not change. By changing to Btrfs the volume will now have a different UUID. Even if you had kept the same file system for the volume on the new NAS it would still have a different UUID.

The only way you could have kept the original UUID would have been to transfer the hard disks from the old NAS to the new NAS and keep the existing storage configuration.

[haggis999]

Many thanks for that explanation. I had already resigned myself to the need for regenerating all my Retrospect duplication jobs from scratch, but at least I now know that this is unavoidable.

[haggis999]

Recopying all my digital images to the NAS when they already exist there is a real pain. Is there no way to trick Retrospect into accepting the new UUID?

[Lennart_T]

28 minutes ago, haggis999 said:

Recopying all my digital images to the NAS when they already exist there is a real pain. Is there no way to trick Retrospect into accepting the new UUID?

UUID= Universal Unique IDentifier

Retrospect needs to make a difference between two different volumes (even if they use the same hardware).

So the answer is probably "no".

[Scillonian]

Is the share name and folder structure EXACTLY the same on your new NAS as they were on your old NAS? In order to convince Retrospect that not much has changed you need to make the new configuration match the old configuration as much as possible.

Also it is worth checking is what option are set for the Destination of the Duplicate script. If this has been set/reset to Replace Entire Volume then every time the script is run Retrospect will delete everything on the Destination and copy it again.

When you copied the digital images from your backup NAS to your new NAS what method did you use? This could also have an effect on your chances of success.

[haggis999]

 

42 minutes ago, Scillonian said:

Also it is worth checking is what option are set for the Destination of the Duplicate script. If this has been set/reset to Replace Entire Volume then every time the script is run Retrospect will delete everything on the Destination and copy it again.

I think you might have highlighted my real problem! As I mentioned in my OP, the digital images have been moved to a new folder on my NAS. I therefore had to modify the destination in my duplicate script. The last time I modified or created a duplicate script was some years ago, so my skills are a little rusty. I didn't notice the Replace Entire Volume setting when I defined the new destination. Grrrrrrrr.

I have now changed this setting to Replace if Source is Newer and will give it try after transferring the previously deleted files from the Synology recyle bin.

All my recent traumas with backup hardware and software have been thoroughly documented to reduce the chances of me making the same stupid mistakes again.

[haggis999]

After reinstating the files that were previously deleted in error on my NAS, I restarted my Retrospect duplication script with the Replace if Source is Newer setting. All seemed well at the beginning, as all the source files were scanned and then all the existing files on the destination were scanned. However, it then started once again to try and delete files from the NAS, except that I could see no evidence of such deletions. I kept on pausing the script and checking the NAS, but nothing was transferred to the Synology recycle bin and the size of the Photos folder never reduced (despite several refreshes of the file listing). This went on for at least 5 minutes until I chose to stop the script.

After stopping the script, I checked the NAS again and found that over 9GB of files had indeed been deleted. The process had obviously been inhibiting any update of the NAS file listing until it stopped.

I am now trying again using the Duplicate Missing Files Only option.

 

 

[haggis999]

This time, the duplication script is NOT doing any deletions. It appears to be reinstating all the previously deleted files (even those that I had reinstated manually from the recycle bin on the NAS).

 

 

 

[haggis999]

The job has now completed without any errors. The log is shown below. As mentioned in my last post, it seems to have copied both batches of previously deleted files, despite me having reinstated the first batch of around 153GB. It also suggests that something different was done with the second batch of 9.7GB, which was not manually reinstated.

However, I don't quite understand why it would copy files that then failed the comparison test, so maybe my interpretation is not quite correct. I now plan to try again with the Replace if Source is Newer setting, just to see if it still wants to delete any of the files.

 

+    Duplicate using MyPicsDuplicateToNAS at 22/03/2018 16:05
    To volume My Pictures - DUPLICATE NAS1 on Photos on Bonzonas...

-    22/03/2018 16:05:01: Copying My Pictures on MyData1 (G:)
        22/03/2018 16:13:34: Copying: 52,950 files (165.6 GB) and 0 hard links
        22/03/2018 17:18:23: Creating folders
        22/03/2018 17:18:26: Finished creating 19,437 folders
    22/03/2018 17:18:26: Comparing My Pictures - DUPLICATE NAS1 on Photos on Bonzonas
        22/03/2018 17:18:26: Comparing: 1,197 files (9.7 GB)
    22/03/2018 17:26:02: Execution completed successfully
        Completed: 1197 files, 9.7 GB
        Performance: 2215.0 MB/minute (2309.4 copy, 1301.4 compare)
        Duration: 01:21:01 (00:00:01 idle/loading/preparing)

 

 

[haggis999]

Unfortunately, when I returned to using the Replace if Source is Newer setting for my duplication script, it still wants to delete lots of existing files from the destination. All the deleted files checked so far have later time stamps than the source versions. What might be going on here?

[Scillonian]

I've checked the log of a small Duplicate script that runs daily (see below). Once Retrospect has scanned the Source it identifies which files have changed then deletes the old versions on the Destination before copying the new versions to the Destination. Retrospect does't overwrite a changed file on the Destination — it deletes the old version  then creates the new version. Unchanged files are skipped. In the script the Destination is set to Replace Entire Volume and there are 23 files and 35 folders on the Destination.

When you copied the digital images from your backup NAS to your new NAS what method did you use? This could also have an effect on your chances of success.

 

Quote

+    Duplicate using Duplicate Retrospect Files to Boxer at 2018-03-18 23:00
    To volume Ziggy-RBS Retrospect Files on BackupSets on Boxer...
    Warning: The destination volume Ziggy-RBS Retrospect Files on BackupSets on Boxer does not support extended file attributes.

-    2018-03-18 23:00:00: Copying Ziggy Retrospect Backup Server (C:)
        2018-03-18 23:01:06: Finished deleting 6 unnecessary files and 1 unnecessary folders on destination
        2018-03-18 23:01:06: Copying: 6 files (4.4 GB) and 0 hard links
        2018-03-18 23:02:07: Creating folders
        2018-03-18 23:02:08: Finished creating 36 folders
    2018-03-18 23:02:08: Comparing Ziggy-RBS Retrospect Files on BackupSets on Boxer
        2018-03-18 23:02:08: Comparing: 6 files (4.4 GB)
    2018-03-18 23:03:13: Execution completed successfully
        Completed: 6 files, 4.4 GB
        Performance: 2804.1 MB/minute (2137.0 copy, 4076.8 compare)
        Duration: 00:03:13 (00:00:03 idle/loading/preparing)
 

 

[haggis999]

1 hour ago, Scillonian said:

I've checked the log of a small Duplicate script that runs daily (see below). Once Retrospect has scanned the Source it identifies which files have changed then deletes the old versions on the Destination before copying the new versions to the Destination. Retrospect does't overwrite a changed file on the Destination — it deletes the old version  then creates the new version. Unchanged files are skipped. In the script the Destination is set to Replace Entire Volume and there are 23 files and 35 folders on the Destination.

I don't quite follow what you are saying there. How can "unchanged files are skipped" coexist with "replace entire volume"?

1 hour ago, Scillonian said:

When you copied the digital images from your backup NAS to your new NAS what method did you use? This could also have an effect on your chances of success.

I used one of the methods available on Synology NAS devices. First, I opened up File Station on the destination NAS (this is the Synology equivalent of Windows File Explorer). Within File Station, I mounted a remote folder, which gave me a view of the files I wished to copy from the source NAS. I was then able to run a simple copy operation to a local folder on the destination NAS. Around 3 hours later, all my digital images had been restored to my rebuilt NAS.

[haggis999]

On checking the User Guide, I see that the duplication options are defined as follows.

• Replace Entire Volume replaces the entire contents of the destination volume with the selected files and folders from the source volume. Identical files already present on the destination are not duplicated.
• Replace Corresponding Files copies the selected files and folders to the destination volume. When Retrospect finds a file that exists on both the source and destination, the destination file is always overwritten. Retrospect leaves files untouched if they are identical to files marked for duplication or if the file names and locations do not match those marked.
• Replace if Source is Newer copies the selected files and folders to the destination volume. When Retrospect finds a file that exists on both the source and destination, the destination file is overwritten only if the source file is newer. Retrospect leaves files untouched if they are identical to files marked for duplication or if the file names and locations do not match those marked.
• Duplicate Missing Files Only, copies only the selected files and folders that don’t already exist on the destination volume. Other files and folders on the destination are left untouched.

I still find the wording of Replace Entire Volume rather contradictory.

[Scillonian]

3 minutes ago, haggis999 said:

I still find the wording of Replace Entire Volume rather contradictory

It could do with a better name. In use it means the Destination is always an exact copy of the Source. You add a file to the Source it gets added to the Destination. You change a file on the Source it gets updated on the Destination. You delete a file on the Source it gets deleted on the Destination. It will also delete any other files, however they got there, that are on the Destination that are not on the Source.

The other options do not delete any deleted or extraneous files.

16 minutes ago, haggis999 said:

I used one of the methods available on Synology NAS devices.

That is what I was hoping you did but this may also be what caused the problem. I don't know about the Synology File Station but the QNAP equivalent has a habit of changing the Owner:Group information on the NAS file system for the copied files. If the Owner:Group on the NAS file system are different to the Owner:Group that Retrospect presents then the files will be seen as different.

How were the files duplicated to the backup NAS from the old primary NAS?

[haggis999]

7 minutes ago, Scillonian said:

It could do with a better name.

The writers of the User Guide could do a much better job of explaining those duplication options. A check on my earlier copies of the User Guide show that the wording has remained unchanged since version 8.

7 minutes ago, Scillonian said:

I don't know about the Synology File Station but the QNAP equivalent has a habit of changing the Owner:Group information on the NAS file system for the copied files. If the Owner:Group on the NAS file system are different to the Owner:Group that Retrospect presents then the files will be seen as different.

How were the files duplicated to the backup NAS from the old primary NAS?

I don't recognise the term Owner:Group and have failed to see anything with a similar name in the properties of files stored on my Synology kit.

The files were copied on a daily basis from the primary NAS to the backup NAS using an old Synology package called Backup and Replication. This is no longer provided by Synology, but my installation continued to work until the primary NAS crashed earlier this month.

[haggis999]

Correction. I have just found out how to display Owner and Group on the NAS. Everything seems to have Owner = admin and Group = users.

On my PC, Windows File Explorer shows my image files with Owner = Dell_T7500/David, while Group is blank. As far as I am aware, these differences between the PC and NAS have always been present.

[Scillonian]

What shown for Owner and Group on the backup NAS?

If the Synology Backup and Replication was doing its job then the Owner, Group and Permissions should be what they were on the old NAS. The Backup and Replication would have been the best option to restore the files to the new NAS. Has Synology created a replacement for Backup and Replication?

Have you ever accessed your NAS via SSH and do you have any experience of command line Linux?

[haggis999]

There was no direct replacement for the Backup and Replication package. Several other Synology solutions exist for replicating files in their native format, which I have been checking out over the past couple of weeks. My chosen option is called Shared Folder Sync.

Once upon a time, I looked after a Unix server for a few months and ran a variety of tasks via the command line. However, that was over 20 years ago and I have little memory of the details. I can't remember ever using SSH, but I'm willing to give it a go.

 

 

[Scillonian]

If the Owner, Group and/or Permissions have been changed in the copy from the backup NAS the idea would be to login to the new NAS via SSH and use the chown command to change the Owner and Group and the chmod command to change the file permissions as necessary.

 

 

[haggis999]

On 23/03/2018 at 10:17 PM, Scillonian said:

If the Owner, Group and/or Permissions have been changed in the copy from the backup NAS the idea would be to login to the new NAS via SSH and use the chown command to change the Owner and Group and the chmod command to change the file permissions as necessary.

The only user account I have ever created on my NAS boxes is the admin account, so I don't think that my problem can be associated with Owner or Group settings. I am guessing that this also means that there cannot be an issue with file permissions. Is that correct? 

However, I am getting close to giving up on trying to cure this problem. It may be easier to just let the duplication script recreate the backup copy from scratch using the Replace Entire Volume setting (which I now realise is the setting that best fits my needs, and which was the setting I must have used in the past). 

[haggis999]

I went ahead and regenerated my duplication script for these digital images. The copies have now been completed and it has made a start on comparing the copies with the originals. It looks like the total job will take well over 8 hours. I'd forgotten how long this could take!

There is another batch of images that were backed up via another Retrospect duplication script. I haven't tested that one yet, but I hope it doesn't suffer the same issues as the first, as I'd prefer to avoid all this wear and tear on my hard disks.