File compare problem

[tbr00]

I have just upgraded from Retrospect 12 to Retrospect 16 on Windows 7.

I started to create a new backup set, on LTO4 tape with full verification, but I am getting "File didn't compare" errors.  For example, in one partition, out of 288285 files written 119232 did not compare.  I ran the backup again, and Retrospect decided these 119232 files needed to be written again, and did so, but again all 119232 failed to compare.

I extracted both copies of one of the files from the tape and manually compared it to the source and all three versions are identical, so they appear to be written correctly but for some reason not comparing.

This partition had previously been backed up without this problem with Retrospect 12 on a different backup set.

Is there any way to get more information from the logs as to why it thinks these files are failing to compare?

[DavidHertzberg]

tbr00,

See this post and the ones after it in that thread. 

Are you using OneDrive?  Why are you still on Windows 7, since extended support ended on January 14, 2020?

[tbr00]

No OneDrive, but the partitions are on a Netgear ReadyNas.  I have not seen a similar issue backing up the local drive, but there  is not much data there, so I can't be sure it does not happen in that case.

Regarding Windows 7, all our other machines have been upgraded to Macs but I keep this one around because it has a bunch of old hardware devices (not least my tape drives) which I cannot easily move to a Mac.  Extended security updates for Windows 7 are available to 2022.

[DavidHertzberg]

tbr00,

So, you're going to do a "strip tease" by gradually revealing more information about your problem.🤣  How about "taking some more off" by actually reproducing the "file compare errors" you are getting—as shown in the logs?  That would help us Forums volunteers to figure out what the problem is.

It sounds to me as if you are having a problem described in this 2004 thread.  See the last three posts in the thread.  Is there a recent Windows-7-compatible driver update for your ReadyNAS RAID card?

If you are satisfied that your manual compares show the data is being backed up correctly, how about switching to Media Verification—as described on page 354 of the Retrospect Windows 16 User's Guide?  Page 459 says "In certain circumstances, Retrospect does not have access to MD5 digests generated during backup. This is true for all backups created using versions of Retrospect prior to Retrospect 16.0, as well as backups that took place when Retrospect’s “Generate MD5 digests during backup operations” preference was disabled."  But at worse that means IMHO your data backed up with Retrospect Windows 12 will be backed up again—once.

 

 

[DavidHertzberg]

tbr00,

I also think it would aid us in suggesting s solution to your problem if you clarified your Retrospect-related installation configuration. 

My understanding from what you've said so far is that all your Retrospect "client" machines are Macs, but your "backup server" is a Windows 7 machine that also has a Netgear ReadyNAS.  It is the files on the ReadyNAS that are giving "file compare errors"—about which you have so far revealed no further information—when you back them up.  You are devoted to whatever data is already backed up to LTO4 tape, but (AFAICT) that generation is so old you can't buy a tape drive that will read it and can be attached to a more-modern machine.  You are as devoted to your ReadyNAS server as to the "bunch of old hardware devices", but that server also cannot be moved to a newer machine.

Is my understanding essentially correct?

P.S.: If you upgraded to Retrospect Windows 16 within the last 30-45 days, you are entitled to personalized assistance from Retrospect Tech Support.  If—as I suspect—your "file compare errors" have something to do with your ReadyNAS characteristics, you're going to need that assistance.  Here's why and how to file a bug-report Support Case; note that the Support Case will need to reveal to R.T.S. the information you won't reveal to us.

Edited February 15, 2020 by DavidHertzberg
Revised to remove _further_ "strip tease" allusion, for which I apologize; P.S. on filing bug-report Support Case

[Nigel Smith]

On 2/15/2020 at 1:38 AM, tbr00 said:

but the partitions are on a Netgear ReadyNas

How is this formatted? How are you connecting to it to do the backups?

Most likely explanation is that the metadata doesn't match for some reason, eg the file is being "touched" but not changed between backup and verification, but without examples from your error log it's difficult to tell.

[tbr00]

DavidHerzbeg is correct.  Retrospect is running on the windows machine, which has partitions mounted (CIFS) from the Netgear Readynas. The macs are not involved in this issue.

I did not include the log file originally because all it contains (see below, with paths/filenames elided) is the fact that  a bunch of files failed to compare.  In my original posting instead I asked the question:

  "Is there any way to get more information from the logs as to why it thinks these files are failing to compare?"

In the hope that someone could tell me how to get Retrospect to write more detailed information to the log when it fails to match to get some insight as to the real cause.

I have now opened a support ticket with Retrospect and will report further if it leads to a solution.

Thanks for the help.

From the logfile:

-    2/14/2020 8:17:31 AM: Copying orion on Fileserver
        2/14/2020 8:32:45 AM: Found: 285,888 files, 9,651 folders, 6.6 GB
        2/14/2020 8:32:48 AM: Finished matching
    2/14/2020 8:32:49 AM: A custom selector was used to select 285,888 files out of 285,888.
        2/14/2020 8:32:52 AM: Copying: 119,232 files (1.4 GB) and 0 hard links
    2/14/2020 9:15:22 AM: Building Snapshot...
        2/14/2020 9:15:22 AM: Copying properties for 9,651 folders
        2/14/2020 9:16:34 AM: Finished copying properties for 9,651 folders and 0 files
        2/14/2020 9:16:36 AM: Copying Snapshot: 2 files (60.9 MB)
        2/14/2020 9:16:41 AM: Snapshot stored, 60.9 MB
    2/14/2020 9:16:41 AM: Comparing orion on Fileserver
        File "\\FILESERVER\orion\<file1>": didn't compare
        File "\\FILESERVER\orion\<file2>": didn't compare
        File "\\FILESERVER\orion\<file3>": didn't compare
        File "\\FILESERVER\orion\<file4>": didn't compare
        File "\\FILESERVER\orion\<file5>": didn't compare
        File "\\FILESERVER\orion\<file6>": didn't compare
        File "\\FILESERVER\orion\<file7>": didn't compare
        File "\\FILESERVER\orion\<file8>": didn't compare
        File "\\FILESERVER\orion\<file9>": didn't compare
        File "\\FILESERVER\orion\<file10>": didn't compare
        File "\\FILESERVER\orion\<file11>": didn't compare
        File "\\FILESERVER\orion\<file12>": didn't compare
        File "\\FILESERVER\orion\<file13>": didn't compare
        File "\\FILESERVER\orion\<file14>": didn't compare
        File "\\FILESERVER\orion\<file15>": didn't compare
        File "\\FILESERVER\orion\<file16>": didn't compare
        File "\\FILESERVER\orion\<file17>": didn't compare
        File "\\FILESERVER\orion\<file18>": didn't compare
        File "\\FILESERVER\orion\<file19>": didn't compare
        File "\\FILESERVER\orion\<file20>": didn't compare
        File "\\FILESERVER\orion\<file21>": didn't compare
        and 119,211 others
    2/14/2020 9:57:41 AM: Execution completed successfully        Completed: 119232 files, 1.4 GB
        Performance: 32.4 MB/minute (31.8 copy, 33.1 compare)
        Duration: 01:40:10 (00:17:03 idle/loading/preparing)

 

[DavidHertzberg]

tbr00 (at least I can spell your "handle" correctly🤣),

Here's an apparently-relevant Forums post I came across a few minutes ago in searching for a solution for someone else's backup problem described on an Ars Technica forum.  Its thread—which I suggest you read in its entirety—concerns a Synology NAS instead of a Netgear NAS (can I name non-StorCentric brands?).  The thread is on the Mac 9+ Forum instead of the Windows Professional Forum, so YMMV—but the underlying Engine is the same even though the UI is different.

P.S.: The corresponding-to-link Macintosh Client option to uncheck for Retrospect Windows—even though this applies to a shared volume rather than a "client" machine—is Use attribute modification date when matching on page 371 of the Retrospect Windows 16 User's Guide. The Unix Client option to uncheck—even though this applies to a shared volume rather than a "client" machine—is Use status modified date when matching on UG pages 372-373.

 

Edited February 21, 2020 by DavidHertzberg
Add P.S.

[Nigel Smith]

18 hours ago, tbr00 said:

"Is there any way to get more information from the logs as to why it thinks these files are failing to compare?"

Probably not -- I think that's the entire message. You could mess around with log levels in "secret preferences" (Ctrl-Alt-P-P on a PC) but I don't think you'll get anything more useful.

But what you've given us may be enough. Appended/deleted data would give you "<Path>: different data size" in the log, connections errors are more explicit, etc. A plain "didn't compare" is generally database-type files whose contents can change without the size altering (sounds like too many for it to be that) or differences in metadata (reinforced by the fact that your restored files are, data-wise, identical to the originals). David's link above is for Macs, but suggests another approach -- it may be problems with security information, so try turning off "Back up folder security information from workstations" (the backup set's Options->Windows->Security).

That assumes that you are mounting the NAS as a network share on the RS PC -- you haven't said if you're doing that or have installed RS Client on the NAS -- and, hopefully, you're using SMB rather than CIFS.

Is there any correlation between files which did successfully compare and their server-paths, vs those that failed and their paths? Special character in a folder common to the failures but not the successes? To have exactly the same file-set fail comparison twice in a row suggests some commonality -- you've just got to find it!

If you can't, well... there's a lot that can go wrong here -- a btrfs-formatted volume presented by a Linux OS over SMB (and maybe also AFP/CIFS) to clients with wildly varying security models... It's amazing it works at all! If you've recently bought RSv16 you could raise a support ticket -- the engineers may be able to tell what changed between v12 (which worked) and v16 (which doesn't entirely work). But the good news is that your data is safe, as you've proved, even if you are re-backing up files unnecessarily.

[DavidHertzberg]

Nigel Smith,

tbr00 confirmed in this preceding post what I had said in this previous post.  "Retrospect is running on the windows machine, which has partitions mounted (CIFS) from the Netgear Readynas."  I am a NAS innocent, but this WP article says CIFS is another name for SMB1; maybe the problem is tbr00's using that obsolete protocol version.  It appears to my Windows-innocent eyes that Windows 7 can handle SMB2—but not SMB3.  That's a subject for the Support Case that tbr00 has filed.

I linked to a solution involving Macs, because I didn't then know the corresponding settings for a Windows client—but further investigation led me me to the settings described in the P.S. added to this preceding post.  Maybe my predicted rewrite of the Retrospect Windows GUI will make such settings more visible, even though the use of the setting in the post I linked to was surprising because it worked for a NAS instead of a Retrospect Mac "client" machine.

 

Edited February 19, 2020 by DavidHertzberg
Revise first sentence of second paragraph, because I've _found_ the corresponding Retrospect Windows Backup options to uncheck

[Nigel Smith]

People often use CIFS and SMB interchangeably -- vaguely correct originally, but SMB2 (and now 3) are different beasts altogether. OP may just be loose with his terminology, but it may be that he's forcing a CIFS connection either deliberately (old software that requires it) or accidentally (old settings that can now be changed). If it's a metadata issue, the connection protocol might matter...

The PS you added still doesn't apply to OP's situation. That setting is for Mac clients -- "No option in this category affects Windows or Linux clients" (last line of p370). The closest similar option I could find for mounted shares was the Windows Security was the one mentioned.

I still think the clue is in the consistency -- it looks like none of the previously successfully compared files were backed up again, while all the "didn't compare" files were both backed up again and still didn't compare. Find the commonalities within, and the differences between, the two groups and we could solve this.

[DavidHertzberg]

NigelSmith,

I know the settings in the P.S. I added are supposed to apply only to "clients".  But an administrator extremely familiar to you posted on 1 November 2019:

Quote

I always thought those options applied to sources with the RS Client installed, rather than non-client shares -- or does it consider the share to be mounted on a client which also happens to be the server? I'd certainly never think of changing options in the "Macintosh" section when backing up a Synology, however it was attached!

Every day's a learning day 🙂 

I'm pretty sure this is in fact a metadata or connection issue.  But then so was Lennart_T's problem discussed in the October 2019 thread.  If a solution that shouldn't have worked then did in fact work, maybe it'll work for tbr00.

[tbr00]

Thank you for all the useful suggestions which I am working though.  First off I used ctrl-alt-P-P to change the logging levels and I found that setting "Trees and Volumes" to 7 gave some additional output in the log.  First it explicitly listed every file that failed to compare rather than just the first 21, and second, for each file I have an entry:

        TPCFile::BackupReadOpen: OriginalFilePath = \\FILESERVER\orion\<path/filename>
        TPCFile::BackupReadOpen: ppath = \\FILESERVER\orion\<path/filename>, isDedup = 0
        TPCFile::BackupReadOpen: \\FILESERVER\orion\<path/filename>
        BackupParseStreams: stream header read: (dsi. 256 256)    0    3    2    152    0
        BackupParseStreams: stream header read: (dsi. 256 256)    0    1    0    15,807    0
        BackupParseStreams: stream header read: (dsi. 256 256)    1    0    0    0    0
        bkstreamParseRead: id = 3, flags = 0, size = 128, name =
        bkstreamParseRead: id = 1, flags = 0, size = 15,807, name =
        bkstreamParseRead: id = 0, flags = 1, size = 0, name =
        File "\\FILESERVER\orion\<path/filename>": didn't compare

but to the untrained eye at least there is still nothing there that is helpful.

Next in order to speed up experimenting I defined a sub-volume pointing to a directory containing just one file from the list of persistent failures.  When I ran the backup on that sub-volume, as expected, it then wrote just the one file to tape, but that file then verified OK. So it cannot be related to the attributes of the individual files.

Just to be sure nothing else had changed I ran the backup of the full partition again expecting to see one less than the prior number of persistent failures (119231) but to my surprise it this time only found 6900 files it said it needed to write.  So it seems that when I had the logging level turned right up, a large number of the files must have compared OK.  I could not tell the actual number of failures because the voluminous output overran the depth of the log file (I have now increased the maximum size).  And on this pass now all 6900 files verified OK.  So retrospect now thinks the backup is complete.

I am starting a full backup after recycling the scratch backup set to see if it reproduces the original problem . . .

[Nigel Smith]

1 hour ago, DavidHertzberg said:

I know the settings in the P.S. I added are supposed to apply only to "clients".

Sorry -- bolded the wrong word. I'll try again -- Mac clients.

Lennart's issue was with a Mac server backing up a mounted SMB share and it looks like, in that situation, the server considers itself a client with respect to RS's client options.

OP has a PC server backing up a mounted SMB share, so Mac client options won't help. But, assuming the same mechanisms that Lennart revealed also apply to Windows, we could look at the Windows client options. There's no obvious option matching the Mac's "Use attribute mod...", but the most similar is the "security information" option.

Of course, this is Retrospect -- so a Mac-client option specifically stated not to affect Windows may just solve the problem!

[Nigel Smith]

25 minutes ago, tbr00 said:

So it seems that when I had the logging level turned right up, a large number of the files must have compared OK.

There's no sensible reason why changing the log level would change how the comparisons would work -- it's more likely that whatever was causing the "change" didn't happen this time (unless you also changed some other options?).

Again, an obvious thing would be the differences between Mac and Windows permission schemes, and how the NAS manages and presents them. Simplified, even when using ACLs rather than POSIX mode bits, your NAS can't accurately replicate the Windows security schema. How this is handled depends on implementation, but I've seen before where a share accessed by both Macs and PCs can end up in the middle of a tug-of-war, the metadata changing depending on which platform last accessed it. Hence the advice about turning off the security info option, and also the question about CIFS (less likely to happen if everything is actually using SMB2 or 3).

[tbr00]

OK, so I go back to a new backup set and run the full partition backup again, and I am back to a huge number of miss-compares, (183263) which is comparable to but not the same as I had before, and further the first 21 miss-compares (which is all that's explicitly listed in the log) are not the same as the first 21 the last time I did a clean run.

In answer to some of the other questions:

I can say with confidence nothing else has been accessing this partition between these various retrospect runs.  But there is activity on other partitions on the same NAS.

Second as far as I can tell the NAS is serving SMB1. 

Next I will experiment with the permission options.

[Nigel Smith]

14 hours ago, tbr00 said:

Second as far as I can tell the NAS is serving SMB1.

Win 7 can do SMB2.1, IIRC. For the NAS, it will depend on which version of ReadyNAS OS you are running. The easiest way I know for you to check is, madly, to connect to the share with one of your Macs and then in the Terminal type:

smbutil statshares -m /Volumes/sharename

or, to list them all:

smbutil statshares -a

Since your Mac will start high and negotiate down to find a match, you'll see the best the NAS can provide in the "SMB_VERSION" line for that share.

I think NAS OS 4 and 5 only support SMB1 (SMB2 was experimental, and could be worth a try). You could consider an unsupported upgrade to NAS OS 6 -- but that seems a huge amount of work (and risk, given the current state of your backups!) for something that might not help...

However...

18 hours ago, tbr00 said:

When I ran the backup on that sub-volume, as expected, it then wrote just the one file to tape, but that file then verified OK. So it cannot be related to the attributes of the individual files.

Which suggest it might be something along the path to that directory that's causing the problem. Have a careful look at all the directory names above, and make sure there are no strange characters -- easiest done by ssh-ing to your NAS and using command line tab completion to list the whole path. Reason being that it is very easy to create SMB-invalid names on a Mac, eg a trailing space in a folder name, which the Mac SMB client then cleverly encodes so it doesn't cause problems on the server -- but it can cause problems further on, if a later process/client can't cope with the encoding. (Springs to mind because I'm currently having problems with a data migration where some users have used spaces/periods at the end of folder names, along with angle brackets, symbols, forward slashes... Grrrr!)

[tbr00]

The 4 security options were set as:

(checked) Back up file security information from servers
(unchecked) Back up file security information from workstations
(checked) Back up folder security information from servers
(checked) Back up file security information from workstations

which according to the documentation are the defaults.  I don't remember ever changing any of these and they are set that way in all my scripts.  I unchecked them all and ran a full backup.  Once again miss-compares, but once again  a different  number (this time 96126) and different files.

Nigel,

smbutil on the MAC confirms it is SMB1.  While it is possible there could be a peculiar filename somewhere, there is nothing odd about any of the filenames Retrospect explicitly lists for the first 21 miss-compares in each run.

I have considered the OS upgrade to version 6, not least because a factory reset is required to expand the volume further on version 4.  But as you say that's too risky without 100% confidence in the backup.  In fact the whole reason I upgraded Retrospect from 12 to 16 and then started a new backup set was in preparation for going that path.

 

[Nigel Smith]

Well, I think we've checked all the obvious things -- if nothing else, you've a bunch of answers ready for a Support ticket 😉 I agree that it's really annoying that what worked in v12 is failing in v16, and maybe they can quickly work out why.

Personally, I'd be confident about the backup. You've proven to yourself that at least some of the "didn't compare" files are, in fact, identical in terms of data. While the only way to be almost sure is to restore and manually compare every single one, a reasonable random sample should be enough to quell any disquiet.

But Retrospect themselves say that the Verify step is optional, and tell you to turn if off for Cloud backups. So the main problem at the moment (since you can ignore those annoying error messages by not logging them!) is that the failed compare step means the files will be backed up over and over again even though they haven't changed. Either put up with the wasted resources or turn off the Verify step, problem solved...

[tbr00]

Progress!

I found enough posts in the ReadyNAS forums to suggest the "experimental" SMB2 support in OS 4 is in fact reliable, so I enabled it.  Regular access to the shares is fine from both the MACs and windows with the one caveat that for some reason on Windows, the NAS no longer shows up under "Network" in the explorer, at least not until I type its name into the address bar, at which point it and all the shares on it show up right away.  smbutil on the MAC now says it's using SMB2.02.

I reset the Retrospect security options back to the defaults, recycled the scratch backup set and ran a full backup of the problem share, and . . . zero compare errors.   Just to be sure, I recycled the backup set again and did it a second time to make sure it was not a fluke.  Again zero compare errors.  I will wait till I have run a full backup on all the other shares which did not cause a problem with SMB1 and Retrospect 16 in order to be absolutely sure, but at this point I think this change has solved it.

There does of course remain the issue of why this only showed up after switching from Retrospect 12 to 16, but that is something I think only Retrospect are going to be able to get to the bottom of, and I will update my Support ticket with this new information.

Thank you two gentlemen for all the help, and my apologies to DavidHertzberg for miss-typing his handle earlier 🙂

[DavidHertzberg]

tbr00,

Congrats on getting it working.😄 

As far as mis-typing the last-name part of my "handle" is concerned: I'm used—ever since  childhood—to people leaving letters out of the "hertz" part, but this is the first time anybody has ever converted the "berg" part into a Turkish social title.🤣

Just a definitely-more-significant correction.  MAC as all-upper-case is not an abbreviation for Macintosh.   It is universally used in IT as part of "MAC address", and was invented by Xerox Network Systems when Macintosh computers existed only in the mind of Steve Jobs.  You're not alone; tens of millions of ignorant Windows users make this mistake, and millions of more-knowledgeable Macintosh users try to correct it (guess which OS I use 😉).

Enjoy your Retrospect backups.  As Nigel Smith says, the Verify step should be turned off for Cloud backups.  Verify is really intended to catch either tape-write errors or LAN-transmission errors; with Cloud backup, nothing can go wrong ... go wrong ... go wrong  (this joke really shows its aged origins in the days of vinyl phonograph records).

[tbr00]

I have found one thing which I think may be relevant.  When I backed up the errant partition using SMB1 the log had:

    2/19/2020 5:37:00 PM: A custom selector was used to select 285,888 files out of 285,888.
        2/19/2020 5:37:02 PM: Copying: 256,935 files (6.4 GB) and 28,953 hard links

but with SMB2 it has:

    2/20/2020 3:27:23 PM: A custom selector was used to select 285,888 files out of 285,888.
        2/20/2020 3:27:26 PM: Copying: 285,888 files (6.6 GB) and 0 hard links

Most of the data in this share did originally come from a UNIX system, and it would appear that SMB1 preserves the hard links in some form,
whereas SMB2 is presenting them as distinct files . . .

None of my other shares reported any hard links with SMB1.

[DavidHertzberg]

tbr00,

As to hard links, see for example this 2013 thread (dealing with a Linux Client) and this 2009 thread.  If those aren't relevant enough for you, try using the Forums Search box with "hard links" in quotes.  Linux, Shminux, so long as you've got your health (joke derived from famed 19th-Century Yiddish writer). 🤣

As to "why this only showed up after switching from Retrospect 12 to 16", my impression from several recent Forums threads is that the "backup server" Engine—which is the same for both variants of Retrospect aside from GUI—lost its ability in version 15 to handle certain old versions of network protocols used by NASes.

Edited February 24, 2020 by DavidHertzberg
Added 2nd paragraph

[Nigel Smith]

On 2/22/2020 at 3:12 PM, tbr00 said:

Most of the data in this share did originally come from a UNIX system, and it would appear that SMB1 preserves the hard links in some form,
whereas SMB2 is presenting them as distinct files . . .

I know that soft links are treated differently -- with SMB1 they are resolved by the client, with SMB2+ by the server, and I remember various warnings in our old Isilon docs about unpredictable behaviour when using them with SMB1 clients. Assuming hard links are treated similarly, it looks like you've nailed it.

Would still be interesting to know why things are handled differently in the read and verify phases but, at this point, it may be better to leave it working well and not look too hard at why, just in case...