Normal backups, backing up too many files.

[Sakumbi]

I have recently upgraded from Retrospect Pro 6.5 to 7.5. The configuration is virtually unchanged from the old version to the new version, and most settings are default. I have 3 Windows XP Pro PC that I'm backing up with a set of scripts. Nightly each PC is backed up to one of the other PCs. On Sunday a Recycle Backup is preformed, and a normal backup each day of the week. These backups work as expected. Each night that a "normal" backup is preformed and only a few hundred MB of changed files are backed up.

 

I also have another script which I manually kick-off that backs up all 3 PCs to DVD for long term, and off-site storage. I typically run this script with a recycle backup ever 2 months, and run normal backups about once or twice a week. Until I upgraded to 7.5 this script too behaved as expected. When I'd run a "normal" backup, only the changed files (about 1GB or so) would be added to the DVD backup. But after having updated, the first PC in the script (a client) seems to still behave correctly, but the 2nd (the retrospect server) and the 3rd (another client) both try to backup nearly their entire HDs again.

 

The 2nd PC, for example, contains about 8.1 GB total. On a "Recycle" backup the 8.1 GB is backed up as expected, but if you immediately re-run the same backup script again as a "normal" backup it will try to backup an addition 6.9 GB. There is no way this much data has changed. The other daily scripts confirm this, as if you now re-run the daily script for the PC in question, it will only try to backup the correct few hundred MB.

 

This is a real problem, mostly with the 3rd PC, as it contains nearly 100 GB. This takes a lot of DVDs and a lot of time.

 

The only different between the daily backup scripts and the long term DVD scripts are as follows:

 

Daily -- Each PC backs up to another client PC's HD.

DVD -- Backs up to DVDs.

 

Daily -- Uses a file selector to skip temp/cache files.

DVD -- Uses a copy of the daily selector with *.rbf & *.rbc files added to the exclude list, so as not to backup the daily backup files.

 

Daily -- runs a thorough verification.

DVD -- runs a media verification.

 

Any idea why the "normal" backups want to backup 90% of the HD again. Any help is greatly appreciated.

 

Greg

[Guest]

Sorry I can't really help, but I can report that I'm seeing the same behaviour.

 

I have the same problem - on a simpler setup - just the one PC. Having upgraded to 7.5 from 6.5, Retrospect has now decided that it needs to backup virtually every file (but not all) again, even though very little has changed. I haven't got as far as a 2nd backup on 7.5 yet as the performance is so terrible writing to DVDs (33Mb/s) its taking days to do the first backup!

 

So far, I'm convinced that 7.5 is a backward step from 6.5 - come on Dantz, please sort these issues out.

[michaeln]

Hi,

 

Have you tested your selectors to make sure they are grabbing the correct files?

 

Configure > Selectors > Choose your selector and click "Edit" > click on the blue check icon labeled "Check Selector" and test it against a volume you are backing up.

 

As opposed to your custom created selector, what happens if you try a built-in selector and try backing up these files that are continuously being backed up?

[nekr0phage]

Hi,

 

If you look at the read-me for 7.5 the first item listed under "Tips and Late-Breaking Information" discusses how Retrospect now backs up file security information. The first time you run a backup it might need to back up all files again if it's backing up from a server OS. That is, if you left the options at default to backup file security information from servers and not from workstations. If those are at default and it's backing up more data on XP... I don't know. You could try disabling the option for file security information on servers also.

[Guest]

Yep that works - disabling the backup security info at file level sorts it gets a far more reasonable file selection for backup - thanks

[Sakumbi]

Sorry for taking so long to reply. Each full recycle backup takes a day or two depending on how johnny-on-the-spot I am to switch DVDs. So it takes a while to test each new change I make.

 

Quote:

---

Have you tested your selectors to make sure they are grabbing the correct files?

 

...

 

As opposed to your custom created selector, what happens if you try a built-in selector and try backing up these files that are continuously being backed up?

 

---

 

The selectors have been tested and are working correctly. They are selecting the correct files. Which are basically all of them. The problem is that they are getting backed up again on normal backups when they shouldn't.

 

The selector is relatively simple, with an include everything, followed by a few excludes to reduce unneeded files, like temp/cache, and other backup files. It is close to a built-in selector. I in fact started with "All Files Except Cache Files", removed some options that didn't apply and added excludes for backup files.

 

Quote:

---

If you look at the read-me for 7.5 the first item listed under "Tips and Late-Breaking Information" discusses how Retrospect now backs up file security information. The first time you run a backup it might need to back up all files again if it's backing up from a server OS. That is, if you left the options at default to backup file security information from servers and not from workstations. If those are at default and it's backing up more data on XP... I don't know. You could try disabling the option for file security information on servers also.

 

---

 

I'm not running a server OS, plus I have tried un-checking the option of backing up NTFS file security info for both workstation and server.

 

This is very weired. Like I said the scripts, selectors, backup sets, Volumes, and options are all unchanged from 6.5. I've read through all the docs and info I can find, and nothing I've seen begins to explain it.

 

Weirder, at least the first 2 client volumes in the script behave correctly every time, while at least the 3rd and 6th client volumes in the script always have the problem. But only with this script. (I'm not sure if the 4th and 5th client volumes in the script work correctly or not, they are just too small and not enough files to tell for sure, but I strongly suspect that they do have the problem.)

 

I have other scripts that run correctly, and the only real difference is the backup sets they are going to, (backup files on network shares vs DVD+RWs and through vs media verifications.)

 

I have even just tried completely deleting the problem script, selector, backup set, and volume set for the DVD script in question and completely recreated them from scratch. I then ran a full "recycle" backup, and everything work as expected. I then re-ran a normal backup of the same script. Again, the first 2 client volumes (the first client) worked correctly, the 3rd client volume again wanted to backup nearly the entire drive again. I stopped the script at that point.

 

All PC are running the same OS (Windows XP Pro SP 2) with same patch levels and relatively similar software.

 

I just don't get it, this has got to be some kind of bug!

 

I'm sitting here wondering, what is different about the first client? Why does it work?

The only software running on the other two that isn't running on the first, is BOINC and pageant (part of PuTTY). I can't think of what pageant or BOINC could posibly do to so screw with Retrospect.

 

Then I'm also thinking, wait a minute. The scripts that run nightly and backup the 2nd and 3rd PCs work fine night after night. I can even kick off tonights script right now, and it says that only 352 files (270.7 MB) has changed on the 3rd client volume; while the DVD normal backup I just cancled a few minutes ago said there was 27043 files (4.9 GB) remaing to backup on the 3rd client volume when I canceled it.

 

I just don't get it, this has got to be some kind of bug!

 

The only thing left I can think to try is to completely un-install Retrospect, re-install it, rebuild all my scripts and settings, and see if that fixes it.

 

Unless, I'm hoping, someone out there has a better idea than I.

[AmyJ]

Two things come to mind in terms of investigation:

 

 

 

1. Compare two copies of the exact same file between sessions to see if there are any obvious differences. Go to Configure > Backup Sets and get Properties on the DVD backup set. Go to Sessions and browse a session for the problem computer. Find a file that absolutely should not have changed between backups. Something in your User folder, for example. Right click and get Properties. Leaving that window open, open a different session for the same problem computer. Find the exact same file and get Properties on it. Then compare the two Properties windows side by side. Are there any differences in the times/dates/sizes? Even time differences to the second are relevant.

 

 

 

Note: If you cannot find the file in a session it means it was not "changed" or backed up in that session. Try the test with a different file. Your best bet is to look at a later session first to find a test file, and then compare to the earliest session for that computer.

 

 

 

2. You mentioned the following difference between your scripts:

 

 

 

Daily -- runs a thorough verification.

 

DVD -- runs a media verification.

 

 

 

-- Are there any errors when the backup to DVD is run against the problem comptuer with media verification?

 

 

 

-- Can you try backing up just the problem computer to a new DVD backup set with thorough verification to first see if there are any errors that are somehow not being caught by media verification? If there are no errors, try to Preview an immediate backup to see how many files Retrospect thinks it has to copy. Backup > Backup. Set Source and Destination and then hit Preview. Files with diamonds next to them already exist as exact copies in the backup set (meaning they will not be copied again). If a file does not have a diamond it will be marked for copy because Retrospect detects a difference between the live file and the file in the backup set.

[Sakumbi]

Amy, you definitely had me stumble upon something interesting...

 

Finding a file that absolutely should not have changed isn't a problem. Since just about every file on the system is being affected, any file in \i386 will take care of that.

 

Now I did notice something very interesting. The Created and Modified dates and times are the same. But I notice that in the first backup, the file is marked "Flags: Miscompared" Does this imply that the verify failed? The log from the job didn't indicate any problems.

 

Code:

---

-	4/16/2006 11:44:36 PM: Copying Tiger XP (C:)

- 4/17/2006 12:01:54 AM: Verifying Tiger XP (C:)

- 4/17/2006 12:15:31 AM: Copying Tiger XP (C:)

- 4/17/2006 12:53:16 AM: Verifying Tiger XP (C:)

- 4/17/2006 2:32:07 AM: Copying Tiger XP (C:)

4/17/2006 2:45:48 AM: Snapshot stored, 58.7 MB

4/17/2006 2:46:19 AM: Execution completed successfully

Completed: 43851 files, 7.8 GB, with 22% compression

Performance: 119.1 MB/minute

Duration: 02:40:47 (01:34:18 idle/loading/preparing)

---

 

Quote:

---

-- Are there any errors when the backup to DVD is run against the problem computer with media verification?

 

-- Can you try backing up just the problem computer to a new DVD backup set with thorough verification to first see if there are any errors that are somehow not being caught by media verification? If there are no errors, try to Preview an immediate backup to see how many files Retrospect thinks it has to copy. Backup > Backup. Set Source and Destination and then hit Preview. Files with diamonds next to them already exist as exact copies in the backup set (meaning they will not be copied again). If a file does not have a diamond it will be marked for copy because Retrospect detects a difference between the live file and the file in the backup set.

 

 

---

 

No errors when running against the problem computers with media verification. Prior to upgrading to version 7.5, I ran thorough verifications. I switched to media verification to reduce the number of DVD swaps. (One of the main reasons I upgraded actually.)

 

I'll try to preform a thorough verification tomorrow night, and let you know the results.

 

That "Flags: Miscompared" really has me wondering what's going on.

[AmyJ]

As a shortcut you could probably try backing up a subvolume rather then the entire computer to see if you can replicate the problem. This would be much quicker and less painful then backing up the entire 8 gig to DVD.

 

I haven't done much with Media Verification, and haven't seen the "Flags: Miscompared" error before, but it certainly does seem like there is a problem with the verification. That would definitely explain why the files are being backed up again, but I'm not sure what the root cause would be. Is there any pattern to the files that are being backed up again? Are they all in your user folder, for example, or are the files from all over the system? Take a look at the session to find out.

 

Is there any installed software which monitors the state of that computer or indexes it in real time? Just looking for things that could be changing the files in some subtle way...

[RetroNudge]

This raises an interesting question...which is:

 

Question #1 - What changes to a file should require backing up the data portion again?

 

Should a change of last access date require backup?

 

Change in which flags should require data portion backup.

 

File Size: Yes

 

Archive: Yes

Read Only: No

System: No

Hidden: No

 

File Owner: No

File Security rights: No

 

Access Date: No -- Definitely not, as retrospect erroneously changes this itself.

Creation date: Yes, but only if archive bit and/or file size changed.

Modified date: Yes

 

The changes in the flags should be backed up...and stored in whatever retrospect uses for a database of directory information...but not the file's data again.

 

Question #2 - When GROOMING backup...

Does retrospect compare the MD5 sum of identically named files, and if they are identical, keep only one copy?

 

If not...why not.

[AmyJ]

Hi RetroNudge

 

 

 

1. In Thorough Verification on Windows, Retrospect looks at creation date and time, modified date and time, size and name. If match only in same location option is set, Retrospect matches on the path, volume name and drive letter also. In 7.5 Retrospect also looks at the archive bit. In Media Verification Retrospect compares the MD5 hash.

 

 

 

2. It doesn’t need to because the backup takes care of that. By the time grooming occurs, there should only be one copy of a file that is identical (unless of course it resides in the System folder which we don’t match).

 

 

 

Amy

[Sakumbi]

Okay, I've done some more test with very interesting (but not completely conclusive) results.

 

First, I have restored files that have had the "Flags: Miscompared" in there Properties, and preformed manual MD5 and SHA-1 hash comparisons with the originals. I can confirm that the backed up and then restored files data is 100% identical to the originals, including ACLs and attributes.

 

Second, I have preformed a full recycle backup with thorough verification and with no verification, and in both cases the problem went away.

 

Quote:

---

As a shortcut you could probably try backing up a subvolume rather then the entire computer to see if you can replicate the problem. This would be much quicker and less painful then backing up the entire 8 gig to DVD.

 

---

 

Third, only backing up a small subset revealed another abnormality, and explained why client volumes 1 & 2 always worked correctly, volumes 3 & 6 always had the problem, and 4 & 5 usually seemed to have the problem, but I couldn't tell for sure. The problem only occurs if the entire volume does not fit on to a single DVD. It is only when a client volume's backup must spam multiple DVDs that the problem appears. Since volumes 1 & 2 are small enough to always fit on the first DVD, they never had a problem. Volumes 3 and especially 6 are massive and each take multiple DVDs, and thus always have the problem. Volumes 4 & 5 are each about 2 GB, so sometimes one or the other would end up spanning DVDs and sometimes they wouldn't.

 

Quote:

---

Is there any pattern to the files that are being backed up again? Are they all in your user folder, for example, or are the files from all over the system? Take a look at the session to find out.

 

---

 

There is no pattern in terms of the folder they are in, they are all over the directory tree. However, taking in to account the third observation above, I am strongly suspecting that it is either all the files from the first or the last DVD of a volume that a spans multiple DVDs that are behaving correctly. All the files on all the other DVDs in the volume spanned set are being backed up again on the next normal backup as if they changed or where never backed up.

 

I am thinking that it is the last DVD in the set that is working correctly. I base this on the facts that 1) in my 8.5 GB test, the first DVD holds 5.4 GB of data, the second the remain 3.1 GB, and when the next "normal" backup is run it tries to backup approximately 5.4 GBs of data again. 2) Most of the files backed up on the second "normal" backup are from the "front" of the drive which which gets backed up first.

 

I'm having a little problem getting 100% confirming on this though, because I can find at least a few files on the first DVD that Retrospects thinks are good. (At least I think so.)

 

At any rate this is definitely looking more and more like some kind of funky problem with the media verification scheme.

 

I have always done through verifications when doing the nightly backups which are backed up to a file on one of the other computer's drives since I didn't have to worry about trying to reduce media changes.

 

I'm going to preform a test, doing a backup to a file using media verification instead of through verification and see if I get the same results there too.

 

Quote:

---

Is there any installed software which monitors the state of that computer or indexes it in real time? Just looking for things that could be changing the files in some subtle way.

 

---

 

No indexing software. Only file monitoring software is basic Symantec Anti-virus, and that is installed on all 3 computers, and only two show the symptom, so I don't thing that is it.

[AmyJ]

Hi Greg -

 

Thank you for the detailed testing and analysis. You may very well have stumbled upon somthing that needs to be addressed in media verification. You've got my curiosity piqued - and I'm going to run a few media verification tests to DVD and see if I can replicate the problem. Will let you know how it goes...

 

-Amy

[Sakumbi]

As expected the test backup to a file using media verification worked correctly. It only seems to occur when spanning.

[AmyJ]

Hi Greg -

 

 

 

I saw the problem and a bug has been logged. For the time being the best thing to do would be use thourough verifcation when writing to optical media.

 

 

 

Thanks again for all the help,

 

Amy

[Sakumbi]

Thanks Amy,

 

I have reverted back to using through verification. This works but is a pain do to the number of disc changes needed. Hopefully a fix will be soon in coming.

 

Thanks

Greg

[ctwist]

I too have experienced this problem. Its description and a workaround follow. I sent this note to tech support on 16th June, but have not received a reply.

-------------

 

When doing a full backup to multiple DVDs, if one of the volumes is defective, the entire volume is marked as defective, along with all prior volumes. This causes a lot of data to be backed up unnecessarily during the next incremental backup.

 

Details follow:

- I ran a full backup for a new backup set. The preview reported that the total file size was 17.9GB. Verification was turned on.

- While the 3rd volume was being verified, I noticed that 1.5GB remained to be backed up.

- When the backup was complete, the log reported error -206 on the 3rd volume (i.e. a damaged disk).

- I started an incremental backup, which should have backed up the files that were not readable. However, the preview reported that it was going to back up 16.4GB. Since 17.9 - 16.4 = 1.5, this suggested that all the files on the first 3 volumes were going to be backed up again, even thought the first 2 volumes had no errors. I did not run this backup.

- I then repaired the catalog, rebuilding it from the 4 backup volumes.

- I started the incremental backup again, and this time it only needed to back up 41MB. I allowed the backup to run, and it completed normally.

 

My conclusion is that when the verification process finds errors, far too many backed up files are marked as unusable in the catalog.

 

This problem has happened to me twice. I suggest that you need to improve your error recovery process.