Backup excluding 20GB of source data

[amkassir]

I am backing up one volume to a (hard) disk media set. The source has 210 GB used, according to the Finder's Get Info. The backup rule is All Files Except Cache Files. The backup copies only 190 GB. I looked through all the folders and files excluded by the rule and there is no more than 500MB of data in the cache folders. Why is the backup not backing up nearly 20GB of data? How can I find out which data is not being backed up?

 

Retrospect 9.0.1

Mac OS 10.6.8

 

Thanks.

[Lennart_T]

I assume you have 20 GB of file duplicates.

 

Do a restore to an empty hard drive and see how much is restored. You can then try a comparison (or syncing) program to see the difference between the disks.

[twickland]

To view all the files included in a backup snapshot, initiate a dummy Restore (I recommend choosing to restore to a new folder just in case you make a mistake in the process) and when you get to the Summary window, click on Preview. It will take a number of minutes to prepare the data, at which point you can browse what Retrospect has backed up.

 

There are a lot of system-related temporary and log files that are useless to restore, which Retrospect does not back up. Unfortunately, to my knowledge they have never published a complete listing of these excluded files, so you'll need to manually compare the backed up folders, folder sizes, and files with what's on the source volume.

[Lennart_T]

you'll need to manually compare the backed up folders, folder sizes, and files with what's on the source volume.

You can't do that using the Finder, since Finder excludes invisble files and folders. Many system files are invisible, for instance.

 

Try

man ls

in Terminal.

[prl]

You can't do that using the Finder, since Finder excludes invisble files and folders. Many system files are invisible, for instance.

 

Try

man ls

in Terminal.

something like:

cd /backup/source/path; sudo find -s . -print > ~/sourcelist
cd restore/path; sudo find -s . -print > ~/restorelist
cd ~; diff -u sourcelist restorelist

may do the job you want (but will require a restore as suggested above). It does a sorted recursive listing of the original and backed up data, and then finds the differences between the lists. The sudo makes sure that it will list the contents of directories you don't normally have permission to access. backuplist and restorelist will be put it hour home directory. If the result of the diff -u is big, you may want to redirect it:

cd ~; diff -u sourcelist restorelist > restorediffs

[Lennart_T]

To compare the two files sourcelist and restorelist, you can use the free TextWrangler:

http://www.barebones...s/textwrangler/

 

Open both files and select "Compare Two Front Documents" in the "Search" menu.

[amkassir]

Thanks to everyone for your replies.

 

I'm just about to buy a new external hard drive, and perhaps the easiest first step is to execute a complete restore to the new drive, and see if the 20 missing GB are really missing. I'm not sure how I could have 20GB of duplicates. I thought maybe I had a duplicate copy of my iPhoto library or my iTunes folders, but nope.

[amkassir]

something like:

cd /backup/source/path; sudo find -s . -print > ~/sourcelist
cd restore/path; sudo find -s . -print > ~/restorelist
cd ~; diff -u sourcelist restorelist

may do the job you want (but will require a restore as suggested above). It does a sorted recursive listing of the original and backed up data, and then finds the differences between the lists. The sudo makes sure that it will list the contents of directories you don't normally have permission to access. backuplist and restorelist will be put it hour home directory. If the result of the diff -u is big, you may want to redirect it:

cd ~; diff -u sourcelist restorelist > restorediffs

 

I need a little help with this. How do I create the sourcelist and restorelist files? Do I need to initiate a restore without actually completing it (stopping at the preview files stage)?

 

Thanks.

[amkassir]

Since I have an ASM contract, I thought it would be good to see how Retrospect's support staff responded to this query as well. Their response is interesting, so I'll post it here:

 

Agent Response:

 

Your disk may contain other things that Retrospect does not back up and should not back up. As an example Virtual Memory swap files that are hidden and used by the OS. The finder also calculates file sizes that include the block allocation size. Retrospect looks at the actual total number of bytes for the data and adds them together. Retrospect also calculates the actual individual folder sizes in a different way from the finder. Duplicate files are also tracked in the snapshots and not re-copied. I suspect that all would account for the 20GB of missing data.

 

You can check your Past Backups section of Retrospect. I am sure all of your data will appear.

 

Thank you for using Retrospect.

 

The Retrospect Support Team

 

 

Besides the VM swap files, I also wonder if the directory takes up space on the source disk that is reported by Get Info as part of the "used" space on the disk. That is, the source disk is 210 GB. Is part of that the directory? If so, how big are directories?

 

(Sometimes DiskWarrior's directory rebuild makes more space available on the disk when the new directory is smaller than the original. I've never checked if the "used" space according to Get Info shrinks when the new directory replaces the original.)

 

Addendum:

I just did a DiskWarrior rebuild on a different disk.

"Optimizing the Catalog Tree Directory File has increased available disk space.

"Optimizing the Attributes Tree Directory File has increased available disk space."

Previewing the before and after disks, there is a difference of about 0.08 GB of used space. That suggests the directory is, in fact, part of a disk's "used" space. Still, not sure how big the whole directory is.

Edited April 11, 2012 by amkassir

[Lennart_T]

I need a little help with this. How do I create the sourcelist and restorelist files? Do I need to initiate a restore without actually completing it (stopping at the preview files stage)?

 

Thanks.

You do need to restore the files first.

Then you run the commands in Terminal.

Be sure to change the "cd" command to point out your actual volumes (or folders).

[amkassir]

You do need to restore the files first.

Then you run the commands in Terminal.

Be sure to change the "cd" command to point out your actual volumes (or folders).

Do I need to restore to the original source volume, or to any volume?

Thanks.

[Lennart_T]

Do I need to restore to the original source volume, or to any volume?

Thanks.

In order to have something to compare with, you should restore to any OTHER volume than the original source.

[prl]

What Lennart said.