Jump to content
prl

Copy/overwrite Entire Volume Still Copying Far Too Much In Retrospect 9

Recommended Posts

Copy/Overwrite Entire Volume still seems to copy far more than necessary.

 

Here are some results from a small test.

 

The initial copy, which should copy the whole test folder:

+  Copy using Retro copy test at 28/12/11
   To volume RetroTest...
   -  28/12/11 15:51:27: Copying Computer
   28/12/11 15:54:38: Comparing RetroTest
   28/12/11 15:55:34: Execution completed successfully
   Completed: 2840 files, 490.7 MB
   Performance: 241.3 MB/minute (156.5 copy, 535.2 compare)
   Duration: 00:04:06 (00:00:02 idle/loading/preparing)

Then immediately another copy using the same script, without touching anything in either the source or destination folders:

+  Copy using Retro copy test at 28/12/11
   To volume RetroTest...
   -  28/12/11 15:55:52: Copying Computer
   28/12/11 15:57:00: Comparing RetroTest
   28/12/11 15:57:16: Execution completed successfully
   Completed: 917 files, 107.6 MB
   Performance: 172.1 MB/minute (109.4 copy, 403.4 compare)
   Duration: 00:01:24 (00:00:09 idle/loading/preparing)

So who's gone and changed 107.6MB in 917 files while I wasn't looking?

 

And then the same run again, and again not modifying anything in either the source or destination folders:

+  Copy using Retro copy test at 28/12/11
   To volume RetroTest...
   -  28/12/11 15:58:29: Copying Computer
   28/12/11 15:59:39: Comparing RetroTest
   28/12/11 15:59:55: Execution completed successfully
   Completed: 917 files, 107.6 MB
   Performance: 167.6 MB/minute (104.1 copy, 430.3 compare)
   Duration: 00:01:25 (00:00:07 idle/loading/preparing)

And someone's gone and changed 107.6MB in 917 files on me again!

 

This is what the manual says is supposed to happen (my emphasis):

Overwrite entire volume replaces the entire contents of the destination volume or Favorite Folder with the selected files and folders from the source volume or Favorite Folder. Everything else on the destination volume is deleted. Retrospect saves time by not copying identical files, that is, files that share the same location, name, modification date and time, etc., that are already present on the destination. New files are added, and different versions of files already present on the destination are replaced by the files from the source, even if the file already present on the destination is newer.

That doesn't sound like what's happening to me!

 

Is there some way I can find out which files are being copied? And even better, why?

Edited by prl

Share this post


Link to post
Share on other sites
Guest Steve Maser

If you look in "Past Backups" -- you'll see what files were backed up. You may have to turn on Grooming for the media set to see the results of each past backup.

Share this post


Link to post
Share on other sites

If you look in "Past Backups" -- you'll see what files were backed up. You may have to turn on Grooming for the media set to see the results of each past backup.

It's not a Backup, it's a Copy (with Overwrite Entire Volume set). It doesn't appear in Past Backups. There's no media set.

Share this post


Link to post
Share on other sites

I posted about this problem in Retro 8 in this forum in March 2011.

 

That posts links to posts in the old EMC2 Retrospect forum about the same problem in Retro 8, including a similarly detailed post in the Bugs section. Those posts were made in July and October 2010. There was no response to either of those two posts.

Share this post


Link to post
Share on other sites
Here are some results from a small test

 

Perhaps some more specific information about your test parameters.

 

What is the Source? A logical volume? A Favorites Folder? Locally attached? Network share?

What is the Destination? (see above) ?

 

Specific steps should include configuration information so others can attempt to reproduce.

 

 

-dave

Share this post


Link to post
Share on other sites

Perhaps some more specific information about your test parameters.

 

What is the Source? A logical volume? A Favorites Folder? Locally attached? Network share?

What is the Destination? (see above) ?

 

Specific steps should include configuration information so others can attempt to reproduce.

 

 

-dave

In that instance, the source is a folder as a Favourite, and the destination is a folder as a Favourite. But I have the same problem when the source is a physical volume and the destination is a physical volume (my system volume and my backup system volume respectively).

 

There's another report of the problem here.

Share this post


Link to post
Share on other sites

And someone's gone and changed 107.6MB in 917 files on me again!

My guess is that the 917 files have a modification date in the future. That way, they will always be "newer" than the last copy.

Share this post


Link to post
Share on other sites

My guess is that the 917 files have a modification date in the future. That way, they will always be "newer" than the last copy.

And there are 8GB or so of files on my system volume with future dates, too?

 

I will check the dates in the test example anyway, but I'm skeptical about that as a cause.

Share this post


Link to post
Share on other sites

My guess is that the 917 files have a modification date in the future. That way, they will always be "newer" than the last copy.

And there are 8GB or so of files on my system volume with future dates, too?

 

I will check the dates in the test example anyway, but I'm skeptical about that as a cause.

 

Mayoff seemed to think the problem was with the attribute modification date, and suggested turning matching on that off. Which suggests something quite strange going on. But he was talking about "copying every file every time" which also isn't what's happening here.

Share this post


Link to post
Share on other sites

And there are 8GB or so of files on my system volume with future dates, too?

 

I will check the dates in the test example anyway, but I'm skeptical about that as a cause.

It might be another date, such as creation date that is in the future.

 

We run Retrospect Windows 7.6 at work. We use disk-to-disk-to-tape. From client to disk every night, from disk to new tapes every weekend and new/changed files from disk to tape on Thursday night/Friday morning. There are a couple of client computers that are very rarely used and are turned off weeks in a row. For two of these clients, there are a handful of files that are always transferred to tape Thursday night. This happens even if the client computer has been turned off for more than a week. I checked up on a few of these files and there was indeed a modification date way in the future.

 

Please report what you find.

Share this post


Link to post
Share on other sites

It might be another date, such as creation date that is in the future.

 

Apparently not. For the test example in the OP:

Cambyses:~ prl$ find Computer -mtime -1m -o -atime -1m -o -ctime -1m
Cambyses:~ prl$ 

 

We run Retrospect Windows 7.6 at work. We use disk-to-disk-to-tape. From client to disk every night, from disk to new tapes every weekend and new/changed files from disk to tape on Thursday night/Friday morning. There are a couple of client computers that are very rarely used and are turned off weeks in a row. For two of these clients, there are a handful of files that are always transferred to tape Thursday night. This happens even if the client computer has been turned off for more than a week. I checked up on a few of these files and there was indeed a modification date way in the future.

But it's not a few files in my case. In the test cast I posted, its 917 files out of 2840. In the case of my system volume backup, which is what I really care about, it's about 8GB out of 20. I'm not talking about a few odd files with crazy dates on them, I'm talking about a substantial fraction of all files.

 

Please report what you find.

Done.

 

And I repeat my earlier unanswered question. Is there any way to find out from Retrospect which files are being copied and why?

Edited by prl

Share this post


Link to post
Share on other sites

And I repeat my earlier unanswered question. Is there any way to find out from Retrospect which files are being copied and why?

If you look in "Past Backups" -- you'll see what files were backed up. You may have to turn on Grooming for the media set to see the results of each past backup.

 

 

Apparently not. For the test example in the OP:

Cambyses:~ prl$ find Computer -mtime -1m -o -atime -1m -o -ctime -1m
Cambyses:~ prl$ 

That doesn't mean anything to me. I meant in terms of what Finder says about one or two of the 917 files.

Share this post


Link to post
Share on other sites

That doesn't mean anything to me. I meant in terms of what Finder says about one or two of the 917 files.

It means "find any file or folder with modification time (-mtime), access time (-atime), or inode change time (-ctime) more than one minute in the future, and list it on standard output".

 

If I knew which of the 2800-odd files were in the 917 being re-copied, I'd be in a better position to give you Finder info. Not sure what information you're looking for, exactly.

 

I've already answered Steve Maser's suggestion about looking in Past Backups to see which files were being copied into the "media set". Perhaps you didn't see it:

It's not a Backup, it's a Copy (with Overwrite Entire Volume set). It doesn't appear in Past Backups. There's no media set.

 

I again repeat my earlier unanswered question. Is there any way to find out from Retrospect which files are being copied and why?

Share this post


Link to post
Share on other sites

I've already answered Steve Maser's suggestion about looking in Past Backups to see which files were being copied into the "media set". Perhaps you didn't see it:

 

 

I again repeat my earlier unanswered question. Is there any way to find out from Retrospect which files are being copied and why?

Right, I missed that one.

 

A copy is dependent on the file systems on both the source and destination disks. Are they both Mac OS Extended (HFS+)?   Or is one an AFP or SMB mounted network drive?

 

Perhaps if you watch the copy process and simply write down the file names as they appear on the screen. You should be able to get at least some names.

 

Or if you create a temporary disk media set and perform two backups with that media set as the destination. How many files were copied the second time? You can find out which using Steve Maser's tip.

Share this post


Link to post
Share on other sites

Right, I missed that one.

No problem.

 

A copy is dependent on the file systems on both the source and destination disks. Are they both Mac OS Extended (HFS+)?   Or is one an AFP or SMB mounted network drive?

Source: Mac OS Extended (Journaled)

Destination: Mac OS Extended (Journaled)

 

Perhaps if you watch the copy process and simply write down the file names as they appear on the screen. You should be able to get at least some names.

That's not a bad idea. I'll give it a go.

 

Or if you create a temporary disk media set and perform two backups with that media set as the destination. How many files were copied the second time? You can find out which using Steve Maser's tip.

Backups appear to work just as I expect them to. This problem appears to be related specifically to Copy. I'm not sure why I'd expect to see the same behaviour on a Backup, but I guess it's worth a try.

Share this post


Link to post
Share on other sites

Source: Mac OS Extended (Journaled)

Destination: Mac OS Extended (Journaled)

Are they both local volumes? Not network volumes?

Are they both NOT case sensitive?

Backups appear to work just as I expect them to. This problem appears to be related specifically to Copy. I'm not sure why I'd expect to see the same behaviour on a Backup, but I guess it's worth a try.

That indicates there is no problem with the files dates after all.

 

This problem is more and more puzzling. And also more and more interesting.  :)

Share this post


Link to post
Share on other sites

Are they both local volumes? Not network volumes?

Are they both NOT case sensitive?

...

Both local volumes. Neither is case sensitive (that shows up in the Finder Info window as "Mac OS Extended (Case-sensitive, Journaled)", and what I posted was a Copy/Paste from the respective Info windows).

 

Source is on a HDD volume, dest is on a USB stick volume, but for my System Volume copy script, both are HDD volumes. It's the System Volume copy where I really care about this (mis)behaviour.

Edited by prl

Share this post


Link to post
Share on other sites

...

Perhaps if you watch the copy process and simply write down the file names as they appear on the screen. You should be able to get at least some names.

...

Unfortunately, most of the file names listed in the copy are ones that appear more than once in the sample, and without path info (which isn't visible in the Activities screen), it's not possible to find which instance of the name is being copied.

 

However, I did find one file that is copied over and over and which has a unique name. Here are the Finder Info windows for the original (left window) and the copy (right window). I can't see any reason why Copy keeps copying this file.

 

Share this post


Link to post
Share on other sites
Guest Steve Maser

check the extended attributes between the two files in Terminal to compare them:

 

First do an "ls -lae" on the file -- you'd get something like:

 

Desktop admin$ ls -lae letters.rtf

-rw-r--r--@ 1 admin staff 779 Dec 28 20:36 letters.rtf

 

 

Then check the extended attributes with xattr:

 

Desktop admin$ xattr letters.rtf

com.apple.FinderInfo

com.apple.quarantine

 

 

 

See if there are any differences between the source and target files? I'm wondering if the source file has extended attributes on it that aren't surviving the copy process.

 

 

And then do the same thing for a file that *doesn't* look like it's being recopied? Maybe make just the UPnP folder a Favorite folder and see if you can isolate the differences if you just copy that folder (I'm assuming there is more than one file in that folder...)

Edited by Steve Maser

Share this post


Link to post
Share on other sites

check the extended attributes between the two files in Terminal to compare them:

 

First do an "ls -lae" on the file -- you'd get something like:

 

Desktop admin$ ls -lae letters.rtf

-rw-r--r--@ 1 admin staff 779 Dec 28 20:36 letters.rtf

Cambyses:Computer prl$ ls -lae /Volumes/Backup/RetroTest/src/Beyonwiz/UPnP/upnpresources.zip src/Beyonwiz/UPnP/upnpresources.zip
-rw-r--r--@ 1 prl  admin  30582907 29 Dec  2008 /Volumes/Backup/RetroTest/src/Beyonwiz/UPnP/upnpresources.zip
-rw-r--r--@ 1 prl  admin  30582907 29 Dec  2008 src/Beyonwiz/UPnP/upnpresources.zip
Cambyses:Computer prl$ 

I'm not sure why you wanted the '-a' in that, though.

 

Then check the extended attributes with xattr:

 

Desktop admin$ xattr letters.rtf

com.apple.FinderInfo

com.apple.quarantine

 

See if there are any differences between the source and target files? I'm wondering if the source file has extended attributes on it that aren't surviving the copy process.

 

Cambyses:Computer prl$ xattr /Volumes/Backup/RetroTest/src/Beyonwiz/UPnP/upnpresources.zip src/Beyonwiz/UPnP/upnpresources.zip
/Volumes/Backup/RetroTest/src/Beyonwiz/UPnP/upnpresources.zip: com.apple.quarantine
src/Beyonwiz/UPnP/upnpresources.zip: com.apple.quarantine
Cambyses:Computer prl$ 

 

I've also tried to find any differences using ls -l@eOT, GetFileInfo and mdls, and haven't found any differences.

 

 

And then do the same thing for a file that *doesn't* look like it's being recopied? Maybe make just the UPnP folder a Favorite folder and see if you can isolate the differences if you just copy that folder (I'm assuming there is more than one file in that folder...)

I'd considered doing exactly that. I will do so.

Share this post


Link to post
Share on other sites

... Maybe make just the UPnP folder a Favorite folder and see if you can isolate the differences if you just copy that folder (I'm assuming there is more than one file in that folder...)

This has just thrown up more wierdness. The first Copy:

+  Copy using Retro copy test at 31/12/11
   To volume RetroTest...
   -  31/12/11 18:04:37: Copying UPnP
   31/12/11 18:04:50: Comparing RetroTest
   31/12/11 18:04:56: Execution completed successfully
   Completed: 40 files, 29.2 MB
   Performance: 184.7 MB/minute (134.9 copy, 292.4 compare)
   Duration: 00:00:19

 

The second copy:

+  Copy using Retro copy test at 31/12/11
   To volume RetroTest...
   -  31/12/11 18:05:21: Copying UPnP
   31/12/11 18:05:39: Comparing RetroTest
   *File "/Volumes/Cambyses User/Users/prl/Computer/src/Beyonwiz/UPnP/testMX_findings.zip": didn't compare
   *File "/Volumes/Cambyses User/Users/prl/Computer/src/Beyonwiz/UPnP/testMX_results.zip": didn't compare
   *File "/Volumes/Cambyses User/Users/prl/Computer/src/Beyonwiz/UPnP/testMX_results2.zip": didn't compare
   *File "/Volumes/Cambyses User/Users/prl/Computer/src/Beyonwiz/UPnP/upnpresources.zip": didn't compare
   31/12/11 18:05:41: Execution completed successfully
   Completed: 4 files, 29.2 MB
   Performance: 194.5 MB/minute (109.4 copy, 875.4 compare)
   Duration: 00:00:19 (00:00:01 idle/loading/preparing)

 

The third Copy:

+  Copy using Retro copy test at 31/12/11
   To volume RetroTest...
   -  31/12/11 18:06:50: Copying UPnP
   31/12/11 18:07:03: Comparing RetroTest
   *File "/Volumes/Cambyses User/Users/prl/Computer/src/Beyonwiz/UPnP/upnpresources.zip": didn't compare
   31/12/11 18:07:05: Execution completed successfully
   Completed: 4 files, 29.2 MB
   Performance: 269.3 MB/minute (145.9 copy, 1,750.9 compare)
   Duration: 00:00:14

 

And, what's more, it's true that the two copies of upnpresources.zip are different:

Cambyses:Computer prl$ cmp /Volumes/Backup/RetroTest/upnpresources.zip src/Beyonwiz/UPnP/upnpresources.zip | head
/Volumes/Backup/RetroTest/upnpresources.zip src/Beyonwiz/UPnP/upnpresources.zip differ: char 1, line 1
Cambyses:Computer prl$ cmp -l /Volumes/Backup/RetroTest/upnpresources.zip src/Beyonwiz/UPnP/upnpresources.zip | head
      1   0 120
      2   0 113
      3   0   3
      4   0   4
      5   0  24
      9   0  10
     11   0 110
     12   0 111
     13   0 204
     14   0  66
Cambyses:Computer prl$ 

 

In fact, there's more than just a few bits or bytes not copied correctly here. The backup copy (and I use that term very loosely) looks like this:

Cambyses:Computer prl$ od -x /Volumes/Backup/RetroTest/upnpresources.zip | head -20
0000000      0000    0000    0000    0000    0000    0000    0000    0000
*
0171060      0001    0000    0000    0000    0000    0000    1000    0000
0171100      0000    0000    0000    0000    0000    0000    0000    0000
*
0171160      0000    0000    0000    0000    0000    8000    0000    0000
0171200      0000    0000    0000    0000    0000    0000    0000    0000
*
0171300      0000    0000    0000    0000    0000    0000    0000    0080
0171320      0000    0080    0000    0000    0000    0000    0000    0000
0171340      0000    0000    0000    0000    0000    0000    0000    0000
*
0171500      0000    0000    0000    0100    0000    0000    0000    0000
0171520      0000    0000    0000    0000    0000    0000    0000    0000
0171540      2000    0000    0000    0000    0000    0000    0000    0000
0171560      0000    0000    0000    0000    0000    0000    0000    0000
*
0171640      0000    0000    0080    0000    0000    0000    0000    0000
0171660      0000    0000    0000    0000    0000    0000    0000    0000
*
Cambyses:Computer prl$ 

 

Not at all what I'd expect a copy of a ZIP file to look like. And, not surprisingly, ZIP doesn't like it, either:

Cambyses:Computer prl$ unzip -l /Volumes/Backup/RetroTest/upnpresources.zip Archive:  /Volumes/Backup/RetroTest/upnpresources.zip
 End-of-central-directory signature not found.  Either this file is not
 a zipfile, or it constitutes one disk of a multi-part archive.  In the
 latter case the central directory and zipfile comment will be found on
 the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of /Volumes/Backup/RetroTest/upnpresources.zip or
       /Volumes/Backup/RetroTest/upnpresources.zip.zip, and cannot find /Volumes/Backup/RetroTest/upnpresources.zip.ZIP, period.
Cambyses:Computer prl$ 

 

The original is just fine:

Cambyses:Computer prl$ unzip -l src/Beyonwiz/UPnP/upnpresources.zip 
Archive:  src/Beyonwiz/UPnP/upnpresources.zip
 Length     Date   Time    Name
--------    ----   ----    ----
   54272  04-04-07 09:10   sampledevice_and_servicetemplates/Audio-1-2001Feb.doc
   29696  04-04-07 09:10   sampledevice_and_servicetemplates/CDPlayer-1-2001Feb.doc
  173056  04-04-07 09:10   sampledevice_and_servicetemplates/ChangeDisc-1.doc
   65024  04-04-07 09:10   sampledevice_and_servicetemplates/PlayCD-1 -2001Feb.doc
... more like this ...
   61440  04-04-07 09:10   documents/UPnP_Service_Checklist1.01.doc
   57610  04-04-07 09:10   documents/UPnP_Service_Checklist1.01.pdf
   89696  04-04-07 09:10   documents/UPnP_Service_Checklist1.01.rtf
   31163  04-04-07 09:10   documents/UPnP_Vendor_Implementation_Guide_Jan2001.htm
  621952  11-21-08 13:49   documents/UPnP-arch-DeviceArchitecture-v1 1-20081015.pdf
--------                   -------
36885676                   141 files
Cambyses:Computer prl$ 

 

 

I didn't get these compare errors in Retrospect when the same folder was copied as part of the larger test copy.

 

The more I look into it, the less I like it.

Share this post


Link to post
Share on other sites
In that instance, the source is a folder as a Favourite, and the destination is a folder as a Favourite. But I have the same problem when the source is a physical volume and the destination is a physical volume (my system volume and my backup system volume respectively).

I'm still trying to just get easy confirmation on the hardware involved. Are Cambyses User and Backup both locally-attached non network-shared logical volumes?

Share this post


Link to post
Share on other sites

I'm still trying to just get easy confirmation on the hardware involved. Are Cambyses User and Backup both locally-attached non network-shared logical volumes?

Does either volume have "ignore ownership" set?

What about ACLs (Access Control Lists), are these disabled on either volume?

Share this post


Link to post
Share on other sites

I'm still trying to just get easy confirmation on the hardware involved. Are Cambyses User and Backup both locally-attached non network-shared logical volumes?

I've already answered this. Cambyses User is a volume on a locally attached Firewire HDD internal SATA drive; Backup is a volume on USB memory stick. Both are formatted HFS+ journaled, case insensitive.

 

The two drives I used for my system volume and system volume backup are respectively on internal SATA and Firewire HDDs, formatted as above.

 

Edit: fixed location of Cambyses User volume.

Edited by prl

Share this post


Link to post
Share on other sites

Does either volume have "ignore ownership" set?

No.

 

What about ACLs (Access Control Lists), are these disabled on either volume?

I don't know how to do that, or even how check for it.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×