Copy/overwrite Entire Volume Still Copying Far Too Much In Retrospect 9

[prl]

I've already answered this. Cambyses User is a volume on a internal SATA drive; Backup is a volume on USB memory stick. Both are formatted HFS+ journaled, case insensitive.

...

I've moved the copy destination from the USB memory stick to the Archive volume on the external Firewire HDD, so the copy now is from volume on an internal SATA HDD to a volume on an external Firewire HDD. Both are formatted Mac OS Extended (Journaled) (case insensitive). The source HDD does not have Ignore ownership set. The destination HDD is a Time Machine destination (still has 60GB free), but for some reason, it doesn't have an option for "Ignore ownership".

 

Copy/Overwrite Entire Volume still copies 4 files over and over again each time the Copy script is run after the initial full copy of 41 files, and upnpresources.zip continues to be one of the files that is copied without apparent reason. However, I haven't had any of the data errors that I had when the memory stick was the destination. Looks like I may have to run some tests on the memory stick.

[Guest Steve Maser]

So, is it .zip files -- everywhere -- that are the *only* files that are being recopied each time? (Or is this just what I'm gathering from your recent UPnP folder test only?)

 

If so, how are the .zip files being created?

[prl]

So, is it .zip files -- everywhere -- that are the *only* files that are being recopied each time? (Or is this just what I'm gathering from your recent UPnP folder test only?)

 

If so, how are the .zip files being created?

No, it's not just .zip files that were being recopied in my original test. It just happened that this particular .zip file was the only one I could uniquely identify from the files that appeared in the Activities window while the original test was running. I thought I made that clear when I first posted about the UPnP folder test.

 

Is there any way I can find out from Retrospect which files (full paths) are being re-copied and why? Sorry to harp on about this, but that seems the key to finding out what's going on.

 

Also, has anyone responding here tried to replicate this problem? If not, is anyone willing to try?

[Guest Steve Maser]

I'm willing to try when I'm back in the office (I have no external FW drives at home...)

 

Three other questions before I do this, though:

 

1) What OS are you using for these copy script tests?

 

2) Do you have something non-standard about how you are doing dates? Your log posts show:

 

Copy using Retro copy test at 31/12/11

 

3) In your copy script, did you change any of the default "options"? (And, I'm assuming you've tried starting with a *new* script and are not using a copy of an existing script?)

[Guest Steve Maser]

actually, I had a flash drive at home, so I was able to test this.

 

First copy pass of a folder on my desktop to the flash drive was like this:

 

Copy using Copy test at 1/2/12 (Activity Thread 1)

To volume FLASH...

****** Warning: volume "FLASH" has the "Ignore ownership" setting enabled. ******

- 1/2/12 10:04:19 AM: Copying misc desktop files on Macintosh HD

1/2/12 10:05:15 AM: Comparing FLASH

*File "/Volumes/Macintosh HD/Users/admin/Desktop/misc desktop files/.DS_Store": didn't compare

1/2/12 10:05:32 AM: Execution completed successfully

Completed: 127 files, 162.5 MB

Performance: 267.1 MB/minute (174.1 copy, 573.5 compare)

Duration: 00:01:13

 

 

subsequent copies did show -- like you -- that some files were recopied:

 

+ Copy using Copy test at 1/2/12 (Activity Thread 1)

To volume FLASH...

****** Warning: volume "FLASH" has the "Ignore ownership" setting enabled. ******

- 1/2/12 10:06:11 AM: Copying misc desktop files on Macintosh HD

1/2/12 10:06:12 AM: Comparing FLASH

1/2/12 10:06:15 AM: Execution completed successfully

Completed: 42 files, 6.7 MB

Performance: 267.6 MB/minute (401.4 copy, 133.8 compare)

Duration: 00:00:03

 

+ Copy using Copy test at 1/2/12 (Activity Thread 1)

To volume FLASH...

- 1/2/12 10:06:46 AM: Copying misc desktop files on Macintosh HD

1/2/12 10:06:48 AM: Comparing FLASH

1/2/12 10:06:49 AM: Execution completed successfully

Completed: 41 files, 6.7 MB

Performance: 266.7 MB/minute (200 copy, 400.1 compare)

Duration: 00:00:03

 

(I flipped the ignore ownership box between runs 2 and 3)

 

 

However, I then modified the copy script from the defaults to turn off: Source --> Macintosh --> Use attribute modification date when matching.

 

After doing that, no files were copied again on the 4th run of the script:

 

+ Copy using Copy test at 1/2/12 (Activity Thread 1)

To volume FLASH...

- 1/2/12 10:09:39 AM: Copying misc desktop files

1/2/12 10:09:39 AM: Execution completed successfully

Duration: 00:00:00

 

 

So the attribute modification date is key here -- but it's clearly not something that exists on all files (some of the files in my test folder were downloads from the Internet/Mail.app attachments -- most were not. I'd have to dig into which files have this issue to figure that out.

[Guest Steve Maser]

so, what I've found:

 

1) If you need to retain the extended attributes (visible with the xattr command), then you either suffer through the recopying *or* tell the script not to match against that.

 

2) If you want to leave the default match script option, but don't care about the extended attributes, you can remove the attributes from the files with "xattr -c <file>"

 

ie:

 

before:

 

-rw-r--r--@ 1 admin staff 101728 Nov 29 20:35 Ameritrade Open Your Account.pdf

Maserhome:Ameritrade files admin$ xattr "Ameritrade Open Your Account.pdf"

com.apple.quarantine

 

then: xattr -c *

 

now looks like:

 

-rw-r--r-- 1 admin staff 101728 Nov 29 20:35 Ameritrade Open Your Account.pdf

 

and the file -- with the default script options -- will not be recopied.

 

 

The copy process seems to change the time stamp of the attributes, so Retrospect is doing what is normally done with a UNIX copy process.

[prl]

Steve, thanks for replicating the problem.

 

I'm using OS X Snow Leopard 10.6.8. I've still got some legacy application stuff to change from before I make the switch to Lion.

 

I'm not sure what you're asking about my using an unusual date setup, or why you think that "Copy using Retro copy test at 31/12/11" indicates anything more than that my system preference for date formats is dd/mm/yy. That's just the normal date format for the part of the world where I live (and using mm/dd/yy just hurts my head). My time zone is currently UTC+1100 (Australian Eastern Daylight Time).

 

My test script was simply created by clicking on the Copy button to get the Copy Wizard, selecting "Make an exact copy of source volume of folder", then selecting a Favorite folder for source and destination. I've changed the source and destination folders around for the various other tests I've made, but I haven't changed any of the options.

 

My system volume copy script was made the same way, with the exception that I selected volumes instead of folders, and it has All Files Except Cache Files instead of All Files (the default).

 

My main concern in all of this is that at each copy I do of my system volume, about 8GB of about 20GB is copied every time, whether I've had any major changes (software updates, etc) to the source volume or not. Copying 8GB when a far smaller amount should be copied is bad enough, but additionally Norton AntiVirus's copy protection kicks in and virus checks every one of the quarter-million or so files being copied. This slows the already over-sized copy down to a snail's pace.

 

For my original concern, the time taken to do the system folder backup, I really don't think it's advisable to remove all the extended attributes from a quarter of a million files, for many of which I'd have no idea of whether the attributes were important or not.

 

I've already posted that Mayoff has suggested selecting Source --> Macintosh --> Use attribute modification date when matching (link to his post in my earlier post). However, I've never seen any post about what the consequences of such a choice might be.

 

Really, for me, it boils down to Retrospect not meeting its own stated capability for Copy:

Creating a Copy Script Manually

...

Overwrite entire volume ... Retrospect saves time by not copying identical files, that is, files that share the same location, name, modification date and time, etc., that are already present on the destination. ...

This simply does not seem to be the case.

[Guest Steve Maser]

On Unix systems, dates for Extended Attributes are determined when the file is actually copied (you'd see the same results with a standard "cp" command.) As such, the target file is not 100% "identical" to the source file because of this.

 

Retrospect doesn't act any differently here (that's why they added that check box about the attribute modification date for the script. (NOTE: I have zero clue what mechanism Retrospect uses for writing/copying extended attributes on files in a copy process...)

 

 

There are utilities (CCC and SuperDuper come to mind) that retain the extended attributes *exactly* when you copy them, but unless you need that exact of a match -- for those attribute time stamps (which, note, are not the *file modification* time stamps )-- you may not need to worry about it. If you do, then you might want to "tar" up the directories for archive purposes or use a different tool for this.

[prl]

Hi, Steve. That just sounds like a confirmation that the operation of this part of the Retrospect Copy process is pretty much unknown outside Retrospect.

 

Unfortunately, according to the User Guide, if 'Use attribute modification date when matching' is switched off, changes to extended attributes aren't propagated from the source to the destination.

 

It seems odd that Retrospect decided that if the only difference in a file state was the attribute modification date, that the whole file should be copied, and not just the attributes.

 

As for using tar, I'm not sure what you're getting at. Tar does have the -u option (update files with a newer modification date), but I want a bootable copy. I can't boot a tar file.

 

All I'm asking is that Retrospect Copy performs the way that is described in the documentation, or at least for the documentation to describe what it actually does.

[Guest Steve Maser]

File a feature request with Customer Service, then. This would require an internal rewrite of how they do the "copy" process, I'm sure.

 

After I went through the above, I had some vague memory of that this is pretty much how Retrospect has always done this copy process.

Edited January 4, 2012 by Steve Maser

[prl]

It's a Feature Request to have Retrospect work as the documentation describes? Surely it'd be a Bug Report!

[Guest Steve Maser]

The documentation excerpt that you stated said: Retrospect saves time by not copying identical files...

 

 

When you are doing the second run, the *target* files are no longer identical to the *source* files -- because the extended attribute date is set to the offset of when the files were created (copied) on the target.

 

 

So, from that perspective, your second (and third) runs *are* doing what the manual says it will do -- it's only copying files (again) that are not *identical* to the source. That's why Retrospect gives you that option to not do matching based on the extended attributes.

 

 

The feature request would be to have Retrospect changed so that extended attributes are *exactly the same* (like when doing rsync) on the target. Right now -- and forever? -- it doesn't do that.

[Lennart_T]

NOTE: Most of the file attributes are strictly there for application and Finder support. You can add

your own attributes for whatever reason you want, but for the most part in OS X they are there to

support the Finder

http://my.safaribooksonline.com/book/operating-systems/9781430237624/chapter-23-introducing-darwin-and-the-mac-os-x-command-line/465

 

So I would not care about the attributes being different and have Retrospect not match the extended attributes.

[51ae86ee-d35a-4c46-b879-c3b8399377a3]

Just on a separate note- I do fid some of the other applications do a faster job of scanning, and get the same identical attributes copied. So I often use something else to do the copy specifically for that reason (and then sidestep this issue entirely).

[prl]

Just on a separate note- I do fid some of the other applications do a faster job of scanning, and get the same identical attributes copied. So I often use something else to do the copy specifically for that reason (and then sidestep this issue entirely).

Which applications? In the past I've used Deja Vu, which isn't fast scanning, and seems to only copy files that have actually changed, but doesn't do a check. Carbon Copy Cloner wasn't able to do bootable copies for a while, but I see now that it says it does do them. Perhaps that's the way for me to go.

 

Thanks for prompting me to go back and have another look at CCC.

[Guest Steve Maser]

CCC has done bootable copies since practically the beginning of it's release (AFAIK). I've used it to image machines probably 5000 times now.

[Lennart_T]

There are utilities (CCC and SuperDuper come to mind) that retain the extended attributes *exactly* when you copy them, but unless you need that exact of a match -- for those attribute time stamps (which, note, are not the *file modification* time stamps )-- you may not need to worry about it. If you do, then you might want to "tar" up the directories for archive purposes or use a different tool for this.

Which applications?

See Steve's post.

[prl]

See Steve's post.

Thanks. I was wondering if demani had other suggestions.

[Lennart_T]

Thanks. I was wondering if demani had other suggestions.

Well, both CCC and SuperDuper are highly regarded. Give them a try. 

[prl]

I've just tried CCC. Its initial backup of my system volume to the backup system (previously maintained by using a Copy script in Retro9) volume copied 629MB and took 10min4sec. A second copy soon after copied 11MB and took 8min42sec. This contrasts with Retro9 Copy copying 8GB and taking several hours! Somehow, CCC also appaers to sneak under Nortin AntiVirus's "radar", so the copy wasn't bogged down by NAV racing behind the copy to virus-check the copied data.

 

Until Retrospect's Copy can match this kind of performance, I'll be sticking with CCC.

 

By the way, the only way I could find this topic was by going into My Content. I couldn't another way to find any topics that hadn't been posted to in the last week. I tried changing the filter settings in the Custom link on the top of topic list in the forum page, but that didn't seem to change anything.