prl Posted December 28, 2011 Report Share Posted December 28, 2011 (edited) Copy/Overwrite Entire Volume still seems to copy far more than necessary. Here are some results from a small test. The initial copy, which should copy the whole test folder: + Copy using Retro copy test at 28/12/11 To volume RetroTest... - 28/12/11 15:51:27: Copying Computer 28/12/11 15:54:38: Comparing RetroTest 28/12/11 15:55:34: Execution completed successfully Completed: 2840 files, 490.7 MB Performance: 241.3 MB/minute (156.5 copy, 535.2 compare) Duration: 00:04:06 (00:00:02 idle/loading/preparing) Then immediately another copy using the same script, without touching anything in either the source or destination folders: + Copy using Retro copy test at 28/12/11 To volume RetroTest... - 28/12/11 15:55:52: Copying Computer 28/12/11 15:57:00: Comparing RetroTest 28/12/11 15:57:16: Execution completed successfully Completed: 917 files, 107.6 MB Performance: 172.1 MB/minute (109.4 copy, 403.4 compare) Duration: 00:01:24 (00:00:09 idle/loading/preparing) So who's gone and changed 107.6MB in 917 files while I wasn't looking? And then the same run again, and again not modifying anything in either the source or destination folders: + Copy using Retro copy test at 28/12/11 To volume RetroTest... - 28/12/11 15:58:29: Copying Computer 28/12/11 15:59:39: Comparing RetroTest 28/12/11 15:59:55: Execution completed successfully Completed: 917 files, 107.6 MB Performance: 167.6 MB/minute (104.1 copy, 430.3 compare) Duration: 00:01:25 (00:00:07 idle/loading/preparing) And someone's gone and changed 107.6MB in 917 files on me again! This is what the manual says is supposed to happen (my emphasis): Overwrite entire volume replaces the entire contents of the destination volume or Favorite Folder with the selected files and folders from the source volume or Favorite Folder. Everything else on the destination volume is deleted. Retrospect saves time by not copying identical files, that is, files that share the same location, name, modification date and time, etc., that are already present on the destination. New files are added, and different versions of files already present on the destination are replaced by the files from the source, even if the file already present on the destination is newer. That doesn't sound like what's happening to me! Is there some way I can find out which files are being copied? And even better, why? Edited December 28, 2011 by prl Quote Link to comment Share on other sites More sharing options...
Guest Steve Maser Posted December 28, 2011 Report Share Posted December 28, 2011 If you look in "Past Backups" -- you'll see what files were backed up. You may have to turn on Grooming for the media set to see the results of each past backup. Quote Link to comment Share on other sites More sharing options...
prl Posted December 28, 2011 Author Report Share Posted December 28, 2011 If you look in "Past Backups" -- you'll see what files were backed up. You may have to turn on Grooming for the media set to see the results of each past backup. It's not a Backup, it's a Copy (with Overwrite Entire Volume set). It doesn't appear in Past Backups. There's no media set. Quote Link to comment Share on other sites More sharing options...
prl Posted December 28, 2011 Author Report Share Posted December 28, 2011 I posted about this problem in Retro 8 in this forum in March 2011. That posts links to posts in the old EMC2 Retrospect forum about the same problem in Retro 8, including a similarly detailed post in the Bugs section. Those posts were made in July and October 2010. There was no response to either of those two posts. Quote Link to comment Share on other sites More sharing options...
f30e6e60-a263-4d69-8f60-c92703669308 Posted December 28, 2011 Report Share Posted December 28, 2011 Here are some results from a small test Perhaps some more specific information about your test parameters. What is the Source? A logical volume? A Favorites Folder? Locally attached? Network share? What is the Destination? (see above) ? Specific steps should include configuration information so others can attempt to reproduce. -dave Quote Link to comment Share on other sites More sharing options...
prl Posted December 28, 2011 Author Report Share Posted December 28, 2011 Perhaps some more specific information about your test parameters. What is the Source? A logical volume? A Favorites Folder? Locally attached? Network share? What is the Destination? (see above) ? Specific steps should include configuration information so others can attempt to reproduce. -dave In that instance, the source is a folder as a Favourite, and the destination is a folder as a Favourite. But I have the same problem when the source is a physical volume and the destination is a physical volume (my system volume and my backup system volume respectively). There's another report of the problem here. Quote Link to comment Share on other sites More sharing options...
Lennart_T Posted December 28, 2011 Report Share Posted December 28, 2011 And someone's gone and changed 107.6MB in 917 files on me again! My guess is that the 917 files have a modification date in the future. That way, they will always be "newer" than the last copy. Quote Link to comment Share on other sites More sharing options...
prl Posted December 28, 2011 Author Report Share Posted December 28, 2011 My guess is that the 917 files have a modification date in the future. That way, they will always be "newer" than the last copy. And there are 8GB or so of files on my system volume with future dates, too? I will check the dates in the test example anyway, but I'm skeptical about that as a cause. Quote Link to comment Share on other sites More sharing options...
prl Posted December 28, 2011 Author Report Share Posted December 28, 2011 My guess is that the 917 files have a modification date in the future. That way, they will always be "newer" than the last copy. And there are 8GB or so of files on my system volume with future dates, too? I will check the dates in the test example anyway, but I'm skeptical about that as a cause. Mayoff seemed to think the problem was with the attribute modification date, and suggested turning matching on that off. Which suggests something quite strange going on. But he was talking about "copying every file every time" which also isn't what's happening here. Quote Link to comment Share on other sites More sharing options...
Lennart_T Posted December 28, 2011 Report Share Posted December 28, 2011 And there are 8GB or so of files on my system volume with future dates, too? I will check the dates in the test example anyway, but I'm skeptical about that as a cause. It might be another date, such as creation date that is in the future. We run Retrospect Windows 7.6 at work. We use disk-to-disk-to-tape. From client to disk every night, from disk to new tapes every weekend and new/changed files from disk to tape on Thursday night/Friday morning. There are a couple of client computers that are very rarely used and are turned off weeks in a row. For two of these clients, there are a handful of files that are always transferred to tape Thursday night. This happens even if the client computer has been turned off for more than a week. I checked up on a few of these files and there was indeed a modification date way in the future. Please report what you find. Quote Link to comment Share on other sites More sharing options...
prl Posted December 28, 2011 Author Report Share Posted December 28, 2011 (edited) It might be another date, such as creation date that is in the future. Apparently not. For the test example in the OP: Cambyses:~ prl$ find Computer -mtime -1m -o -atime -1m -o -ctime -1m Cambyses:~ prl$ We run Retrospect Windows 7.6 at work. We use disk-to-disk-to-tape. From client to disk every night, from disk to new tapes every weekend and new/changed files from disk to tape on Thursday night/Friday morning. There are a couple of client computers that are very rarely used and are turned off weeks in a row. For two of these clients, there are a handful of files that are always transferred to tape Thursday night. This happens even if the client computer has been turned off for more than a week. I checked up on a few of these files and there was indeed a modification date way in the future. But it's not a few files in my case. In the test cast I posted, its 917 files out of 2840. In the case of my system volume backup, which is what I really care about, it's about 8GB out of 20. I'm not talking about a few odd files with crazy dates on them, I'm talking about a substantial fraction of all files. Please report what you find. Done. And I repeat my earlier unanswered question. Is there any way to find out from Retrospect which files are being copied and why? Edited December 28, 2011 by prl Quote Link to comment Share on other sites More sharing options...
Lennart_T Posted December 28, 2011 Report Share Posted December 28, 2011 And I repeat my earlier unanswered question. Is there any way to find out from Retrospect which files are being copied and why? If you look in "Past Backups" -- you'll see what files were backed up. You may have to turn on Grooming for the media set to see the results of each past backup. Apparently not. For the test example in the OP: Cambyses:~ prl$ find Computer -mtime -1m -o -atime -1m -o -ctime -1m Cambyses:~ prl$ That doesn't mean anything to me. I meant in terms of what Finder says about one or two of the 917 files. Quote Link to comment Share on other sites More sharing options...
prl Posted December 28, 2011 Author Report Share Posted December 28, 2011 That doesn't mean anything to me. I meant in terms of what Finder says about one or two of the 917 files. It means "find any file or folder with modification time (-mtime), access time (-atime), or inode change time (-ctime) more than one minute in the future, and list it on standard output". If I knew which of the 2800-odd files were in the 917 being re-copied, I'd be in a better position to give you Finder info. Not sure what information you're looking for, exactly. I've already answered Steve Maser's suggestion about looking in Past Backups to see which files were being copied into the "media set". Perhaps you didn't see it: It's not a Backup, it's a Copy (with Overwrite Entire Volume set). It doesn't appear in Past Backups. There's no media set. I again repeat my earlier unanswered question. Is there any way to find out from Retrospect which files are being copied and why? Quote Link to comment Share on other sites More sharing options...
Lennart_T Posted December 29, 2011 Report Share Posted December 29, 2011 I've already answered Steve Maser's suggestion about looking in Past Backups to see which files were being copied into the "media set". Perhaps you didn't see it: I again repeat my earlier unanswered question. Is there any way to find out from Retrospect which files are being copied and why? Right, I missed that one. A copy is dependent on the file systems on both the source and destination disks. Are they both Mac OS Extended (HFS+)? Or is one an AFP or SMB mounted network drive? Perhaps if you watch the copy process and simply write down the file names as they appear on the screen. You should be able to get at least some names. Or if you create a temporary disk media set and perform two backups with that media set as the destination. How many files were copied the second time? You can find out which using Steve Maser's tip. Quote Link to comment Share on other sites More sharing options...
prl Posted December 30, 2011 Author Report Share Posted December 30, 2011 Right, I missed that one. No problem. A copy is dependent on the file systems on both the source and destination disks. Are they both Mac OS Extended (HFS+)? Or is one an AFP or SMB mounted network drive? Source: Mac OS Extended (Journaled) Destination: Mac OS Extended (Journaled) Perhaps if you watch the copy process and simply write down the file names as they appear on the screen. You should be able to get at least some names. That's not a bad idea. I'll give it a go. Or if you create a temporary disk media set and perform two backups with that media set as the destination. How many files were copied the second time? You can find out which using Steve Maser's tip. Backups appear to work just as I expect them to. This problem appears to be related specifically to Copy. I'm not sure why I'd expect to see the same behaviour on a Backup, but I guess it's worth a try. Quote Link to comment Share on other sites More sharing options...
Lennart_T Posted December 30, 2011 Report Share Posted December 30, 2011 Source: Mac OS Extended (Journaled) Destination: Mac OS Extended (Journaled) Are they both local volumes? Not network volumes?Are they both NOT case sensitive? Backups appear to work just as I expect them to. This problem appears to be related specifically to Copy. I'm not sure why I'd expect to see the same behaviour on a Backup, but I guess it's worth a try. That indicates there is no problem with the files dates after all. This problem is more and more puzzling. And also more and more interesting. Quote Link to comment Share on other sites More sharing options...
prl Posted December 31, 2011 Author Report Share Posted December 31, 2011 (edited) Are they both local volumes? Not network volumes? Are they both NOT case sensitive? ... Both local volumes. Neither is case sensitive (that shows up in the Finder Info window as "Mac OS Extended (Case-sensitive, Journaled)", and what I posted was a Copy/Paste from the respective Info windows). Source is on a HDD volume, dest is on a USB stick volume, but for my System Volume copy script, both are HDD volumes. It's the System Volume copy where I really care about this (mis)behaviour. Edited December 31, 2011 by prl Quote Link to comment Share on other sites More sharing options...
prl Posted December 31, 2011 Author Report Share Posted December 31, 2011 ... Perhaps if you watch the copy process and simply write down the file names as they appear on the screen. You should be able to get at least some names. ... Unfortunately, most of the file names listed in the copy are ones that appear more than once in the sample, and without path info (which isn't visible in the Activities screen), it's not possible to find which instance of the name is being copied. However, I did find one file that is copied over and over and which has a unique name. Here are the Finder Info windows for the original (left window) and the copy (right window). I can't see any reason why Copy keeps copying this file. Quote Link to comment Share on other sites More sharing options...
Guest Steve Maser Posted December 31, 2011 Report Share Posted December 31, 2011 (edited) check the extended attributes between the two files in Terminal to compare them: First do an "ls -lae" on the file -- you'd get something like: Desktop admin$ ls -lae letters.rtf -rw-r--r--@ 1 admin staff 779 Dec 28 20:36 letters.rtf Then check the extended attributes with xattr: Desktop admin$ xattr letters.rtf com.apple.FinderInfo com.apple.quarantine See if there are any differences between the source and target files? I'm wondering if the source file has extended attributes on it that aren't surviving the copy process. And then do the same thing for a file that *doesn't* look like it's being recopied? Maybe make just the UPnP folder a Favorite folder and see if you can isolate the differences if you just copy that folder (I'm assuming there is more than one file in that folder...) Edited December 31, 2011 by Steve Maser Quote Link to comment Share on other sites More sharing options...
prl Posted December 31, 2011 Author Report Share Posted December 31, 2011 check the extended attributes between the two files in Terminal to compare them: First do an "ls -lae" on the file -- you'd get something like: Desktop admin$ ls -lae letters.rtf -rw-r--r--@ 1 admin staff 779 Dec 28 20:36 letters.rtf Cambyses:Computer prl$ ls -lae /Volumes/Backup/RetroTest/src/Beyonwiz/UPnP/upnpresources.zip src/Beyonwiz/UPnP/upnpresources.zip -rw-r--r--@ 1 prl admin 30582907 29 Dec 2008 /Volumes/Backup/RetroTest/src/Beyonwiz/UPnP/upnpresources.zip -rw-r--r--@ 1 prl admin 30582907 29 Dec 2008 src/Beyonwiz/UPnP/upnpresources.zip Cambyses:Computer prl$ I'm not sure why you wanted the '-a' in that, though. Then check the extended attributes with xattr: Desktop admin$ xattr letters.rtf com.apple.FinderInfo com.apple.quarantine See if there are any differences between the source and target files? I'm wondering if the source file has extended attributes on it that aren't surviving the copy process. Cambyses:Computer prl$ xattr /Volumes/Backup/RetroTest/src/Beyonwiz/UPnP/upnpresources.zip src/Beyonwiz/UPnP/upnpresources.zip /Volumes/Backup/RetroTest/src/Beyonwiz/UPnP/upnpresources.zip: com.apple.quarantine src/Beyonwiz/UPnP/upnpresources.zip: com.apple.quarantine Cambyses:Computer prl$ I've also tried to find any differences using ls -l@eOT, GetFileInfo and mdls, and haven't found any differences. And then do the same thing for a file that *doesn't* look like it's being recopied? Maybe make just the UPnP folder a Favorite folder and see if you can isolate the differences if you just copy that folder (I'm assuming there is more than one file in that folder...) I'd considered doing exactly that. I will do so. Quote Link to comment Share on other sites More sharing options...
prl Posted December 31, 2011 Author Report Share Posted December 31, 2011 ... Maybe make just the UPnP folder a Favorite folder and see if you can isolate the differences if you just copy that folder (I'm assuming there is more than one file in that folder...) This has just thrown up more wierdness. The first Copy: + Copy using Retro copy test at 31/12/11 To volume RetroTest... - 31/12/11 18:04:37: Copying UPnP 31/12/11 18:04:50: Comparing RetroTest 31/12/11 18:04:56: Execution completed successfully Completed: 40 files, 29.2 MB Performance: 184.7 MB/minute (134.9 copy, 292.4 compare) Duration: 00:00:19 The second copy: + Copy using Retro copy test at 31/12/11 To volume RetroTest... - 31/12/11 18:05:21: Copying UPnP 31/12/11 18:05:39: Comparing RetroTest *File "/Volumes/Cambyses User/Users/prl/Computer/src/Beyonwiz/UPnP/testMX_findings.zip": didn't compare *File "/Volumes/Cambyses User/Users/prl/Computer/src/Beyonwiz/UPnP/testMX_results.zip": didn't compare *File "/Volumes/Cambyses User/Users/prl/Computer/src/Beyonwiz/UPnP/testMX_results2.zip": didn't compare *File "/Volumes/Cambyses User/Users/prl/Computer/src/Beyonwiz/UPnP/upnpresources.zip": didn't compare 31/12/11 18:05:41: Execution completed successfully Completed: 4 files, 29.2 MB Performance: 194.5 MB/minute (109.4 copy, 875.4 compare) Duration: 00:00:19 (00:00:01 idle/loading/preparing) The third Copy: + Copy using Retro copy test at 31/12/11 To volume RetroTest... - 31/12/11 18:06:50: Copying UPnP 31/12/11 18:07:03: Comparing RetroTest *File "/Volumes/Cambyses User/Users/prl/Computer/src/Beyonwiz/UPnP/upnpresources.zip": didn't compare 31/12/11 18:07:05: Execution completed successfully Completed: 4 files, 29.2 MB Performance: 269.3 MB/minute (145.9 copy, 1,750.9 compare) Duration: 00:00:14 And, what's more, it's true that the two copies of upnpresources.zip are different: Cambyses:Computer prl$ cmp /Volumes/Backup/RetroTest/upnpresources.zip src/Beyonwiz/UPnP/upnpresources.zip | head /Volumes/Backup/RetroTest/upnpresources.zip src/Beyonwiz/UPnP/upnpresources.zip differ: char 1, line 1 Cambyses:Computer prl$ cmp -l /Volumes/Backup/RetroTest/upnpresources.zip src/Beyonwiz/UPnP/upnpresources.zip | head 1 0 120 2 0 113 3 0 3 4 0 4 5 0 24 9 0 10 11 0 110 12 0 111 13 0 204 14 0 66 Cambyses:Computer prl$ In fact, there's more than just a few bits or bytes not copied correctly here. The backup copy (and I use that term very loosely) looks like this: Cambyses:Computer prl$ od -x /Volumes/Backup/RetroTest/upnpresources.zip | head -20 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 0171060 0001 0000 0000 0000 0000 0000 1000 0000 0171100 0000 0000 0000 0000 0000 0000 0000 0000 * 0171160 0000 0000 0000 0000 0000 8000 0000 0000 0171200 0000 0000 0000 0000 0000 0000 0000 0000 * 0171300 0000 0000 0000 0000 0000 0000 0000 0080 0171320 0000 0080 0000 0000 0000 0000 0000 0000 0171340 0000 0000 0000 0000 0000 0000 0000 0000 * 0171500 0000 0000 0000 0100 0000 0000 0000 0000 0171520 0000 0000 0000 0000 0000 0000 0000 0000 0171540 2000 0000 0000 0000 0000 0000 0000 0000 0171560 0000 0000 0000 0000 0000 0000 0000 0000 * 0171640 0000 0000 0080 0000 0000 0000 0000 0000 0171660 0000 0000 0000 0000 0000 0000 0000 0000 * Cambyses:Computer prl$ Not at all what I'd expect a copy of a ZIP file to look like. And, not surprisingly, ZIP doesn't like it, either: Cambyses:Computer prl$ unzip -l /Volumes/Backup/RetroTest/upnpresources.zip Archive: /Volumes/Backup/RetroTest/upnpresources.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of /Volumes/Backup/RetroTest/upnpresources.zip or /Volumes/Backup/RetroTest/upnpresources.zip.zip, and cannot find /Volumes/Backup/RetroTest/upnpresources.zip.ZIP, period. Cambyses:Computer prl$ The original is just fine: Cambyses:Computer prl$ unzip -l src/Beyonwiz/UPnP/upnpresources.zip Archive: src/Beyonwiz/UPnP/upnpresources.zip Length Date Time Name -------- ---- ---- ---- 54272 04-04-07 09:10 sampledevice_and_servicetemplates/Audio-1-2001Feb.doc 29696 04-04-07 09:10 sampledevice_and_servicetemplates/CDPlayer-1-2001Feb.doc 173056 04-04-07 09:10 sampledevice_and_servicetemplates/ChangeDisc-1.doc 65024 04-04-07 09:10 sampledevice_and_servicetemplates/PlayCD-1 -2001Feb.doc ... more like this ... 61440 04-04-07 09:10 documents/UPnP_Service_Checklist1.01.doc 57610 04-04-07 09:10 documents/UPnP_Service_Checklist1.01.pdf 89696 04-04-07 09:10 documents/UPnP_Service_Checklist1.01.rtf 31163 04-04-07 09:10 documents/UPnP_Vendor_Implementation_Guide_Jan2001.htm 621952 11-21-08 13:49 documents/UPnP-arch-DeviceArchitecture-v1 1-20081015.pdf -------- ------- 36885676 141 files Cambyses:Computer prl$ I didn't get these compare errors in Retrospect when the same folder was copied as part of the larger test copy. The more I look into it, the less I like it. Quote Link to comment Share on other sites More sharing options...
f30e6e60-a263-4d69-8f60-c92703669308 Posted December 31, 2011 Report Share Posted December 31, 2011 In that instance, the source is a folder as a Favourite, and the destination is a folder as a Favourite. But I have the same problem when the source is a physical volume and the destination is a physical volume (my system volume and my backup system volume respectively). I'm still trying to just get easy confirmation on the hardware involved. Are Cambyses User and Backup both locally-attached non network-shared logical volumes? Quote Link to comment Share on other sites More sharing options...
Lennart_T Posted December 31, 2011 Report Share Posted December 31, 2011 I'm still trying to just get easy confirmation on the hardware involved. Are Cambyses User and Backup both locally-attached non network-shared logical volumes? Does either volume have "ignore ownership" set? What about ACLs (Access Control Lists), are these disabled on either volume? Quote Link to comment Share on other sites More sharing options...
prl Posted December 31, 2011 Author Report Share Posted December 31, 2011 (edited) I'm still trying to just get easy confirmation on the hardware involved. Are Cambyses User and Backup both locally-attached non network-shared logical volumes? I've already answered this. Cambyses User is a volume on a locally attached Firewire HDD internal SATA drive; Backup is a volume on USB memory stick. Both are formatted HFS+ journaled, case insensitive. The two drives I used for my system volume and system volume backup are respectively on internal SATA and Firewire HDDs, formatted as above. Edit: fixed location of Cambyses User volume. Edited January 1, 2012 by prl Quote Link to comment Share on other sites More sharing options...
prl Posted December 31, 2011 Author Report Share Posted December 31, 2011 Does either volume have "ignore ownership" set? No. What about ACLs (Access Control Lists), are these disabled on either volume? I don't know how to do that, or even how check for it. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.