Retrospect 6.5 and Duplicate Replace - why copying existing files?

[Guest]

Greetings,

 

I use Retrospect 6.5 as part of a kit to backup a Fireball music server.

I have read the FAQ's and documents and am pretty familiar with the Retrospect software and other backup software.

 

What has me confused is this. I used the "duplicate" "replace entire volume" option for the second time to backup the unit. Now based on the documentation, I expected it to ONLY have to copy the NEW files. In other words, files that were not there the last time the duplicate was run. If files were exactly the same, it should have left them alone.

But retrospect re-copied everything. The new files (as expected) but also each and every old file that was absolutely there for the last duplicate. Now at the end of the day, I had a good backup, but it took forever vs. just taking a short time to copy the new files...

 

Now I talked to Dantz support.....and their suggestion was to use the "replace corresponding files" option. Now again, after reading the documentation, I understand the difference between the two options. And based on what I read, if the files are the SAME (name, path, date, etc) either option should leave them alon.

 

Is there a setting somewhere that I am missing here? Was phone support totally offbase???

 

Any thoughts or suggestions are most welcome,

 

Thanks,

BJB

[CallMeDave]

Quote:

---

If files were exactly the same, it should have left them alone.

 

---

 

Correct.

 

This would be an excellent time for you to provide a complete description of your hardware and sofware setup, as well as exactly what you did, step-by-step.

 

 

Dave

[Guest]

I am running a Dell XPS 3ghz PC with 1 gig memory running XP Pro backing up an escient fireball E-120 over a wired network to an external Maxtor drive connected to the PC via Firewire. Is there a place in your user setup to allow you to input your equipment etc? I did not see it, just personal stuff.

 

I utilized retrospect express 6.5 duplicate>replace entire volume>

 

I went to source> advanced and set the UNC path for the source content directory.

Selected all files.

 

Started the backup.

 

I saw it scan the drives, and then it started copying. And it obviously copied the entire drive again as the new files should have only been a handful. This takes hours....

 

Based on the reply, I assume I am correct that Retrospect support might have been...well incorrect that using the "replace corresponding files" might have fixed this option.

 

So I guess there is no setting in the backup software that changes this behaviour?

I have the external drive letter hardcoded so that path should not change. I am using the UNC path for the source so that does not change....I guess somehow the fireball might have flipped a bit or something on the files but I doubt it.

 

If there are no ideas I guess I will just try it again with the "replace entire volume" option and see what happens. It is just that support absolutely understood my question, but all the documentation I have read disputes their recommendation.

 

BJB

[Guest]

Is there additional information required to help diagnose this?

 

BJB

[CallMeDave]

Although both the Macintosh and Windows versions of Retrospect are designed to work the same way in this regard, you might have more responses posting your 6.5 (Windows) version question in one of the appropriate Forum catagories; this is the Mac OS X Forum.

 

Dave

[Guest]

CallMeDave,

I aplogize...sorry about that. Will repost there.

 

 

BJB

[Larry Gegel]

I am seeing the same thing in version 6.1.126 on my Mac.

 

I am certain the files are not changing. My main volume contains about .5 million files. About 10K of those are backed up every time the duplicate is done. The 10K files are the same ones every time as best as I can tell. If I execute the script for this duplicate, and once it's finished execute the same script again the same files will be copied. The files are scattered all over the place. Some are header and source files buried deep in the Developer folder that I know I have never touched. Others are disk images that I downloaded months ago and have not touched since. Some Mac OS 9 files and apps get copied and I haven't launched Classic in as long as I can remember. Something odd is definitely going on.

 

Larry

[waltr]

hi yellow,

 

as promised:

 

although you give _no_ info on your system, i think it's safe to guess you are on at least 10.4. since your files could be quite old (as per your post), here is my best guess:

 

http://kb.dantz.com/article.asp?article=7982&p=2

 

if you don't think that is the problem, please post here (not in the windows forum) and give as much info as you can.

[Larry Gegel]

My system:

 

Dual 1 Ghz G4, 1 GB RAM, Mac OS X 10.4.5

 

Duplicate is being done from internal 80 GB drive to a 80 GB partition on an external firewire drive.

 

I checked the modified and creation dates on a few of the files and the creation date was the same as the modified date in each case.

 

There are no error messages in the log with respect to these files. There are a few error messages for other files since the system was being used at the time.

 

Larry

[waltr]

hi yellow,

 

Quote:

---

I checked the modified and creation dates on a few of the files and the creation date was the same as the modified date in each case.

 

---

 

ok, fair enough. what's the filesystem of your external drive?

[Larry Gegel]

Pretty sure it's Mac OS Extended, but I'll double check when I'm at home this evening. Not sure if it's journaled or not. I reformatted it before I started using it for backups and I can't imagine I would have made it anything other than Mac OS Extended, but you never know.

 

Larry

[Larry Gegel]

Confirmed. Both the source and destination volumes are Mac OS Extended (Journaled)

 

Larry

[waltr]

hi yellow,

 

ok, you are going to have to give some examples of files that are being copied, but should not be. not all 10,000, but some example data. please give what type of file, creation and modification dates and path.

 

the more common (ie, system files and such) these files are the easier it will be to replicate the problem.

[Guest]

Well since I started this thread I thought I would chime in . I re-posted on the windows forum noted below but have had no replies:

 

Windows post on same issue

 

 

I have the same issue. I went back and checked the creation, modified, and last accessed date on my source files. The ones I checked were all from 2005! All three date fields were the same. Dantz noted that if time was blank that would be a problem but they had a time in there too.... Even with all of that the same, duplicate/replace all went and re-copied every last one. I have no idea why.... And no error log as noted.

 

I too am copying to an external firewire drive but can't believe that is a contributing factor.

 

Anyway, if this is resolved and it is not a Mac OS-specific issue I would love to understand it...

 

In a related issue, after 10gig of copying the backup slowed to a crawl so I had to reboot and restart to get the speed back up...

 

Thanks,

BJB

[waltr]

hi bjb,

 

i think that yellow is having a Macintosh specific problem (the codebase for the two products is entirely different) and i had thought he might have the creation/modification date problem that is specific to 10.4, but he says that is not the case.

 

i'll try to catch up with your post on the windows side if i can.

[CallMeDave]

I've gotta chime in and report that I've seen this recently too.

 

In my case, it was only a few files that I saw marked but not matched before a Duplicate.

 

Source is internal SATA/HFS+ 10.4.5 boot drive, Destination is external FW/HFS+ partition. Executions are Replace entire disk.

 

When I noticed the files in question, some of which are .dmg files I use specifically to make sure that their contents are unchanged, I compared a couple from within Retrospect this way:

 

- Preflight Duplicate

- Click on "Preview"

- Scroll to one of the files that had a checkmark but _no_ diamond that I was confident had not changed since the last Duplicate

- Get Info (Command+I) to bring up Retrospect's file information window

- Move this window to the side

 

- Configure->Volumes->MyDestinationVolume->Browse

- Scroll to the same file as in the Source above

- Get info

 

- Put two windows side by side and compare

- I noted that all attributes listed, except for the full path, were identical

 

I solved this by moving the Source files into another folder and running another Duplicate. I then setup yet another Duplicate, and after scanning/matching the files were marked/matched as expected.

 

For others who's files can't be moved, this would not be a solution. But there's something wonky going on with Retrospect's Duplicate matching functionality with this app/os configuration (6.1.126).

[waltr]

hi dave,

 

 

 

what you describe as a solution seems to indicate that there is something not the same between the files. perhaps there is some hidden attribute that we don't (and Retrospect does) know about? and that this attribute is changed by moving the file. this is _exactly_ what happens when the Modify date is older than the Creation date (as noted in the KB article i referenced above).

 

 

 

moving the files from one place to another as a fix indicates to me that there is some attribute changed by the OS. if this is so, then a Duplicate action would also cause the attibute to be changed on the destination also, which means the files would not compare, and subsequent Duplicates would recopy the files. if that is true, it does not sound like a Retrospect issue.

[CallMeDave]

Quote:

---

perhaps there is some hidden attribute that we don't (and Retrospect does) know about?

 

---

 

Agreed; but what Retrospect could (and should) do is provide _all_ the matching attributes in the Get Info dialog box.

 

Otherwise, the menu command would be more appropriately named "Get Partial Info!"

 

Dave

[waltr]

hi dave,

 

Quote:

---

what Retrospect could (and should) do is provide _all_ the matching attributes in the Get Info dialog box.

 

---

 

not even Apple does that. why hold a vendor to higher standards than the maker of the OS?

[CallMeDave]

??? I don't understand; what does this have to do with the OS?

 

Retrospect's ability to match files from a Source against files in an existing Destination is one of the things that makes it so powerful. We know the obvious matching criteria, the ones that are listed in the Get Info dialog box (Label, Flags, Kind, Size, Location, Created, Modified, Backed up, Owner, Group and Mode). But if what you say is true (and I have no doubt that it is) then Retrospect has additional matching criteria that are unknown to users.

 

There is no reason not to display all the information the program can discover about a file, so users can see what Retrospect sees.

 

Dave

[waltr]

hi dave,

 

and compare it to what? if the OS is hiding the info, how are we to determine what the information means? and where the problem lies?

 

the OS has EVERYTHING to do with it, because that's where the information originates. i realize i'm playing devil's advocate here, and i don't know what this mysterious attribute might be (yet), but i think seeing something in Retrospect that is not visible in the native OS is confusing.

 

how many home users already become confused when they see the /etc, /var, and other 'invisible' folders--to some people (and yeah, i realize that's not you or me) that is downright scary.

[CallMeDave]

> and compare it to what?

 

Walt,

 

As someone who's most comfortable in the devil role, I profess that I am unclear as to the position you're advocating here.

 

Retrospect, in all flavors and on all operating systems, can compare Source files to Destination files.

 

The current Get Info window already shows quite a few attributes that are not visible in the default Macintosh OS GUI (Type/Creator codes, for example).

 

But if there _is_ a mysterious attribute (and I confess some regret that I didn't do more due diligance when I had two files identified by Retrospect as being unmatched; it's not as if I don't have a copy of Terminal available to use) then what reason could there be for such an attribute to go unlisted in the Get Info window?

 

I know that the Retrospect interface has seen very little change since 4.1/4.3. Some Unix fields added, a few preferences added or reworded, etc. So if our mystery attribute (if there is one) is OS X/Darwin based, then it's not surprising that ther is no field for it. But it would still be grand if the program was more honest in showing the user what it sees.

 

Best,

 

Dave

[waltr]

hi dave,

 

i did some thinking on this problem last night and think i may have come up with a decent solution (which i'll try to code in a AppleScript sometime this week) and a bit more insight into the problem. a few things first:

 

1) i believe there may be more than one problem at work here. i don't think your problem was the same as yellow's.

2) i believe the problems we are seeing will only happen in 10.4 or later.

3) i believe the problem is very similar to the creation/modification date problem that i cited above and that is detailed here:

http://kb.dantz.com/article.asp?article=7982&p=2

4) i believe that the solutions detailed in that article will fix the problem(s). specifically:

 

1) we already both believe that dragging the files to a new location and then back will fix the problem. this has already been covered.

2) the unix program 'touch' is talked about in the article. performing a touch on these files should fix whatever attribute problems are manifested here. this is what i'll try to codify and post here, to make it easier for the home user.

 

finally, i have done some real seaching into the issue. i don't have direct access to Apple's Bug Database (RADAR), but i was able to have a friend help me look at some bugs and have found one related to an attribute called "attributeModDate", which is set when ACL's or Metadata changes. apparently OS X can 'change' this attribute when a file is 'moved' or 'copied' (or Duplicated), which it should not do. i am not convinced, however, that everyone with this problem has either ACL's or Metadata. it will be interesting to see if my 'touch' program will solve the issue. i do _not_ believe that Retospect itself is changing this attribute.

[CallMeDave]

Just a few comments:

 

- My fix was to move the files and leave them moved. The next time Retrospect Duplicated the Source, it deleted the files from the Destination and re-copied them in the new location. Subsequent Duplicates correctly left the unchanged files alone.

 

- My system is 10.4.x desktop, and ACL's proper are not used. Not to say that attributeModDate isn't in play, but I'm not mucking about with metadata with any utilities.

 

- Of the files I saw this on, I can say for sure that the one I focused on/tested with was created on an earlier system then 10.4, and was unchanged since. I cannot say for certain that my files did not suffer from the creation/mod date problem outlined in the KB article. Sure do wish I'd taken a screen shot of the two side-by-side Get Info windows!

 

- Metadata support that was added to Retrospect should have included some sort of an update to the GetInfo dialog (that's the beauty of disclosure triangles). Maybe no one thought of doing it, maybe it was decided against as too programmer-resource-intensive. But users shouldn't be in the dark about why Retrospect doesn't match files that seem to be identical. (if a list of new features for future versions is being kept, I hope that one can be added)

[waltr]

hi dave,

 

i think moving the files back would have worked too. i agree that ACL's and Metadata are probably not in play, but i'm just reporting my research.

 

here is an AppleScript that i think would fix the problem without having to move files back and forth.

 

before i post it, a few caveats:

1) i'm not going to support this. if you don't understand how to compile an AppleScript, there are plenty of resources on the web to study up with. if it doesn't work for you feel free to post here, but i can't make any guarantees.

2) this is not an EMC product. don't call EMC Technical support and ask them questions about it--they won't know what you are talking about.

3) if you have a problem with the way the script is written, please feel free to post your corrections--i do not pretend to be the best AppleScripter out there, and my ego can take it.

 

i'd also like to note that i adapted someone elses script to create this. full credit is at the bottom, as well as some simple instructions on running it.

 

here is the Code:

---

global myPass

global dPath

 

property passQ : "What is the admin password?"

 

set myPass to text returned of (display dialog passQ default answer "")

 

tell application "Finder"

set chosenFolder to choose folder

my folderprocess(chosenFolder)

end tell

 

do shell script "/bin/rm -rf /tmp/myTouch" password myPass with administrator privileges

 

display dialog "Your files and folders have been \"touch\"ed"

 

on folderprocess(chosenFolder)

set theTmp to "/tmp/myTouch"

set currFolder to chosenFolder as alias

set myTouch to "/usr/bin/touch " & quoted form of POSIX path of currFolder

do shell script "/bin/echo" & space & quoted form of myTouch & space & ">" & theTmp password myPass with administrator privileges

set doMyTouch to "/bin/sh" & space & theTmp

do shell script doMyTouch password myPass with administrator privileges

tell application "Finder"

set myFiles to every file of chosenFolder

if (count of myFiles) ­ 0 then

my finderprocess(myFiles)

end if

set subFolders to folders of chosenFolder

if (count of subFolders) ­ 0 then

repeat with thisFolder in subFolders

my folderprocess(thisFolder)

end repeat

end if

end tell

end folderprocess

 

on finderprocess(myFiles)

set theTmp to "/tmp/myTouch"

repeat with thisfile in myFiles

set currFile to thisfile as alias

set myTouch to "/usr/bin/touch " & quoted form of POSIX path of currFile

do shell script "/bin/echo" & space & quoted form of myTouch & space & ">" & theTmp password myPass with administrator privileges

set doMyTouch to "/bin/sh" & space & theTmp

do shell script doMyTouch password myPass with administrator privileges

end repeat

end finderprocess

---

 

to compile, open your 'Script Editor' and cut and paste the code there. then you can 'Compile' or just save the file as an 'Application'. i call it 'touchStuff' but you can call it what you want. when you double click the application (or run it from the 'Script Editor') it will ask you to chose the folder you'd like to 'touch'. it will 'touch' that folder, and all files and folders recursively in the folder. it sucks that i have to get your admin password in clear text, but the alternative was to have to ask for it each time we need it, which i think is a big pain. i promise the password is not emailed to me

 

finally, this script was adapted from a script posted by Jim Neumann (BLUEFROG) on the 'MacScripter' website. you can check out his original code here.