Jump to content
MrPete

How to restore files on top of what's there?

Recommended Posts

I am surprised this is not an obvious functionality.

Scenario:
- Existing drive, good backups
- For !@#$% reasons, a chunk of an NTFS partition was literally zero'd out at a low level
- Result: files are all in place but I guarantee their contents have changed. Consider it bit rot on a massive scale :)

What I need:
- Restore essentially all backed-up files, ignoring the fact that the metadata matches perfectly
- In other words, I want to do a restore-with-overwrite.
- Important note: I do NOT want to simply restore the volume if at all possible, because there are many new/changed files not in the backup.

I can't find a way to do this. What am I doing wrong?
Mostly, it auto-deselects all the files that appear to already be there (matching in size and date stamps).

Any ideas much appreciated.

I'm about ready to do the obvious/pain-in-the-neck workaround...
- Find all new files and copy elsewhere and/or
- Delete all files that aren't new

THANKS!

Pete

Share this post


Link to post
Share on other sites

Pete,

Yes you can do this.  Setup your restore job under Manage Scripts. On the Destination Selection window, there's a list box at the top of the window "Retrieve Files & Folders".  Click the down arrow and choose the option you want.  If you can, you should restore all files to a new location.

My advice, if this is at all possible, is to replace the drive and then restore.  What you're doing is super risky as I'm sure you're aware.  Make a Disaster Recovery disk to do this, and run it on the problem system.

My further advice is to go to http://grc.com and purchase Spinrite and run it on your system first. If your drive isn't healthy and is repairable, Spinrite will do it.  Your system must be able to run 'chkdsk /f' first before you use Spinrite.  (I can't type the Cee-colon in that command, the stupid forum software keeps inserting a smiley face for some reason.)  You should also have SMART turned on so you're warned if the drive is close to failure.

Good luck, Mark

Edited by mbennett
Forum software glitch

Share this post


Link to post
Share on other sites

Sigh. Mark, thanks for confirming what I sensed. This really is a bug in Retrospect. I'm about to report it.
Let's say you choose "Replace Corresponding Files" from the dropdown.

My expectation: it will replace the corresponding files. The documentation is clear about this.
Unfortunately, it doesn't work. Retrospect ASSUMES that the data content matches, if the metadata matches. Boo.
I guarantee that the "already copied" files aren't the only ones... and in fact, if I prepare a *backup* of the bad data, it's going to copy way more than a few hundred files. :(

1000195867_RetrospectRestoreBug.PNG.3746e47177f8ddd87942b14be8bf45bf.PNG

Time to report this bug.

========
A few quick comments on Mark's other suggestions:


Preserving Data: In my situation, I've already replicated the drive. Nothing will be lost.
SpinRite: I know exactly how/why the drive was partially zero'd. (It was my mistake :) )... no need for SpinRite this time. (YES I highly recommend SpinRite... Look for Pete H here :) - https://web.archive.org/web/20040825043909/https://www.grc.com/sr/testimonials.htm

Bit Rot and Drive Replacement: In reality, bit rot can happen on ANY drive. In no way does bit rot imply a drive that needs to be replaced!

Just for example: Consumer SSD drives retain data "perfectly" for approximately one year (if you've not overwritten a sector) before bit rot begins to show. It's worse for enterprise SSD's (because they are focused on performance over data retention.) This is not exactly advertised by the industry. Dig into the latest research and you'll discover the reality..) NOTE: some mfg's MAY be compensating for this and taking care of it in firmware... but I have yet to see such workarounds documented. And YES I have been harmed by this. Consider: what is the one sector on a drive or partition that NEVER is overwritten?... :)

(How to avoid this: regularly rewrite every sector of an SSD. I do it about 3-4 times a year.


I could point you at the (very!) technical research work that underlies what I wrote above. I agree that it's more than a little surprising.

Share this post


Link to post
Share on other sites
16 hours ago, MrPete said:

....

....

....


....
Bit Rot and Drive Replacement: In reality, bit rot can happen on ANY drive. In no way does bit rot imply a drive that needs to be replaced!

Just for example: Consumer SSD drives retain data "perfectly" for approximately one year (if you've not overwritten a sector) before bit rot begins to show. It's worse for enterprise SSD's (because they are focused on performance over data retention.) This is not exactly advertised by the industry. Dig into the latest research and you'll discover the reality..) NOTE: some mfg's MAY be compensating for this and taking care of it in firmware... but I have yet to see such workarounds documented. And YES I have been harmed by this. Consider: what is the one sector on a drive or partition that NEVER is overwritten?... :)

(How to avoid this: regularly rewrite every sector of an SSD. I do it about 3-4 times a year.


I could point you at the (very!) technical research work that underlies what I wrote above. I agree that it's more than a little surprising.

Thanks for the info, MrPete,

For hysterical reasons I have been alternating between the use of two drives for Retrospect versions on my "cheesegrater" Mac Pro "backup server", one HDD and one SSD.  ATM I have Retrospect Mac 16 on my SSD, with Retrospect Mac 15 on my HDD.  However I've elected to keep my Catalog Files on the HDD, just in case I have to switch back or upgrade forward.  Retrospect Inc. (should I still call them that?) is kind enough to provide new point-releases of Retrospect Mac 16 about every 3 months, which I'm sure they do just so I can refresh my SSD. ;)

I was vaguely aware that SSDs had a shorter lifetime than HDDs, but I thought progress had been made on that problem.   Now you're saying it hasn't, so what was good enough in the 1970s is still better over the long term than its replacement.  Spinning rust forever! :)

 

Share this post


Link to post
Share on other sites

:)
Remember, it's not that the SSD fails. It's simply not an archival storage medium.

Not so sure we can say HDD's are *that* much better today. My work on early drives (yes I've been doing HDD's since the 1970's ;) around that long) ) was simpler, because the stored bits were monstrously big compared to today. (Anyone here remember the technique of literally using a pencil eraser tip to shove an HDD into spinning? They were truly not sealed!!!

Meanwhile, today we no longer store bits in the HDD magnetic wiggles. We don't even store encoded bits.

Today, it's more like a Douglas Adams sci fi joke! Adams talked about the "improbability drive"... well, how about a "probability curve"?! To compress the data, we calculate an exponential function that represents the bits in a sector, and store the parameters of the function. If one bit is wrong, you just lost an entire sector. That's "PRML" (Partial Response, Maximum Likelihood") ... such fun 😄

Share this post


Link to post
Share on other sites
On 8/22/2019 at 2:50 AM, MrPete said:

My expectation: it will replace the corresponding files. The documentation is clear about this.
Unfortunately, it doesn't work. Retrospect ASSUMES that the data content matches, if the metadata matches. Boo.

The documentation could be clearer -- "Retrospect leaves files untouched if their metadata is identical...".

That's good enough for the vast majority of use-cases. What you want would require checksum comparisons and a whole lot of extra processing. Could they offer that as an option? Maybe.

But "Replace corresponding", even if it worked as hoped, wouldn't work for you anyway since you said there are changed files which you *don't* want replacing -- in fact, you want "files whose metadata matches but checksums don't". In this particular case I'd guess your best bet is to full-restore to another disk then walk the original's tree, copying across files that match your criteria -- a nice little scripting project 🙂

Since you appear to have a time-point, you might also be able to build a copy with a backup restore followed by a time-filtered Duplicate from original disk to the copy. You could then replace the original with the copy. No easier than your proposed workrounds, but you could let Retrospect get on with it while you enjoyed your weekend...

Share this post


Link to post
Share on other sites

Thinking about this, your recovery might be easier than I thought...

  1. Make a new backup, to include everything created/modified since the last of the old backups
  2. Restore Volume using the old backups
  3. Replace Corresponding using the new backup

Assuming the matrix is correct, that should mean that "corrupted" files are returned to their uncorrupted form then new/changed files are overlayed, replacing previous versions where appropriate (assuming you weren't doing something silly like editing file data without updating the metadata to suit). All you should lose is data created/modified *after* the last old backup but *before* the borkage *if* it was affected by said event -- that would require data recovery of some sort.

Share this post


Link to post
Share on other sites
20 hours ago, MrPete said:

:)
Remember, it's not that the SSD fails. It's simply not an archival storage medium.

Not so sure we can say HDD's are *that* much better today. My work on early drives (yes I've been doing HDD's since the 1970's ;) around that long) ) was simpler, because the stored bits were monstrously big compared to today. (Anyone here remember the technique of literally using a pencil eraser tip to shove an HDD into spinning? They were truly not sealed.

 

IBM 2311 or was it 2314?  :)  I go back to the days of 026 card punches myself.

HDDs have improved since the days I worked for Shugart Associates.

Share this post


Link to post
Share on other sites
On 8/20/2019 at 9:28 PM, MrPete said:

I am surprised this is not an obvious functionality.

Scenario:
- Existing drive, good backups
- For !@#$% reasons, a chunk of an NTFS partition was literally zero'd out at a low level
- Result: files are all in place but I guarantee their contents have changed. Consider it bit rot on a massive scale :)

What I need:
- Restore essentially all backed-up files, ignoring the fact that the metadata matches perfectly
- In other words, I want to do a restore-with-overwrite.
- Important note: I do NOT want to simply restore the volume if at all possible, because there are many new/changed files not in the backup.

I can't find a way to do this. What am I doing wrong?
Mostly, it auto-deselects all the files that appear to already be there (matching in size and date stamps).

Any ideas much appreciated.

I'm about ready to do the obvious/pain-in-the-neck workaround...
- Find all new files and copy elsewhere and/or
- Delete all files that aren't new

THANKS!

Pete

Pete,

Frustrating situation I agree.  Can you detect which files are new/changed on the source drive?

If so, may I suggest that you get a new drive that will serve as the source.  Copy over all files from the current source in a way that preserves file dates, to a new location of course.  Then use this utility http://scootersoftware.com/ to compare your current files with the restored ones.  Beyond Compare will automatically highlight files that are newer/changed, so you could ignore those.  I use this utility on a daily basis.

Share this post


Link to post
Share on other sites
On 8/23/2019 at 4:09 PM, x509 said:

IBM 2311 or was it 2314?  :)  I go back to the days of 026 card punches myself.

HDDs have improved since the days I worked for Shugart Associates.

It could have been either model; the IBM 2314 was basically a larger-capacity version of the 2311, with 11 platters instead of 6.

My first job as a programmer (initially a trainee after two weeks of reading)—starting in June 1964—was with C-E-I-R Inc., the pioneer of non-manufacturer service bureaus.  We had professional operators in the air-conditioned computer room, who would have broken my fingers if I dared to touch hardware—with a pencil eraser tip or even pushing a switch.

IBM 026 card keypunches were used for input to 2nd-generation computers such as the IBM 1401, beloved because it was so easy to program in assembly language (as in this example whose statement labels and field names are in programmer-chosen abbreviated German but the operation codes are IBM-standardized abbreviated English).  IBM 029 keypunches were a later model used for input to IBM System/360 computers, to which 2311s and 2314s were attached.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×