Jump to content
MrPete

How to restore files on top of what's there?

Recommended Posts

I am surprised this is not an obvious functionality.

Scenario:
- Existing drive, good backups
- For !@#$% reasons, a chunk of an NTFS partition was literally zero'd out at a low level
- Result: files are all in place but I guarantee their contents have changed. Consider it bit rot on a massive scale :)

What I need:
- Restore essentially all backed-up files, ignoring the fact that the metadata matches perfectly
- In other words, I want to do a restore-with-overwrite.
- Important note: I do NOT want to simply restore the volume if at all possible, because there are many new/changed files not in the backup.

I can't find a way to do this. What am I doing wrong?
Mostly, it auto-deselects all the files that appear to already be there (matching in size and date stamps).

Any ideas much appreciated.

I'm about ready to do the obvious/pain-in-the-neck workaround...
- Find all new files and copy elsewhere and/or
- Delete all files that aren't new

THANKS!

Pete

Share this post


Link to post
Share on other sites

Pete,

Yes you can do this.  Setup your restore job under Manage Scripts. On the Destination Selection window, there's a list box at the top of the window "Retrieve Files & Folders".  Click the down arrow and choose the option you want.  If you can, you should restore all files to a new location.

My advice, if this is at all possible, is to replace the drive and then restore.  What you're doing is super risky as I'm sure you're aware.  Make a Disaster Recovery disk to do this, and run it on the problem system.

My further advice is to go to http://grc.com and purchase Spinrite and run it on your system first. If your drive isn't healthy and is repairable, Spinrite will do it.  Your system must be able to run 'chkdsk /f' first before you use Spinrite.  (I can't type the Cee-colon in that command, the stupid forum software keeps inserting a smiley face for some reason.)  You should also have SMART turned on so you're warned if the drive is close to failure.

Good luck, Mark

Edited by mbennett
Forum software glitch

Share this post


Link to post
Share on other sites

Sigh. Mark, thanks for confirming what I sensed. This really is a bug in Retrospect. I'm about to report it.
Let's say you choose "Replace Corresponding Files" from the dropdown.

My expectation: it will replace the corresponding files. The documentation is clear about this.
Unfortunately, it doesn't work. Retrospect ASSUMES that the data content matches, if the metadata matches. Boo.
I guarantee that the "already copied" files aren't the only ones... and in fact, if I prepare a *backup* of the bad data, it's going to copy way more than a few hundred files. :(

1000195867_RetrospectRestoreBug.PNG.3746e47177f8ddd87942b14be8bf45bf.PNG

Time to report this bug.

========
A few quick comments on Mark's other suggestions:


Preserving Data: In my situation, I've already replicated the drive. Nothing will be lost.
SpinRite: I know exactly how/why the drive was partially zero'd. (It was my mistake :) )... no need for SpinRite this time. (YES I highly recommend SpinRite... Look for Pete H here :) - https://web.archive.org/web/20040825043909/https://www.grc.com/sr/testimonials.htm

Bit Rot and Drive Replacement: In reality, bit rot can happen on ANY drive. In no way does bit rot imply a drive that needs to be replaced!

Just for example: Consumer SSD drives retain data "perfectly" for approximately one year (if you've not overwritten a sector) before bit rot begins to show. It's worse for enterprise SSD's (because they are focused on performance over data retention.) This is not exactly advertised by the industry. Dig into the latest research and you'll discover the reality..) NOTE: some mfg's MAY be compensating for this and taking care of it in firmware... but I have yet to see such workarounds documented. And YES I have been harmed by this. Consider: what is the one sector on a drive or partition that NEVER is overwritten?... :)

(How to avoid this: regularly rewrite every sector of an SSD. I do it about 3-4 times a year.


I could point you at the (very!) technical research work that underlies what I wrote above. I agree that it's more than a little surprising.

Share this post


Link to post
Share on other sites
16 hours ago, MrPete said:

....

....

....


....
Bit Rot and Drive Replacement: In reality, bit rot can happen on ANY drive. In no way does bit rot imply a drive that needs to be replaced!

Just for example: Consumer SSD drives retain data "perfectly" for approximately one year (if you've not overwritten a sector) before bit rot begins to show. It's worse for enterprise SSD's (because they are focused on performance over data retention.) This is not exactly advertised by the industry. Dig into the latest research and you'll discover the reality..) NOTE: some mfg's MAY be compensating for this and taking care of it in firmware... but I have yet to see such workarounds documented. And YES I have been harmed by this. Consider: what is the one sector on a drive or partition that NEVER is overwritten?... :)

(How to avoid this: regularly rewrite every sector of an SSD. I do it about 3-4 times a year.


I could point you at the (very!) technical research work that underlies what I wrote above. I agree that it's more than a little surprising.

Thanks for the info, MrPete,

For hysterical reasons I have been alternating between the use of two drives for Retrospect versions on my "cheesegrater" Mac Pro "backup server", one HDD and one SSD.  ATM I have Retrospect Mac 16 on my SSD, with Retrospect Mac 15 on my HDD.  However I've elected to keep my Catalog Files on the HDD, just in case I have to switch back or upgrade forward.  Retrospect Inc. (should I still call them that?) is kind enough to provide new point-releases of Retrospect Mac 16 about every 3 months, which I'm sure they do just so I can refresh my SSD. ;)

I was vaguely aware that SSDs had a shorter lifetime than HDDs, but I thought progress had been made on that problem.   Now you're saying it hasn't, so what was good enough in the 1970s is still better over the long term than its replacement.  Spinning rust forever! :)

 

Share this post


Link to post
Share on other sites

:)
Remember, it's not that the SSD fails. It's simply not an archival storage medium.

Not so sure we can say HDD's are *that* much better today. My work on early drives (yes I've been doing HDD's since the 1970's ;) around that long) ) was simpler, because the stored bits were monstrously big compared to today. (Anyone here remember the technique of literally using a pencil eraser tip to shove an HDD into spinning? They were truly not sealed!!!

Meanwhile, today we no longer store bits in the HDD magnetic wiggles. We don't even store encoded bits.

Today, it's more like a Douglas Adams sci fi joke! Adams talked about the "improbability drive"... well, how about a "probability curve"?! To compress the data, we calculate an exponential function that represents the bits in a sector, and store the parameters of the function. If one bit is wrong, you just lost an entire sector. That's "PRML" (Partial Response, Maximum Likelihood") ... such fun 😄

Share this post


Link to post
Share on other sites
On 8/22/2019 at 2:50 AM, MrPete said:

My expectation: it will replace the corresponding files. The documentation is clear about this.
Unfortunately, it doesn't work. Retrospect ASSUMES that the data content matches, if the metadata matches. Boo.

The documentation could be clearer -- "Retrospect leaves files untouched if their metadata is identical...".

That's good enough for the vast majority of use-cases. What you want would require checksum comparisons and a whole lot of extra processing. Could they offer that as an option? Maybe.

But "Replace corresponding", even if it worked as hoped, wouldn't work for you anyway since you said there are changed files which you *don't* want replacing -- in fact, you want "files whose metadata matches but checksums don't". In this particular case I'd guess your best bet is to full-restore to another disk then walk the original's tree, copying across files that match your criteria -- a nice little scripting project 🙂

Since you appear to have a time-point, you might also be able to build a copy with a backup restore followed by a time-filtered Duplicate from original disk to the copy. You could then replace the original with the copy. No easier than your proposed workrounds, but you could let Retrospect get on with it while you enjoyed your weekend...

Share this post


Link to post
Share on other sites

Thinking about this, your recovery might be easier than I thought...

  1. Make a new backup, to include everything created/modified since the last of the old backups
  2. Restore Volume using the old backups
  3. Replace Corresponding using the new backup

Assuming the matrix is correct, that should mean that "corrupted" files are returned to their uncorrupted form then new/changed files are overlayed, replacing previous versions where appropriate (assuming you weren't doing something silly like editing file data without updating the metadata to suit). All you should lose is data created/modified *after* the last old backup but *before* the borkage *if* it was affected by said event -- that would require data recovery of some sort.

Share this post


Link to post
Share on other sites
20 hours ago, MrPete said:

:)
Remember, it's not that the SSD fails. It's simply not an archival storage medium.

Not so sure we can say HDD's are *that* much better today. My work on early drives (yes I've been doing HDD's since the 1970's ;) around that long) ) was simpler, because the stored bits were monstrously big compared to today. (Anyone here remember the technique of literally using a pencil eraser tip to shove an HDD into spinning? They were truly not sealed.

 

IBM 2311 or was it 2314?  :)  I go back to the days of 026 card punches myself.

HDDs have improved since the days I worked for Shugart Associates.

Share this post


Link to post
Share on other sites
On 8/20/2019 at 9:28 PM, MrPete said:

I am surprised this is not an obvious functionality.

Scenario:
- Existing drive, good backups
- For !@#$% reasons, a chunk of an NTFS partition was literally zero'd out at a low level
- Result: files are all in place but I guarantee their contents have changed. Consider it bit rot on a massive scale :)

What I need:
- Restore essentially all backed-up files, ignoring the fact that the metadata matches perfectly
- In other words, I want to do a restore-with-overwrite.
- Important note: I do NOT want to simply restore the volume if at all possible, because there are many new/changed files not in the backup.

I can't find a way to do this. What am I doing wrong?
Mostly, it auto-deselects all the files that appear to already be there (matching in size and date stamps).

Any ideas much appreciated.

I'm about ready to do the obvious/pain-in-the-neck workaround...
- Find all new files and copy elsewhere and/or
- Delete all files that aren't new

THANKS!

Pete

Pete,

Frustrating situation I agree.  Can you detect which files are new/changed on the source drive?

If so, may I suggest that you get a new drive that will serve as the source.  Copy over all files from the current source in a way that preserves file dates, to a new location of course.  Then use this utility http://scootersoftware.com/ to compare your current files with the restored ones.  Beyond Compare will automatically highlight files that are newer/changed, so you could ignore those.  I use this utility on a daily basis.

Share this post


Link to post
Share on other sites
On 8/23/2019 at 4:09 PM, x509 said:

IBM 2311 or was it 2314?  :)  I go back to the days of 026 card punches myself.

HDDs have improved since the days I worked for Shugart Associates.

It could have been either model; the IBM 2314 was basically a larger-capacity version of the 2311, with 11 platters instead of 6.

My first job as a programmer (initially a trainee after two weeks of reading)—starting in June 1964—was with C-E-I-R Inc., the pioneer of non-manufacturer service bureaus.  We had professional operators in the air-conditioned computer room, who would have broken my fingers if I dared to touch hardware—with a pencil eraser tip or even pushing a switch.

IBM 026 card keypunches were used for input to 2nd-generation computers such as the IBM 1401, beloved because it was so easy to program in assembly language (as in this example whose statement labels and field names are in programmer-chosen abbreviated German but the operation codes are IBM-standardized abbreviated English).  IBM 029 keypunches were a later model used for input to IBM System/360 computers, to which 2311s and 2314s were attached.

Share this post


Link to post
Share on other sites

Such fun :)

I basically ended up accomplishing the purpose manually:

1) Restore all data to a separate drive
2) Overwrite all files on the drive of interest

The files with mangled data can't be detected except by comparing actual content. The metadata is unchanged, because the changes were made lower than OS level.

As for computer history... I'm a bit younger than y'all 😄

* I played with IBM 026 card punches while in elementary school at Stanford (my dad was a grad student.) (Amazing class! One friend's dad had the first desktop calculator, HP9100a. Another brought her dad for Show & Tell: Arthur Schawlow, coinventor of the LASER. His hour long demo changed my life...) https://news.stanford.edu/news/1999/photos/schawlow200.jpg
* Dad then became a research scientist for GE. I had access at home to the GE 635 mainframe (same computer used for Dartmouth Time Sharing System)... MD-20 drum storage and all. We had a teletype, then a TI Silent 700. In our house! Whoo! I was probably one of the first kid-hackers :) ... all for the good. I even got a letter of commendation from the R&D Center head, and a gift of a week-long professional simulation course -- a whole week out of high school :)(https://upload.wikimedia.org/wikipedia/commons/2/28/Silent-700.jpg)
* In college I helped build our student computing center, based on DECSystem20 (never forget the JFFO instruction... and the almost-apocryphal HCF ;) )
* Ultimately, I spent years as a SiValley consultant, including early HDD test equipment etc.

Nope, never worked on IBM drives. My first professional HDD work was on the Shugart ST-506. My home computer in 1981 had somebody's 14" hard drive. Sounded like a jet engine... I wasn't allowed to turn it on if our baby daughter needed a nap!

Share this post


Link to post
Share on other sites

MrPete,

In the interest of history.

If I remember correctly, the "Multics" system at MIT when I was a grad student there was a Honeywell 635 system with virtual memory.

I worked at Shugart Associates during the time that they started to build low-cost "Winchester" drives, all modeled after some IBM drive that had sealed head-disk assemblies.  Thanks to a very friendly salesperson, I "got" a customer sample SA 1004, which was an 8" HDD with all of 10 MB.  I hooked it up to my CP/M system, and it was like a whole new world, compared with the typical dual-floppy system.

Shugart Associates was late to the 5 1/4" HDD market, and they went out of business a few years later; fortunately I had left by then.  But while it lasted, it was a great time.

Share this post


Link to post
Share on other sites
On 10/8/2019 at 3:35 PM, x509 said:

MrPete,

In the interest of history.

If I remember correctly, the "Multics" system at MIT when I was a grad student there was a Honeywell 635 system with virtual memory.

....

x509,

Your attempted correction of MrPete is wrong as to manufacturer name.  Multics was developed for the GE 645, which was a GE 635 with "a configurable hardware protected memory system".  General Electric later sold its computer business to Honeywell, and the Wikipedia articles say a GE 645 follow-on was sold as the Honeywell 6180.   "The bulk of these computers running time-sharing on Multics were installed at the NSA and similar governmental sites. Their usage was limited by the extreme security measures and had limited impact on subsequent systems, other than the protection ring."

IIRC, Fernando Corbató—lead developer of Multics—delivered the introductory IBM 7090 lecture in a 1963 post-summer week-long programming course at MIT somebody got me into.  I chose to study Fortran II (go-to-only logic  :o ) for the rest of the week; the course, followed by self-study, changed my life.

Edited by DavidHertzberg
Re-word first paragraph to replace my bad jokes with quotes from a Wikipedia article

Share this post


Link to post
Share on other sites

David,

I stand properly corrected.  I will only plead a failing memory based on getting older. 

I also did Fortran II, on an IBM 1130.  Punch cards, removeable 1-platter disk cartridge of about 1 MB, if I remember correctly (which I may not ...)

Share this post


Link to post
Share on other sites

x509,

I only once ran a program on an IBM 1130; it was IBM's Project Control System at an IBM service bureau in suburban Boston.  Your memory about the hardware is correct; the removable disk cartridge was an IBM 2315.   That cartridge mounted in an IBM 2310 disk drive, whose distinctive characteristic—to make it inexpensive—was that it used a slow voice-coil actuator instead of a faster rack-and-pinion one.  I remember standing around (there were several hundred activities in my client's project-scheduling data) and chuckling while the disk drive's access arm did buzzing seeks to the proper track.

However you are incorrect in remembering Fortran II on the 1130; it had a Fortran IV compiler.  I never programmed for the 1130, but I wrote programs for a year using my service-bureau employer's IBM 7094 Fortran II (+) compiler.  A major problem was that Fortran II did not have logical operators and a logical IF statement—which were added in Fortran IVPerhaps the lack of these caused me to make a truly stupid coding error in enhancing my NASA PERT "C" summarization program (for which I independently re-invented depth-first search) used for RCA Burlington's Apollo LM electronics subsystems.  This led to the project manager for the Rendezvous Radar and Transponder losing faith in my program (although I found the error in 4 15-hour days), and doing his bi-weekly project network summarization for two years by hand.  He did that well; LM Eagle rendezvoused with Command Module Columbia on 21 July 1969.

P.S.: It's been 54 years, and I never preserved a listing of my NASA PERT "C" summarization program because I couldn't follow it any more (see the last sentence in this paragraph).  However here follows what I now recall as a more-precise explanation of the coding error I made: Fortran II, even with the enhanced—denoted by "(+)" above—compiler my employer developed with the Smithsonian Astrophysical Observatory, which made the language barely suitable for character-comparison business applications (with an optional 'B' in card column 1 that converted the arithmetic IF on page 34 of this manual

   IF (a) x1,x2,x3 
   Where: a is any arithmetic expression except complex. 
	      x1,x2,x3 are executable statement numbers.
     The arithmetic IF statement causes control to be transferred to the statement numbered x1,x2,or x3 
   when the value of the arithmetic expression (a) is less-than, equal to, or greater than zero, respectively. 

into a kludgey logical IF—by making (a) into a Boolean expression with AND and EXCLUSIVE OR operators and making the values of (a) for the 3 statement numbers be zero for x2 and non-zero for x1 and x3—plus adding a DATA statement per page 69 of the manual), but had no capability for nested IFs—which weren't added until Fortran 77.   I had a pair of tests that were—I now recall—conceptually a nested IF, and I bollixed up the statement numbers.  I spent four 15-hour days searching for a compiler Boolean bug (because I'd found one soon after I started using the compiler), before realizing while brushing my teeth on Saturday night that the bug was in my code.  Like most programmers then and now, I was dismal at following control-transfer-via-branching logic.

Edited by DavidHertzberg
P.S. with more-complete explanation of the coding error I made; explanation progressively improved; change (++) to (+); clarify second-paragraph _preliminary_ mention of coding error

Share this post


Link to post
Share on other sites

David,

I guess memories do get "modified" over time.  What I clearly remember about doing Fortran on the IBM 1130 was that it was limited compared to the mainframe 360 Fortran I had done in college.  And certainly more limiting that PL/1.

 

That's a great story about how limitations in a programming language lead to errors in program logic.

Share this post


Link to post
Share on other sites
On 10/8/2019 at 1:35 PM, x509 said:

MrPete,

In the interest of history.

If I remember correctly, the "Multics" system at MIT when I was a grad student there was a Honeywell 635 system with virtual memory.

I worked at Shugart Associates during the time that they started to build low-cost "Winchester" drives, all modeled after some IBM drive that had sealed head-disk assemblies.  Thanks to a very friendly salesperson, I "got" a customer sample SA 1004, which was an 8" HDD with all of 10 MB.  I hooked it up to my CP/M system, and it was like a whole new world, compared with the typical dual-floppy system.

Shugart Associates was late to the 5 1/4" HDD market, and they went out of business a few years later; fortunately I had left by then.  But while it lasted, it was a great time.

Well, there's a lot of overlap I guess :)

(Sorry for delays... Real Life kicks in for me a lot... I'm now the proud owner of a big boot on my right foot, after some extensive ankle surgery. Looking forward to long walks for the first time in many years ;) )

David H is correct: what I played with was the GE 635... later renamed Honeywell 635.

As for the SA1004... OK, I peeked in my 1970-90 archive box. Yep, I worked on those as well as the SA 1002 (5MB version) :) ... I no longer remember all of the names, but vaguely recall strong long-term friendship relationships among Shugart founders (possibly Al himself) and others I worked with/for back then: Dr David H Chung (also formerly w/ Fairchild, inventor of F8 micro, and of the first retail microcomputer...); And other names -- Dash Chang and more... I was a firmware dev/performance/test/consultant for Qubex, which made one of the first HDD testers - the QA2000, with both ST506 and SA100x interfaces. I can't find info on who led Qubex. (The only hint I see is about a guy I vaguely remember, Mike Juliff, formerly of Memorex... oh well. The mists of time do tend to fade things out!)

 

Share this post


Link to post
Share on other sites

MrPete,

The mists of time have hidden the names of most of the people I worked with at Shugart.  I knew mostly sales and marketing people, with a few in engineering.

Also, something else we have in common, which I wish we didn't.  I am about to have the same surgery as you.  Getting older stinks!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×