ADMIN GUIDE: Hints for success with Disaster Recovery

[MrPete]

(Feb 2022: Retrospect has recently done a LOT of work on Bare Metal Restore. I've not yet absorbed it all. Some portions of this guide may now be outdated... HOWEVER: note that my warnings to be fully aware of your endpoint boot environments still holds. Some setups are well supported, others have caveats, some are not supported at all. Please look at this recent update note: https://www.retrospect.com/en/support/kb/bmr_with_multiple_critical_disks )

(Jul 2021: It appears this info is still valuable for RS 18. Please tell me about any hiccups you find! I'm happy to correct this document.)
(Sep 2020: I've begun updating for RS 17, and for Win10 1809 and beyond...)

This document (as of July 31 2018, Retrospect Desktop 15.1.2.100) is intended to augment the information in the official Retrospect 15 User Guide. All errors are my responsibility. I do not guarantee that this applies to any other version of Retrospect; in fact, I don't guarantee anything about this at all! ? YMMV. Buyer Beware. Etc.

A few items highlighted below are not certain for me at this time. Insight welcome!

Preparing for Disaster

A. Crucial Attributes To Record About Each Client/Host System

Several crucial attributes must be recorded about any client or host system that you wish to later restore with a DRD (Disaster Recovery Disk):

1) Disk Layout

Why: the DRD is currently unable to fully auto-create this info. It's up to you to do so. Get it wrong and Things Can Go Badly

• Partition Table Type (MBR or GPT)
• Number and sequence of partitions. (MOST important: is there a "System Reserved" partition, is there a WinRE (Recovery) partition, which partition is Active, and what's the sequence?)
• (Nice to have: the name of the 'C:' windows partition)

2) Boot method

Why: The boot method for recovery must match that of the system that was backed up. The DRD is currently not aware of this when regenerating a system.

• BIOS or UEFI? (MBR partition tables support both BIOS and UEFI boot. GPT partition tables only support UEFI boot, with a few rare exceptions.)
• Where is the boot BCD info? (From experience: Retrospect will NOT complain if your boot info is not on the C partition... and it may not be backed up!)
  • For BIOS boot, the BCD info is typically either c:\boot\BCD or on the system reserved partition, at \boot\BCD. For UEFI boot, it's typically in one of those partitions, at \EFI\Microsoft\Boot\BCD or \EFI\boot\BCD.
  • Hint: It's a good idea to save an exported copy of the BCD store while the system is in good shape. From Admin Cmd prompt: bcdedit /export c:\bcd-yymmdd will save it.
  • [7/31/18 update] There is some indication that UEFI but not BIOS boot information is backed up from any appropriate partition. We're in discussion on this.
  • If you have a complex multi-boot (eg using GRUB or even manually-added BCD entries), I would suggest keeping an image of your boot disk. Retrospect uses Microsoft Windows tools for recovery; recovering non-Windows boot information is (quite reasonably) beyond the product's scope.

3) 64 bit drivers required

Why: Many environments do not require 64 bit drivers. Some do. If so, you'll need a 64 bit DRD rather than the default 32 bit.

• I have one: unless extreme measures are taken (see below), access to our Catalog Files is on a RAID 1 internal drive pair, managed by IRST (Intel RAID Storage Tech) which uses 64 bit drivers on 64 bit Windows.

4) Custom drivers required

Why: If recovery requires access to devices that need nonstandard drivers, you'll need to prepare ahead. Example: my IRST setup.

• Typically, custom disk drivers that can be used at boot time are downloadable either in normal "installable" form, or in what is known as "F6 Floppy" form (refers to pre-boot interruptable driver-load... TMI )
• The DRD creation instructions tell  you to copy these drivers to a particular place on your Retrospect Desktop machine before creating the DRD. Do it.
  (currently they go in <Retrospect Install Folder>/drsupp/drivers )

5) Non-hard-drive boot methods fully supported for system recovery

Why: Not all machines support USB memory key boot. Windows 7 does not fully support USB for recovery operations. You may need a DVD (even a USB DVD, strangely).

 

B. Crucial Things to Know About the Disaster Recovery Disk

This information is not documented elsewhere, AFAIK, other than the first line below

1) The DRD...

Why: These attributes determine how many DRD's you may want to create and maintain. AND, you'll want to update the DRD after significant system or Retrospect config changes.

• Is either 32 or 64 bit, and recovers a certain range of OS versions (eg seven varieties of Win10, etc)
• Assumes the boot style of the host system (it appears the DRD is intended to boot both UEFI and BIOS. Not yet clear if this works properly. For now I would not make assumptions.)
• Contains all Retrospect configuration as of when it is created, including Devices, Clients, Backup Sets, Volumes, Selectors, Preferences, Licenses, and Automation Settings
• Has built-in drivers for network, USB and many other devices

Why: These attributes are unknown to the DRD. You'll need to maintain this knowledge separately, available for use in case of disaster

• Does not know how to auto-restore system Partition Table types (Reserved, Recovery, etc), partition settings (sizes, etc), have access to catalog files on other disks, or login info to access network shares

2) Where do you keep your Catalog File?

Why: Be sure you can get to the catalog file while recovering from a disaster!

• It's easy to move the catalog file off of your boot drive. Do it. (Or, make a copy as part of your backup strategy)
• In our case, to avoid other hassles, we host DRD recovery using a copy of the catalog files loaded into a USB stick. Easy-peasy.

C. Before Creating the DRD

• Do you need custom drivers? Make sure they are in place already (see above)!
• For Windows 10, you need to download and install WinPE, which is part of the ADK as described in the DRD documentation. These items are not yet documented:
  • For Windows 7, a different kit is needed, the "AIK"
  • PRE-Win10-1809: You don't need to install the whole kit. When running the ADK setup, uncheck everything other than "Windows Preinstallation Environment (Windows PE)"... which will auto-check "Deployment Tools"
  • Win10 1809 and beyond: You only need the WinPE addon to the ADK. (See here: https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/download-winpe--windows-pe) When running the ADK setup, uncheck everything other than "Deployment Tools." Then then install the WinPE addon as described here: https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/winpe-create-usb-bootable-drive
• Highly recommended: just before you create the DRD, do something to disable or pause all auto-run scripts! The DRD recovery environment is a "real" Retrospect environment, and will attempt to run any active scripts! (I introduced an N month delay in all scripts as a workaround, then removed it) Yes, it is possible to cancel all scripts once the DRD is running, but that can take quite a while as Retrospect goes through "preparing for open file backup" on the active scripts...)

D. Creating the DRD

• The DRD tool wants you to locate a file, "copype.cmd" . The Retrospect team intends to auto-find this, but that's not yet implemented. The file is found in <Kit install dir>/Assessment and Deployment Kit/Windows Preinstallation Environment

Bare Metal Recovery

A. Preparing the System

Ensure the system is set up in a similar fashion to the original system that will be restored. In particular:

• Boot settings so the system boots into the correct boot type (UEFI vs Legacy/BIOS. For one of our machines that supports both, I had to force it to Legacy to get everything working.)
• The correct Partition Table Type and Partition Layout (The DRD can create partitions, but has no ability to set Partition Types, special partition formats, detail-level partition sizes, etc.)

(At first I reloaded Win10 on a system to be reloaded. The partitions were laid out very differently, and in particular a different count and sequence. Result: the recovery process wanted to wipe the contents of other data drives on the machine! Another time, the recovery immediately failed with strange VSS errors. Only when I correctly pre-set all of these elements did recovery go reasonably well. Tech Support has informed me these are known bugs (eg Bug 6109 about an invalid disk erasure warning)... however, if you have any concern for preserving drives, I urge care in restoring to ensure you don't accidentally make things worse than they already are )

B. Doing the Restore

The recovery process involves several steps. Remember, I'm just giving additional notes and hints. The primary steps involve: pre-setup, run retrospect, post-restore

• Pre-setup: if you've predefined the partitions, you'll mostly just want to erase and reformat the 'C:' windows partition. Give it the same name that it had before to make life simple.
• Pre-setup: If you'll need network access to your backup sets, this is a good time to do a network-use of any needed shares. The DRD process will remember you're logged in from this point on
• Pre-setup: if you have other disks (eg USB stick) to attach to the system, eg containing Catalog files, now's the time to plug them in.
• In retrospect: check the needed catalog file(s) / backup sets. Can they be accessed (double click in 'Backup Sets'). If not, click "More..." then "Open..." to open the catalog file. Drag-and-drop of a catalog file does not work at this point to attach it.
• In retrospect: to do the recovery, go through the "Restore" process. I find it helpful to click on "Switch to Advanced Mode", and go through the steps one by one to be sure everything is as desired.
• In retrospect: before rebooting, be sure to remove the DRD disk! You don't want to just run the recovery again
• Post-restore: (if using Dissimilar Hardware Restore, don't leave the DRD script after finishing with Retrospect!)

C. After Restore

When my main host restore was complete, after reboot I got "no operating system found"... a bit scary.

Solution for my situation: Boot with a Windows Recovery CD, get to a command prompt, and use these commands...

• bcdedit   (shows boot information setup, if any. My system had none! bcdedit /store x:\boot\BCD is good to know about...)
• bootrec /rebuildbcd   (finds windows and builds the correct boot environment)
• bcdboot c:\Windows /s b: /f BIOS /v   [where the drive letters are what's valid in your recovery environment; "c:" is your windows volume, and "b:" is your boot volume, which could be the same as the windows volume, or could be the System Reserved partition.)
• ALSO of note: for bcdboot to work, you need a valid copy of the following file from the bootable normal windows environment: c:\windows\system32\config\BCD-Template
• If you get further errors, you're beyond the scope of this hints-doc. Lots of material is out there to assist you. All is NOT lost.

Building A DRD After The Fact

I didn't have a chance to build a DRD before the boot disk on our primary backup system died. Here's what worked to get around that not-so-little problem:

1. Downloaded a Windows 10 Pro installer from Microsoft (yes, it's free... controlled by license codes and activation keys)
2. Used a separate tool set to predefine the partition structure of the replacement drive, to match the old one. MBR disk with System, C:, WinRE in my case. Didn't put any data in the partitions.
3. Installed Win10 Pro into the C partition. Told it "no license key" since it was already activated. It really did auto-activate when the time came.
4. Downloaded and installed Retrospect. used the c:\ProgramData\Retrospect\Config77.dat file from my almost-totally-dead drive. This gave me a very nice working environment
5. Installed the ADK as described above
6. Modified a few Retrospect scripts as described above
7. Installed the IRST drivers. Then (due to other problems I'll not discuss here) switched tactics and recovered the current catalog files from my RAID to a USB key
8. With the USB key catalogs in place, and all other drives disconnected, created the DRD

Bottom Line

This all sounds so neat and tidy... I have done this writeup because my actual recovery process involved discovering the hard way that there are many undocumented aspects to the DRD process!

I suspect with these notes, a Retrospect Desktop system could be easily recovered in a matter of hours. Mine... well let's just say I began recovery Sunday evening and finished Wednesday morning... (One of those times when I wish I could get paid by vendors who benefit from my bug-sleuthing skills?

Edited February 19, 2022 by MrPete
Feb 2022: New note about recent BMR changes

[DavidHertzberg]

This is wonderful, MrPete!  I strongly urge you to create a Support Case linking to your post above, suggesting as a Documentation product improvement that the post be turned into a Knowledge Base article.  Here's why and how to do that.

I suggest that your Support Case link to the post, rather than trying to copy the post as an "Additional" Note.  That is for two reasons: [1] Retrospect Inc.'s Support Case software limits each Additional Note to about 2000 bytes, so you'd have to create a new Additional Note for each 15-20 lines.  [2] The Support Case software has no capability for bolding, underlining, or italicizing text; when I write an Additional Note for a Support Case, I have to make do with underscore characters on each end of a word I want to emphasize.

Again, thanks a lot!

[MrPete]

Thanks, David. We'll see how it goes...

[MrPete]

I've updated the Admin Guide with new information.
- More details on recovery in tough situations
- clarified GPT/MBR partitions vs BIOS/UEFI boot support
- clarified my latest understanding of what boot info is saved, backed up, known to Retrospect and the DRD.

LATEST:

1) It now appears that the DRD is intended to function correctly with both BIOS and UEFI boot environments. Not sure if it does.
2) Retrospect is supposed to back up the System Reserved partition (but not the System Recovery)... however...

a) There's VERY little indication that this happens (I see some info in some log files)
b) I don't think it actually works for BIOS boot environments

Yes, this has all been reported and I'm in discussion with support.

(I'm providing this info in a very timely manner because I know for myself how frustrating it is when you lack information in a disaster recovery situation!)

OK, back to my normal work.

[MrPete]

I've begun to update the Admin Guide for Retrospect 17, and for Windows 10 1809 and beyond.

Key new info added:

As of Win10 1809, installing the needed WinPE files (in the ADK) has significantly changed.

[MrPete]

I'm updating this guide for RS 18. See any errors/glitches? Please tell me and I'll fix the document. THANKS!

[MrPete]

I've added a link to a new note from Retrospect, about changes they've made to Bare Metal Restore.