Jump to content


  • Content count

  • Joined

  • Last visited

  • Days Won


Everything posted by MrPete

  1. Very interesting... * The AA*.rdb files are from mid- May 2018 and today * The AB*.rdb files are from June 2018 all the way through until today * Somehow they were both in the same folder and Retrospect was not upset... but it sure was upset now! AA rebuild finished AB rebuild under way...
  2. I wonder if anybody has an idea I can try. I'm stumped and support's closed for the day. I have a backup set with : * (1.4GB) catalog file. * 1.7GB of backups in 2.3 million files, 527 sessions, 132 snapshots. * The backup set has around 5000 *.RDB files. I went to groom and the groom failed. Now I need to rebuild the catalog. PROBLEM: no matter what I do, the catalog rebuild only scans the first 262 RDB files?!!! I have: * Rebooted * Deleted the old catalog file * Checked all permissions Any ideas MOST welcome! pete
  3. I solved it, and learned a few things along the way. I finally examined to see what was so special about 262 files... and discovered something I probably should have noticed before: the folder contains TWO sets of *.RDB files! AA*.rdb and AB*.rdb -- AA*.rdb is 262 files. I am guessing it is what had been built during the failing groom, **even though** most of the files have a create and modify date that's quite old! So: * I split out the AA*.rdb files into a separate folder and am rebuilding a catalog on those just to see what's there * Then i'll rebuild the AB*.rdb files
  4. MrPete

    Retrospect 15.6 status?

    According to the Blog, 15.6 was announced on Tuesday. https://www.retrospect.com/en/blog/2018/10/16/retrospect_15_6 It's available for download now. https://www.retrospect.com/en/support/downloads However: My 15.5 Desktop/Pro does not see an update available Mayoff hasn't announced it here QUESTION: Is the new version safe to download and use?
  5. What I see here is: - Someone is attempting to use WikiPedia as a wiki platform for writing Retrospect user-generated info of various kinds I would suggest backing off from that. As others have noted, Wikipedia isn't a particularly trusted platform anyway, and they dislike misuse/abuse of the platform. Alternatives include: - Perhaps Retrospect could implement wiki.retrospect.com - Or, just make a shared Google Drive folder. Easy peasy.
  6. (At long last, I have filed a support case with extensive info on how/why I recommend this change...) Here's what I wrote: Hi! In 2014 KB article https://www.retrospect.com/en/support/kb/file_vs_block_deduplication your team discusses why you've chosen File-based dedupe, based on name/size/attributes. I strongly recommend revisiting this decision, due to changes in technology, typical usage, and data volume. AND, there's at least one good implementation in a competitor that demonstrates the advantages of a move to block-based dedupe. Tech change: hash calcs are *extremely* fast in modern CPU's. Both traditional and new ones (cf CityHash). Multiple GB/sec! Storage technology now easily exceeds 100MB/sec, often to 500MB/sec. Data volume: with multi-TB drives, large SSD's, 4k video and large photo sensors, it's quite common to toss around many many GB of *noncompressible* files. Many TB in fact. Compression has little value as we move toward photo/video media. Typical usage: File-metadata dedupe is woefully inadequate today. The 2014 article doesn't cover many common scenarios: the exact same 20MB photo is often renamed as it gets copied to dropbox, shared with others, etc. Multi-MB audio files can have header info significantly changed, while the modify timestamp is retained. (eg every play causes the play count to update in the file! Yes, the access timestamp is updated. Photo and video collections are typically duplicated, renamed, and minor metadata header info is updated, while the bulk of the data blocks are not touched at all. A simple partition copy can cause Retrospect to make a new backup of every file in the partition. Many modern apps use the same DLL. Unfortunately, filenames and timestamps *often* vary (for identical content.) (A great, paid, tool I use for content-dupe-finding is SpaceMan99. Run it against a ? drive on a well-used computer!) C:\Windows\Installer contains duplicates of installed executables, with different names. Many hundred MB. Right now, all of the above scenarios cause anything from waste to havoc in Retrospect. One commercial example that handles all of this quite well: CrashPlan. (They left the home and SMB market last year, so not really competing anymore...) The client maintains a local database of file metadata and block hashes. Across the board, a "file" is a [metadata packet] plus N [content blocks]. Metadata can be quickly scanned for grooming, storage, recovery purposes Content block references can be efficiently rebuilt as needed via a scan Deterioration of both source and backed-up blocks is easily detected via a scan, and re-backup solves it - Up to N copies of any given content block can be retained for redundancy.
  7. I've updated the Admin Guide with new information. - More details on recovery in tough situations - clarified GPT/MBR partitions vs BIOS/UEFI boot support - clarified my latest understanding of what boot info is saved, backed up, known to Retrospect and the DRD. LATEST: 1) It now appears that the DRD is intended to function correctly with both BIOS and UEFI boot environments. Not sure if it does. 2) Retrospect is supposed to back up the System Reserved partition (but not the System Recovery)... however... a) There's VERY little indication that this happens (I see some info in some log files) b) I don't think it actually works for BIOS boot environments Yes, this has all been reported and I'm in discussion with support. (I'm providing this info in a very timely manner because I know for myself how frustrating it is when you lack information in a disaster recovery situation!) OK, back to my normal work.
  8. This document (as of July 31 2018, Retrospect Desktop is intended to augment the information in the official Retrospect 15 User Guide. All errors are my responsibility. I do not guarantee that this applies to any other version of Retrospect; in fact, I don't guarantee anything about this at all! ? YMMV. Buyer Beware. Etc. A few items highlighted below are not certain for me at this time. Insight welcome! Preparing for Disaster A. Crucial Attributes To Record About Each Client/Host System Several crucial attributes must be recorded about any client or host system that you wish to later restore with a DRD (Disaster Recovery Disk): 1) Disk Layout Why: the DRD is currently unable to fully auto-create this info. It's up to you to do so. Get it wrong and Things Can Go Badly Partition Table Type (MBR or GPT) Number and sequence of partitions. (MOST important: is there a "System Reserved" partition, is there a WinRE (Recovery) partition, which partition is Active, and what's the sequence?) (Nice to have: the name of the 'C:' windows partition) 2) Boot method Why: The boot method for recovery must match that of the system that was backed up. The DRD is currently not aware of this when regenerating a system. BIOS or UEFI? (MBR partition tables support both BIOS and UEFI boot. GPT partition tables only support UEFI boot, with a few rare exceptions.) Where is the boot BCD info? (From experience: Retrospect will NOT complain if your boot info is not on the C partition... and it may not be backed up!) For BIOS boot, the BCD info is typically either c:\boot\BCD or on the system reserved partition, at \boot\BCD. For UEFI boot, it's typically in one of those partitions, at \EFI\Microsoft\Boot\BCD or \EFI\boot\BCD. Hint: It's a good idea to save an exported copy of the BCD store while the system is in good shape. From Admin Cmd prompt: bcdedit /export c:\bcd-yymmdd will save it. [7/31/18 update] There is some indication that UEFI but not BIOS boot information is backed up from any appropriate partition. We're in discussion on this. If you have a complex multi-boot (eg using GRUB or even manually-added BCD entries), I would suggest keeping an image of your boot disk. Retrospect uses Microsoft Windows tools for recovery; recovering non-Windows boot information is (quite reasonably) beyond the product's scope. 3) 64 bit drivers required Why: Many environments do not require 64 bit drivers. Some do. If so, you'll need a 64 bit DRD rather than the default 32 bit. I have one: unless extreme measures are taken (see below), access to our Catalog Files is on a RAID 1 internal drive pair, managed by IRST (Intel RAID Storage Tech) which uses 64 bit drivers on 64 bit Windows. 4) Custom drivers required Why: If recovery requires access to devices that need nonstandard drivers, you'll need to prepare ahead. Example: my IRST setup. Typically, custom disk drivers that can be used at boot time are downloadable either in normal "installable" form, or in what is known as "F6 Floppy" form (refers to pre-boot interruptable driver-load... TMI ) The DRD creation instructions tell you to copy these drivers to a particular place on your Retrospect Desktop machine before creating the DRD. Do it. (currently they go in <Retrospect Install Folder>/drsupp/drivers ) 5) Non-hard-drive boot methods fully supported for system recovery Why: Not all machines support USB memory key boot. Windows 7 does not fully support USB for recovery operations. You may need a DVD (even a USB DVD, strangely). B. Crucial Things to Know About the Disaster Recovery Disk This information is not documented elsewhere, AFAIK, other than the first line below 1) The DRD... Why: These attributes determine how many DRD's you may want to create and maintain. AND, you'll want to update the DRD after significant system or Retrospect config changes. Is either 32 or 64 bit, and recovers a certain range of OS versions (eg seven varieties of Win10, etc) Assumes the boot style of the host system (it appears the DRD is intended to boot both UEFI and BIOS. Not yet clear if this works properly. For now I would not make assumptions.) Contains all Retrospect configuration as of when it is created, including Devices, Clients, Backup Sets, Volumes, Selectors, Preferences, Licenses, and Automation Settings Has built-in drivers for network, USB and many other devices Why: These attributes are unknown to the DRD. You'll need to maintain this knowledge separately, available for use in case of disaster Does not know how to auto-restore system Partition Table types (Reserved, Recovery, etc), partition settings, have access to catalog files on other disks, or login info to access network shares 2) Where do you keep your Catalog File? Why: Be sure you can get to the catalog file while recovering from a disaster! It's easy to move the catalog file off of your boot drive. Do it. (Or, make a copy as part of your backup strategy) In our case, to avoid other hassles, we host DRD recovery using a copy of the catalog files loaded into a USB stick. Easy-peasy. C. Before Creating the DRD Do you need custom drivers? Make sure they are in place already (see above)! For Windows 10, you need to download and install the ADK as described in the DRD documentation. These items are not yet documented: For Windows 7, a different kit is needed, the "AIK" You don't need to install the whole kit. When running the ADK setup, uncheck everything other than "Windows Preinstallation Environment (Windows PE)"... which will auto-check "Deployment Tools" Highly recommended: just before you create the DRD, do something to disable or pause all auto-run scripts! The DRD recovery environment is a "real" Retrospect environment, and will attempt to run any active scripts! (I introduced an N month delay in all scripts as a workaround, then removed it) Yes, it is possible to cancel all scripts once the DRD is running, but that can take quite a while as Retrospect goes through "preparing for open file backup" on the active scripts...) D. Creating the DRD The DRD tool wants you to locate a file, "copype.cmd" . The Retrospect team intends to auto-find this, but that's not yet implemented. The file is found in <Kit install dir>/Assessment and Deployment Kit/Windows Preinstallation Environment Bare Metal Recovery A. Preparing the System Ensure the system is set up in a similar fashion to the original system that will be restored. In particular: Boot settings so the system boots into the correct boot type (UEFI vs Legacy/BIOS. For one of our machines that supports both, I had to force it to Legacy to get everything working.) The correct Partition Table Type and Partition Layout (The DRD can create partitions, but has no ability to set Partition Types, special partition formats, detail-level partition sizes, etc.) (At first I reloaded Win10 on a system to be reloaded. The partitions were laid out very differently, and in particular a different count and sequence. Result: the recovery process wanted to wipe the contents of other data drives on the machine! Another time, the recovery immediately failed with strange VSS errors. Only when I correctly pre-set all of these elements did recovery go reasonably well. Tech Support has informed me these are known bugs (eg Bug 6109 about an invalid disk erasure warning)... however, if you have any concern for preserving drives, I urge care in restoring to ensure you don't accidentally make things worse than they already are ) B. Doing the Restore The recovery process involves several steps. Remember, I'm just giving additional notes and hints. The primary steps involve: pre-setup, run retrospect, post-restore Pre-setup: if you've predefined the partitions, you'll mostly just want to erase and reformat the 'C:' windows partition. Give it the same name that it had before to make life simple. Pre-setup: If you'll need network access to your backup sets, this is a good time to do a network-use of any needed shares. The DRD process will remember you're logged in from this point on Pre-setup: if you have other disks (eg USB stick) to attach to the system, eg containing Catalog files, now's the time to plug them in. In retrospect: check the needed catalog file(s) / backup sets. Can they be accessed (double click in 'Backup Sets'). If not, click "More..." then "Open..." to open the catalog file. Drag-and-drop of a catalog file does not work at this point to attach it. In retrospect: to do the recovery, go through the "Restore" process. I find it helpful to click on "Switch to Advanced Mode", and go through the steps one by one to be sure everything is as desired. In retrospect: before rebooting, be sure to remove the DRD disk! You don't want to just run the recovery again Post-restore: (if using Dissimilar Hardware Restore, don't leave the DRD script after finishing with Retrospect!) C. After Restore When my main host restore was complete, after reboot I got "no operating system found"... a bit scary. Solution for my situation: Boot with a Windows Recovery CD, get to a command prompt, and use these commands... bcdedit (shows boot information setup, if any. My system had none! bcdedit /store x:\boot\BCD is good to know about...) bootrec /rebuildbcd (finds windows and builds the correct boot environment) bcdboot c:\Windows /s b: /f BIOS /v [where the drive letters are what's valid in your recovery environment; "c:" is your windows volume, and "b:" is your boot volume, which could be the same as the windows volume, or could be the System Reserved partition.) ALSO of note: for bcdboot to work, you need a valid copy of the following file from the bootable normal windows environment: c:\windows\system32\config\BCD-Template If you get further errors, you're beyond the scope of this hints-doc. Lots of material is out there to assist you. All is NOT lost. Building A DRD After The Fact I didn't have a chance to build a DRD before the boot disk on our primary backup system died. Here's what worked to get around that not-so-little problem: Downloaded a Windows 10 Pro installer from Microsoft (yes, it's free... controlled by license codes and activation keys) Used a separate tool set to predefine the partition structure of the replacement drive, to match the old one. MBR disk with System, C:, WinRE in my case. Didn't put any data in the partitions. Installed Win10 Pro into the C partition. Told it "no license key" since it was already activated. It really did auto-activate when the time came. Downloaded and installed Retrospect. used the c:\ProgramData\Retrospect\Config77.dat file from my almost-totally-dead drive. This gave me a very nice working environment Installed the ADK as described above Modified a few Retrospect scripts as described above Installed the IRST drivers. Then (due to other problems I'll not discuss here) switched tactics and recovered the current catalog files from my RAID to a USB key With the USB key catalogs in place, and all other drives disconnected, created the DRD Bottom Line This all sounds so neat and tidy... I have done this writeup because my actual recovery process involved discovering the hard way that there are many undocumented aspects to the DRD process! I suspect with these notes, a Retrospect Desktop system could be easily recovered in a matter of hours. Mine... well let's just say I began recovery Sunday evening and finished Wednesday morning... (One of those times when I wish I could get paid by vendors who benefit from my bug-sleuthing skills?
  9. AFAIK you don't necessarily need separate client vs server DRD's. I'm working to get clarification on some of these things from the tech support team. They are definitely aware of the confusing documentation etc.
  10. Please read my Admin Guide - it explains the kinds of things that cause you to need multiple DRD's (AFAIK). Assuming these are all client machines, the varying hardware is not the issue. The real issue is that at least for now you may have different partition layouts you need to know... different disk partition tables (MBR vs GPT), and possibly different boot methods (UEFI vs BIOS.) Restore to dissimilar hardware refers to restoring a single computer: if the original computer dies and you don't have an exact match replacement, you need to restore that computer to "dissimilar hardware." ?
  11. Are you restoring a client or the (Retrospect) host? If a client, why not just try it? You have nothing to lose but some time. You're starting from scratch anyway
  12. Thanks, David. We'll see how it goes...
  13. Answers to the rest... and a new document coming #1 No, a DRD really is needed. #2 True. Only need WinPE environment from the ADK (#3, #4 see above) #5 DRD's need to be carefully maintained. It's likely you will not be creating just one. Certainly not one "for all time." ...
  14. I'm recovering from a boot drive disaster on my Retrospect Desktop system Didn't have time to make the DRD before the disaster, but apparently I can do so afterwards. I have several crucial questions about undocumented and/or missing features. The most basic: 1) In Win10, can I do a correct bare metal restore from retrospect backup *without* a DRD? If so, is there any document describing or giving hints on this? 2) From forum comments (but nothing I see in the documentation) I am guessing that I don't actually need the full Win ADK installed, just the Windows Preinstallation Environment? Is this true? 3) It appears that there was intent (in 2015) to auto-find the copype.cmd file; that did not happen. It appears to be in "<install dir>\Assessment and Deployment Kit\Windows Preinstallation Envrironment" - correct? 4) There is a bland reference to "any drivers or packages that you would like to add to the ISO image should be placed in the following directory before creating the ISO image." There are no hints on what drivers might be needed. I use IRST (Intel Rapid Storage Technology) control over built-in firmware RAID1 for a system data drive... that contains the Retrospect Catalog Files. - Do I need to discover what drivers or packages are needed to make this work during disaster recovery? - The drive letters for these devices are NOT the default drive letters that would show up if just installing devices in random order. Do I need to find a way to assign drive letters during disaster recovery? 5) From forum comments, it's inferred that one disaster recovery disk could suffice for an entire network. Is this true? If not, what other considerations are needed? Thanks for any hints or pointers. Much pain here until I work this out!
  15. I've solved all of these issues, and succeeded in restoring. MANY lessons learned. I'm creating a new writeup to provide additional info on bare metal restore setup, management and implementation...
  16. I'm getting an error similar to what was reported here ... BUT I'm on Retrospect Desktop 15, latest version across the board. - I'm doing a bare metal restore of the boot drive (a small SSD with C drive and whatever other system partitions Win10 uses) Ignoring other bugs and missing docs for the moment, here's what has me stumped: - My catalog files (rather important) are in a partition on a second (RAID 1, Intel RST drivers) drive, which actually has two partitions. for now they are D and E drives - I set up the restore and told it to start. I immediately got a warning: WARNING: All data will be lost on the following volumes (and then it listed info for my D and E drives) Why in the world would it wipe those partitions on a separate disk? Any idea how to work around this? Obviously I don't want those drives wiped! (4TB of valuable files...)
  17. Answering some of my own questions: #3 - I was correct on the location. Looks like the devs never got around to fixing this #4 - I downloaded the latest "F6 floppy boot" version of IRST drivers, which is used for various pre-config environments. It appears to have worked nicely. Just drop the ZIP file contents into the folder recommended by retrospect docs, and the drivers get loaded. My network drivers and config appear to have correctly auto-config'd and auto-loaded
  18. Yikes. I'm new to Retrospect 15, although was a long time user of early versions. Unless my memory is failing me, Retrospect has always been quite smart about avoiding needless copies of the same file content. The same binary file content only stored once for multiple copies of the original file (even with varying ownership and attributes), for copies on the same or different computers. So, imagine my surprise and actually horror, to see the following on three successive backups: a) Day 1, G: drive, 2TB of files stored b) Day 2, G: drive, nothing new needs to be stored c) Day 3, G: drive, the SAME files are stored again. I checked. The stored file information says the same files, with the same timestamp, were backed up on two separate occasions. What causes this, and how do I eliminate it? Yes, it is using up Real Disk Space. I don't have THAT much backup space! Any hints much appreciated, Pete
  19. MrPete

    Why is it backing up the EXACT same file?

    Tech support has been very helpful. It turns out the Backup Script "Options" settings do not do what they say. I don't think I could have guessed most of these without a ton of trial and error. Here's an Approximation of Reality 1) Macintosh Client "Use attribute modification date": - Actually applies to BOTH Mac and Windows! - It is not discussing FILE modification date but potentially a change timestamp on any file attributes. - SAFEST to leave this unchecked for all Windows backups Windows Security "Back up XYZ security information from XYZ" - These do NOT control whether file/folder security info is backed up! File/folder security info is ALWAYS backed up, whenever contents are backed up. - These control whether the info is *compared* to decide if a new backup copy is needed. - Thus, changing any of this will tend to force a new backup of a file/folder. Even going from "on" to "off." - Except for exceptional circumstances, it ought to be safe to leave these turned off because the security info is backed up anyway. Since we want the ability to successfully do bare-metal restores, I had turned on "Back up file security." So: Now, I have File security backup OFF, and Mac client "Use attrib mod date" OFF. ... ...And everything is working as I would have expected in the first place. (I also asked if there is any diagnostic allowing us to see the attributes that are stored... to verify that change-comparisons are actually working correctly. Sadly, not yet.
  20. Thanks for the link. Makes total sense. When I get a few Round Tuits I'll assemble some info to make it easy.
  21. Thanks for that link, David. Sadly, their 2014 reasoning is quite dated only four years later. Client system throughput improvements, hash calculation speedups, and the low cost of maintaining hash lists all combine to make block-based deduplication almost a no brainer today. This is particularly true in light of: * Greatly increased data file sizes * Greatly increased use of VM's with common content yet rarely exactly common file attributes For the last several years we used a backup solution that did block deduplication in the client (sadly, they have left the SMB market.) Such methods are wonderfully fast and efficient. Not only that: because all metadata is maintained separate from content, the metadata can be very efficiently scanned for reporting, filtering and recovery purposes. The time required in Retrospect to calculate grooming and restoration jobs would become miniscule.
  22. I will +1 the feature suggestion. In fact, I want to take it further, down a path that other backup software systems HAVE successfully implemented: - Create a hash for each block of each file (They can define the block size :) ) - Store unique content-blocks. - All files with identical content will only be stored once. Doesn't matter what filename, date, attributes, etc.
  23. MrPete

    Why is it backing up the EXACT same file?

    It is an internal drive. Yes, file and folder security info is being backed up. If that causes this, it's an obvious bug: * Retrospect is supposed to save metadata separately, so that it doesn't save duplicate copies of files from different clients, just because they have different metadata * In this case, none of the files on the drive have been touched. It is a large photo archive.
  24. I've suddenly started getting this error on my primary home backup. The backup device is not exhibiting any issues. I've tested the drives, the interface, chkdsk, file copies, etc. All is well. A clue: it's a raid box that got low on space and a backup failed. I expanded the available capacity. Yet Retrospect is upset. Any hints on what error -105 really means? I've searched the forums etc... people seem to get random results, and often give up, reformatting and starting all over. That's a pretty extreme result. Blessings, Pete
  25. Hiya, I've run into a new one. I'm rebuilding a catalog file (after groom failed) for one of our backup drives. I'm getting this error: ...no matter what I do. The catalog should be a bit over 2 GB. It gets several hours into the rebuild, then fails (with several dozen of these errors.) The local drive where catalogs live has 30+GB free. C: has 5+ GB free. The backup drive has 5+ GB free. What do I need to do to get past this?