Restore entire disk from backup of subvolumes

[abg]

Please excuse any incorrect version numbers or terminology. I'm presently restoring my hard drive from a Time Machine backup and don't have immediate access to my files.

 

I managed to hose my hard drive while attempting a win7 Bootcamp update. After erasing the drive, I restarted the machine from another disk with OSX on it (regular machine runs 10.5.8 and Retrospect 6.1.138, or whatever version precedes V8). While I have a very recent Time Machine backup, my most recent backup (by about 4 days) is my trusty Retrospect backup. I've always found it very straightforward to dig files or directories out of it. This time, however, I wanted to restore the entire disk, and selected that option.

 

Let me first explain that I discovered a couple of years ago that I couldn't simply backup all my data by selecting the hard drive; there were too many files and I needed to create subvolumes and to then back up a collection of those. Arcane, but it appears to work (subvolumes were basically created for all the top-level directories on my hard drive).

 

I connected my external drive with the backup, selected the catalog file, which showed the snapshot I wanted (early on 2-Mar). There were subvolumes listed in a window BELOW the snapshot, but I was unable to select more than one at a time. When I tried to restore "the entire disk", I basically could only get one subvolume restored at a time. Not at all what I had in mind, and since each "restore entire disk" ERASES the disk first it makes it impossible to restore all my data in a single shot.

 

Is there something obvious that I missed in the restore process? In a related question, since I'm just planning to augment my Time Machine backup with the individual files/directories in my Retrospect backup added in the 4 days in-between, is there a straightforward way to have Retrospect select files on the basis of "date added to backup set"?

 

Thanks in advance,

Alan

 

Edited March 4, 2010 by Guest
Corrected version of Retrospect

[rhwalker]

Most recent Retrospect 6.x is 6.1.230. Update is here. That may be important if you run into bugs in an earlier version. Don't forget the RDU update too. Downloads are here:

Retrospect 6 updates

 

Is there something obvious that I missed in the restore process?

I think so, but it's not quite clear to me what you mean. I understand what you did with this part:

 

Arcane, but it appears to work (subvolumes were basically created for all the top-level directories on my hard drive).

I think you did what was needed when you did your backup, considering the issues you were facing at the time.

 

The part that is unclear is this bit:

 

There were subvolumes listed in a window BELOW the snapshot, but I was unable to select more than one at a time. When I tried to restore "the entire disk", I basically could only get one subvolume restored at a time. Not at all what I had in mind, and since each "restore entire disk" ERASES the disk first it makes it impossible to restore all my data in a single shot.

Here's the mindset that you need, followed by what I would suggest, considering that it's not clear exactly what you mean by "basically":

 

subvolumes were [color:red]basically[/color] created for all the top-level directories on my hard drive

There are three things that need to happen, assuming that you backed up the subvolumes correctly:

 

(1) you need to get the complete and correct root structure for your drive ABOVE all the subvolumes, including any stuff that might have been missed in the "basically" part;

 

(2) you need to get the permissions and ownership correct for the various subvolume subtrees; and

 

(3) you need to get the files into the subvolumes.

 

For (1), if you left anything out in your "basically" step, and I don't know exactly what you did, the way to ensure that this step is correct would be to erase the disk again (sorry, but the current state is unknown) then install from your 10.5.8 (or whatever you had as a starting point) install DVD (it's unclear from your problem statement whether you have two machines, the hosed one running some unknown version of Mac OS X, and the machine running Retrospect 6.1.whatever, or whether the hosed machine ran Retrospect too), then apply all Apple updates and security patches to bring that system to the level it was at when you did the backup that you are about to restore.

 

If you plan to do the Bootcamp update eventually, you might as well set things up and do that after this part, while you have a clean 10.x install, as a part of the erase / partition step, before you start doing the subvolume restores, discussed below.

 

For step (2), as a quirk of Retrospect (I've tried to understand the "why" for this, but it escapes me - I am sure it's some metaphysical truth of the universe that I can't grasp), in order to get the ownership / permissions right, you have to do an "entire disk" restore. Well, not quite - "entire subvolume" counts too, you just have to understand the chroot stuff that Retrospect does for an "entire whatever" restore.

 

What you were missing, I believe, is that you need to set up a subvolume for the restore destination, and then do an "entire subvolume" (entire disk restore) with the target as the subvolume.

 

If you do step (1) first, then, if you have any special filesystem subtrees that aren't a part of the standard OS X install (e.g., /garbage or some such), create those directories at the root level so that you have empty "subvolumes".

 

The next part of the trick, and you seem to have grasped this, is not to be running on the target disk. You did this right when you booted the machine from another disk with OS X on it. It can be done if you are careful, but issues can develop if you try to restore on top of a live OS. A discussion of those issues is beyond this reply.

 

Then, restore your subvolumes on to subvolume targets. That won't erase the "entire disk", but will erase the "entire subvolume" targets.

 

It will also set permissions / ownership right, and will also do step (3) at the same time.

 

Now you should be back to where you were, and can boot off the restored disk.

 

Clear?

 

Russ

[abg]

Most recent Retrospect 6.x is 6.1.230. Update is here. That may be important if you run into bugs in an earlier version. Don't forget the RDU update too. Downloads are here:

Retrospect 6 updates

 

First, let me thank you for a very detailed and useful reply. Let me try to answer your questions. First, I need to correct my Retrospect version; it's 6.1.230, I had looked back to where I first created my subvolumes and I was running 6.1.138 at that time.

 

Is there something obvious that I missed in the restore process?

I think so, but it's not quite clear to me what you mean. I understand what you did with this part:

 

Arcane, but it appears to work (subvolumes were basically created for all the top-level directories on my hard drive).

I think you did what was needed when you did your backup, considering the issues you were facing at the time.

 

My saga in trying to get a catalog file created w/o running out of memory is detailed here:

 

http://forums.dantz.com/showtopic.php?tid/26542/

 

I created a separate subvolume for each top-level directory on my hard drive, and then collected those subvolumes into a Source Group which is what gets backed up.

 

The part that is unclear is this bit:

 

There were subvolumes listed in a window BELOW the snapshot, but I was unable to select more than one at a time. When I tried to restore "the entire disk", I basically could only get one subvolume restored at a time. Not at all what I had in mind, and since each "restore entire disk" ERASES the disk first it makes it impossible to restore all my data in a single shot.

Here's the mindset that you need, followed by what I would suggest, considering that it's not clear exactly what you mean by "basically":

 

subvolumes were [color:red]basically[/color] created for all the top-level directories on my hard drive

There are three things that need to happen, assuming that you backed up the subvolumes correctly:

 

(1) you need to get the complete and correct root structure for your drive ABOVE all the subvolumes, including any stuff that might have been missed in the "basically" part;

 

This is something that I probably messed up on, but I'm not sure it affects the behavior I'm seeing.

 

When I launch Retrospect and click on Restore, followed by 'restore entire disk', I get a window to select a Source Snapshot for restore. This new window has two sections; the top one has backsets. Once I click on the one I want, it prompts me for the catalog file password and then the lower window is populated with the "current snapshots" in my backup source. These are the sub-volumes in my backup set, some 17 of them. Only one of these can be clicked on at a time.

 

(2) you need to get the permissions and ownership correct for the various subvolume subtrees; and

 

(3) you need to get the files into the subvolumes.

 

For (1), if you left anything out in your "basically" step, and I don't know exactly what you did, the way to ensure that this step is correct would be to erase the disk again (sorry, but the current state is unknown) then install from your 10.5.8 (or whatever you had as a starting point) install DVD (it's unclear from your problem statement whether you have two machines, the hosed one running some unknown version of Mac OS X, and the machine running Retrospect 6.1.whatever, or whether the hosed machine ran Retrospect too), then apply all Apple updates and security patches to bring that system to the level it was at when you did the backup that you are about to restore.

 

At the moment I'm running off a restore from Time Machine, which predates my latest Retrospect backup by some 4 days. I'm happy enough to continue from this state, but I'd like to understand just what I did wrong. It's starting to sound like my mistake was at the very beginning of the backup source group/subvolume creation (omitting some important files) but I'm still confused why Retrospect won't restore more than 1 subvolume during the "entire disk restore".

 

If you plan to do the Bootcamp update eventually, you might as well set things up and do that after this part, while you have a clean 10.x install, as a part of the erase / partition step, before you start doing the subvolume restores, discussed below.

 

For step (2), as a quirk of Retrospect (I've tried to understand the "why" for this, but it escapes me - I am sure it's some metaphysical truth of the universe that I can't grasp), in order to get the ownership / permissions right, you have to do an "entire disk" restore. Well, not quite - "entire subvolume" counts too, you just have to understand the chroot stuff that Retrospect does for an "entire whatever" restore.

 

What you were missing, I believe, is that you need to set up a subvolume for the restore destination, and then do an "entire subvolume" (entire disk restore) with the target as the subvolume.

 

If you do step (1) first, then, if you have any special filesystem subtrees that aren't a part of the standard OS X install (e.g., /garbage or some such), create those directories at the root level so that you have empty "subvolumes".

 

The next part of the trick, and you seem to have grasped this, is not to be running on the target disk. You did this right when you booted the machine from another disk with OS X on it. It can be done if you are careful, but issues can develop if you try to restore on top of a live OS. A discussion of those issues is beyond this reply.

 

Yes, I was running off a clone of my hard drive (I actually use 4 different backup strategies, but Retrospect 6.1 is the one I use on a daily basis) so it was pretty straightforward to play around and see which method was going to work best for a complete disk restore. So far, Time Machine is the simplest. While I was very comfortable using Retrospect 6.1 for various backups, and it had saved my bacon numerous time on a mac and on windows (V6.5), it occurs to me that I had never actually attempted a complete disk restore with 6.1 until today.

 

Then, restore your subvolumes on to subvolume targets. That won't erase the "entire disk", but will erase the "entire subvolume" targets.

 

It will also set permissions / ownership right, and will also do step (3) at the same time.

 

Now you should be back to where you were, and can boot off the restored disk.

 

Clear?

 

Russ

 

 

Again, thanks so much for attempting to set me straight. One last question: is there a simple way to restore files from a backup set that were added to it after a certain date? That would certainly simplify my remaining task, restoring the data files from the 4 days they were only being backed up in Retrospect 6.1

 

Thanks again,

ABG

[rhwalker]

First, I need to correct my Retrospect version; it's 6.1.230, I had looked back to where I first created my subvolumes and I was running 6.1.138 at that time.

Ok. That's what I had guessed, but I just wanted to make sure. I assumed that you were in the midst of a crisis, and were scrambling to do what you could. I seem to recall that the Retrospect 2.0 User Guide (or Quick Start, or whatever) had a section at the front about doing a restore in the midst of a crisis which, for many people, is the first time they opened the manual following installation.

 

I seem to recall that it started out with something like: "Calm down, take a deep breath, if your data was backed up you will get it back." That used to be true prior to Retrospect 8, and some day it may again be true.

 

My saga in trying to get a catalog file created w/o running out of memory is detailed here:

 

http://forums.dantz.com/showtopic.php?tid/26542/

 

I created a separate subvolume for each top-level directory on my hard drive, and then collected those subvolumes into a Source Group which is what gets backed up.

Ok. It wasn't clear to me whether you had backed up "most" top-level directories, perhaps missing some standard Mac OS X stuff, relying on being able to restore those from the install DVD, or whether you had gotten them all.

 

In that case, all you would have needed to do for the "Step (1)" that I discussed would be to create the top-level directories (so that you could designate them as "subvolume" destinations for the restore.

 

Even so, I think it might have been (or might be) safer and easier to do the install disk and updates to ensure that some special stuff (/dev nodes, volume partitioning, etc.) got done correctly.

 

When I launch Retrospect and click on Restore, followed by 'restore entire disk', I get a window to select a Source Snapshot for restore. This new window has two sections; the top one has backsets. Once I click on the one I want, it prompts me for the catalog file password and then the lower window is populated with the "current snapshots" in my backup source. These are the sub-volumes in my backup set, some 17 of them. Only one of these can be clicked on at a time.

Yes, I think that is the expected behavior. To Retrospect's "chroot" behavior that is used to do subvolume restore, it can only work on a single volume restore at a time, hence the selection of a single source for restore (and destination).

 

It many also have something to do with unix's prohibition (because of underlying design) of cross-volume hard links (but not cross-volume symbolic links, which are simply the text of the pathname).

 

At the moment I'm running off a restore from Time Machine, which predates my latest Retrospect backup by some 4 days. I'm happy enough to continue from this state, but I'd like to understand just what I did wrong. It's starting to sound like my mistake was at the very beginning of the backup source group/subvolume creation (omitting some important files) but I'm still confused why Retrospect won't restore more than 1 subvolume during the "entire disk restore".

Again, see above. Short answer, that's the way it is.

 

Long answer, it has to do with Unix chroot behavior. You might want to do a "man chroot" in Terminal to read the man page. Basically, the chroot capability was set up to permit web access for a contained filesytem (as if the filesystem really started at a lower level "root") without permitting any way to go out of that box. Needed for things like FTP, etc., so that malicious hackers couldn't scribble all over other areas of the disk, but Retrospect takes advantage of this to provide "subvolume" behavior.

 

While I was very comfortable using Retrospect 6.1 for various backups, and it had saved my bacon numerous time on a mac and on windows (V6.5), it occurs to me that I had never actually attempted a complete disk restore with 6.1 until today.

Well, the middle of restore crisis is not the time to learn / test this behavior. Really, it should be done as part of the installation / qualification / testing of any backup program, because that's when you discover the shortcomings of a backup program.

 

One last question: is there a simple way to restore files from a backup set that were added to it after a certain date? That would certainly simplify my remaining task, restoring the data files from the 4 days they were only being backed up in Retrospect 6.1

Yes, there are several ways.

 

One way, because you really don't seem to be going back that far, would be to restore the files in those sessions (not snapshots) from that date forward. That's the approach that would be used with other backup programs, basically restoring the incrementals forward from a given date.

 

Or, you could craft a Selector (in Retrospect 6 language) to select all files modified from a given date forward. Note that such a process would mean that you will have to go through all of your subvolume sources, separately targeting all of your subvolume destinations.

 

Good luck.

 

I still think that the procedure I outlined is the most foolproof.

 

Russ

[abg]

First' date=' I need to correct my Retrospect version; it's 6.1.230, I had looked back to where I first created my subvolumes and I was running 6.1.138 at that time.[/quote']

Ok. That's what I had guessed, but I just wanted to make sure. I assumed that you were in the midst of a crisis, and were scrambling to do what you could. I seem to recall that the Retrospect 2.0 User Guide (or Quick Start, or whatever) had a section at the front about doing a restore in the midst of a crisis which, for many people, is the first time they opened the manual following installation.

 

I seem to recall that it started out with something like: "Calm down, take a deep breath, if your data was backed up you will get it back." That used to be true prior to Retrospect 8, and some day it may again be true.

 

My saga in trying to get a catalog file created w/o running out of memory is detailed here:

 

http://forums.dantz.com/showtopic.php?tid/26542/

 

I created a separate subvolume for each top-level directory on my hard drive, and then collected those subvolumes into a Source Group which is what gets backed up.

Ok. It wasn't clear to me whether you had backed up "most" top-level directories, perhaps missing some standard Mac OS X stuff, relying on being able to restore those from the install DVD, or whether you had gotten them all.

 

In that case, all you would have needed to do for the "Step (1)" that I discussed would be to create the top-level directories (so that you could designate them as "subvolume" destinations for the restore.

 

Even so, I think it might have been (or might be) safer and easier to do the install disk and updates to ensure that some special stuff (/dev nodes, volume partitioning, etc.) got done correctly.

 

When I launch Retrospect and click on Restore, followed by 'restore entire disk', I get a window to select a Source Snapshot for restore. This new window has two sections; the top one has backsets. Once I click on the one I want, it prompts me for the catalog file password and then the lower window is populated with the "current snapshots" in my backup source. These are the sub-volumes in my backup set, some 17 of them. Only one of these can be clicked on at a time.

Yes, I think that is the expected behavior. To Retrospect's "chroot" behavior that is used to do subvolume restore, it can only work on a single volume restore at a time, hence the selection of a single source for restore (and destination).

 

It many also have something to do with unix's prohibition (because of underlying design) of cross-volume hard links (but not cross-volume symbolic links, which are simply the text of the pathname).

 

At the moment I'm running off a restore from Time Machine, which predates my latest Retrospect backup by some 4 days. I'm happy enough to continue from this state, but I'd like to understand just what I did wrong. It's starting to sound like my mistake was at the very beginning of the backup source group/subvolume creation (omitting some important files) but I'm still confused why Retrospect won't restore more than 1 subvolume during the "entire disk restore".

Again, see above. Short answer, that's the way it is.

 

Long answer, it has to do with Unix chroot behavior. You might want to do a "man chroot" in Terminal to read the man page. Basically, the chroot capability was set up to permit web access for a contained filesytem (as if the filesystem really started at a lower level "root") without permitting any way to go out of that box. Needed for things like FTP, etc., so that malicious hackers couldn't scribble all over other areas of the disk, but Retrospect takes advantage of this to provide "subvolume" behavior.

 

While I was very comfortable using Retrospect 6.1 for various backups, and it had saved my bacon numerous time on a mac and on windows (V6.5), it occurs to me that I had never actually attempted a complete disk restore with 6.1 until today.

Well, the middle of restore crisis is not the time to learn / test this behavior. Really, it should be done as part of the installation / qualification / testing of any backup program, because that's when you discover the shortcomings of a backup program.

 

One last question: is there a simple way to restore files from a backup set that were added to it after a certain date? That would certainly simplify my remaining task, restoring the data files from the 4 days they were only being backed up in Retrospect 6.1

Yes, there are several ways.

 

One way, because you really don't seem to be going back that far, would be to restore the files in those sessions (not snapshots) from that date forward. That's the approach that would be used with other backup programs, basically restoring the incrementals forward from a given date.

 

Or, you could craft a Selector (in Retrospect 6 language) to select all files modified from a given date forward. Note that such a process would mean that you will have to go through all of your subvolume sources, separately targeting all of your subvolume destinations.

 

Good luck.

 

I still think that the procedure I outlined is the most foolproof.

 

Russ

 

Thanks again for your help.

 

Rather than create a selector I may just go ahead and restore my user directory to a new location (without Bootcamp I now have sufficient room) and just use the UNIX find command to delete everything older than 26-Feb in the restored directory. That, at least to me, seems like a more straightforward task than thoroughly understanding source groups, sub-volumes and selectors :tongue2: