Issues with using optical disks in Retrospect 15

[Rhyslin]

Just a warning, this is going to be rather long because I want to outline why I'm doing what I'm doing, and then lay out the issues.

I'm a personal user, and I've always loved Retrospect for the control and flexibility it gave me for my backups. So for a long time now, I have been doing an "all files" backup to a external USB HDD, and then doing a transfer snapshot to my DVD writer of just what I call my User files (mostly under \Users with a few extra locations as well as a few excluded locations). I do this because I wanted a slightly more "permanent" archive of files over time, in case I needed to go back to an older revision of a file (I've almost never done this, but I'm paranoid...). Given how HDDs do fail (and I don't want the expense of a RAID or tape just for backups, and I don't trust the cloud security-wise), and since I've contemplated turning on grooming, I didn't want to rely on just a HDD for my backup sets for any sort of historical data.

This has worked fine for many, many years. So, I just installed Retrospect 15, and I've run into some problems. I actually didn't have any problems to start, but then I made what is probably a mistake--I thougth, just in case, I'll reconfigure the optical drive just to update the custom config (it's not one that Retrospect knows about, so I've always had to create a custom config for it). And this fails for my choice of DVD+R DL media. And after this, I could no longer read the backup set I had written just the day before on these types of disks. FYI, the failure occurs after the first round of write tests, after it ejects the disk and asks for it to be re-inserted. There's hardly any info given about why it was "unable to configure", and this message doesn't even get logged permanently. If anyone knows of any more extensive logs, I could look at them. I have actually run my system's read/write diagnostic for DVD+R DL disks, and it said it passed. And like I said, it had worked on R15 until I tried to reconfig.

I ended up trying to configure DVD+R (non-DL), and this worked. And then I retried the DL, and it got to the second re-insertion (I think that's after append tests) before failing. I read some posts here that questioned using DL media in the first place, so I figured I'd try it with non-DL since that might be more reliable (FYI, it's not cheaper as some posts I've read because nowadays I can get DL media for about the same cost per GB, and it's a lot more convenient to change disks about half as often). So, here I am, transferring snapshots and I notice I seem to be using more disks than I figure I should be, based on the sizes I am transferring.

So, I look at the members of my DVD set, and I see that while the first few disks say they are holding 4 GB or so, by my 14th disk, that's down to 2 GB. I then look at my catalog directory, and see that my new DVD set is storing a 2+ GB catalog file, where as my old DVD set was less than 300 MB. Since I have fast catalog file rebuild set (and non-compressed, but that is true for my older set as well), I figure that's why the space is going down, in order to store that after each transfer.

I then looked at the DVD set I had made recently for my wife's computer, and its catalog file was at 14 GB! I remember trying to rebuild that catalog file on a different system at one point, and it failed to do so from the last member and asked to do it the slow way. Maybe it's because it was too big and doesn't get saved in this case?

I wonder if the catalog files are so large because I have the block-level incremental backups enabled? While I would be OK with disabling that for the transfer, I don't see that as an option--it's only settable on the actual backup from what I see.

So to sum it all up:

• With R15, I can no longer create a custom config for my DVD writer
• The size of catalog files have ballooned recently (I think I can see it had increased in R12, maybe coincidental with block-level backups)
• Fast catalog rebuild no longer seems to work on DVD, though that may be because of the catalog file size?

I'm in a bit of a quandry about how to proceed. Getting only 2 GB on my DVDs (which may  go even lower as I put more snapshots on it) seems suboptimal. Maybe I should get a HDD to replace what I use optical for? But HDDs can fail, and if I enable grooming then I won't be able to recreate my historical archive. I've thought of getting a BD writer at some point, are those reliable for backup media? I suppose I could turn off fast catalog rebuild on my DVDs, but then recreating them will take forever if I have to do it.

Any other thoughts about what I should do, or what I shouldn't do?

Thanks in advance if you've read through this all....

[Scillonian]

Would your "all file" backup include the Retrospect configuration files in it. If it does then you may be able to recover your configuration from before you made the change.

[Lennart_T]

5 hours ago, Rhyslin said:

I didn't want to rely on just a HDD for my backup sets for any sort of historical data.

I always buy my disks in sets of three. I rotate them regularly and the disks I'm not currently using are stored off site. When they are full, I retire them and buy a new set. That happens about every three years or so. My current set on three 4TB drives started in 2016 and I still have 500 GB free on each of them.

Nowadays you can buy large disks for pocketmoney, compared to just a few years ago.

How much is a 4TB disk per GB compared to your DVDs?

And is your time really worthless? (You must spend more time struggling with and shifting of optical discs than you would be using hard drives.)

[Rhyslin]

6 hours ago, Scillonian said:

Would your "all file" backup include the Retrospect configuration files in it. If it does then you may be able to recover your configuration from before you made the change.

That's an idea, but I'd have to learn/figure out what config files contains these settings.

However, that's not the only problem. Given how large the catalog files are getting (14 GB for one of my computer's DVD backup set), it seems like using fast catalog rebuild is really a no-go anymore for these anyways. And given the custom config failure in any case, I'm not sure if I should trust R15 writing to DVD+R DLs anyways. However, it did seem to write and verify a few snapshots.

[Rhyslin]

2 hours ago, Lennart_T said:

I always buy my disks in sets of three. I rotate them regularly and the disks I'm not currently using are stored off site. When they are full, I retire them and buy a new set. That happens about every three years or so. My current set on three 4TB drives started in 2016 and I still have 500 GB free on each of them.

Nowadays you can buy large disks for pocketmoney, compared to just a few years ago.

How much is a 4TB disk per GB compared to your DVDs?

And is your time really worthless? (You must spend more time struggling with and shifting of optical discs than you would be using hard drives.)

The system I had in place used 1 2TB HDD ($100+ at the time), and around 15 DVD+R DLs (about $15) over the span of almost 5 yrs, so it was pretty economical compared to the $300+ and offsite storage. I was kinda hoping that the DVDs would also last many years without worry, something I'd not be sure of for HDDs left stored. I don't have any off-site storage--I'm not a business and I don't even rent any safe deposit boxes or the like. And I don't really need to keep history on all of my system, just my personal files.

I'm not looking for anything new to be implemented, just wanting what used to work (my DVD+R DLs) to work in R15. And to understand why catalog files ballooned from 300MB (my almost 5 yrs of DVD backup set) to over 2 GB for about 3-4 yrs of that time span (when I went to retransfer the snapshots from my HDD set to my new DVD+Rs). I guess I could live without using fast catalog rebuild since I keep those catalogs backed up (though it would be one session behind, but I could live with it, I guess).

[Lennart_T]

I'm not a business, either. I have one of my personal backup drives in a drawer at work, the other in a small self-storage facility where I have lots of other stuff. I could also store it at a friend's house.

Long term storage is a real problem. The only solution is either to carve the information in stone or to copy to newer media once every few years.  

[Scillonian]

8 hours ago, Rhyslin said:

That's an idea, but I'd have to learn/figure out what config files contains these settings.

If you would still like to recover the old configuration for the drive I believe the custom configuration files have a .rdi file extension.

The .rdi files shoul be located in one of the Retrospect folder in Program Files or Program Data. I'm not near a machine running Retrospect at present so can't be certain which folder.

[Rhyslin]

On 3/23/2018 at 9:32 AM, Scillonian said:

If you would still like to recover the old configuration for the drive I believe the custom configuration files have a .rdi file extension.

The .rdi files shoul be located in one of the Retrospect folder in Program Files or Program Data. I'm not near a machine running Retrospect at present so can't be certain which folder.

Found and restored them, and R15 now at least recognizes my old sets. Now to see if I  can actually write new ones...

[Scillonian]

13 hours ago, Rhyslin said:

Found and restored them, and R15 now at least recognizes my old sets. Now to see if I  can actually write new ones...

Thank you for the feedback.

For the benefit of future reader the device configuration files are located at: C:\ProgramData\Retrospect\. The .rdi file will be named devicenn.rdi  where nn is a two digit number starting from 00 for the first configured device.