Easily reduce tape backup/verify times in half.

[ShadeTek]

Is it possible at some time in the future that Retrospect could talk to a tape drive manufacturer to make a drive that can verify the data that has just been put on tape as it does the backup.  This was available on Sony  Betacam video tape recorders 25 years ago and called confidence playback.  There was an extra set of playback heads just after the record heads that played back the just recorded data off the tape.  Could cut the time of a backup in half!

[Lennart_T]

I think that is part of the LTO tape technology.

[DavidHertzberg]

Lennart_T is, as usual, correct (second paragraph of the section).

[Lennart_T]

1 hour ago, DavidHertzberg said:

Lennart_T is, as usual, correct (second paragraph of the section).

Thanks, you found what I was looking for, but didn't find.

A bit further down in that article:

"LTO uses an automatic verify-after-write technology to immediately check the data as it is being written,[42][43] but some backup systems explicitly perform a completely separate tape reading operation to verify the tape was written correctly. This separate verify operation doubles the number of end-to-end passes for each scheduled backup, and reduces the tape life by half."

[ShadeTek]

Thanks for the replies.  This would explain why Retrospect displays errors before it re-verifies.

[DavidHertzberg]

On 10/13/2018 at 4:02 PM, ShadeTek said:

Thanks for the replies.  This would explain why Retrospect displays errors before it re-verifies.

ShadeTek,

When Retrospect does that, does it at least occasionally catch additional errors on the verify pass?  I ask because IM long-ago E a significant percentage of read errors for tapes were caused by something happening to a tape surface after it had been recorded on.

I should point out that my most recent ten years of tape experience was with DDS tape drives based on Digital Audio Tape technology.  Although Retrospect verify passes reported these as quite reliable—unlike previous tape drives I had used, the fact that DAT technology was fundamentally designed by Sony as a not-too-much-more-expensive replacement for the analog audio compact cassette meant it did not have any frills such as readback-after-write AFAIK.

I should also point out that my mainframe computer experience goes back to the mid-1960s, with 2400-foot half-inch-wide 7-track reel-to-reel tapes on drives that cost many tens-of-thousands of dollars.  You don't want to hear about that; it would spoil your digestion.

[Craigo5000]

Personally, I'd not use thorough verification. Instead, use MD5 checksum. Much quicker. (Media Verification which eliminates re-read of LTO media)

[ShadeTek]

LTO tape drives have an auto verification using confidence read heads that read the data that has just beeen recorded to tape.

[SunbeamRapier]

It seems to me that the major issue here is that Retrospect creates all the .rdb files first, instead of creating them as it writes to tape.  In my case, I am backing up a large set of files from a portable USB drive for the first time.  After 4 hours, Retrospect is only one third of the way through creating the .rdb files with 8 hours left to go before it even think about writing anything to tape.

This approach might be sensible when backing up to a disk drive but writing to tape is quite a slow process and there ought to be plenty of cpu cycles available while this is happening to generate the .rdb files and the write to tape could be happening immediately instead of 12 hours later.

Retrospect v16.6 is running on a Mac Pro (2015) with 64Gb ram, 3.5Ghz 6 Core Xeon E5.  It is using everything!  Doing anything else on the Mac while the .rdb files are being created is almost impossible.

 

[DavidHertzberg]

15 hours ago, SunbeamRapier said:

It seems to me that the major issue here is that Retrospect creates all the .rdb files first, instead of creating them as it writes to tape.  In my case, I am backing up a large set of files from a portable USB drive for the first time.  After 4 hours, Retrospect is only one third of the way through creating the .rdb files with 8 hours left to go before it even think about writing anything to tape.

This approach might be sensible when backing up to a disk drive but writing to tape is quite a slow process and there ought to be plenty of cpu cycles available while this is happening to generate the .rdb files and the write to tape could be happening immediately instead of 12 hours later.

Retrospect v16.6 is running on a Mac Pro (2015) with 64Gb ram, 3.5Ghz 6 Core Xeon E5.  It is using everything!  Doing anything else on the Mac while the .rdb files are being created is almost impossible.

 

SunbeamRapier,

Where exactly is Retrospect creating your .rdb files?  According to this post and the previous discussion in that thread , tapes do not have .rdb files—which are instead in the Catalog File on disk for the tape Media Set.  For disk Media Sets, the destination disk files mostly consist of the .rdb files—which are the containers for the backed-up data.  If your 2015 Mac Pro is "using everything" creating a Catalog File for upwards of 40 million files, that's probably because doing so eats up all available CPU cycles.

If you want to create a Support Case for this feature request, here's why and how to do it.  I suspect Tech Support will reply that, when you took your Mac Pro to Europe 3 years ago, you should have installed a copy of Retrospect Desktop on it for use with your portable 4 TB USB HDD, instead of making do with Time Machine.  They may also reply that, having failed to do so, you should at least be running your Retrospect backup to tape from your restored TimeMachine disk as a series of script runs—each specifying a non-overlapping Date Modified Within as a Rule or non-overlapping Favorite Folder sources.

BTW, I've used Retrospect Mac terminology in this post because your post is blatantly about that variant.  It doesn't make any difference, because your complaint is about the underlying Engine—which is the same for both variants.  However your complaint has nothing to do with the topic of the preceding posts in this thread; you should have started a new thread in the Product Suggestions-Mac OS X sub-forum—which has a large square green Start New Topic button.  To create a link to a post in another thread: right-click the doohicky on the top right of that post that says "Share Link" when you mouse over it, then choose Copy Link Location form the resulting drop-down, then select the text in your own post that you want the link to be in, then click the Link tool in the editing toolbar and paste the copied Link Location into the URL box in the Link dialog.

[SunbeamRapier]

Hi David.  You are quite right, the media set was a disk set - I thought I had selected a tape backup.  It created a new folder "retrospect" on the same drive.  Would I have been prompted for the destination volume?  I don't recall seeing that but I had a lot of trouble upgrading from v12 to v16 so I was somewhat frazzled at the time and, after 3 years, still struggling to get used to the interface.

It took 17 hours to create this backup which seems excessive for 844Gb of data (The new retrospect copy is 1Tb).

As for replacing Time Machine with Retrospect, now that I have all my kit I can look at that.  I note, however, that there is nothing in the retrospect manual about any issues backing time machine backups.  Perhaps if I had known this was going to be an issue I could have set up retrospect backups instead.

The 40 million files seems absurd but I have no idea how TimeMachine manages its storage and perhaps it does indeed create tens of millions of little files to keep track of the backups.

Thank you for the reference to the  Product Suggestions-Mac OS X sub-forum.  When I have completed the backups I will assemble my thoughts and make some suggestions.

Best regards

Stephen

[DavidHertzberg]

SunbeamRapier,

If you used the Backup Assistant on pages 89-92 of the Retrospect Mac 16 User's Guide, you would have been prompted to create a Media Set destination and would have created one per pages 86-89—which includes specifying a Media Set Type.  I'm not going to claim the UI is as simple as TimeMachine's, but I didn't have particular trouble learning it for backing up local disk sources in 2015—"client" machine sources were somewhat more complicated for a couple of installation-specific reasons.  However from 1995-2010 I'd used the old UI, a non-multithreaded version of what Retrospect Windows'  UI still is.

I happen to do a Recycle backup on Saturday mornings that includes a source HDD that is installed inside my 2010 Mac Pro "backup server".  The rate for its backup phase is around 2.3GB/minute for source data ("''B' is for bytes 'b' is for bits"), so for 1TB that would take about 7 hours.  If you used the Backup Assistant it seems that'd automatically turn on the Thorough Verification option described on page 97; since it is a byte-by-byte comparison, that would add about another 7 hours.  So IMHO 17 hours fits into the speed range for Retrospect, considering your source HDD is connected with USB.  The product in column 2 of this Retrospect Knowledge Base article gives faster speed at a much higher price.  See  here for Retrospect backup of TimeMachine backups.

Now you can proceed to create a tape Media Set, and then create a Copy Media Set script to copy the disk Media Set set you have already created to that tape Media Set.  A Copy Media Set script is described on pages 135-137; it may take a few hours to run, but shouldn't put as much load on your CPU.😁