Jump to content

AWS virtual tape library


blm14

Recommended Posts

An interesting question, to which I don't know the answer.  Someone (I see now it was blm14) asked the same question about 2 years ago, but didn't get an answer.  However here is a list of iSCSI-interfaced VTLs which have been qualified to work with Retrospect Windows.

However another interesting question is whether blm14 should do this, assuming he/she can.  There is a non-trivial hardware cost associated with it, as described under "Storage Requirements: Tape gateway" here.  And what blm14 would be getting for that cost is "low-latency access to data through transparent local caching".   If I understand it correctly, that "transparent local caching" would be Amazon's tape gateway automatically deciding which files should be backed up onto local disk vs. backed up to the AWS cloud.   Certainly that would be less work for blm14's organization than managing a disk-to-disk-to-(virtual) tape backup system, in which the decision of which files would remain on local disk would be left up to the grooming strategy.  But would the organization really be served well, especially for less-than-catastrophic disaster recovery?

There is also the question of cloud storage cost associated with Amazon's VTL.  After an AWS S3 (and Google Cloud)-compatible cloud storage capability was introduced with Retrospect Windows 11, there was a distinct falling-off of interest after administrators discovered how much it would cost for a large installation to store data using it.  Not surprisingly, Retrospect Windows 12.0 added a Backblaze B2 capability with a storage cost less than 20% of that of Amazon's.  Moreover Retrospect Windows 12.5 certified Wasabi, a provider whose cloud storage costs are nearly as low as Backblaze B2's but which—unlike Backblaze B2—has API compatibility with AWS.  blm14 certainly couldn't use Amazon's VTL with Backblaze B2; I don't know whether he/she could use it with Wasabi.

P.S.: For those administrators who don't understand what blm14 is asking about, here is a video about the use of the Amazon VTL with another enterprise client-server backup application.  I found it as a reference in the LAN/WAN/Cloud subsection of DovidBenAvraham's now-separated Wikipedia "Enterprise client-server backup" article.  My point in the second paragraph of this post about non-trivial hardware cost is briefly alluded to as "on-premise appliance" at minute:second 0:25 in the video.

P.P.S.: Here is a more-detailed YouTube video about the use of a third client-server backup application with the AWS Storage Gateway VTL.  If you look shortly after minute:second 3:50 in the video, you'll see it talks about an "unique inquiry string " that is coded into both this app and the one covered by the video in the preceding paragraph for use in the Medium Changer Type field of the Storage Gateway VTL.  I'd suggest that blm14 ask Retrospect Sales (they'll undoubtedly check with Tech Support) if this "inquiry string" is an option in the Retrospect "backup server"; if it is then he/she can use it, if not then probably not.

Edited by DavidHertzberg
Added link to Wikipedia article on Cloud storage gateway; added third prgf. on storage costs; it was blm14 who asked question two years ago; added P.S. linking to video about Amazon VTL; added P.P.S. linking to more-detailed video; modernized link in P.S.
Link to comment
Share on other sites

  • 1 year later...

blm14,

I still suggest contacting Retrospect Sales.  Did you do that, as I suggested in the P.P.S. of my preceding post in this thread?   Also, I said in the P.S. of that same post, AFAIK there would be a non-trivial hardware cost; however a re-listen to the brief mention in the NB video and Google search suggests that may be replaced by a virtual appliance in an AWS EC2.

Link to comment
Share on other sites

Why use a VTL instead of doing a "normal" disk backup set to an S3 bucket or similar cloud service? (I can think of some reasons, but would be interested to hear yours.)

Regardless, Retrospect isn't listed in Amazon's supported applications table and neither media changer is listed by Retrospect (unless the STK-L700 is close enough to e.g. the SL500 that it'll work), so I think you're still left waiting. Or very much on your own if you try to make something work.

As David suggested, try contacting Sales. You may get lucky, and at least you'll flag it as something for them to consider in the future.

Link to comment
Share on other sites

4 minutes ago, Nigel Smith said:

Why use a VTL instead of doing a "normal" disk backup set to an S3 bucket or similar cloud service? 

 

The short answer is that I am backing up data which for various regulatory reasons must be retained for very long periods (minimum 7 years) and I would like to be able to segment this stuff into "recent" backups that are available more readily, and "archival" data which is stored and housed in glacier...

Link to comment
Share on other sites

But S3 would also offer you Standard, Glacier and even Deep Archive. So you could use Retrospect's Cloud Backup or Disk Set(s). I've never used Cloud (some who has might want to chip in) so I'd probably default to Disk Sets with reasonably small member sizes, generated on-prem then uploaded to S3 Standard once each is "full", then migrating from Standard to Glacier after a year or so.

There are many ways to skin this cat -- how exactly you do it will be prompted by your data, how you manage your backups, how the regulations say you *must* manage them, how much money you're allocated for the job, etc...

The use-case for VTL I thought of first was "lots of physical tapes which I can put into cloud storage by duplicating through the Gateway" i.e. you need to generate an exact virtual replica of your current physical holdings. If that isn't a regulatory requirement and you can manage the migration using Transfer operations from tape set to Cloud/disk set, VTL support may not be needed at all.

Link to comment
Share on other sites

blm14,

What Nigel Smith said.  Let me add two pieces of information.  First, it is a not-evident fact (see page 225 of the Retrospect Windows 11 User's Guide; last I heard you were still using that version) that Transfer Backup Sets operations can use Selectors.  You could use Selectors in running Transfer Backup Sets operations whose source is a regular Amazon S3 bucket and whose destination is a Glacier bucket, specifying a date to separate the "archival" backups from the "recent" backups.

But that would still leave you with the "archival" backups also in regular S3.  It is a fact that, if you upgraded to at least Retrospect Windows 15, you could use Selectors in Grooming scripts to groom out the "archival" backups from regular S3 after you have migrated them to Glacier.   This Knowledge Base article talks about using Selectors in Grooming scripts specifically for satisfying GDPR requirements, which is why the Selector capability was added to Grooming scripts in Retrospect 15.  My guess is that you could specify the faster and cheaper (no Amazon download fees) Performance-Optimized Grooming in those scripts, since your grooming Selector would specify a date—not the name of an individual customer having a "right to be forgotten".

In any case, my impression from this article is that you would have to have an on-premises appliance in order to use an AWS VTL.  because that would require some kind of "intermediary" local storage.  Since you seem to have an aversion to taking local tapes or disks off-site, how would that differ from Nigel's proposal?

P.S.: Here's rforgaard's 2016 post on how to setup AWS S3 cloud backup for Retrospect Windows 11.  Here's the only-slightly-later KB article; here and here are other applicable KB articles.

P.P.S.: Just to make it crystal-clear, blm14, I don't think you need an Amazon VTL.  Transfer Backup Sets in Retrospect Windows 11 gives you the capability to transfer "archival" backups from a local tape Backup Set directly to Amazon Glacier (after a 1-day stay in regular AWS).  Retrospect Windows 15's enhanced Grooming scripts would give you the capability to keep the "recent" backups in regular AWS, without duplicating what you have in Retrospect Glacier.  If you need multiple  "conceptual tape" Backup Sets to keep the "recent" vs. "archival" dates straight, you'll probably have to pay for extra hardware even if Retrospect can interface with an Amazon VTL—and I don't know if it can (or needs to).

P.P.P.S.: Here and here are articles discussing VTLs.  The bottom of the second page of the first article discusses a solution that  "is available either as a fully-built appliance or as a software component of ***'s Virtual SAN SDS solution that can run on a customer's existing hardware."  The STK L700 is a physical tape library originally from StorageTek; my impression is that its instruction set is widely emulated.  Again, I don't see why you can't simply run Transfer Backup Sets scripts to copy  your "recent" backups to either local HDDs or regular AWS,  and then run Transfer Backup Sets scripts to move the "archival" data to Glacier—followed by Grooming it from the "recent" backups.

Edited by DavidHertzberg
Reworded second paragraph for clarity; added P.S.; added P.P.S.; added P.P.P.S.
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...