How I Added Cloud Backup Capability to Retrospect

[mbennett]

I have documented this process thoroughly and carefully, because there either is no specific information regarding backing up to the cloud on the Retrospect website and in the documentation, or I’ve been unsuccessful at finding it.

 

I have tried to be extremely thorough and detailed in documenting this, and it’s  intentional.  Get impatient and start trying to skip steps, and you’re on your own.  I know this works. Here is what I did, and what you will need to do to duplicate my project:

 

Retrospect 9.5  [Please add replies as to whether it works with earlier versions.]

A third party application called TNT Drive [see notes]

An Amazon S3 cloud account [see notes]

 

These are the steps I followed.

 

1.      Sign up for Amazon Web Service.  If you already have an Amazon account, you’re ready to add the AWS service. [see note at bottom for the link.]  In any event, you  must supply a credit card number.

2.      Install TNT Drive on the Retrospect server system.  TNT Drive runs as a service, so you’ll need Administrator credentials on most Windows systems when you install it.

3.      Once your AWS account is established, login to the Amazon Management Console.  You’ll get an email from Amazon with links.

4.      Go to the S3 section.

5.      Create a bucket, which must be a globally unique name, in much the same way a domain name is unique.  I used the Oregon repository, which is several thousand miles from my physical location.  Use your own judgement here.  [in my example, I’ll use a bucket named ‘wxyzwxyz’ which isn’t a bucket that I own.  Please don’t use that name in real life.  Make up a name that’s meaningful to you.] Write down the name of your bucket.  You’re going to need to know that.

6.      Back to the Amazon Console Home, start the Identity & Access Management program.

7.      Work through the following sections to secure your account.

a.      Click the down arrow next to “Delete root access keys”, then click the “Manage Security Credentials” button.

b.      On the box that pops up, click “Get Started with IAM Users”

c.      Click “Create new users” and create at least one user.  For this example I’ll call the user “Backup”.  Make sure you check the box to generate an access key for each user.  Click “Create”.

d.      Click “Download Credentials” in the lower right corner.  Make sure you can open the CSV file that is downloaded to your system, then Close that window.

e.      For each user you create, create a secure password.  Amazon will generate a very good password for you, and I recommend you use it, or use a program such as LastPass that generates a complex password.  Simple passwords  make it easy for the bad guys to break into your account and steal your confidential information.  Again download those credentials in a CSV file, then close the window.

8.      I chose NOT to create roles, which I believe are unnecessary for our purposes.

9.      I also chose NOT to setup MFA (multi-factor authentication).  I suspect enabling MFA would break the ability of TNT Drive to successfully connect to the cloud server.

10.    You must setup a Group, and assign the user you created as the Retrospect backup user to have “AmazonS3FullAccess” rights, by checking the box next to that policy.  For this example, I call the group “BGroup”.

a.      If you have more than one user (for example an Administrator and the Backup user) you can fine tune who is allowed specific functions.

b.      Now go back to the Users screen to assign users to the Groups which have rights to functions to which they should be allowed.

11.    I did create a password policy just to make sure a policy exists.

12.    To summarize this section: At minimum you need one user who belongs to one group.  In our example the user Backup belongs to group BGroup. That group must have AmazonS3FullAccess rights.  You must generate and download security credentials and a password.

13.    Log out of the Amazon site.

 

14.    Now start the TNT Drive program.  Click the “Add a new mapped drive” button.

a.      Open the CSV credentials file you downloaded and select the account (“Backup” in our example) that has permissions for full access to the S3 service.  Copy and paste the Access ID Key and the Secret Access Key to the setup screen.

b.      The “Amazon S3 Bucket” will be the name you assigned in Step #5.  In our example this is “wxyzwxyz”.

c.      I assigned drive R:, but it is important to note that this drive will NOT be used when you create the Retrospect backup job.  Just assign a drive letter and move on.

d.      Click “Add New Drive” and if you did everything right, you should see a box indicating that you’re connected to the drive.  You now have a cloud file repository.  Celebrate if you wish, then continue.

e.      You may close the TNT Drive program.  Unless something changes regarding your credentials, it’s unlikely you’ll need to run the program again.

f.       You may now look in “My Computer” to see the Amazon drive, which is mapped to drive R: This drive will show free space of 16 terabytes. Yikes!  Review the Amazon price list link in the Notes below to see if you want to pay for this much storage, but we’re addressing that issue later.

 

15.    Start the Retrospect program and take a deep breath.

16.    Click Configure, then open the Volumes selection.

a.      Click the “Network Places” name in the tree one time, so it turns blue.

b.      Click the “My Network” button on the right.

c.      Click the “Advanced” button.

d.      You should see a box and the instructions “Please enter the UNC or HTTP path for the volume or subvolume.”

e.      The name should be the UNC of the Amazon S3 bucket, relative to your computer system. See the Notes if you don’t know the name of your computer.  If your PC is named “WorkPC” you will enter \\WORKPC\wxyzwxyz

f.       Click OK, and you will return to the Volumes Database with your drive displayed.

g.      Note that if the folder symbol has a red dot you’ve done something wrong, and it won’t work.  Delete the folder with a red dot and carefully repeat this step.  You must be successful before you continue.

17.    Now select the Backup Sets configuration program.

a.      Select Disk as the type of media.

b.      Give the set a name, such as “S3 Cloud”.

c.      Click the Select button to choose the disk.

d.      Select My Computer.

e.      Click the Advanced button, and enter exactly the same UNC as you did in the above step.

f.       A new window will open displaying the properties of the backup device, including the maximum volume size of 16,384 GB.  Truncate this, depending on your needs and how much you want to pay Amazon.  This will effectively impose a quota on your S3 storage.

g.      A window opens prompting for backup set security.  I personally set my security to the highest level, AES-256.  This insures that all data transmitted across the internet is highly encrypted, and is also encrypted on the server.  Using AES-256 requires you to create a long password, and whatever you do in this entire process, DON’T LOSE THAT PASSWORD, or your backups will be impossible to restore in a disaster.  Be careful and deliberate when creating and documenting this password.  Allow Retrospect to store the password, but be forewarned that any time you edit this Backup Set, you must know and enter that password manually.  Click Next.

h.      The next window will ask about the policy to follow if the drive fills.  I recommend that you select the  “Keep according to Retrospect Policy” button.  Read the manual about Grooming for more information.  Click Next.

i.       Click the catalog file location.  I used the default.

j.       Click Finish.

 

Now create a backup script to utilize this resource, which I’m not going to try to document.  I figure you’ve already done that at least once.  I will note that I used data compression as well as Block Level Incremental Backup, which I did to minimize storage usage as well as bandwidth.  I strongly encourage you to run the backup right now by leaving that option checked.  The reason for this is that Retrospect is going to balk the first time and give the “Waiting for media” message and stop.  Force it to run once.   Setup the schedule for execution and you’re done.  See the additional Notes at the bottom.

 

That concludes today’s lesson.  If I documented everything well, and if you followed my instructions correctly, we have jointly passed the test.  Make lots of backups, because you can never have too many, and good luck.

 

Mark

 

 

 

Notes:

 

Disclaimer: I know this all works now, on 3/1/2015, but can’t guarantee that it will work if features and functionality change in any of the three programs and services which are used. I’ll try to revisit this if something changes, as soon as I figure it out.

 

There may be more that can be done to secure this service, but I can’t see what that might be.  I welcome suggestions and comments from those who are more well-versed and/or paranoid on the subject of security.

 

TNT Drive can be downloaded here: http://tntdrive.com/

At this time, the personal use license is $39.95 and the commercial use license is $59.95.  There are pretty good volume discounts. I wish this functionality was built into Retrospect, but as far as I can tell it is not.  There are also competing services if you want to shop, but setup is on you.

 

Amazon Web Services is here: http://aws.amazon.com/

Amazon’s S3 price list is here: http://aws.amazon.com/s3/pricing/

 

I have experienced a fairly frequent occurrence of getting bogus messages when the cloud backup job starts, saying the job is waiting for media.  I believe this is a network latency issue.  After the job has run the first time, in nearly every subsequent run the backup job has thrown the message, then continued and completed successfully.  Don’t panic the first time you see it.

 

Realizing that some people don’t know the name of their computer, the simple method is to find the “My Computer” icon on your desktop.  Right-click and select Properties from the context menu.  The computer name will be shown there.  Windows 8 systems will probably not be able to use this method.  Google for instructions.

 

[alexkorenori]

Thank you very much for the detailed explanation of how to add Amazon S3 backup capability to Retrospect!  I followed your method to the letter and I too now have my home PC backing up via TNTDrive to my personal AWS account.  I had tried other cloud storage vendors via Webdav but this never really worked for me mainly because of the slow upload speeds caused by the distant locations of the cloud vendor's servers.  With Amazon, I am able to select a server near my location so upload speeds are pretty good even via the VDSL broadband connection that I have in my apartment. 

 

Since this worked so well, I have now signed up for an AWS account at my work place, and am going to do some trials with backing up from my Retrospect Multi-Server 10 set up at work.  Hopefully, the faster broadband connection at my work will let me get some good backup speeds compared to my home.

 

Thanks again for the detailed how-to. 

[mbennett]

You're more than welcome.  It's gratifying to know that somebody is taking advantage of this.  There are no Amazon service points that are close to me.  I'm in Missouri hitting the Oregon server.  It would be interesting to find out if the latency for S3 is indeed affected by distance.

 

I've learned a couple of things after using this technique for a few months.

1. TNTDrive runs as a service, and as with all services under Windows it can die suddenly without explanation.  So when troubleshooting, the service is the first thing to check.

2. Amazon S3 runs pretty reliably, but infrequently it will crash and leave a deep trench.  (Figuratively speaking.)  If your cloud backup fails, and you can't seem to resolve it, my advice is to leave it alone for 24 hours.  I have this running on three systems, and one day all three of them failed!  I wasted time trying to figure out what was wrong, changed some settings, and finally decided nothing on my end was wrong.  I canceled the jobs and that night it ran perfectly on all three systems.

 

Good luck.

[alexkorenori]

I've found that I'm incurring unexpectedly high "data transfer" charges from my retrospect backups to Amazon s3.  I originally thought that it was because I did a rebuild catalog from the cloud backupset on s3, but looking at the Billing & Cost Management>Cost Explorer>Custom View (by day) on AWS, it looks as though I incur data transfer charges during normal backups too.  Perhaps it's because I have the option for "Thorough verification on" in my S3 backup script. 

 

Perhaps what I need to do is to do a staged backup first to local NAS (with thorough verification on) and then from there to S3 (with thorough verification off).  Bit scary though, since there's no guarantee that the backupset on s3 will be a perfect copy of the one on the NAS.

 

Are you or anyone else doing Retrospect cloud backups to s3 experiencing high data transfer charges?

[mbennett]

No, I'm getting nominal monthly charges from Amazon.  I don't think the charges have even exceeded $1/month, but I'm only backing up extremely critical files to the cloud, and compressing them.  About the only options for you would be to change your settings.  You could also scout around for a service other than Amazon's which should also work if you can come up with a technique to use the cloud drive as a URI on your system.

 

Within Retrospect there's a utility to copy a backup.  I haven't played with it or even read up on it.  You might explore simply copying a backup that is already made and verified to the S3 service, which should reduce the network traffic significantly.

 

Sorry I can't be of more help.  Let us know what solution you arrive at.