Jump to content

How to Set Up Cloud Backup for Retrospect Desktop v.11 for Windows with Amazon AWS S3 Services


Recommended Posts

[NEW: These steps have been updated to enable Amazon S3's "Infrequent Access" storage class, for cheaper cloud backups.]
 
It took me a few hours of trial-and-error to get Retrospect's new cloud backup feature working with a cloud services provider.  I wanted to share what I learned in the hope that it will save others a little time.
 
Be sure you are running Retrospect v.11.0.1 or later for Windows, or v13.0.1 or later for Mac, for full support of both cloud storage and Amazon S3's "Infrequent Access" storage class.  The instructions below were written specifically for Windows and for Amazon S3, but for Mac users or other cloud providers, some of the instructions below might still be useful to you.  Retrospect provides a list of supported cloud storage providers.
 
I found the Retrospect page on how to setup an Amazon S3 account to be useful, but missing some details and slightly out-of-date.  So, below are updated instructions for setting up your Amazon S3 account, creating a bucket to hold your Retrospect cloud data, setting the storage class of that bucket, creating a user for your S3 account, obtaining the Access Key and Secret Key for that user, and setting permissions for that user.
 
Create your Amazon AWS S3 account:
  1. If you don't already have a regular Amazon shopping account, go to www.amazon.com and create one.
  2. Go to http://aws.amazon.com/s3.
  3. Click "Create an AWS Account" in the upper-right corner.
  4. Sign in using your normal Amazon shopping email address and password.
  5. Follow the instructions to create your Amazon AWS account.
  6. Provide credit card information and complete the identify verification to finish the account creation.
Going forward, to login to your Amazon S3 console, go to http://aws.amazon.com/s3, click "Sign in to the Console" in the upper-right corner, and sign in using your normal Amazon email address and password.  If you don't see a "Sign in to the Console" button, click instead on the "My Account" menu in the upper-right corner and select "AWS Management Console."
 
Create a bucket in your Amazon S3 account to hold your Retrospect backups:
  1. If you haven't already, login to your Amazon S3 console, as described above.
  2. From the "Services" menu at the top, click "S3" (on the right toward the middle).
  3. Click the blue "Create Bucket" button in the upper-left.
  4. For the "Bucket Name," type a name that consists entirely of lower case letters, digits, hyphens, and periods.  Note that it must be globally unique across everyone that uses S3.  For example, a good bucket name might be "myname-retrospect", where "myname" is your name or a shortened form of your name.
  5. For "Region," choose the Amazon S3 data center that is closest to you, or the one that meets your needs.  The region called "US Standard" is currently the one data center that is located on the US East Coast (in Virginia).
  6. Click the blue "Create" button to create that bucket.

If you use cloud backup for multiple servers, you might want to create a separate Amazon S3 bucket for each server you want to back up.

 

Set the storage class of the bucket to be "Infrequent Access" for cheaper cloud storage charges:
  1. If you haven't already, login to your Amazon S3 console, as described above.
  2. From the "Services" menu at the top, click "S3" (on the right toward the middle).
  3. On the left, click on your Retrospect bucket.
  4. Toward the upper-right, click the "Properties" button.
  5. On the right, click on "Lifecycle."
  6. Beneath that, click "Add rule."
  7. On the right, leave the option "Whole Bucket" selected.
  8. Click "Configure Rule" in the lower-right corner.
  9. On the right, under "Action on Objects," put a checkmark next to "Transition to the Standard - Infrequent Access Storage Class".
  10. To the right of that, leave "30" as the value of "Days after the object's creation date" (this is the minimum value).
  11. Click "Review" in the lower-right corner.
  12. You can leave the "Rule Name" blank; a name will then be generated automatically.
  13. Click "Create and Activate Rule" in the lower-right corner.
  14. Note: There is no need to click the "Save" button toward the lower-right.  The rule is already saved.
If you haven't done so already, create a user that can access your Amazon S3 bucket, obtain its Access Key and Secret Key, and give it appropriate permissions.  NOTE: You can use the same Amazon S3 user for providing access to multiple buckets; all you need to do is add an additional Policy for that user to access the additional bucket(s).  See below for either creating a new user, or adding a Policy to an existing user:
  1. The instructions on Retrospect's Amazon S3 setup page seem to imply that you can either follow the instructions in the "Simple Access Setup Guide" section or the "Advanced Access Setup Guide" section.  However, it turns out that you must follow the "Advanced Access Setup Guide" instructions.  If you don't, you will get a permissions error when you try to create the cloud backup set in Retrospect.  Also, the "Advanced Access Setup Guide" instructions are slightly out-of-date, so follow the remaining steps below instead.
  2. If you haven't already, login to your Amazon S3 console, as described above.
  3. From the "Services" menu at the top, click "IAM" (toward the middle of the column that is second from the right).
  4. In the navigation bar on the left, click "Users".
  5. NOTE: If you have already created a user for another cloud backup bucket, you can use that same user (skipping the "Create New Users" instructions below), and just add the Policy for the additional bucket to it (below),
    using the same Access Key and Secret Key you previously saved for that user when creating the Retrospect cloud backup set.
  6. Otherwise, click the blue "Create New Users" button.
  7. Type an appropriate user name; e.g., "retrospect".  NOTE: You won't actually directly use this user name with Retrospect; instead, you'll just use the Access Key and Secret Key (see below). This user name is just used within Amazon S3.
  8. Make sure there is a checkmark next to "Generate an access key for each user".
  9. Click the blue "Create" button in the lower-right corner.
  10. IMPORTANT: At this point, copy and paste and save the Access Key and Secret Key for this user.  This is the only time you will be able to obtain these two keys, and you need to copy these into Retrospect itself, below.
  11. You should be back on a screen of users that now shows your newly-created user.  (If not, click on "IAM" from the "Services" menu then click "Users" on the left.)
  12. Regardless of whether you just created this user or are re-using a previously-created user, click directly on the user name you just created (NOT on the box to the left of it).
  13. Beneath the Summary you should see four tabs.  Click the "Permissions" tab.
  14. Under "Inline Policies," click "Create User Policy".
  15. Click the radio button next to "Custom Policy".
  16. Click the "Select" button toward the right.
  17. For the "Policy Name," type any name you like; e.g., "retrospect-backup".  You won't really use this name for anything.
  18. Under "Policy Document," cut and paste the following text:   { "Statement": [ { "Effect": "Allow", "Action": [ "s3:*" ], "Resource": [ "arn:aws:s3:::your_bucket_name", "arn:aws:s3:::your_bucket_name/*" ] } ] }
  19. Carefully replace both occurrences of "your_bucket_name" in the text above with the name of the bucket you created earlier in these instructions.  Be sure NOT to remove any punctuation in the above text as you do this.
  20. Click the "Validate Policy" button in the lower-right corner of the screen, and make sure the policy validates.
  21. Click the blue "Apply Policy" button in the lower-right corner of the screen.
Your Amazon S3 account is now ready to receive Retrospect backups.
 
Create a cloud-based backup set in Retrospect to access your Amazon S3 account:
  1. Make sure you are running Retrospect v.11 (or later) for Windows.  (For the Mac, the cloud backup feature was added in v.13, and these instructions will be slightly different.)
  2. Start up Retrospect.
  3. You will want to throttle down the maximum speed of cloud backups; otherwise, they will easily saturate all of your available Internet bandwidth.  To do this, in the navigation bar on the left, under "Configure," click "Preferences".
  4. In the "Preferences" dialog, under "Execution", click "Cloud".
  5. On right side of dialog under "Cloud preferences...", put checkmarks next to both "Limit backup bandwidth" and "Limit restore bandwidth".
  6. Depending on your available Internet upload/download speed, a good number for "Maximum Backup Mbps" is 10 and for "Maximum Restore Mbps" is 20.  If you find that your Internet connection abruptly becomes unusable by other users when doing a cloud backup, it means you set these numbers too high; reduce them until you still have acceptable Internet performance for other users.
  7. Click "OK" to close the "Preferences" dialog.
  8. In the navigation bar on the left, under "Configure," click "Backup Sets".
  9. In the "Backup Sets" dialog, click "Create New...".
  10. Choose "Cloud" and click "Next".
  11. Choose "Cloud Storage".
  12. For the "Name," type any name you like for this backup set; e.g., "Cloud Backup".
  13. For the "Path", type "s3.amazonaws.com/your_bucket_name", replacing "your_bucket_name" with the name of the Amazon S3 bucket you created above.
  14. For the "Access Key" and the "Secret Key", copy and paste the Access Key and Secret Key you obtained and saved above when you created the Amazon S3 user, above.
  15. UNcheck "Use SSL" because you'll generally want Retrospect to encrypt the data it sends to the cloud server (below), so using SSL on top of Retrospect's encryption will unnecessarily slow things down.
  16. Toward the bottom of the dialog, you probably want to specify the maximum value of "8192" in the "Use at most 8,192 GB" setting.  This allows Retrospect to store up to 8 TB in your Amazon S3 bucket, the maximum allowed by Retrospect v.11 for Windows.  If you need additional space, you can add additional Amazon S3 buckets as members of this cloud-based backup set.
  17. Click "Next."
  18. Generally, you will want your backups to be encrypted on the cloud server.  So, for "Backup Set Security", use one of the available Retrospect encryption methods.  Make sure you remember whatever password you choose for that encryption method, because if you lose your computer catastrophically and need to retrieve your backups from the cloud, you won't be able to restore your data unless you remember that password.  NOTE: It's tempting to rely on server-side AES-256 encryption provided by Amazon S3, instead of having Retrospect do the encryption.  But this doesn't work.  If you try and set the properties of the Amazon S3 bucket folder to (say) AES-256, it encrypts all the files that are currently there, but any new files that get added by Retrospect to the folder are unencrypted (unless you manually encrypt again).  Better to leave server-side encryption turned off, and just have Retrospect itself do the encryption.
You can now use your Amazon S3 bucket as a backup set target for your Retrospect backups.
 
On the Retrospect best practices for cloud storage page, the section "Backup Strategy: Disk-to-Disk-to-Cloud" describes a backup strategy where you have one script that does your normal Retrospect backup to a backup disk, and then have a separate transfer script that performs an incremental backup of the contents of your Retrospect backup disk (containing your ".RDB" files) to the cloud.  That is the strategy that I use.  However, it might not be clear how to create this transfer (incremental backup) script.  If you simply create a normal backup script that backs up your backup disk to the cloud, it won't work; it skips right over the ".RDB" files.  Instead, you need to create a transfer script.  The instructions for doing this are below.  NOTE: This is effectively an incremental backup; only the Retrospect ".RDB" files that have changed or added will be transferred to the cloud, so it won't try and transfer the whole disk-based backup set each time.  Also, this "transfer" script is really a "copy" script; it leaves the original disk-based backup set in place, of course.
 
Create a transfer (incremental backup) script in Retrospect to copy your disk-based backup set to your cloud-based backup set (e.g., to your Amazon S3 account):
  1. Start up Retrospect.
  2. In the navigation bar on the left, under "Tools," click "Transfer Backup Sets".
  3. Click "Sources" and choose the disk-based backup set that you want to transfer to the cloud.
  4. Click "Destination" and choose your cloud-based backup set (e.g., your Amazon S3 account).
  5. Leave "Selecting" unchanged as "All Files".
  6. Click "Options" and REMOVE the checkmark from "Media Verification".  For the explanation of why you should do this, see "Disable Verification" in the "Technical Notes" section at the bottom of the Retrospect best practices for cloud storage page  Basically, the verification is unnecessary because the cloud service already does the verification, the verification will add extra time to the transfer process, and most importantly, it will cost you extra money because cloud services charge for each GET and PUT request.
  7. While still in Options, click "More Choices," then click "Schedule," and specify a suitable schedule window for this transfer script to occur.  If the script is not finished when the time window elapses, Retrospect will stop where it is, run other scripts that might be pending (like your normal disk backup script), then restart this transfer script according to the schedule below.
  8. Click "Schedule," and specify a suitable schedule for transferring (copying) the disk backup set to the cloud-based backup set.  You do not need to worry that there will be a collision between your normal backup script to your disk-based backup set, and the transfer script for your disk-based backup set to the cloud-based backup set, if the timing of these two scripts happens to overlap.  Retrospect automatically delays a script from starting if it needs a backup set (in this case, the disk-based backup set) that is already in use by another script.
That's it.  I hope the above instructions are useful.
  • Like 1
Link to comment
Share on other sites

  • 1 year later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...