Jump to content
Sign in to follow this  
emulator

FEATURE REQUST: Hardware compression+AES encryption

Recommended Posts

To anyone from EMC reading this (Robin!!!), I would like to make the following feature request:

 

the ability to encrypt data from within Retrospect (AES 256-bit) while still being able to use a backup devices hardware based compression.

 

Here's why:

 

1. Backing up to tape using the tape drive's native hardware compression takes us a little over two days on an LTO3 autoloader. We can fit the entire backup set on six tapes.

 

2. Backing up to tape using Retrospect's software based compression with AES 256-bit encryption takes over SIX days to complete, and uses EIGHT tapes.

 

I have been told by EMC representatives that it is impossible to use a device's hardware compression with Retrospect's encryption because of data loss. But what about encrypting it FIRST, and then allow the tape drive to compress LAST? I can encrypt a file using any third party encryption tool (such as PGP), and then ZIP the encrypted file, with no data loss. Why can't Retrospect do this?

Share this post


Link to post
Share on other sites

If the encryption is good, the resulting stream should be close to pseudo-random. If it was not, then patterns would be clues to decryption. Such encrypted streams are thus not amenable to compression.

 

But, going in the other order, compressing first, then encrypting, ends up by encrypting

a smaller near-random data stream.

 

Think a bit about the entropy of the data stream at various points.

 

Russ

Share this post


Link to post
Share on other sites

Russ explained it very well. However there might be some options for you to consider.

 

You might want to experiment with a (way) faster CPU to speed up the compression and encryption process. This is kind of a brute force option but actually the only one I can think of. Make sure you have disabled your devices hardware compression settings.

 

And maybe you can save time during the verify run, if you are using that. We nowadays backup using the option 'Media verification', which creates an MD5 digest during backup. During the verify run Retrospect compares the backup medium with this digest instead of re-reading the original files. Maybe somebody can confirm this should work faster with your setup/problem.

Share this post


Link to post
Share on other sites

As of today it is not possible to compress encrypted data. You can only encrypt compressed data. It is a chicken and the egg thing.

 

The solution to this is support for Hardware based Encryption within the tape drive. Some tape devices have a built in AES encryption. We are looking at ways to add support for these storage devices.

Share this post


Link to post
Share on other sites

Thank you for answering me.

 

>You might want to experiment with a (way) faster CPU to speed up the compression and encryption >process. This is kind of a brute force option but actually the only one I can think of. Make sure you >have disabled your devices hardware compression settings.

 

We upgraded our hardware to a dual 3Ghz Xeon system with 4GB RAM. The numbers that I gave are the numbers from this system.

 

>And maybe you can save time during the verify run, if you are using that. We nowadays backup using >the option 'Media verification', which creates an MD5 digest during backup....

 

The numbers that I gave do not include the verification times (!!). Imagine how much longer that might take!

 

I understand that you need a certain amount of entropy to perform the encryption. But, and please forgive this as I am not a math major, but why can I compress an encrypted PGP file using ZIP? Why does that not corrupt the data?

 

And, Robin (thank you for writing!!), are we to understand that, as of right now, there is not support in Retrospect for hardware-based encryption (such as the LTO4 devices)?

 

Thanks again!

Share this post


Link to post
Share on other sites

Quote:

I understand that you need a certain amount of entropy to perform the encryption.

 


No, that wasn't what I said. High entropy = random data. Encryption increases the entropy (randomness) of the output stream. A data stream has whatever entropy it has. You can encrypt data with no or almost no entropy (i.e., a single bit). There is no "entropy minimum" to be able to encrypt data.

 

Quote:

But, and please forgive this as I am not a math major, but why can I compress an encrypted PGP file using ZIP?

 


You should be able to. What makes you think that you can't? After unzipping, it should be the same data. ZIP is lossless compression of whatever file it is given (but there have been some releases of ZIP compression that had bugs).

 

Quote:

Why does that not corrupt the data?

 


Why do you think it should?

Share this post


Link to post
Share on other sites

Quote:

But, and please forgive this as I am not a math major, but why can I compress an encrypted PGP file using ZIP?

 

You should be able to. What makes you think that you can't? After unzipping, it should be the same data. ZIP is lossless compression of whatever file it is given (but there have been some releases of ZIP compression that had bugs).

 


 

So why can't I encrypt first, and then compress second? This is the same situation as above, isn't it? We encrypt the data FIRST, and then compress is LAST? The compression on the LTO3 drive is lossless as well, is it not?? If you say that I can compress data after it has been encrypted (PGP encrypt first, ZIP second), why can't Retrospect do this?

 

How about this situation:

 

1. Back up to hard drive with encryption and no compression

2. do a "raw copy" of the data to a tape drive, using the drive's built in compression

 

Just as I can ZIP a PGP encrypted file, can't we compress an AES256 encrypted file to tape using the drive's built-in compression?

Share this post


Link to post
Share on other sites

Compression removes redundant data. Encryption scrambles the data and removes the redundancy, making it impossible to compress.

Share this post


Link to post
Share on other sites

Quote:

Compression removes redundant data. Encryption scrambles the data and removes the redundancy, making it impossible to compress.

 


 

Thank you!!! This actually makes sense to me!! And this simple, clear statement gets to the point without the dissertation!!

 

And Robin, thanks for working the weekend shift :-)

Share this post


Link to post
Share on other sites

Quote:

This actually makes sense to me!! And this simple, clear statement gets to the point without the dissertation!!

 


It's the same answer I gave.

Share this post


Link to post
Share on other sites

Quote:

Quote:

This actually makes sense to me!! And this simple, clear statement gets to the point without the dissertation!!

 


It's the same answer I gave.

 


 

But Robin's is the ABRIDGED version ;-)

 

I think we can close this topic. Thanks all who posted for your help.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×