emulator Posted September 21, 2007 Report Share Posted September 21, 2007 To anyone from EMC reading this (Robin!!!), I would like to make the following feature request: the ability to encrypt data from within Retrospect (AES 256-bit) while still being able to use a backup devices hardware based compression. Here's why: 1. Backing up to tape using the tape drive's native hardware compression takes us a little over two days on an LTO3 autoloader. We can fit the entire backup set on six tapes. 2. Backing up to tape using Retrospect's software based compression with AES 256-bit encryption takes over SIX days to complete, and uses EIGHT tapes. I have been told by EMC representatives that it is impossible to use a device's hardware compression with Retrospect's encryption because of data loss. But what about encrypting it FIRST, and then allow the tape drive to compress LAST? I can encrypt a file using any third party encryption tool (such as PGP), and then ZIP the encrypted file, with no data loss. Why can't Retrospect do this? Quote Link to comment Share on other sites More sharing options...
rhwalker Posted September 21, 2007 Report Share Posted September 21, 2007 If the encryption is good, the resulting stream should be close to pseudo-random. If it was not, then patterns would be clues to decryption. Such encrypted streams are thus not amenable to compression. But, going in the other order, compressing first, then encrypting, ends up by encrypting a smaller near-random data stream. Think a bit about the entropy of the data stream at various points. Russ Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted September 21, 2007 Report Share Posted September 21, 2007 Russ explained it very well. However there might be some options for you to consider. You might want to experiment with a (way) faster CPU to speed up the compression and encryption process. This is kind of a brute force option but actually the only one I can think of. Make sure you have disabled your devices hardware compression settings. And maybe you can save time during the verify run, if you are using that. We nowadays backup using the option 'Media verification', which creates an MD5 digest during backup. During the verify run Retrospect compares the backup medium with this digest instead of re-reading the original files. Maybe somebody can confirm this should work faster with your setup/problem. Quote Link to comment Share on other sites More sharing options...
Mayoff Posted September 21, 2007 Report Share Posted September 21, 2007 As of today it is not possible to compress encrypted data. You can only encrypt compressed data. It is a chicken and the egg thing. The solution to this is support for Hardware based Encryption within the tape drive. Some tape devices have a built in AES encryption. We are looking at ways to add support for these storage devices. Quote Link to comment Share on other sites More sharing options...
emulator Posted September 21, 2007 Author Report Share Posted September 21, 2007 Thank you for answering me. >You might want to experiment with a (way) faster CPU to speed up the compression and encryption >process. This is kind of a brute force option but actually the only one I can think of. Make sure you >have disabled your devices hardware compression settings. We upgraded our hardware to a dual 3Ghz Xeon system with 4GB RAM. The numbers that I gave are the numbers from this system. >And maybe you can save time during the verify run, if you are using that. We nowadays backup using >the option 'Media verification', which creates an MD5 digest during backup.... The numbers that I gave do not include the verification times (!!). Imagine how much longer that might take! I understand that you need a certain amount of entropy to perform the encryption. But, and please forgive this as I am not a math major, but why can I compress an encrypted PGP file using ZIP? Why does that not corrupt the data? And, Robin (thank you for writing!!), are we to understand that, as of right now, there is not support in Retrospect for hardware-based encryption (such as the LTO4 devices)? Thanks again! Quote Link to comment Share on other sites More sharing options...
Mayoff Posted September 22, 2007 Report Share Posted September 22, 2007 Right now Retrospect does not support hardware based encryption but it is high on the feature list right now. Quote Link to comment Share on other sites More sharing options...
rhwalker Posted September 22, 2007 Report Share Posted September 22, 2007 Quote: I understand that you need a certain amount of entropy to perform the encryption. No, that wasn't what I said. High entropy = random data. Encryption increases the entropy (randomness) of the output stream. A data stream has whatever entropy it has. You can encrypt data with no or almost no entropy (i.e., a single bit). There is no "entropy minimum" to be able to encrypt data. Quote: But, and please forgive this as I am not a math major, but why can I compress an encrypted PGP file using ZIP? You should be able to. What makes you think that you can't? After unzipping, it should be the same data. ZIP is lossless compression of whatever file it is given (but there have been some releases of ZIP compression that had bugs). Quote: Why does that not corrupt the data? Why do you think it should? Quote Link to comment Share on other sites More sharing options...
emulator Posted September 22, 2007 Author Report Share Posted September 22, 2007 Quote: But, and please forgive this as I am not a math major, but why can I compress an encrypted PGP file using ZIP? You should be able to. What makes you think that you can't? After unzipping, it should be the same data. ZIP is lossless compression of whatever file it is given (but there have been some releases of ZIP compression that had bugs). So why can't I encrypt first, and then compress second? This is the same situation as above, isn't it? We encrypt the data FIRST, and then compress is LAST? The compression on the LTO3 drive is lossless as well, is it not?? If you say that I can compress data after it has been encrypted (PGP encrypt first, ZIP second), why can't Retrospect do this? How about this situation: 1. Back up to hard drive with encryption and no compression 2. do a "raw copy" of the data to a tape drive, using the drive's built in compression Just as I can ZIP a PGP encrypted file, can't we compress an AES256 encrypted file to tape using the drive's built-in compression? Quote Link to comment Share on other sites More sharing options...
Mayoff Posted September 22, 2007 Report Share Posted September 22, 2007 Compression removes redundant data. Encryption scrambles the data and removes the redundancy, making it impossible to compress. Quote Link to comment Share on other sites More sharing options...
emulator Posted September 23, 2007 Author Report Share Posted September 23, 2007 Quote: Compression removes redundant data. Encryption scrambles the data and removes the redundancy, making it impossible to compress. Thank you!!! This actually makes sense to me!! And this simple, clear statement gets to the point without the dissertation!! And Robin, thanks for working the weekend shift :-) Quote Link to comment Share on other sites More sharing options...
rhwalker Posted September 23, 2007 Report Share Posted September 23, 2007 Quote: This actually makes sense to me!! And this simple, clear statement gets to the point without the dissertation!! It's the same answer I gave. Quote Link to comment Share on other sites More sharing options...
emulator Posted September 23, 2007 Author Report Share Posted September 23, 2007 Quote: Quote: This actually makes sense to me!! And this simple, clear statement gets to the point without the dissertation!! It's the same answer I gave. But Robin's is the ABRIDGED version ;-) I think we can close this topic. Thanks all who posted for your help. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.