Advice on remote rSync backup storage

[Richy_Boy]

Hi all;

 

I have changed the way in which our backups take place and part of this project is to grab a copy of the data and store offsite.

 

I have RS 7.6 Multi Server with the value addon pack and have a pretty standard Windows 2000 Domain with Exchange, SQL server, about 50 desktops and few random Windows, Linux servers.

 

I'm now using a Netgear ReadyNAS box for my backup repository, all backups are copied to this device into grouped shares/backup sets (i.e. DCs, file servers, exchange, laptops, desktop etc.

 

Currently, in the evenings these backups are duplicated to SATA a 1TB drive for offsite storage using the 'copy latest snapshot' function.

 

This seems to work fine, but due to the manual intervention with swapping SATA drives daily, I'm looking at getting automated offsite backups working for DR purposes.

 

I'm planning on using a second ReadyNAS device, which will initially be placed in my house (like you do in a SMB!). These ReadyNAS devices have an rSync-over-ssh function which is extrememly good at copying data from one device to the other in a block-level, compressed, encrypted manner. Once this device is up and running and tested, I plan to duplicate the backups to this remote device, instead of office-based SATA drives.

 

What I'm not 100% sure of yet is how well the RS storage mechanism will work with rSync as it appears to write its new data to ~614MB chunks.

 

Usually with rSync it's recommended that you don't compress the data before copying as it stops the block-level fucntion from being able to find where in a file the data was updated, so it'll copy the whole file again.

 

i.e. if you have a large Word file and make changes, rSync will find where the new data is in the file and only copy the changed blocks, grafting new and old together on the target machine.

 

With RS, I understand that new snapshots (i.e. changed data since last backup) is saved a NEW RDB file and not appended to the old one? Can anyone confirm this?

 

Of course, if this IS how RS data storage works, then RS will remove some of the functionality which makes rSync so efficient, as it will copy duplicate data, rather than block due to RS recopying the whole changed file.

 

Anyway, sorry this is a bit long winded, but I was interested in your thoughts whether you think compressing the RS data will have a good or bad effect in this instance. Also whether you see a more efficient way of doing this...?

 

Right now, I think RS will make rSync way less efficient by using RDB files to store the data, yet compression will have no detrimental effect in this instance.

 

In theory, I'd be better off if RS could write the backup data in the original file formats, so rSync could fully utilise it's block-level functionality, I think..

 

Rich

Edited November 25, 2009 by Guest

[phlampe]

Hi Richy !

 

I've been using such a setup for over 2 years now, with a Synology NAS I have in a remote location. I wrote a script that scans continuously my rdb files and sends them over 2 ADSL lines whenever they change, using rsync over ssh as you want to do.

 

It works very well, and as you said, Retrospect (I'm using RS 7.6 on a Windows 2003 SBS server) fills up files up to 600 Mb. The only times they get changed by Retrospect, is whenever you do an optimization of your backup set, and usually the old rdb files get cropped down, and rsync resends them.

 

I see a real benefit to rsync's optimizations only whenever the link goes down, and that's when you're glad you don't have to resend those 595 Mb (worse case :-)) of data over the Internet. That's also a good reason to keep those files "small" and 600 Mb is a good compromise.

 

So go ahead with rsync,

Paul-Henri

 

[Richy_Boy]

Great phlampe, just what I wanted to hear. Just brought the NAS box home with me tonight for a test run - all has gone to plan thus far, but I'll find out tomorrow how the compressions/speeds have worked out.

 

INCREMENTAL Backup started. Tue Dec 8 18:16:49 WET 2009

 

Job: 001

Protocol: rsync

Source: 8*.**.50.**:/DesktopBU

Destination: [DesktopBU]/

 

 

Warning: Permanently added '8*.**.50.**' (RSA) to the list of known hosts.

receiving incremental file list

Retrospect/NAS Desktop/1-NAS Desktop/

Retrospect/NAS Desktop/1-NAS Desktop/AA000348.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000349.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000350.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000351.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000352.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000353.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000354.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000355.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000356.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000357.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000358.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000359.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000360.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000361.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000362.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000363.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000364.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000365.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000366.rdb

Retrospect/NAS Desktop/1-NAS Desktop/AA000367.rdb

 

sent 397 bytes received 349620611 bytes 225780.44 bytes/sec

total size is 55413541904 speedup is 158.50

 

Backup finished. Tue Dec 8 18:42:37 WET 2009

 

Not too bad over a 2mb link...

 

Rich