Best Practice question

[bbutle01]

Ok, we have a 4TB raid on a Drobo Pro. It's expandable to 16TB.

 

Is it better for retrospect to have 2 backup sets on this, so it can perform 2 different operations at the same time (say, groom one, backup another) or is that pointless.

 

I have had a backup set become corrupt once before and we lost data, so I'm thinking about having my backups alternate between these two sets... Although that looses a lot of de-dupe space savings...

 

Yes, I know it's better to have 2 drobo, and 2 backup sets.

[Lennart_T]

That should work fine. We have two backup sets on one volume with no problems.

[Ramon88]

In my experience Drobo Pro's performance is very good until you let it do some concurrency (more than one writing and/or reading thread). Also make sure you run the latest firmware and Drobo Dashboard.

 

We store a lot of backup sets on a Drobo Pro. Make sure you set it up as a single (virtual) 16TB volume and format it NTFS with the default block size. Use a bit of planning to schedule your scripts so they run in succession rather than parallel. That way it runs much smoother and the net result is faster as well.

 

Also make sure you have quality cables and very good switches if you use it in a network topology rather than using a dedicated point-to-point ethernet cable. What also helps is to use a server grade network card, unless you use a server mainboard which should already provide for this. We have found performance easier to monitor when your server is attached to the unit with its own network card even when sharing the same switch with the rest of the network as this gives you the opportunity to monitor iSCSI performance much better. Giving the unit it's own subnet seems to slightly improve performance as well.

 

Don't worry about space too much. Just add more/switch to larger capacity drives. Space shouldn't be a problem with this device and SATA disks are too cheap nowadays. Using disks form Drobo's approved list will keep you on the save side. Special RAID disks are not needed. I can recommend Western Digital WD15EADS 1.5TB and WD20EADS 2TB disks. I wouldn't mix older and newer disks too much altough that is a nice feature of BeyondRAID, using same performing disks should be more efficient.

 

In reality I'm seeing performance for Retrospect equals and even exceeds a local RAID-10 storage system. But we use the unit only for Retrospect storage.

 

Oh, and switch of defragmentation for the unit. It's been said that the unit does that itself. The process will only slow down the unit when it's defragmented. Same goes for indexing, it's not needed for Retrospect Backup Set storage.

 

Make sure your backup server has plenty of free RAM and a fast multi-core CPU. Retrospect often is the real bottleneck when working with this unit. Windows 2008 and Vista are the best Windows OS'es you can use with the unit, as they have more efficient network technology.

 

Adding a UPS to protect the Drobo Pro would be a smart thing to do as well.

 

Hope this helps you.

[sbeliever]

hi ramon88,

 

wow...that is a major drag about the concurrency, though that is what i am finding here too. i have 3 running concurrently pretty easily but any more than that starts to get VERY iffy. i did just update the drobo firmware this morning and i think that has made some difference as well.

 

i was just curious if you have anything to add on this topic since your last post. and thanks for it...it really helped out figuring out where my problems were.

 

 

[Ramon88]

Not really,

 

We still really like the unit as storage target for Retrospect. We were planning to upgrade to the 1.1.5 firmware (currently using 1.1.3) but the system engineer has been busy with other stuff, so it's not high on our priority list.

 

Usually we run between 1 to 3 execution units at the same time. More is not really efficient. But keep in mind we use encryption for all our backups, so that has something to do with it too.

 

This Retrospect server (we use a couple) is quite powerful with 24GB RAM and two quad core Xeon E5520's (That means 16 HT 'cores' for the OS). It also has four server grade network ports that we all use.

 

We were thinking of taking a look at the new Drobo Elite. But is is quite expensive compared to the Pro (about 2,5 times as much - but that is cheap for a SAN). It has two network interfaces instead of one and can provide multiple LUNs instead of just one (apparently 16 according to the specs). Quite possible it has superior concurrency performance. But so far we have not been able to obtain a test unit.

 

In the end I think the Elite is too expensive for usage as a Retrospect storage target. Retrospect in its current incarnation is not really taxing the server hardware, so in al honesty I don't think you'll improve much with such an investment when the bottleneck is Retrospect itself.

 

What IS interesting is the fact that SAN grade storage will become very cheap in the foreseeable future. All this because of the magic of consumer grade equipment evolution starting to touch the lower spectrum of enterprise grade equipment. The same has happened with, for example, 3D video hardware and video equipment. Some consumer video camera's now exceed pro specs from a couple of years ago. The magic of numbers can do that.

 

Still an afterthought...

 

When the DroboPro fills up, we see performance degradation as well. So to avoid that we made sure to add as much capacity HDD's was needed to keep it between 50-70% of our storage needs.

 

And I didn't get it if using 1.1.5. firmware improved or degraded concurrency performance for you.

Edited February 11, 2010 by Guest
Added afterthought note.

[jwright]

Ramon88:

 

I hadn't considered a drobo unit as an storage option for Retrospect. Just running the numbers... It looks like from their capacity calculator, if you max out the Drobo Pro with 8 of the WD 2 TB drives and enable Dual Disk Redundancy, that gives you 10.89 TB of disk space. If I understand your estimate of performance, that means I would want to keep my backups around ~6-7.5 TB (50-70% of capacity) for best performance. Is that right? I was just trying to do the cost analysis for best performance. It looks like I could put this together for about $2,600.

 

Our storage needs are closer to 6 TB, but not growing, so this could be a good option for us.

 

Thanks,

 

Jeff

[Ramon88]

We do not have dual disc redundancy activated. This is because, when we initially set up the unit, some people on the Drobo forum reported problems when having that activated. Can't really confirm that though. We figured we didn't need double protection for a backup that is, in our case, already a mirror for another backup server. And the source data was on a double RAID already. But that's just our case... Did I mention we also transfer data from the DroboPro to an LTO4 drive and keep all that in three geographically different locations? *L*

 

We just keep a spare disk next to the unit and thus can replace a disk very quickly. This yields an extra 2TB capacity.

 

To save money you could put WD15EADS 1.5TB HDD's in the unit instead of the 2TB version. These are quite a lot cheaper and have the same performance and specs (apart from the capacity). If you need at least 6TB of real life storage, get eight of these. Put seven in the unit and keep one as a spare. You now have one slot free for future expansion. You might want to test that spare on a testing rig beforehand. I've had DOA's in the past and it's better to know that when buying.

 

Seven 1.5 TB drives yields ±8.2 TB useable storage. Times ±70% means you'll almost have that 6TB of 'fast' storage. It's not really crawling at 70%. It's just quicker when its about empty. We never did extensive performance testing to get this clear. It's just what we see when using Retrospect. It gets slower when it fills up. But it is no problem for us.

 

I heard 3TB drives will be available later this year. Not sure if DroboPro will support these. That would be quite interesting. But probably (relatively) expensive.

 

On a sidenote, there are quicker options out there, and even more advanced ones, like some Thecus models. What WE like about Drobo is its simplicity. Their engineers only have to concentrate on it being an iSCSI target. Nothing else (The Thecus and similar are complete media servers with a lot of 'goodies'). It's probably a product Steve Jobs could have created: "It just works". However, if you need performance, there are better products on the market, but they are also more sophisticated and more expensive... But sometimes, depending on what you actually want to do with it, they may be a better choice. For US the DroboPro as a Retrospect storage target works like a charm.

Edited February 11, 2010 by Guest
Added sidenote.

[sbeliever]

ramon,

 

fantastic; thanks for your input and info. it has just been really difficult to diagnose where our breakdown was occurring and your info was helpful.

 

we have a new quad core xeon 2.4ghz dell poweredge w/8gb RAM, win 2008 x64, two server grade network cards and two drobo pro's (though using just one currently). we were getting spontaneous reboots with nary a clue about what the culprit was (nothing in the windows logs, retro logs, etc). i did go back to retro 7.6 and it seems much more stable at this point.

 

i am wondering if we could get more executions working if we assigned executions per specific drobo unit (3x to each), which might help our throughput. haven't gotten that far yet, as i still think we have a bad RAM scenario preventing us from using all 8GB (but the dell guy should be here monday to take care of that).

 

you didn't mention which version of retro you are using. we have used 7.6 on our previous system w/almost never an issue. due to some of the posts on the forum i decided to run it instead of 7.7 (which i had originally been testing). thoughts?

 

thanks,s

Edited February 11, 2010 by Guest

[utig]

a word of caution: when you rebuild a catalog, retrospect 7.6 and 7.7 scan every file on a disk,

not just the ones in the particular backup set. If you have a large disk with many backup sets, this can be quite annoyingly slow.

[Ramon88]

you didn't mention which version of retro you are using. we have used 7.6 on our previous system w/almost never an issue. due to some of the posts on the forum i decided to run it instead of 7.7 (which i had originally been testing). thoughts?

We are still running our boxes with 7.6.123 Multi Server. Stability is important for us and we think there are still a couple of bugs in 7.7 to be ironed out by EMC (which they will eventually do). The only reason WE would have for switching to 7.7 is better client side Windows 7 compatibility. 7.6.123 does have a couple of issues with Windows 7 clients, but we can still live with that as the alternative of switching to 7.7, at this time, probably will give us more problems.

 

Keep us posted if that Dell engineer makes a difference.

[Ramon88]

It could be a coincidence, but after updating the DroboPro with the 1.1.5 firmware our backups took a lot more time. We noticed by monitoring the data flow for the iSCSI NIC there were severe drops in the link.

 

So we had a look in the Event Viewer and saw a lot of Errors with iScsiPrt. Now, you will get those once in a while and that's not really a problem. But these were a lot!

 

Our solution was to disable Receive Side Scaling (RSS) on the NIC providing the server with the iSCSI connection. It so happens a lot of people using iSCSI devices are having this problem and it is not limited to DroboPro hardware. The iSCSI protocol seems to tax all network hardware and RSS seems to have an adverse result.

 

Maybe this is useful information for people having trouble with their iSCSI devices.

Edited February 18, 2010 by Guest

[sbeliever]

apparently i do have a defective dell server, but at least the replacement is on the way. ugh...what a waste of time...

 

but, that being said, i am also going to stay on 7.6.123 for the time being. i was also (finally) able to get both drobo's to show up correctly via iSCSI (though beware there is a defect in the dashboard software wherein they won't both show up using iSCSI; they will w/usb and FW). i have the latest 1.1.5 firmware on teh drobo and haven't noticed a slowdown but then i haven't had it up long enough to have a good baseline. i did, however, disable RSS on the interfaces as that does sound like a good solution.

 

btw, what kind of performance are you seeing in retrospect in your setup? i am curious to try to gauge where people are w/their particular systems (understanding the enormous differences).

 

thanks,

sj

[Ramon88]

Well, at least you know what was wrong!

 

We had two system crashes in two days after we updated the network drivers. We rolled them back yesterday evening and reconfigured the NIC's to their old settings. Now we see the problems again. But RSS is switched on (default setting after driver up/downgrade). We'll test for a couple of days if the system is stable again (it did do all backups correctly last night) and if so, switch RSS off.

 

As for the problem of it not showing up. It does that, as far as I know, only when setting up the first time. It's easily remedied by the 'USB trick' though.

 

Performance is highly 'erratic' It depends on the type of data we backup. But we also use data encryption on all our backups, so that slows things down. There are two switches between the server and DroboPro. There is approximately 70 meters of Cat6 S/FTP cable between the server and DroboPro and it is actually in the next building for security purposes. To give you a more reliable estimate, a normal file copy is about 65MB/sec (but we have seen >80MB/sec). But performance in Retrospect can be much less. For regular machines we usually get ± 1 GB per minute (including media verify). For developer machines and servers it can be less, depending on the number of (small) files and the matching time THAT takes (a real drawback of Retrospect imho). But we sometimes also see figures of more than 2 GB per minute.

 

I don't have exact numbers, but if I would need to estimate it, running two execution units simultaneously maybe has a 25% performance hit. Three maybe another 25% based on the figure of two, and so on. It depends on a lot of factors. One of these is Retrospect itself. I think it could be made a lot faster by using more resources. But that is another discussion altogether.

 

DroboPro is what it is. It probably could be created with faster components, but for what it is and costs it is a nice deal. But I feel Retrospect itself is the bottleneck at this time. For now we can live with that.