Ramon88 Posted August 21, 2009 Report Share Posted August 21, 2009 I've connected an external HP Storageworks RDX drive to one of our backup servers (Vista Ultimate 32 bit / SP2). This is to facilitate mandatory offline backups as required by a client (it's their database). The backup server runs Retrospect Multi Server 7.6.123 + Driver Update and Hot Fix 7.6.2.101. Within Windows we can write to the RDX device (drag & drop copy) with ± 24-25 MB/s flat. However when we use the drive as a backup target the speed takes a dive to ± 7 MB/s. This is just a regular backup without even encryption and compression yet. Btw, we use the drive in "Removable Disk" mode. We need to add about 150 GB daily to the backup, so with only 7 MB/s this takes about 6 hours, which is too much time. Questions: - Is this 'normal' Retrospect behaviour for RDX? - If not, any ideas how we can improve this? The device has a "Qualified" status: http://www.retrospect.com/supportupdates/technical/retrospect/detail/?id=9851 Seems like the Law of Perpetual Problems is at it again... This kind of -you know what- takes just sooo much time... :rollie: Ah well, I guess there are worse things. Quote Link to comment Share on other sites More sharing options...
Mayoff Posted August 21, 2009 Report Share Posted August 21, 2009 Please provide the exact performance data as displayed in the operations log for total file count, speed, etc. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 21, 2009 Author Report Share Posted August 21, 2009 Will do, you'll get it on Monday. The system is now running regular backups and I'm getting tired (10:25 PM here, it's been a long day). Did try a "Disk" type backup though. That was around 1000 MB/min before the regular backups kicked in and I had to stop it. I Googled a review that also complained about Retrospect speed of around 7MB/s: http://www.pcpro.co.uk/reviews/backup-devices/254836/hp-storageworks-rdx-removable-disk-backup-system "Real-world speeds were slightly slower, as copying a 2.52GB video clip returned read and write speeds of 25.8MB/sec and 20MB/sec. The RDX is compatible with any backup software that supports removable drives, and we tested with EMC Retrospect. Securing a 13.5GB test folder with nearly 15,000 files in it returned a much lower write speed of 7.5MB/sec." Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 22, 2009 Author Report Share Posted August 22, 2009 Okay slightly sooner than Monday, here goes: This one is using the RDX drive with a "Disk" type Backup Set. As you can see its speed is okay: + Recycle backup using RDX Backup test at 21-8-2009 23:19 (Execution unit 1) To Backup Set RDX Test... 21-8-2009 23:19:11: Recycle backup: The Backup Set was reset - 21-8-2009 23:19:11: Copying IIS Log Archief on iSCSI (X:) 21-8-2009 23:39:54: Snapshot stored, 6 KB 21-8-2009 23:39:55: Execution completed successfully Completed: 4 files, 24,3 GB Performance: 1200,1 MB/minute Duration: 00:20:43 (00:00:03 idle/loading/preparing) - 21-8-2009 23:39:55: Verifying RDX Test 21-8-2009 23:54:59: Execution completed successfully Completed: 4 files, 24,3 GB Performance: 1651,6 MB/minute Duration: 00:15:04 (00:00:02 idle/loading/preparing) And here is the same test run, with a "Removable Drive" type Backup Set. As you can see it is much slower: + Recycle backup using RDX Backup test at 21-8-2009 23:59 (Execution unit 1) To Backup Set RDX Test... 21-8-2009 23:59:28: Recycle backup: The Backup Set was reset - 21-8-2009 23:59:28: Copying IIS Log Archief on iSCSI (X:) 22-8-2009 0:49:12: Snapshot stored, 6 KB 22-8-2009 0:49:12: Execution completed successfully Completed: 4 files, 24,3 GB Performance: 499,0 MB/minute Duration: 00:49:43 (00:00:01 idle/loading/preparing) - 22-8-2009 0:49:12: Verifying RDX Test 22-8-2009 1:31:10: Execution completed successfully Completed: 4 files, 24,3 GB Performance: 591,2 MB/minute Duration: 00:41:58 Because the source is located on an iSCSI appliance I can actually monitor how Retrospect reads. When using the "Drive" type Backup Set we see enormous bursts of reading speeds (more than 100 MB/s) for seconds followed by periods of nothing for about the same length of time. This seems to be due to buffering. With the "Removable Drive" Backup Set as a target we see a much slower read in a flat line at about 8-10 MB/s. Verification was "Media verification" and no encryption or compression were used. During Media verification, when monitoring Vista resources, the difference was the RDX drive reading speeds of ± 24 MB/s versus ± 8 MB/s Anyway, we want to use the device as a Removable Drive Backup Set. So performance in that intended mode is subpar to say the least... Quote Link to comment Share on other sites More sharing options...
Mayoff Posted August 24, 2009 Report Share Posted August 24, 2009 1200 and 1600 MB/Minute seem like good speeds to me. That is similar to other devices when you consider that a possible bottleneck could be the read speed of the source data leaving the disk and not necessarily the write speed. Maybe other users of this hardware can share real life numbers too. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 24, 2009 Author Report Share Posted August 24, 2009 Hi robin, It's not the 1200 and 1600 MB/min that are the problem. In the second example is the performance in Retrospect's "Removable Drive" mode. In that case we only see 500 and 600 MB/min, which is very different. So: - "Disk" Backup Set = fast (as fast as a regular copy in the OS) - "Removable" Backup Set = slow What is the reason the preferred "Removable" Backup Set is so much slower? It's the same device and system. Why does Retrospect use a completely different pattern to read the source? I seriously think this is a Retrospect BUG. Nothing less... Question for a workaround: What would we 'miss' when using the Backup Sets in "Disk" mode instead of "Removable" mode? Take note the Backup Set needs to be able to expand beyond the RDX's cartridge capacity, so span multiple disks. Quote Link to comment Share on other sites More sharing options...
Mayoff Posted August 24, 2009 Report Share Posted August 24, 2009 What file system is the RDX using? Make sure it is NTFS Removable Disk sets write to a different size data file, which could be a factor You really don't miss much by using the Disk Sets. The media request dialog boxes are a little different. For years the removable disk set had been removed from the product. We added it again for the REV autoloader devices. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 24, 2009 Author Report Share Posted August 24, 2009 What file system is the RDX using? Make sure it is NTFS Retrospect itself formats the drive (It behaves like it's a tape drive more or less I suppose). When opening the drive in Windows itself, data on the RDX (in "Removable" mode) is written in multiple 'blocks' of 1,99 GB (2.146.435.072 bytes), the last numbered block is smaller. All 'blocks' are placed in the root and are named "Retrospect Data", Retrospect Data 002, Retrospect Data 003, etc. According to Windows the drive has an NTFS format. You really don't miss much by using the Disk Sets. The media request dialog boxes are a little different. For years the removable disk set had been removed from the product. We added it again for the REV autoloader devices. I'll give it a try (Disk 'mode'), we have a 21 day period arranged to test the unit. We wanted something that would behave exactly like tape, but in a disk form for speed considerations (we handle quite large amounts of data daily). So we are a little disappointed as it doesn't match our expectations/plans. Still, the behaviour we're seeing is strange/unexpected to say the least. I would suggest handling this as an official issue/bug. Thanks for your feedback. It's appreciated. Quote Link to comment Share on other sites More sharing options...
Mayoff Posted August 24, 2009 Report Share Posted August 24, 2009 The disk maintains some type of Windows file system. What file systems is being used? It is displayed in Windows disk properties. Retrospect never formats the disk, it just erases them using a simple erase command. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 24, 2009 Author Report Share Posted August 24, 2009 (edited) It says NTFS. Don't know if this was standard, as I formatted it before using it for any of the tests. If I want to reformat it, it wants to do it with a clustersize of 4096 bytes. Probably what it is now. Capacity of this 320GB cartridge is 298GB in 'real life'. B.t.w. I just found out I can only format it NTFS and exFAT. Nothing else is possible. Note: Drive firmware is 2040 Cartridge firmware is 13.01A13 HP RDX Utility is version 2.20 / DLL version is 2.28 Edited August 24, 2009 by Guest Added Note. Quote Link to comment Share on other sites More sharing options...
Mayoff Posted August 24, 2009 Report Share Posted August 24, 2009 ntfs is best Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 26, 2009 Author Report Share Posted August 26, 2009 Just FYI, Inserted a brand new cartridge. They are already factory formatted as NTFS (at least this HP version is). Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 26, 2009 Author Report Share Posted August 26, 2009 Okay, running as "Disk" Backup Set... I purposely fed it a transfer of a Backup Set exceeding the drive's storage capacity of 320GB, to test what handling was required. It filled up the cartridge with about 18 MB/sec (1100MB/min). This took about 4,5 hours. Then it stopped transferring and started media verification. This took about (I didn't pay exact attention to the unit) 3,5 hours. Then it requested new media, but didn't eject the first cartridge by itself (the script was not ended, so that setting would not effect this anyway). Then something strange; I needed to press the eject button twice before it ejected (± 30 seconds between presses)... I inserted a new cartridge and pointed Retrospect to it. I feel this is somewhat of a dangerous thing, as I could have pointed it to a regular drive by mistake and it might have erased that... It correctly named the cartridge and continued the backup transfer script. Still running at the moment. Robin, what would happen if I would have pointed the media requester to the wrong fixed drive? Of course I didn't try this, but it has me a bit worried as this would be something one easily can do by mistake... Quote Link to comment Share on other sites More sharing options...
Mayoff Posted August 26, 2009 Report Share Posted August 26, 2009 Removable and local disks are treated differently. We will not erase a local disk with data, we will write to a new folder. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 26, 2009 Author Report Share Posted August 26, 2009 :thanks: Okay, thanks for the clarification. This does indeed make sense, glad the engineers thought of that one! Would have loved Retrospect to be able to eject the cartridge when it was full though. But that's actually not really important. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 27, 2009 Author Report Share Posted August 27, 2009 Okay, the script says to eject media from the drive, but at the end this doesn't happen. Besides that, I get the following: + Executing RDX Transfer Snapshot test at 26-8-2009 8:44 (Execution unit 1) To Backup Set RDX Test... - 26-8-2009 8:44:58: Transferring from TC SQL DB Backup Set Backup (26-8-2009 1:30) - 26-8-2009 13:21:44: Verifying RDX Test - 26-8-2009 16:51:23: Transferring from TC SQL DB Backup Set 26-8-2009 20:11:40: Execution completed successfully Completed: 2219 files, 474,9 GB Performance: 1027,6 MB/minute Duration: 08:16:07 (00:22:58 idle/loading/preparing) - 26-8-2009 20:11:40: Transferring from TC SQL DB Backup Set Backup (25-8-2009 1:30) 26-8-2009 20:30:58: Execution completed successfully Completed: 93 files, 19,8 GB Performance: 1067,7 MB/minute Duration: 00:19:17 (00:00:20 idle/loading/preparing) - 26-8-2009 20:30:58: Transferring from TC SQL DB Backup Set Backup (24-8-2009 1:30) No files need to be transferred 26-8-2009 20:31:00: Execution completed successfully Duration: 00:00:02 - 26-8-2009 20:31:00: Transferring from TC SQL DB Backup Set Backup (23-8-2009 1:30) No files need to be transferred 26-8-2009 20:31:02: Execution completed successfully Duration: 00:00:01 - 26-8-2009 20:31:02: Transferring from TC SQL DB Backup Set Backup (22-8-2009 1:30) No files need to be transferred 26-8-2009 20:31:04: Execution completed successfully Duration: 00:00:01 - 26-8-2009 20:31:04: Transferring from TC SQL DB Backup Set Backup (21-8-2009 1:30) No files need to be transferred 26-8-2009 20:31:06: Execution completed successfully Duration: 00:00:01 - 26-8-2009 20:31:06: Transferring from TC SQL DB Backup Set Backup (20-8-2009 1:30) No files need to be transferred 26-8-2009 20:31:08: Execution completed successfully Duration: 00:00:01 - 26-8-2009 20:31:08: Transferring from TC SQL DB Backup Set Backup (19-8-2009 1:30) No files need to be transferred 26-8-2009 20:31:11: Execution completed successfully Duration: 00:00:01 - 26-8-2009 20:31:11: Verifying RDX Test 26-8-2009 22:38:11: Execution completed successfully Remaining: 1 files, 26,2 GB Completed: 2311 files, 464,7 GB Performance: 1504,4 MB/minute Duration: 05:17:35 (00:01:16 idle/loading/preparing) I get the eight "in between" reports in the log as they probably reflect the eight snapshots from the source backup set. However I don't get why it ends with "Remaining: 1 files, 26,2 GB". In the activity log > history it is graphically represented with the bar not indicating total completion and even claiming " Remaining: 00:06:09" and reporting 0,0 errors/warnings in the listing... The script used is a default "Transfer Snapshots" script configured with "Transfer all active Snapshots". The rest is default. Quote Link to comment Share on other sites More sharing options...
Mayoff Posted August 27, 2009 Report Share Posted August 27, 2009 The remaining is under he verify section. That seem to indicate that you have one very large file that Retrospect didn't attempt to verify and I am not sure why. You could try a tools>verify operation and see if the same issue is reported. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 27, 2009 Author Report Share Posted August 27, 2009 Will do that within a couple of days as I erased this set for other tests. Large files is easily possible, this script's source is a SQL database backup dump folder filled with BAK files and Transaction logs. Is there a limit when Retrospect can't read a certain file size anymore? Quote Link to comment Share on other sites More sharing options...
Mayoff Posted August 27, 2009 Report Share Posted August 27, 2009 Retrospect should be able to handle any file size that the file system itself can handle. I have worked with multi-GB files before. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 30, 2009 Author Report Share Posted August 30, 2009 I've found another problem. Actually I've reported it earier, but I now really can confirm it. Eject between cartridge swapping (during a backup larger than a single cartridge can hold) takes very long (like five minutes). Pressing Retrospect's Eject button doesn't do anything. Ejecting the media from Windows doesn't work (Vista says the cartridge is being used by another program). The only thing that works is to press the eject button. The drive's activity LED starts to blink and that remains until the drive ejects the cartridge. This however takes several minutes. As this is not a (rewinding) tape drive, it shouldn't act like one in this respect. I dunno... I'm starting to think I should send this unit back and get an LTO4 drive... :rollie: Seems like Mauricev has shown me the light regarding LTO... Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted September 1, 2009 Author Report Share Posted September 1, 2009 To conclude my non-exhausitive 'review' of using RDX with Retrospect, here is a summary of my findings: Summary - Using the unit as a Removable Backup Set, works, but extremely slow performance (7 MB/sec) is the result. - Using the unit as a Disk Backup Set, works faster (Write 18-20 MB/sec - Read 24-25 MB/sec - Stand alone Verify ± 27 MB/sec). However there are now two 'problems': - When doing a backup that requires a new cartridge during the backup, the media eject button of Retrospect doesn't work, and eject from Windows is blocked by the OS. To eject the media one has to press the eject button on the unit itself and it will take between 1 - 5 minutes (observed a couple of times - no exceptions, this happened always) for the unit to eject the cartridge. - During verify of a backup operation that included a cartridge swap, one encounters a strange behaviour in the log file, where the execution completes successfully however there is always mentioned that there is one file remaining to be verified. System used: - Intel based 3GHz Dual Core CPU with 4GB RAM (3.5GB addressable). - Vista (32 bits version) Ultimate. - Backup source is iSCSI based storage (peak at 115MB/sec). - Retrospect Multi Server 7.6.123 + Driver Update and Hot Fix 7.6.2.101. - HP Storageworks RDX Drive (USB-2 based) firmware is 2040. - Cartridge firmware is 13.01A13. - Cartridge size 320GB (300GB NTFS formatted - two used). - HP RDX Utility is version 2.20 / DLL version is 2.28. - Backup type used was a "Transfer Snapshots" (Transfer all active Snapshots), default settings with Media verification ON. Conclusion (based on our setup) Retrospect isn't 100% Qualified for this product, even if their website states it is. See: http://www.retrospect.com/supportupdates/technical/retrospect/detail/?id=9851 The problems, I reported above, have been consistently detected during several test runs. I deem them enough of an unknown (the verify problem) and unsatisfactory (media swapping and removable performance versus disk performance) factor that I have decided to return the unit. Of course, and like always, your mileage may vary... Quote Link to comment Share on other sites More sharing options...
Mayoff Posted September 1, 2009 Report Share Posted September 1, 2009 I wrote a bug for your concerns. It is bug 23234 Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted September 1, 2009 Author Report Share Posted September 1, 2009 That's good to know. My next stop will be LTO4 though, I'm 'converted' already. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted September 2, 2009 Author Report Share Posted September 2, 2009 Robin, while preparing the cartridges for return shipment (secure erase) and subsequential formatting. I noticed the unit had the same eject problem as within Retrospect. Right after formating, we 'opened' the cartridge in Windows explorer to check its content (it was empty of course). We then pressed the eject button on the unit (while leaving the window open). The cartridge LED started to blink (never stopped), but no eject was produced. A second button press a couple of minutes later didn't seem to help either. However when we closed the still open window, it ejected. Maybe it was a coincidence it happend at the same time, maybe it was ejected by the command of the first button press. We are currently processing the second cartridge and will report back tomorrow if this is repeated. Retrospect was running in the background though, however it was not using the RDX drive. So the eject problem might not be a Retrospect bug after all. It's a bit weird though, as this was reproducible within Retrospect and we didn't have this problem before. But we did format the cartridges now and then, so it might be format-related. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted September 3, 2009 Author Report Share Posted September 3, 2009 Okay,we've figured out the eject problem. It still might be a problem FOR Retrospect, but it can also be triggered from within Vista. For the below mentioned we didn't have Retrospect running. Just insert a cartridge. Let it spin up and eject it by pressing the unit's eject button. It will immediately eject. Now do the same, but open the cartridge's contents in Windows. Leave the window open and press the eject button on the unit. It will blink its LED's, but it will not eject. Only after closing the open window will it eject (after a couple of seconds). Presumably Retrospect, when asking for new media, keeps the drive 'occupied' in the same fashion. Resulting in long eject times. Hope this helps the bug report. Besides that there are the other problems I mentioned, they don't seem related. But at least now you have a little bit more information about the eject problem. The unit is now packaged and will be picked up by the courier tomorrow. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.