Jump to content


Photo

Terrible RDX Performance


  • Please log in to reply
31 replies to this topic

#1 Ramon88

Ramon88

    Occasional forum poster

  • Members
  • 425 posts
  • LocationThe Netherlands

Posted 21 August 2009 - 08:07 PM

I've connected an external HP Storageworks RDX drive to one of our backup servers (Vista Ultimate 32 bit / SP2). This is to facilitate mandatory offline backups as required by a client (it's their database).

The backup server runs Retrospect Multi Server 7.6.123 + Driver Update and Hot Fix 7.6.2.101.

Within Windows we can write to the RDX device (drag & drop copy) with ± 24-25 MB/s flat.

However when we use the drive as a backup target the speed takes a dive to ± 7 MB/s. This is just a regular backup without even encryption and compression yet.

Btw, we use the drive in "Removable Disk" mode.

We need to add about 150 GB daily to the backup, so with only 7 MB/s this takes about 6 hours, which is too much time.

Questions:

- Is this 'normal' Retrospect behaviour for RDX?
- If not, any ideas how we can improve this?

The device has a "Qualified" status: http://www.retrospec...detail/?id=9851

Seems like the Law of Perpetual Problems is at it again... This kind of -you know what- takes just sooo much time... :rollie: Ah well, I guess there are worse things.

#2 Mayoff

Mayoff

    Retrospect Backup Commando

  • Administrators
  • 11,997 posts

Posted 21 August 2009 - 08:21 PM

Please provide the exact performance data as displayed in the operations log for total file count, speed, etc.
Robin Mayoff
Director, Retrospect Support Services
Retrospect Employee #63. Since 1994

#3 Ramon88

Ramon88

    Occasional forum poster

  • Members
  • 425 posts
  • LocationThe Netherlands

Posted 21 August 2009 - 08:29 PM

Will do, you'll get it on Monday. The system is now running regular backups and I'm getting tired (10:25 PM here, it's been a long day).

Did try a "Disk" type backup though. That was around 1000 MB/min before the regular backups kicked in and I had to stop it.

I Googled a review that also complained about Retrospect speed of around 7MB/s: http://www.pcpro.co....k-backup-system

"Real-world speeds were slightly slower, as copying a 2.52GB video clip returned read and write speeds of 25.8MB/sec and 20MB/sec. The RDX is compatible with any backup software that supports removable drives, and we tested with EMC Retrospect. Securing a 13.5GB test folder with nearly 15,000 files in it returned a much lower write speed of 7.5MB/sec."

#4 Ramon88

Ramon88

    Occasional forum poster

  • Members
  • 425 posts
  • LocationThe Netherlands

Posted 22 August 2009 - 05:17 AM

Okay slightly sooner than Monday, here goes:

This one is using the RDX drive with a "Disk" type Backup Set. As you can see its speed is okay:

+	Recycle backup using RDX Backup test at 21-8-2009 23:19 (Execution unit 1)
	To Backup Set RDX Test...
	21-8-2009 23:19:11: Recycle backup: The Backup Set was reset

-	21-8-2009 23:19:11: Copying IIS Log Archief on iSCSI (X:)
	21-8-2009 23:39:54: Snapshot stored, 6 KB
	21-8-2009 23:39:55: Execution completed successfully
		Completed: 4 files, 24,3 GB
		Performance: 1200,1 MB/minute
		Duration: 00:20:43 (00:00:03 idle/loading/preparing)

-	21-8-2009 23:39:55: Verifying RDX Test
	21-8-2009 23:54:59: Execution completed successfully
		Completed: 4 files, 24,3 GB
		Performance: 1651,6 MB/minute
		Duration: 00:15:04 (00:00:02 idle/loading/preparing)
And here is the same test run, with a "Removable Drive" type Backup Set. As you can see it is much slower:

+	Recycle backup using RDX Backup test at 21-8-2009 23:59 (Execution unit 1)
	To Backup Set RDX Test...
	21-8-2009 23:59:28: Recycle backup: The Backup Set was reset

-	21-8-2009 23:59:28: Copying IIS Log Archief on iSCSI (X:)
	22-8-2009 0:49:12: Snapshot stored, 6 KB
	22-8-2009 0:49:12: Execution completed successfully
		Completed: 4 files, 24,3 GB
		Performance: 499,0 MB/minute
		Duration: 00:49:43 (00:00:01 idle/loading/preparing)

-	22-8-2009 0:49:12: Verifying RDX Test
	22-8-2009 1:31:10: Execution completed successfully
		Completed: 4 files, 24,3 GB
		Performance: 591,2 MB/minute
		Duration: 00:41:58
Because the source is located on an iSCSI appliance I can actually monitor how Retrospect reads. When using the "Drive" type Backup Set we see enormous bursts of reading speeds (more than 100 MB/s) for seconds followed by periods of nothing for about the same length of time. This seems to be due to buffering. With the "Removable Drive" Backup Set as a target we see a much slower read in a flat line at about 8-10 MB/s.

Verification was "Media verification" and no encryption or compression were used.

During Media verification, when monitoring Vista resources, the difference was the RDX drive reading speeds of ± 24 MB/s versus ± 8 MB/s

Anyway, we want to use the device as a Removable Drive Backup Set. So performance in that intended mode is subpar to say the least...

#5 Mayoff

Mayoff

    Retrospect Backup Commando

  • Administrators
  • 11,997 posts

Posted 24 August 2009 - 12:56 PM

1200 and 1600 MB/Minute seem like good speeds to me. That is similar to other devices when you consider that a possible bottleneck could be the read speed of the source data leaving the disk and not necessarily the write speed.

Maybe other users of this hardware can share real life numbers too.
Robin Mayoff
Director, Retrospect Support Services
Retrospect Employee #63. Since 1994

#6 Ramon88

Ramon88

    Occasional forum poster

  • Members
  • 425 posts
  • LocationThe Netherlands

Posted 24 August 2009 - 03:41 PM

Hi robin,

It's not the 1200 and 1600 MB/min that are the problem. In the second example is the performance in Retrospect's "Removable Drive" mode. In that case we only see 500 and 600 MB/min, which is very different.

So:

- "Disk" Backup Set = fast (as fast as a regular copy in the OS)
- "Removable" Backup Set = slow

What is the reason the preferred "Removable" Backup Set is so much slower? It's the same device and system. Why does Retrospect use a completely different pattern to read the source?

I seriously think this is a Retrospect BUG. Nothing less...

Question for a workaround: What would we 'miss' when using the Backup Sets in "Disk" mode instead of "Removable" mode? Take note the Backup Set needs to be able to expand beyond the RDX's cartridge capacity, so span multiple disks.

#7 Mayoff

Mayoff

    Retrospect Backup Commando

  • Administrators
  • 11,997 posts

Posted 24 August 2009 - 04:57 PM

What file system is the RDX using? Make sure it is NTFS

Removable Disk sets write to a different size data file, which could be a factor

You really don't miss much by using the Disk Sets. The media request dialog boxes are a little different. For years the removable disk set had been removed from the product. We added it again for the REV autoloader devices.
Robin Mayoff
Director, Retrospect Support Services
Retrospect Employee #63. Since 1994

#8 Ramon88

Ramon88

    Occasional forum poster

  • Members
  • 425 posts
  • LocationThe Netherlands

Posted 24 August 2009 - 05:18 PM

What file system is the RDX using? Make sure it is NTFS


Retrospect itself formats the drive (It behaves like it's a tape drive more or less I suppose).

When opening the drive in Windows itself, data on the RDX (in "Removable" mode) is written in multiple 'blocks' of 1,99 GB (2.146.435.072 bytes), the last numbered block is smaller. All 'blocks' are placed in the root and are named "Retrospect Data", Retrospect Data 002, Retrospect Data 003, etc. According to Windows the drive has an NTFS format.

You really don't miss much by using the Disk Sets. The media request dialog boxes are a little different. For years the removable disk set had been removed from the product. We added it again for the REV autoloader devices.


I'll give it a try (Disk 'mode'), we have a 21 day period arranged to test the unit. We wanted something that would behave exactly like tape, but in a disk form for speed considerations (we handle quite large amounts of data daily). So we are a little disappointed as it doesn't match our expectations/plans.

Still, the behaviour we're seeing is strange/unexpected to say the least. I would suggest handling this as an official issue/bug.

Thanks for your feedback. It's appreciated.

#9 Mayoff

Mayoff

    Retrospect Backup Commando

  • Administrators
  • 11,997 posts

Posted 24 August 2009 - 05:20 PM

The disk maintains some type of Windows file system. What file systems is being used? It is displayed in Windows disk properties.

Retrospect never formats the disk, it just erases them using a simple erase command.
Robin Mayoff
Director, Retrospect Support Services
Retrospect Employee #63. Since 1994

#10 Ramon88

Ramon88

    Occasional forum poster

  • Members
  • 425 posts
  • LocationThe Netherlands

Posted 24 August 2009 - 05:25 PM

It says NTFS.

Don't know if this was standard, as I formatted it before using it for any of the tests.

If I want to reformat it, it wants to do it with a clustersize of 4096 bytes. Probably what it is now. Capacity of this 320GB cartridge is 298GB in 'real life'.

B.t.w. I just found out I can only format it NTFS and exFAT. Nothing else is possible.

Note:
Drive firmware is 2040
Cartridge firmware is 13.01A13
HP RDX Utility is version 2.20 / DLL version is 2.28

#11 Mayoff

Mayoff

    Retrospect Backup Commando

  • Administrators
  • 11,997 posts

Posted 24 August 2009 - 06:28 PM

ntfs is best
Robin Mayoff
Director, Retrospect Support Services
Retrospect Employee #63. Since 1994

#12 Ramon88

Ramon88

    Occasional forum poster

  • Members
  • 425 posts
  • LocationThe Netherlands

Posted 26 August 2009 - 06:31 AM

Just FYI,

Inserted a brand new cartridge. They are already factory formatted as NTFS (at least this HP version is).

#13 Ramon88

Ramon88

    Occasional forum poster

  • Members
  • 425 posts
  • LocationThe Netherlands

Posted 26 August 2009 - 03:07 PM

Okay, running as "Disk" Backup Set...

I purposely fed it a transfer of a Backup Set exceeding the drive's storage capacity of 320GB, to test what handling was required.

It filled up the cartridge with about 18 MB/sec (1100MB/min). This took about 4,5 hours. Then it stopped transferring and started media verification. This took about (I didn't pay exact attention to the unit) 3,5 hours. Then it requested new media, but didn't eject the first cartridge by itself (the script was not ended, so that setting would not effect this anyway). Then something strange; I needed to press the eject button twice before it ejected (± 30 seconds between presses)...

I inserted a new cartridge and pointed Retrospect to it. I feel this is somewhat of a dangerous thing, as I could have pointed it to a regular drive by mistake and it might have erased that...

It correctly named the cartridge and continued the backup transfer script. Still running at the moment.

Robin, what would happen if I would have pointed the media requester to the wrong fixed drive? Of course I didn't try this, but it has me a bit worried as this would be something one easily can do by mistake... :(

#14 Mayoff

Mayoff

    Retrospect Backup Commando

  • Administrators
  • 11,997 posts

Posted 26 August 2009 - 03:35 PM

Removable and local disks are treated differently. We will not erase a local disk with data, we will write to a new folder.
Robin Mayoff
Director, Retrospect Support Services
Retrospect Employee #63. Since 1994

#15 Ramon88

Ramon88

    Occasional forum poster

  • Members
  • 425 posts
  • LocationThe Netherlands

Posted 26 August 2009 - 03:54 PM

:thanks: Okay, thanks for the clarification. This does indeed make sense, glad the engineers thought of that one!

Would have loved Retrospect to be able to eject the cartridge when it was full though. But that's actually not really important.

#16 Ramon88

Ramon88

    Occasional forum poster

  • Members
  • 425 posts
  • LocationThe Netherlands

Posted 27 August 2009 - 08:19 AM

Okay, the script says to eject media from the drive, but at the end this doesn't happen.

Besides that, I get the following:

+	Executing RDX Transfer Snapshot test at 26-8-2009 8:44 (Execution unit 1)
	To Backup Set RDX Test...

-	26-8-2009 8:44:58: Transferring from TC SQL DB Backup Set
		 Backup (26-8-2009 1:30)
-	26-8-2009 13:21:44: Verifying RDX Test
-	26-8-2009 16:51:23: Transferring from TC SQL DB Backup Set
	26-8-2009 20:11:40: Execution completed successfully
		Completed: 2219 files, 474,9 GB
		Performance: 1027,6 MB/minute
		Duration: 08:16:07 (00:22:58 idle/loading/preparing)


-	26-8-2009 20:11:40: Transferring from TC SQL DB Backup Set
		 Backup (25-8-2009 1:30)
	26-8-2009 20:30:58: Execution completed successfully
		Completed: 93 files, 19,8 GB
		Performance: 1067,7 MB/minute
		Duration: 00:19:17 (00:00:20 idle/loading/preparing)


-	26-8-2009 20:30:58: Transferring from TC SQL DB Backup Set
		 Backup (24-8-2009 1:30)
		No files need to be transferred
	26-8-2009 20:31:00: Execution completed successfully
		Duration: 00:00:02


-	26-8-2009 20:31:00: Transferring from TC SQL DB Backup Set
		 Backup (23-8-2009 1:30)
		No files need to be transferred
	26-8-2009 20:31:02: Execution completed successfully
		Duration: 00:00:01


-	26-8-2009 20:31:02: Transferring from TC SQL DB Backup Set
		 Backup (22-8-2009 1:30)
		No files need to be transferred
	26-8-2009 20:31:04: Execution completed successfully
		Duration: 00:00:01


-	26-8-2009 20:31:04: Transferring from TC SQL DB Backup Set
		 Backup (21-8-2009 1:30)
		No files need to be transferred
	26-8-2009 20:31:06: Execution completed successfully
		Duration: 00:00:01


-	26-8-2009 20:31:06: Transferring from TC SQL DB Backup Set
		 Backup (20-8-2009 1:30)
		No files need to be transferred
	26-8-2009 20:31:08: Execution completed successfully
		Duration: 00:00:01


-	26-8-2009 20:31:08: Transferring from TC SQL DB Backup Set
		 Backup (19-8-2009 1:30)
		No files need to be transferred
	26-8-2009 20:31:11: Execution completed successfully
		Duration: 00:00:01

-	26-8-2009 20:31:11: Verifying RDX Test
	26-8-2009 22:38:11: Execution completed successfully
		Remaining: 1 files, 26,2 GB
		Completed: 2311 files, 464,7 GB
		Performance: 1504,4 MB/minute
		Duration: 05:17:35 (00:01:16 idle/loading/preparing)
I get the eight "in between" reports in the log as they probably reflect the eight snapshots from the source backup set. However I don't get why it ends with "Remaining: 1 files, 26,2 GB". In the activity log > history it is graphically represented with the bar not indicating total completion and even claiming " Remaining: 00:06:09" and reporting 0,0 errors/warnings in the listing...

The script used is a default "Transfer Snapshots" script configured with "Transfer all active Snapshots". The rest is default.

#17 Mayoff

Mayoff

    Retrospect Backup Commando

  • Administrators
  • 11,997 posts

Posted 27 August 2009 - 01:14 PM

The remaining is under he verify section. That seem to indicate that you have one very large file that Retrospect didn't attempt to verify and I am not sure why.

You could try a tools>verify operation and see if the same issue is reported.
Robin Mayoff
Director, Retrospect Support Services
Retrospect Employee #63. Since 1994

#18 Ramon88

Ramon88

    Occasional forum poster

  • Members
  • 425 posts
  • LocationThe Netherlands

Posted 27 August 2009 - 03:06 PM

Will do that within a couple of days as I erased this set for other tests.

Large files is easily possible, this script's source is a SQL database backup dump folder filled with BAK files and Transaction logs.

Is there a limit when Retrospect can't read a certain file size anymore?

#19 Mayoff

Mayoff

    Retrospect Backup Commando

  • Administrators
  • 11,997 posts

Posted 27 August 2009 - 03:33 PM

Retrospect should be able to handle any file size that the file system itself can handle. I have worked with multi-GB files before.
Robin Mayoff
Director, Retrospect Support Services
Retrospect Employee #63. Since 1994

#20 Ramon88

Ramon88

    Occasional forum poster

  • Members
  • 425 posts
  • LocationThe Netherlands

Posted 30 August 2009 - 03:49 PM

I've found another problem. Actually I've reported it earier, but I now really can confirm it.

Eject between cartridge swapping (during a backup larger than a single cartridge can hold) takes very long (like five minutes).

Pressing Retrospect's Eject button doesn't do anything. Ejecting the media from Windows doesn't work (Vista says the cartridge is being used by another program). The only thing that works is to press the eject button. The drive's activity LED starts to blink and that remains until the drive ejects the cartridge. This however takes several minutes. As this is not a (rewinding) tape drive, it shouldn't act like one in this respect.

I dunno... I'm starting to think I should send this unit back and get an LTO4 drive... :rollie:

Seems like Mauricev has shown me the light regarding LTO...




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users