Very Slow Performance under Retro 8

[grofvert]

Yeah, the speed isn't w/in the realm of even considering sticking with it. We are back to 6 until some major speed increases are achieved.

 

 

[Mayoff]

We take the speed problems very seriously and we have engineers making code changes and QA does speed tests every day. We are looking at multiple phases within a backup that are too slow.

[wooster101]

[color:black]

I just want to share some numbers for the two types of uses I have put Retrospect 8 thru. I run it in parallel with Retrospect 6 to gain confidence with the new version.

 

First I do a simple copy of data from the servers to the Retrospect servers attached RAID and then I backup the copied data to a HP Ultrium LTO2 tape drive.

 

Retrospect Server is a old G4 867 with 1,5GB (I know it is really a low end machine but has worked very well during Retrospect 6.1 use for years)

Attached to this is a hardware based RAID 5 where the data copied from the client´s are stored. This Retrospect server is going to be replaced with Xserve G5 soon.

 

The Retrospect clients are Xserve G5´s

 

Below is the log for the copying of the 56GB mailbackup with Retrospect 8.1:

 

[/color]

 

+  Duplicate using kk-mlm-srv03_Mailbackup at 7/6/2009 10:53 PM
   7/6/2009 10:53:37 PM: Connected to Mail
   To volume Mailbackup...
   ****** Warning: volume "Mailbackup" has the "Ignore ownership" setting enabled. ******
   7/6/2009 10:53:37 PM: Copying Mailbackup on Mail
   7/7/2009 1:13:03 PM: Comparing Mailbackup
   7/8/2009 3:02:47 AM: Execution completed successfully
   Completed: 34 files, 56.2 GB
   Performance: 68.1 MB/minute (67.0 copy, 69.2 compare)
   Duration: 1 04:09:08 (00:00:08 idle/loading/preparing)
 SMTP: opening addr 192.168.55.3

 

This is 1 day and 4 hours - 28 hours for a simple copy operation and this is without any compresion and encryption - the same hardware manage to copy this over AFP filesharing under 2 hours.

 

Below is the log for the copying of the same 56GB mailbackup with Retrospect 6.1:

 

+	Duplicate using Mail mailbackup alla dagar at 2009-06-28 22.00
2009-06-28 22.00.22: Connected to Mail
Warning: volume backup_data has the Ignore ownership setting enabled.

-	2009-06-28 22.00.22: Copying Mailbackup on Mail…
2009-06-28 23.51.06: Comparing mailbackup_alla_dagar on backup_data…
2009-06-29 01.15.48: Execution completed successfully.
	Completed: 34 files, 56,2 GB
	Performance: 610,9 MB/minute (561,1 copy, 670,5 compare)
	Duration: 03.15.24 (00.00.02 idle/loading/preparing)

 

This is 3 hours and 15 minutes - almost 1 day and 1 hour less than Retrospect 8.1 - 10 times the performance

 

After the copying is done all the data is backed up to tape on a HP Ultrium LTO2 tape device

 

Below is the log for the backup of the 56GB mailbackup with Retrospect 8.1 to the tape:

 

+  Normal backup using kk-mlm-srv07_tapebackup at 7/10/2009 10:53 AM
   To Media Set KK_tapebackup_R8...
   ****** Warning: volume "backup_to_tape" has the "Ignore ownership" setting enabled. ******
   7/10/2009 10:53:25 AM: Copying backup_to_tape
   7/10/2009 7:33:52 PM: Snapshot stored, 144 KB
   7/10/2009 7:34:10 PM: Comparing backup_to_tape
   7/11/2009 0:16:42 AM: Execution completed successfully
   Completed: 314 files, 56.6 GB
   Performance: 144.6 MB/minute (111.5 copy, 205.6 compare)
   Duration: 13:23:16 (00:02:16 idle/loading/preparing)

 SMTP: opening addr 192.168.55.3

 

Retrospect 8.1 is actually almost 2 times faster backing up to tape than just copying the data from one disk to another

 

I get the following with Retrospect 6.1 Performance: 693,0 MB/minute (807,8 copy, 606,9 compare) with the same tape device and computer - 5 times the performance.

 

I just want to share the numbers that I get using Retro 6.1 compared to 8.1 - the point is that many customer use old hardware for backup including myself for my own company. And after reading the performance users on this forum of the latest Intel MacPros and Xserve are getting the only solution does not seem to be faster hardware. One user "Christiaan" is reporting the same 10 - 20 slower performance on the latest hardware - same as I am getting on old G4´s / G5´s

 

I'll chime in here too re slow client backups.

 

Clients - Mac Pro's - Quad Core 2x2.8Ghz - 10GB RAM

 

Server - latest xServe Quad Core Xeon 2x2.26Ghz - 12GB RAM

 

Backing up to SCSI Tape - Tandberg LTO-4 HH Ext LVD via Dual Channel Ultra320 SCSI Card

 

Retrospect 6.1 = averages 1GB - 2GB / min

Retrospect 8.1 = averages 150MB / min

 

Latest version of Retrospect, all under 10.5.7 (server + clients)

 

Surely, you can't blame slow hardware here...?

 

Back to 6.1 for client backups...

 

Christiaan

 

I get almost the same average to tape with a old G4 / LTO2 (144 MB/min) as Christiaan gets with a fully loaded Xserve / LTO4 (150 MB/min)

 

 

I really want to start using Retrospect 8 but the slow backup of 56GB over a Gigabit network compared to 6.1´s speed is not good.

 

I understand that G4´s are slow but as quoted above the performance on latest hardware has the same speed ratio of slowdowns and that is running Retrospect 6.1 under Rosetta translation. The conclusion at the moment seems to be that faster hardware does not help that much.

 

Keep up the good work - but please have a serious look at the speeds both from clients and to tape!

 

Best regards!

[GMRMacBackup]

I would like to add my voice to the slow performance bandwagon. Since moving to Retro 8 I have had excellent results in the backup of data but the performance is sub-par.

 

Performance with Retro 6.1 (Driver 6.1.15.101):

 

+ Recycle backup using LTO4Weekend at 5/9/2009 4:00 PM

To backup set LTO4Weekend-F…

5/9/2009 4:00:12 PM: Recycle backup: The backup set was reset

 

- 5/9/2009 4:00:12 PM: Copying Production on SAN…

5/10/2009 2:21:47 PM: Execution completed successfully.

Completed: 458716 files, 4.1 TB

Performance: 3236.7 MB/minute

Duration: 22:21:35 (00:24:48 idle/loading/preparing)

 

 

Performance with Retro 8.0 (8.0.733):

 

+ Recycle backup using LTO4Weekday at 5/13/2009 6:00 PM (Execution unit 1)

To Media Set LTO4Weekday-A...

5/13/2009 6:00:00 PM: Recycle backup: The Media Set was reset

5/13/2009 6:00:00 PM: Copying Production on SAN

5/14/2009 11:12:44 AM: Execution stopped by operator

Remaining: 236431 files, 2525.7 GB

Completed: 226165 files, 1726.6 GB

Performance: 1730.0 MB/minute

Duration: 17:12:43 (00:10:49 idle/loading/preparing)

 

 

 

System specifics:

• Intel Xserve (10.5.6 Server, 2 x 2Ghz, Apple Raid card, 3 x 750GB Raid 5, 10GB RAM)
  EMC Retrospect Single Server Unlimited 8.0.733
  Apple 4Ghz 4 Port FC Card (driver 2.02)
  Promise 610f 9TB (Raid 5, 4GB FC controller, connected to Apple FC card)
  HP StorageWorks 4048 LTO4 dual drive w/4GB FC controller (connected to Apple FC card)

 

I have just downloaded the new 8.1.148 update so my fingers are crossed.

[cristiano.cumer]

Hello,

 

the slowdowns are only on remote clients, at least on my server:

 

I get over 1GB/min if I perform a local backup, from internal disk to internal disk. Performance drops tenfold to 100MB/min (or even less) if I perform backups over the network.

 

Since I'm connected to a Gbit switch the issue must be on how retrospect handles the network.

 

Bye

 

Cristiano

[grofvert]

We take the speed problems very seriously and we have engineers making code changes and QA does speed tests every day. We are looking at multiple phases within a backup that are too slow.

 

Any idea when a solution might be available?

[denis81]

Hi,

 

I just burnt 1500€ for the multi server edition and I am currently running a 100 Go backup at 55Mb/min over the network with no compression/no verification. I am just talking about a basic copy over gigabit network to Raid FW disks.

 

This is just crazy !!!

 

Interface is great but performance is a nightmare.

I got the feeling I am doing a backup over Apple Talk, like in the 80s

 

V8 should just be faster than v6 and not 10-20 times slower.

 

V8 is just impossible to use in production at this stage and I just wasted 10 hours of my time ( + the time to reinstall/reconfigure V6).

 

@Mayoff, can we have an update about performances in next releases ?

When do you think something is going to come out ?

 

Just give us some hope

 

Thanks,

 

Denis

[CallMeDave]

V8 is just impossible to use in production at this stage and I just wasted 10 hours of my time.

 

some might consider the time necessary to qualify (or disqualify) a technology solution to be a waste. Personally I'd call it a better use of time then to drop something into a production environment without due diligence and then have to scramble to deal with any potential fallout (or loss of data).

 

Of course we hope that our proposed solutions will be adequate; but even if they're teh suck it's critical to find that out in advance.

 

( + the time to reinstall/reconfigure V6)

 

Why did you uninstall v6 if it was working as you wanted? The presence of a configured Retrospect Classic has no effect/interaction with the presence of a configured install of Retrospect 8. And certainly for testing you could have/should have left your working backup in place as you qualified the potential new one.

 

There are few who support this software endeavor more then I do, but I'm unlikely to recommend/install it for any client at the present time.

[Mayoff]

We know some speed issues exist. We have fixed a few really nasty problems already (confirmed by private testers) for the next scheduled update. We still need to fix the network transfer speeds.

[cristiano.cumer]

Those are good news! Can you tell us the planed timeframe for the release?

 

Cristiano

[Mayoff]

I think it is this month or early next month. I don't know exactly yet.

[grofvert]

Any idea when a speedy (at least comparable to v6) network fix is in the offing?

[cristiano.cumer]

Hello Robin,

 

could you also have your support team to look into this topic?

 

http://forums.dantz.com/showtopic.php?fid/47/tid/30229/pid/126332/post/last/#LAST

 

thank you

 

Cristiano

[Graham]

Hi Robin,

 

Any news on when this update is likely to happen?

 

I am nearing the end of my demo period and need to make a decision soon. I have used Retrospect for many years and would like to stick with it due to it's functionality, but Bru's performance figures are impressive (see below).

 

To add another data point, here are my experiences;

 

Retrospect 8.1.150.1 with client 6.3.019

Server - Xserve Intel Quad Core, OSX Server 10.5.8, 16GB RAM

Clients - All Mac Pro Intel Quad Core (so far), OSX 10.5.8

Backup Device - Tandberg Data T40+ library, LTO 4 drive, connected by Fibre Channel.

 

Retrospect 6.1 (AIT 5 Library, SCSI) - 1.5 GB/m clients and local disks (inc. Xsan)

Retrospect 8.1 - 2GB/m for local disks, approx. 100 - 200MB/m for clients

Bru - 7GB/m clients and local disks (inc. Xsan)

 

I can live with the local backup speed (just), but the client speed is a killer.

 

Regards,

Graham.

[Mayoff]

We hope to have an update by some time next week.

[Graham]

Thanks for the update, as documented elsewhere the console is much more responsive.

 

However it has done nothing for my speed problems as above. I have updated the engine, console and clients to 8.1.525 and restarted both client and server.

 

I am I missing anything? Any suggestions on how to get closer to Bru's speed?

 

Thanks,

Graham.

[jel]

Thanks for the update, as documented elsewhere the console is much more responsive.

 

However it has done nothing for my speed problems as above. I have updated the engine, console and clients to 8.1.525 and restarted both client and server.

 

I am I missing anything? Any suggestions on how to get closer to Bru's speed?

 

Thanks,

Graham.

I think there may be three main areas leading to a substantial difference in speed between Retrospect and BRU.

 

1. Retrospect has had a form of 'data deduplication' since well before that term was ever thought up. This inherently requires more time. Tolis have chosen not to implement this even though it is an increasingly popular feature of backup software.

 

However the above is only a small portion of the total difference, since even if you turn it off Retrospect is still glacial in its speed. (Actually with global warming Glaciers have sped up more than Retrospect.)

 

2. Retrospect completely fails to take advantage of multiple CPUs or CPU cores, not just for a single task, some of which like scanning are inherently linear and not possible to share amongst CPUs but others could be, but it also fails to overlap independent steps (e.g. compressing one file/block while encrypting a different file/block, while storing another file/block, while updating the catalog, while scanning the next volume. I get the impression BRU is far better at this.

 

3. Frankly one also gets the impression that the Retrospect software is just slow (i.e. grossly inefficient), for example merely writing out the catalog (snapshot) takes aeons. With backup media continuing to get faster (both tape and disk), these days it is quite likely that the actual process of writing the backup takes less time than Retrospect spends pontificating over scanning and cataloging (individually let alone in total).

 

 

I have spoken to Tolis and they don't seem likely to add data deduplication, but the other handicap they have of poor support for using hard disks as backup media is on their list. Using hard disks as media is something Retrospect has done well for a long time (if slowly ).

 

Note: The 'grooming' feature of Time Machine seems far faster than Retrospect, and more reliable in that it (in Time Machine) successfully prevents disk full errors (the whole point of grooming).

[rhwalker]

3. Frankly one also gets the impression that the Retrospect software is just slow (i.e. grossly inefficient), for example merely writing out the catalog (snapshot) takes aeons.

You are confusing the mere "writing out" of the catalog with the "updating" of the catalog. The catalog is an index into Retrospect's database (the media set, as it is now called). There's a lot of stuff that has to happen to update the catalog and change all of the database index pointers. Not that it couldn't be done more efficiently, and I suspect that an investment in algorithm improvement would reap wonders.

 

Optimization can become a focus once the software works reliably. It's not helpful if the software "crashes fast" or quickly gives wrong results.

 

Russ

[Maser]

I'd agree, though, that it's "slow" -- and gets progressively slower as the catalog gets larger.

 

The significant part of that is how it affects the reporting of performance when this happens.

 

For example, I watch a client back up it's incremental daily backup.

 

As the backup finishes (1472 files/255M), this report 224M/min

 

After the 15 minute updating of the catalog finishes, this drops down to 8.2M/min as the *final* reporting of performance.

 

 

And don't get me started with the "scanning" interference with Sophos 7 adding an hour to the whole backup.

 

 

[rhwalker]

It may just be that the EMC programmers are not familiar with modern sorting and matching algorithms.

 

Russ

[jel]

3. Frankly one also gets the impression that the Retrospect software is just slow (i.e. grossly inefficient)' date=' for example merely writing out the catalog (snapshot) takes aeons.[/quote']

You are confusing the mere "writing out" of the catalog with the "updating" of the catalog. The catalog is an index into Retrospect's database (the media set, as it is now called). There's a lot of stuff that has to happen to update the catalog and change all of the database index pointers. Not that it couldn't be done more efficiently, and I suspect that an investment in algorithm improvement would reap wonders.

 

Optimization can become a focus once the software works reliably. It's not helpful if the software "crashes fast" or quickly gives wrong results.

 

Russ

The status messages in Retrospect are themselves very poor and do not give a good picture of what is going on, indeed initially with Retrospect 8, one thought it had simply locked up since 'nothing was visibly happening' for hours.

 

However your argument is slightly flawed in that Retrospect is not a brand new product and therefore using brand new routines that still need optimising, it is version 8.1 and has been using the same cataloging design for over a decade and has been offering poor performance for many, many years. People have been complaining about this poor performance also for many years (mainly in relation to version 6 and 6.1) and I am sure I am not alone in having hoped that the supposedly completely re-written version 8 would have addressed this well known issue.

 

As I said it has now reached the absurd situation that catalog processing operations take longer than the backup itself and increasingly make it almost impossible to do backups in a practical timescale. It can take more than 24 hours to do a set of backups, most of which Retrospect spends thinking (slowly)!

 

I am not aware of any other backup software that takes longer over catalog related operations than the actual backing up.

 

Note: I have used Retrospect for over 20 years.

[Graham]

Just to add some more performance detail for a client backup;

 

1. The copy part of the backup operation is consistently running at 110 - 120 MB/min, irrespective of back up device (i.e. tape or disk backup). The compare part can be over 4GB/min.

 

2. Different types of backup (e.g. to tape and disk) can run simultaneously, with no effect on performance (i.e. both can run at 120 MB/min).

 

Whilst I understand the differences between Bru and Retrospect, the performance figures above are approx. a factor of 10 slower than v6.1.

 

A response from Retrospect would be appreciated!

[rhwalker]

A response from Retrospect would be appreciated!

Graham, I think you misunderstand.

 

These forums are user-to-user. From the Retrospect Forum Rules / Terms of Use:

 

This forum is a community based self help tool for users of the Retrospect Backup Software and other EMC Insignia Products.

 

...

 

While this forum is monitored by members of the EMC Technical Support team, it is not possible to reply to all questions and threads. This forum is not an official method for contacting technical support. EMC employees are under no obligation to reply to individual forum posts. If you need immediate technical support, you can contact technical support directly at http://www.emcinsignia.com/contactsupport

You might be waiting forever for an official response here.

 

Russ

[pfauland]

What you are saying is interesting : around 110 MB/m seems to be the max. speed. I wonder where this "limitation" is coming from ? Version 6 under Windows was doing around 500 MB/m ..... both to tape and to disk.

[pfauland]

What is a "normal speed" (with version 8.1.526) for a backup to NAS !? Just to have an idea ...