Jump to content
muttel

very very slow backup

Recommended Posts

Regardless of who owns what, we purchased a product that is not performing at reasonably-expected levels on hardware that was previously "'good enough" (8.1 is *1/16th* the speed that 6.1 was on our hardware. A sixteenth!). If they're selling it with official support for a particular platform, it better function at expected (and one would hope competitive!) levels on that platform. It straight up doesn't "work" on PPC servers. It's unusable. So when I call them on Thursday and make use of my Annual Support & Maint., what are they going to tell me? "Sorry, the PPC version is only a *fraction* of the speed of the Intel version"? If there's THAT much overhead in the new version, well, they need to find some new code monkeys.

 

Also, am I *paying* for a year of ASM to essentially be a guinea pig for Retrospect to help EMC fix their own software? I don't like the sound of that.

 

Clearly, there is something wrong with the software... be it client side or server side, it doesn't matter. Personally, I'm thinking it's server side, as Retro was showing 60-100MB/min backup rates with both our Mac OS X Server clients (mix of Intel and PPC), and Windows Server 2003 clients. Like I said, I'll be calling them on Thursday (possibly sooner, I'm just waiting to get a full backup completed with 6.1 before calling), and I'll post back with what comes of it. Perhaps that'll help the greater cause.

Share this post


Link to post
Share on other sites

FWIW...

 

Not that I've done this with *recent* builds, but I did a bunch of tests a number of months ago (May 09) that -- at least to me -- it showed:

 

1) If you turn MD5 off -- Retrospect 8 goes faster. Significantly faster.

 

2) If you are backing up a *mix* of files (and have MD5 turned off on 8) -- 8 *was* about as fast as 6 (!)

 

3) Backing up a lot of tiny files (like a "PubSub" directory with all those .xml files, or an iCal directory with thousands of tiny .ics file) will make the backups speeds drop *dramatically*.

 

http://forums.dantz.com/showtopic.php?tid/30638/

 

 

At some level it might be interesting to try my tests again under 10.6 (and with the later builds). Assuming Retro 6 even runs under 10.6?

 

Of course, I could never get anything to backup a network client in the neighborhood of 600M/min like another forum user reported.

 

 

FWIW. YMMV and all that...

 

 

Share this post


Link to post
Share on other sites
Regardless of who owns what, we purchased a product that is not performing at reasonably-expected levels on hardware that was previously "'good enough" (8.1 is *1/16th* the speed that 6.1 was on our hardware. A sixteenth!). If they're selling it with official support for a particular platform, it better function at expected (and one would hope competitive!) levels on that platform. It straight up doesn't "work" on PPC servers. It's unusable. So when I call them on Thursday and make use of my Annual Support & Maint., what are they going to tell me? "Sorry, the PPC version is only a *fraction* of the speed of the Intel version"? If there's THAT much overhead in the new version, well, they need to find some new code monkeys.

 

Also, am I *paying* for a year of ASM to essentially be a guinea pig for Retrospect to help EMC fix their own software? I don't like the sound of that.

 

Clearly, there is something wrong with the software... be it client side or server side, it doesn't matter. Personally, I'm thinking it's server side, as Retro was showing 60-100MB/min backup rates with both our Mac OS X Server clients (mix of Intel and PPC), and Windows Server 2003 clients. Like I said, I'll be calling them on Thursday (possibly sooner, I'm just waiting to get a full backup completed with 6.1 before calling), and I'll post back with what comes of it. Perhaps that'll help the greater cause.

Please don't misunderstand my post. I'm in complete agreement with you.

 

The software was released long before it was ready, apparently untested. It's been out for over a year and is still very unreliable, very buggy, still has no manual, and cannot read backups made by any older version of Retrospect. To me, it's not ready for production deployment.

 

It's just that none of us users in this user-to-user support forum can do much about it.

 

Russ

Share this post


Link to post
Share on other sites

So, because I was curious (and I could do this while doing other things), I tried Retro 6 vs. Retro 8 running on 10.6 backing up the same (basically) set of files on a client computer.

 

What I found 9 months later is pretty much the same:

 

1) If you are backing up lots of *tiny* files (ie, the "PubSub" directory on a client's "Users" folder) -- Retrospect 6 is *miles* faster than Retrospect 8.

 

In my case, I backed up my "Pubsub" folder only (subvolume or Favorite Folder as the case may be).

 

All things being relatively equal, Retro 6 backed up the 50,962 files (159M) -- at a speed of 112.2M/min.

 

Retro 8 backed up the same folder at a speed of 10.8M/min

 

 

2) Eliminating the "tiny files" -- speeds were much closer (but Retro 6 is still faster).

 

Backing up the *rest* of my Users folder (except Cache files and my "iTunes Music" folder), about 78K files/16.7G of data:

 

Retro 6: 617 M/min

Retro 8: 488 M/min

 

However, the "rest of my data" also includes lots of other tiny files (like my iCal "events" files, etc...).

 

 

However, this doesn't necessarily mean Retro 8 is *always* slower.

 

I then backed up two other favorite folders/subvolumes -- my desktop folder and a folder containing about 3600 small quicktime movies.

 

The desktop folder backed up faster with 6 (920M/min vs. 589M/min).

 

But my QT folder backed up faster in *8* (547M/min with 6 vs. 594 M/min with 8).

 

 

So there's probably really a "client file size" issue here that causes 8 to be slower for whatever reason.

Share this post


Link to post
Share on other sites
So, because I was curious (and I could do this while doing other things), I tried Retro 6 vs. Retro 8 running on 10.6 backing up the same (basically) set of files on a client computer.

 

Was Retro 6 running under 10.6 Server? If that's the case, that's great news for us as I didn't think it would work?

 

Share this post


Link to post
Share on other sites
Was Retro 6 running under 10.6 Server? If that's the case, that's great news for us as I didn't think it would work?

It all depends on whether your definition of "work" includes the ability to restore and get back what you had.

 

Steve was only testing the ability to get through the backup phase, and timing that. And I believe he has reported that he has reverted to 10.5.x on his Retrospect server because of issues. He can address his test conditions.

 

Russ

Share this post


Link to post
Share on other sites

I was working with the client version of 10.6 and did nothing but test backups (and modifying a copy of a selector).

 

I had a couple of things that caused me to Force Quit retrospect 6 while doing this (primarily when creating new "File" backup destinations for my test destination.) Frequently, a dialog would pop up that I couldn't dismiss without Force Quitting.

 

Don't count on what I was doing to be of any use apart from the timing stuff.

 

In fact, I would not be surprised to set Retro 6 under 10.6 to have the exact same problems it does under 10.5

Share this post


Link to post
Share on other sites
In fact, I would not be surprised to set Retro 6 under 10.6 to have the exact same problems it does under 10.5

And more. There is the issue about compressed/uncompressed system files, and the extraneous /dev nodes. There's also the issue of poorer Rosetta support in Snow Leopard than in Leopard, and the different model for Finder program launching database in Snow Leopard. Except for Rosetta, all are restore issues.

 

Russ

Share this post


Link to post
Share on other sites

Update on my continuing experience with Retrospect 8.1 and slow network backups.

 

My server has been doing daily incrementals since the full backup finished Friday evening. There were some very encouraging numbers over the weekend, client performance peaked up in the 600-800 MB/minute range for individual clients, and that was backing up a significant amount of data, 2-3 GB. One of the backups had an overall performance of almost 600 MB/minute! But that now appears to have been an isolated occurrence, and in general backup speeds from network clients is under 200 MB/minute. However those peaks indicate the performance potential is there, but something is getting in the way.

 

I've looked at the file size issue, and I don't find much correlation. I'm sure the backup speed slows significantly if a backup has nothing but small files. But the difference between a 150 MB/minute backup of 21,000 files totalling 12 GB from a client, and a 790 MB/minute backup of 5,600 files totalling 3 GB of data from that same client a few days later, was not file size. I looked at both of those backups in detail and both had a mix of small and large files, with about the same average size.

 

I've also tested turning off data compression and MD5 digests. Although the fast backups over the weekend did come after I turned off compression, making me think for a while that I had found the problem, backups since then have been slow again despite compression still being off. Disabling MD5 digests had no noticeable effect.

 

The only correlation I've found in the data I have so far doesn't make a lot of sense. If the total size of a given backup is very small or very large, it will be slow. A backup of less than 100 MB will have low performance, but that's kind of obvious because the overhead of individual files being processed begins to dominate the total time. The strange result is that a backup of more than 5 GB will also be slow. The only fast backups I've seen are in between, from 500 MB to 3 GB is where the 500-600-700 MB/minute performance numbers can occur; but not always, just sometimes.

 

Tonight is another full backup, and I've set up another test to try to isolate the problem. I've split the client into three groups, and created three media sets and three scripts scheduled to run at the same time. Although the media sets are all on the same physical hard drive, I think I have everything set up so Retrospect will still run those scripts in parallel. If it does, the results should be very informative. If the performance bottleneck is in the client software, overall server performance should increase as it backs up from multiple clients in parallel. If overall speed is still slow, and the individual client speeds actually drop even more, then the bottleneck is on the server end.

 

Share this post


Link to post
Share on other sites

If you are like me, you'll see that when you are doing *concurrent* backups, that the speeds for those concurrent backups will be less than non-concurrent backups.

 

There's only so much data that can flow through the network "pipe" at any one time.

 

 

This is probably some weird "packet size" issue or "flow control" issue when it comes down to it.

Share this post


Link to post
Share on other sites
If you are like me, you'll see that when you are doing *concurrent* backups, that the speeds for those concurrent backups will be less than non-concurrent backups.

 

That would be true if the server is the limiting end; but it turns out that doesn't seem to be the case, at least not for me.

 

My full backup that started last night at 10 pm was done before noon today (last week it ran until Friday at 6 pm). Overall performance more than tripled when running three parallel backup threads! The individual network clients were still backing up at speeds around the same 150 MB/min I've been seeing for single-threaded backups. But the server was maintaining that speed on each of the three threads.

 

And it was maintaining that speed consistently. Very, very consistently. The high was 167 MB/min and the low was 116 MB/min, but both of those extremes occurred after one of the three threads was done and the server was only running one or two threads. During three-thread operations, the speeds spanned a very tight range from 146 to 161 MB/min. Almost like the server was deliberately throttling the speed, some sort of load balancing? Hmmm.

 

There is an upper limit to network backup speed, but in my case I know it is a whole lot higher than 150 MB/min, higher even than three times that. I've intermittently seen much faster speeds. Over the past week there have been 6 backups from network clients with speeds above 600 MB/min, and the highest was 796 MB/min! But there doesn't yet seem to be any strong correlation to specific client machines, number of files, amount of data, etc.

 

I'll continue gathering data.

Share this post


Link to post
Share on other sites

I've gathered a bit more information. Information gathered Tuesday, 16 February 2010.

 

Scenario 1: ABT Sydney Office

Server SW: Retrospect Server 6.1.230 (driver 6.1.16.100)

Server OS: Mac OS 10.5.8

Server hardware: PowerMac G4 dual 1GHz, 1GB RAM

Server tape hardware: LTO-2 (SCSI)

Client SW: Retrospect client 6.2.234

Client OS: Mac OS 10.6.2

Client hardware: MacBook Pro

Network: Gigabit switched

Performance: 350-400MB/min

 

Scenario 2: ABT Melbourne Office

Server SW: Retrospect Server 8.1.626

Server OS: Mac OS 10.6.2 Server

Server hardware: Xserve (Intel) 2.8GHz Quad-Core Intel Xeon, 4GB RAM

Server tape hardware: LTO-4 (SCSI)

Client SW: Retrospect client 6.3.028

Client OS: Mac OS 10.6.2

Client hardware: MacBook Pro

Network: Gigabit switched

Backup Performance: 140-150MB/min

 

The issue is the slower performance using Retrospect 8 software (vs Retrospect 6). The evidence suggests (to me) that the slow performance is linked with Retrospect 8 server or client software.

 

I will now email EMC2/Dantz directly in an effort to get a response. I'm also very interested to see if Mac OS 10.6.3 (not available at this time but expected soon) makes a difference.

Share this post


Link to post
Share on other sites
Scenario 1: ABT Sydney Office

Server SW: Retrospect Server 6.1.230 (driver 6.1.16.100)

Server OS: Mac OS 10.5.8

Server hardware: PowerMac G4 dual 1GHz, 1GB RAM

Server tape hardware: LTO-2 (SCSI)

Client SW: Retrospect client 6.2.234

Client OS: Mac OS 10.6.2

Client hardware: MacBook Pro

Network: Gigabit switched

Performance: 350-400MB/min

Note that this is not a supported configuration because Retrospect 6.x and Mac OS 10.6.2 are involved, so EMC Support may not be concerned about this input. See the Retrospect Snow Leopard Compatibility Statement

 

I agree, though, that the Retrospect 8 times are much too slow with your given hardware. There is something very wrong.

 

Russ

Share this post


Link to post
Share on other sites

The article you refer to states that Retrospect 6.1* Server* isn't supported on Snow Leopard.

 

The article doesn't mention problems running Retrospect 6.1 Server on Mac OS 10.5.x working with Retrospect 6.2 Clients on Mac OS 10.6.

 

Unless EMC/Dantz can fix the problems with Retrospect 8 Server and Retrospect 6.3 Client (slow backups between server and client software over network) I will have to revert to using Retrospect 6 Server (running on Mac OS 10.5.x) and Retrospect 6.2 Client (running on Mac OS 10.6.x) for backup.

Share this post


Link to post
Share on other sites
The article you refer to states that Retrospect 6.1* Server* isn't supported on Snow Leopard.

Nope. Doesn't say that.

 

The article doesn't mention problems running Retrospect 6.1 Server on Mac OS 10.5.x working with Retrospect 6.2 Clients on Mac OS 10.6.

Right. No combination of Retrospect 6.1 [color:red]with Snow Leopard[/color] is supported. Not Retrospect 6.1 server, not Retrospect 6.1 server chatting to a Retrospect client on Snow Leopard. No combination [color:red]with Snow Leopard[/color] involved.

 

Unless EMC/Dantz can fix the problems with Retrospect 8 Server and Retrospect 6.3 Client (slow backups between server and client software over network) I will have to revert to using Retrospect 6 Server (running on Mac OS 10.5.x) and Retrospect 6.2 Client (running on Mac OS 10.6.x) for backup.

Whatever. Keep making backups. Just don't count on a successful restore.

 

Russ

Share this post


Link to post
Share on other sites

I'm afraid I can't wait any longer for EMC2/Dantz to fix the problems with Retrospect 8 (specifically the slow network performance). I regret I'll have to abandon Retrospect 8.

Share this post


Link to post
Share on other sites

Just as an addition to this I have found that I experience backups at around the 200MB/minute mark if I add the server in as an actual server and use up a server license, but if I mount the share that I am backing up it then runs at between 1.5GB and 2GB a minute?! I am using the latest engine/client etc and my engine is sat on an Intel xserve with 10.6.2 server on it and the data I am trying to back-up is on a PPC xserve with 10.4.11 on it. I have tried backing up data from a 10.5 server and it is the same, slow when I add it as a server but fine when I back-up exactly the same data via attaching it as a share!

Share this post


Link to post
Share on other sites

EMC knows the speed sucks.

 

There's nothing *wrong* with backing up as a share -- except you lose the file ACLs. If that isn't important to you (and for a server, I think it would be), then you can certainly backup the data that way.

Share this post


Link to post
Share on other sites
EMC knows the speed sucks.

 

Do they? Have they actually said anywhere that they are working on speeding up network backups? I want to be clear that this is not sarcastic at all. I just haven't seen an official acknowledgment from them that the network backup speed is something that will be fixed. It's been this way for so long through so many updates that I just wanted to verify that this is actively being worked on.

Share this post


Link to post
Share on other sites
EMC knows the speed sucks.

 

Do they? Have they actually said anywhere that they are working on speeding up network backups? I want to be clear that this is not sarcastic at all. I just haven't seen an official acknowledgment from them that the network backup speed is something that will be fixed. It's been this way for so long through so many updates that I just wanted to verify that this is actively being worked on.

See this post on July 10' date=' 2009:

Very Slow Performance under Retrospect 8

 

and this post from August 7, 2009:

Very Slow Performance under Retrospect 8

 

with an update to fix "sometime next week" as of September 8, 2009:

Update some time next week

 

That good enough?

 

Russ

Share this post


Link to post
Share on other sites

Here an other post that might be the issue:

http://forums.dantz.com/showtopic.php?tid/32516/

 

Anyway, I regret that I spent money for Retro 8. That was a total waste! :angryred:

 

By the way, Retro 6.1 server runs great on my 10.6.2 server it is even capable of doing bootable backups of 10.6 clients and is able to do working backups with ACLs of my secondary server.... But hey wait, Retro 6.1 isn't supported anymore.....

Edited by Guest

Share this post


Link to post
Share on other sites
Here an other post that might be the issue:

http://forums.dantz.com/showtopic.php?tid/32516/

 

Anyway, I regret that I spent money for Retro 8. That was a total waste! :angryred:

 

 

I'm not going to make apologies for the speed issue (though, for me, I think the actual *backup* performance is probably fine -- it's the reported performance calculated after the "snapshot" is created that drags things down, etc...)

 

But Retrospect 8.x brought two things that are (for me) hands-down worth the money: The ability to do concurrent backups and grooming.

 

Those are worth the speed hit at this point. YMMV.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×