Jump to content

very very slow backup


muttel

Recommended Posts

Hi,

 

I have a (pretty new) OS/X 10.6.2 server (Xserve with 12GB RAM, connected iSCSI devices) as backup server.

I did some tests and I'm really disappointed by the speed of the backup.

Best speed is on the server itself (to a 'local' iSCSI device) - about 550MB/min.

iMac Clients (even the new i7) won't get faster than 150MB/min. Win7 Client was 30MB/min only!!!

I've already tried the renice tip from this forum (doesn't really speed anything up).

The strange thing is that the cpu time of the retrospect client never exceeds 5-8% (on the win7 machine it never reaches 1%).

Our network is gigabit.

 

Does anybody have a clue?

Link to comment
Share on other sites

  • Replies 80
  • Created
  • Last Reply

Top Posters In This Topic

Some details might help.

 

(1) what version of Retrospect engine? (8.x.x)

(2) what version of Retrospect client on each client machine? (x.x.x)

(3) What version of OS on each client machine?

(4) Architecture of OS on each Mac client (PPC? Intel?)

 

The renice thing really only helps the GUI interface responsiveness.

 

The consensus seems to be that the problem is at the client. Others have noted exactly what you are seeing, no solutions yet. Perhaps if you could provide some version specifics, etc., some pattern might emerge.

 

Russ

Link to comment
Share on other sites

I am also experiencing very, very slow backups.

 

Retrospect engine (Version 8.1 build 626).

Retrospect client version 6.3.028

Retrospect engine running on Mac OS 10.6.2 Server, Xserve (Intel) to LTO-4 (SCSI).

Retrospect clients running on Mac OS 10.6.2.

Retrospect clients all running on Mac (Intel) hardware.

 

I'm backing up staff home folders on their desktop/notebook computer via gigabit ethernet.

 

Performance seems very, very slow at 120MB/sec.

Link to comment
Share on other sites

i am also seeing very very very slow network backups.

 

i have latest retro client sw installed. running on a mix of fast and slow clients both intel and ppc (doesn't seem to matter a lot in terms of speed). using switched gigabit ethernet connections. cpu not fully taxed on client, nor on server.

 

data compression is off. thorough verification is on. byte-by-byte file comparison is off. very little other traffic on the network.

 

 

this is slower than can easily be explained - how can it be this slow when it's not taxing the processor on either box and when there's nothing but a gig-e switch and about 200 µsec of latency?

 

the only things i can think of are things like:

 

1 maybe the client is reading from disk in an extremely inefficient way that's causing it to wait for disk a lot.

 

2 maybe the client and server are communicating in some very inefficient way that causes them to wait for each other a lot.

 

it's getting to be a long time since i used 6.1, but i believe i was getting faster speeds then (and back then i thought IT was terribly slow!).

 

neither of these explanations makes a lot of sense given that 6.1 was faster with (almost) the same client, though i admit that its possible that other things have changed in my environment in the many months since i was last running 6.1.

 

my instincts as a software developer tell me there is a ton of room for improvement here - i can't figure out why it's soooo slow. this kind of performance is what i would expect to see in alpha phase of development, but not a year after release.

 

...and let's not even get started on the topic of retrospect's requirement for a full scan vs. time machine's use of fsevents.

 

 

 

 

Link to comment
Share on other sites

Sorry, forgot to mention that I'm using the latest version of Retrospect (8.1.626), Mac Clients are 6.3.027, Win Client 7.6.107.

All Macs are Intel (latest iMacs/Xserve).

What kind of pattern could that be? The newer the hard- and software the slower the backup speed?

Link to comment
Share on other sites

I've done some serious testing, its the client software, it sucks. Take the same machine, with filesharing instead you get better speed, not best, but better.

 

Same machine and afp-copy, max speed.

 

Same machine with Retro 6, very good speed.

 

Posted some charts about this.

Link to comment
Share on other sites

Take the same machine, with filesharing instead you get better speed, not best, but better.

Yep, but doing it that way (backing up a mounted share) only permits access for the credentials of the filesharing connection, not access to all files (the client runs setuid root when it does file access so that it can back everything up), and ACLs may not be right on restore, etc.

 

Your tests showed where the problem is, and show that something is very, very wrong.

 

Russ

Link to comment
Share on other sites

For what it's worth, I have Retrospect 8.1.626 with the sever running on 10.5.8 on a early 2009 Mac Pro backing up various machines running 10.5.8 (and one PC) and I don't see this problem. The Mac clients are version 6.3.028 and the Windows client is 7.6.106. They all back up several hundred MB/minute.

Link to comment
Share on other sites

No, we're running 10.5.8 here and get the slow speeds for clients.

Sigh. Just when a pattern seemed to be developing.

 

Ok. On all clients or just some?

Are the clients PPC or Intel?

What client versions are slow? All versions?

 

Note that above, the slowness is shown with Mac OS 10.6.2 with both Retrospect client 6.3.027 and 6.3.028.

 

I wonder what the difference is between your configuration and MAlexander's configuration.

 

What is the Network infrastructure? GigE ? any routers in between the client and Retrospect engine?

 

Russ

Link to comment
Share on other sites

I gave my engine and client version numbers above (engine 8.1.626, Mac clients 6.3.028 and Windows client 7.6.106). The engine has two network interfaces, both are connected to gigabit switches. One Mac client is on the primary interface and all the others are on the secondary interface. There are no routers involved. The Windows machine is the only one that doesn't have a gigabit interface, but even it is reasonably fast (around 300 MB per minute in the best case).

Link to comment
Share on other sites

I gave my engine and client version numbers above (engine 8.1.626, Mac clients 6.3.028 and Windows client 7.6.106).

Yes, I noticed that and understood. Perhaps you didn't notice that I wasn't responding to you and wasn't asking the questions of you:

 

[color:red]In response to[/color] [color:orange]Disegno Group[/color] <<<-----====

 

No, we're running 10.5.8 here and get the slow speeds for clients.

Sigh. Just when a pattern seemed to be developing.

 

Ok. On all clients or just some?

Are the clients PPC or Intel?

What client versions are slow? All versions?

Russ

 

 

Link to comment
Share on other sites

I'm also seeing this problem, very slow backups of network clients. I'm transitioning from an old G5 Xserve with OS X 10.3 and Retrospect 6.1, to a quad-core Xserve with OS X 10.5 and Retrospect 8.1. The old box did the last full backup a week ago, and logged client speeds of 600-700 MB/minute. The new box is now running a full backup but logging only 150-160 MB/minute from the same clients. Backup of local volumes on the new server logged 1000-1500 MB/minute. Nothing changed in the network, and the only change on the clients was upgrading the Retrospect client version. Details below.

 

Server:

Xserve Xeon 2.8 GHz quad-core

8 GB RAM

OS X Server 10.5.8

Retrospect 8.1 build 626

backing up to FireWire 800 hard drive

 

Clients:

mix of G4, G5, and Intel

OS X 10.5.8

Retrospect Client 6.3.027

 

network is switched gigabit ethernet

Link to comment
Share on other sites

Follow-up here, I was just poking around in log files and the /var/log/retroclient.log contains this:

 

Main: Client version is 6.3.027

1265318715: bindToValidBootPort: Unable to bind to valid boot port

 

with that second line repeating many, many times with the leading number incrementing (looks like a timestamp in raw clock seconds?). Don't know if this means anything but it is showing in the log on all the clients, even ones that haven't been backed up yet.

 

Link to comment
Share on other sites

That's useful. Because not everyone is seeing this, you might want to contact EMC Retrospect support and work with them to try to figure out what is going on. Because you have an identifiable issue, it might provide insight. It's really not something that we are going to be able to resolve in these user-to-user forums.

 

Contact EMC Retrospect support

 

That might lead to a solution that would help everyone.

 

Russ

Link to comment
Share on other sites

I will be contacting support about this. I'm still waiting for my first backup using 8.1 to actually finish! That started at 10pm last night, backup of about 550 GB total, most of it from the 16 client machines. With 6.1 it would have been done this morning, about 12 hours elapsed. Looks like 8.1 will finish late tomorrow night, about 50 hours. Unacceptable.

 

Link to comment
Share on other sites

It's done! A bit sooner than expected, because inexplicably it got faster during the night (although it didn't do that last night). I summarized the log files of the last full backup on the old server, and the new one that just finished, to get a clear look at the numbers. Logs are attached below. Note the backups did the clients in slightly different order and the filtering rules are a little different, but otherwise these are nearly identical backups. The CPU types of the clients are noted, but there doesn't seem to be much correlation between that and speed in either backup. Bottom line, the old server is more than 3x faster overall. This sucks.

 

Last full backup on old 6.1 server:

 

1/28/2010 10:00:00 PM:

1/28/2010 10:26:54 PM: 239363 files 8.2 GB 379.6 MB/min server local

1/28/2010 10:39:16 PM: 4618 files 10.7 GB 896.9 MB/min server local

1/28/2010 11:07:57 PM: 50960 files 13.7 GB 515.5 MB/min server local

1/29/2010 12:16:48 AM: 342473 files 50.3 GB 812.0 MB/min server local

1/29/2010 12:23:04 AM: 11871 files 3.7 GB 627.3 MB/min G4 533 MHz

1/29/2010 1:08:18 AM: 131073 files 20.3 GB 482.2 MB/min G4 1.25 GHz

1/29/2010 1:15:32 AM: 30522 files 3.3 GB 499.2 MB/min G5 1.6 GHz

1/29/2010 1:47:38 AM: 30035 files 17.0 GB 555.4 MB/min G4 1.25 GHz

1/29/2010 3:09:32 AM: 107344 files 53.1 GB 672.2 MB/min Intel 2.4 GHz dual

1/29/2010 4:20:19 AM: 67880 files 45.8 GB 671.6 MB/min G5 1.8 GHz dual

1/29/2010 4:28:08 AM: 19727 files 2.1 GB 292.3 MB/min G4 1.25 GHz

1/29/2010 5:15:59 AM: 59273 files 32.3 GB 699.7 MB/min Intel 2.4 GHz dual

1/29/2010 5:48:28 AM: 31780 files 25.5 GB 814.6 MB/min Intel 2.4 GHz dual

1/29/2010 6:18:13 AM: 36073 files 16.7 GB 587.9 MB/min G4 1.25 GHz

1/29/2010 7:16:32 AM: 63720 files 21.8 GB 392.6 MB/min G5 1.8 GHz

1/29/2010 8:00:25 AM: 46638 files 21.8 GB 523.0 MB/min G5 1.8 GHz dual

1/29/2010 9:08:08 AM: 72344 files 43.2 GB 659.8 MB/min Intel 2.4 GHz dual

1/29/2010 10:08:42 AM: 99631 files 48.3 GB 831.4 MB/min Intel 2.4 GHz dual

1/29/2010 10:56:17 AM: 150702 files 30.1 GB 666.0 MB/min Intel 2.33 GHz dual

1/29/2010 11:12:59 AM: 57217 files 6.9 GB 446.1 MB/min G4 1.25 GHz

1653244 files 474.8 GB 633.4 MB/min

 

First full backup on new 8.1 server:

 

2/3/2010 10:00:00 PM:

2/3/2010 10:40:38 PM: 488692 files 22.8 GB 641.0 MB/min server local

2/3/2010 11:22:57 PM: 131339 files 58.2 GB 1481.1 MB/min server local

2/3/2010 11:25:41 PM: 6537 files 2.6 GB 1023.8 MB/min server local

2/4/2010 1:53:08 AM: 44763 files 21.2 GB 149.0 MB/min G5 1.8 GHz dual

2/4/2010 2:15:52 AM: 6256 files 3.5 GB 159.7 MB/min G4 533 MHz

2/4/2010 4:25:51 AM: 121770 files 19.9 GB 160.3 MB/min G4 1.25 GHz

2/4/2010 4:43:59 AM: 27862 files 2.7 GB 161.0 MB/min G5 1.6 GHz

2/4/2010 6:08:25 AM: 21195 files 12.3 GB 152.5 MB/min G4 1.25 GHz

2/4/2010 1:56:36 PM: 93725 files 72.6 GB 159.7 MB/min Intel 2.4 GHz dual

2/4/2010 6:57:59 PM: 57821 files 44.2 GB 150.8 MB/min G5 1.8 GHz dual

2/4/2010 7:09:56 PM: 17790 files 1.4 GB 127.0 MB/min G4 1.25 GHz

2/5/2010 12:41:21 AM: 54673 files 59.6 GB 185.5 MB/min Intel 2.4 GHz dual

2/5/2010 2:09:22 AM: 29160 files 25.4 GB 301.0 MB/min Intel 2.4 GHz dual

2/5/2010 3:06:19 AM: 32018 files 15.7 GB 289.5 MB/min G4 1.25 GHz

2/5/2010 4:33:35 AM: 57118 files 21.2 GB 256.5 MB/min G5 1.8 GHz

2/5/2010 8:52:54 AM: 68150 files 51.5 GB 204.9 MB/min Intel 2.4 GHz dual

2/5/2010 10:01:44 AM: 52949 files 9.9 GB 150.5 MB/min G4 1.25 GHz

2/5/2010 2:47:25 PM: 86663 files 44.3 GB 159.7 MB/min Intel 2.4 GHz dual

2/5/2010 6:07:32 PM: 144225 files 29.8 GB 154.2 MB/min Intel 2.33 GHz dual

1542706 files 518.8 GB 203.3 MB/min

Link to comment
Share on other sites

Actually, that's worth a great deal. Note that all of the people who are having this problem are running Mac OS 10.6.2, and you are running Mac OS 10.5.8.

 

Perhaps a pattern is developing that could be investigated at EMC.

 

Russ

 

Im running only 10.5.8 on server. Having problems with both 10.5.8 and 10.6.2/10.6.1 clients. Same problems on all variation you can do with above. Specially the setting the wrong time, 1 hour forward on clients. Happens every time.

Link to comment
Share on other sites

I'm having the exact same issue as mentioned in this thread. I just moved from 6.1 to 8.1 on an Xserve G5, running 10.5.8, backing up to an Overland LoaderXpress w/LTO2 via SCSI. With 6.1, I was achieving 1GB/minute locally and from most server client machines, but now I'm getting 60-90MB/minute... completely unacceptable, especially when our full backups, which happen over the weekends, is upwards of 4-5TB. That would take WEEKS to complete with 8.1, so needless to say I'm heading back to 6.1 for the time being (::shudder::).

 

I'll be contacting support this week, and I'll write back with what we find (if anything!).

Edited by Guest
Link to comment
Share on other sites

See the note in the Retrospect 8 Read Me:

 

EMC Retrospect 8.1 Read Me

 

Known issues with this release

 

Retrospect 8.1 performs slowly on PowerPC-based Macs. For the best performance, EMC strongly recommends that users upgrade their Retrospect servers to Intel-based Macs. For those who cannot upgrade at this time, EMC software engineers are continuing to improve Retrospect 8’s performance on all processor types, and performance boosts are likely in a future release.

You are being hit by the double whammy.

 

(1) Retrospect 8 will always be slow on a PPC, and this won't improve much. The design decision was made to keep the backup set (um, media set) format in Intel-native (little-endian) format rather than in PPC-native (big-endian) format, for performance reasons going forward on Intel Macs and also so that the holy grail of cross-platform independence with Retrospect Windows could be achieved (for reading/writing of backup sets made on one platform with the other). So, Retrospect on PPC spends much of its time swapping bytes.

 

(2) There seems to be some known slowdown problem on the client side with Retrospect 8. Read the threads in this forum.

 

It's not going to improve on a PPC.

 

Russ

Link to comment
Share on other sites

We're running OS X Server 10.5.8 on the Server (Intel) and latest clients 6.3.027 (all Intel Mac Pro's, MacBooks, Mini's) via gigabit cat 6 network and see the slow speeds. It was the same with the previous version of the client as well.

 

Still sticking with 6.1 for client backups, but not sure what we'll do next month when we upgrade to 10.6.x...

 

Christiaan

 

edit: just realised there was client v 6.3.028 - I've been using .027 from the last Retrospect "engine" update. Will see if .028 is any different.

 

edit: Doesn't seem to be a remote update option for .028 - only a direct local install option?

Edited by Guest
Link to comment
Share on other sites

We really need a response from Dantz on this... and soon.

(1) Well, Dantz will never respond because Dantz hasn't existed for over five years. See the press release:

EMC Acquires Dantz Development Corporation (Oct. 12, 2004)

 

(2) Perhaps you misunderstand. These are user-to-user support forums. From the Retrospect Forum Rules/Terms of Use

 

This forum is a community based self help tool for users of the Retrospect Backup Software and other EMC Insignia Products. ...

 

While this forum is monitored by members of the EMC Technical Support team, it is not possible to reply to all questions and threads. This forum is not an official method for contacting technical support. EMC employees are under no obligation to reply to individual forum posts. If you need immediate technical support, you can contact technical support directly at http://www.emcinsignia.com/contactsupport

Another bugfix update for Retrospect 8 is expected some time within the next few months:

Next Retrospect 8 bug fix update should be later this quarter

 

No indication has been given when another Retrospect client update may occur, and the problem seems to be at the Mac client side.

 

I've been trying my best to try to figure out any common causes for this slowdown so as to help EMC track down the cause.

 

Russ

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...