Network Backups are Slow

[rroselavy]

Hello-

 

We are evaluating Retrospect 8.0733 Trial and are experiencing very slow backups of both PPC and Intel Mac Clients over our Gb network. We are not using encryption or compression for these tests. Each test involves the same 20GB(6,000 files) of data. It does not seem to matter whether or not rules are applied.

 

Network backup from Xserve 10.4.11 PPC/Xserve RAID Client to Intel 10.4.11 Mac Pro Engine (Exabyte Magnum 1x7 Autoloader via SCSI) = 168 MB/m

 

Network backup from Intel 10.5.6 Mac Pro Client/startup drive to Intel 10.4.11 Mac Pro Engine (Exabyte Magnum 1x7 Autoloader via SCSI) = 167.4 MB/m

 

Local backup of same data on Intel 10.4.11 Mac Pro Engine (Exabyte Magnum 1x7 Autoloader via SCSI) = 1.7 GB/m

 

 

Our existing backup solution (BRU Server) can perform network backups over the same Gb network at a rate of 2 GB/m.

 

 

Please advise.

Edited May 14, 2009 by Guest

[Mayoff]

If you select a client under Sources and click Refresh, what type of connection speed do you see?

[Maser]

I'll answer this one for one of my clients.

 

Just finished his first "full" backup to one of my media sets (another backup is running now)

 

about 20G of data (94K files) were backed up to my set. Reported as 336.1MB/min

 

if I do a "refresh" on this client source, it reports (now) 2519kbps.

 

Refresh again -- 7748kbps (!)

 

 

 

 

Checked another client that did an incremental backup after that one (627 files, 129M). This performance was 242MB/min

 

Refresh on that client -- speed -- 1262 kbps

 

 

 

Did another refresh of that first client. 30kbps (!!!)

 

 

All over the map, apparently...

 

 

[Maser]

And, for the client that just finished backup:

 

16.8G (78K files) -- reported performance 386MB/min

 

refresh speed is: 2591 kbps

 

 

Is this about expected?

 

Client is a MacBook Pro 2.4 core2duo with 4G RAM. Clearly running other stuff during the backup (which took 46min according to the log...)

[Mayoff]

800 to 1000 is 10baseT speeds

8000 to 10,000 is 100 100baseT

10,000 and higher is Giga speeds.

[rroselavy]

Thanks Mayoff for the reply,

 

Something is definitely wrong here. Both Clients are 6.3.019

 

Retrospect results:

 

Xserve Speed = 2398 kbps; echo time = 50ms

Mac Pro Speed = 249 kbps; echo time = 33ms

 

Please note that the Mac Pro Client is the same machine that I am running Console on.

 

BTW: Ping executed from the Engine's terminal has an echo average of 0.29ms and 0.27ms for these same clients

 

Could this be a DNS issue? We rely on /etc/hosts.

 

 

[Maser]

I agree. Something has to be wrong then either with the app or the client. Everything is gigabit here (which we can confirm with javascript speed tests out to the Internet, etc...)

 

There are no 100Base-T "office" switches in place throttling down traffic -- everything goes right to the Gigabit routers in the network closets, etc...

 

All clients are at 6.3.019

 

 

I have a colleague on another part of campus -- also on Gig Ethernet -- only seeing backups in the 200M/min range (faster on verification) -- with no compression.

 

 

The *only* thing that might be an issue on the clients is they have Sophos Anti-Virus installed on them, so maybe the client "touching" the files is scanning them before backup and that's triggering the "on-access" virus scanner? (And I don't even have Sophos on the backup "server" computer at all...)

 

But I have this turned off on a couple of machines and I'm not seeing any significant difference in speeds when I do that.

 

 

Oh, and the Windows clients would have McAfee Virus Scan 8.5 on them. But there's no significant speed difference between Mac or Windows clients...

Edited May 14, 2009 by Guest

[Maser]

And, all my clients are added by hostname.

 

 

I'm assuming EMC -- in house -- is seeing faster speeds for Gig network backups? Is that the case? If so, what speeds are you seeing and how are you adding your clients to your system?

[Maser]

I confirmed (?) that if I go to http://www.speedtest.net on both my "server" mini and my client 3G iMac, I get download speeds of 100Mb/s and upload speeds of around 22Mb/s

 

 

Yet if I "refresh" my client from my server, the "speed" shows 1672kbps (echo time 50ms)

 

This isn't consistent. It *should* be a lot faster.

[rroselavy]

Mayoff-

 

Interestingly, adding the ENCRYPT_NETWORK_LINK to the Clients improved both the speed test results and the backup speed (275 kbps) by over 50%.

 

Still slow...but a tiny bit better.

 

Please advise.

Edited May 14, 2009 by Guest

[rroselavy]

Not sure if this is related, but on the Xserve Client I am getting repeated errors in the retroclient.log file:

 

bindToValidBootPort: Unable to bind to valid boot port

 

 

 

On my Console machine, I am receiving the repeating message in Console Messages when Console is running:

 

Retrospect[813] connectViaQueue: exception "MacTCPconnection::Connect: connect() failed with error 61: Connection refused" connecting to 127.0.0.1:22024

 

 

Edited May 15, 2009 by Guest

[Maser]

I'm wondering if this speed issue is related to the "backup of small files" that was supposed to be improved in 608, but maybe really wasn't?

 

I just did a quick test here this morning. I have a folder full of about 1400 .mp3 files I set as a "favorite" folder on my client.

 

This folder is about 3.8G -- (so each file is approx. 3M in size).

 

If I do an AFP connection from my server machine to my client and just drag this folder to my external FW 400 hard disk -- the Finder drag-copy ran at 1.3M/min

 

So I set up Retro 8 to just do a backup of this favorite folder (3 times -- with compression and without -- and backing up to "clean" disk media sets and my 240G "dirty" set.) -- no passwords on the media sets.

 

Those 3 backups ran at 948M/min, 1022.9M/min, and 1.1G/min -- which is fully respectable/acceptable.

 

But this is *not* what I see doing incremental backups of my "users" folder on the same client. That would backup (as an example): 1153 files (247.4Mb) -- 243.3M/min

 

 

So maybe the problem is still the speed of backup of *smaller* files?

 

 

 

[Mayoff]

Does this help with your speed?:

 

Go to Preferences>Media. Uncheck "Generate MD5 digests during backup operations".

 

Run another full backup and compare the speed.

 

[Maser]

I will test this shortly:

 

Question, though: On my test machine, I only have one proactive script (with only 3 "sources" in it).

 

On my production machine, I have 3 proactive scripts with 40 clients total. From what I can tell in logging, the "polling" for proactive clients goes on *constantly*. Would/should this have any affect on speeds of backup of *proactive* clients?

[Mayoff]

Polling will not impact backup performance.

[rroselavy]

Does this help with your speed?:

 

Go to Preferences>Media. Uncheck "Generate MD5 digests during backup operations".

 

Run another full backup and compare the speed.

 

I tested disabling the MD5 option yesterday before trying the encryption option. It did not seem to have much if any effect. The copy happened at about 168kbps. I'll try again today just for kicks.

 

We are also going to try bypassing the network by connecting the engine and a client together, just to eliminate the possibility of conflicting services or other bottlenecks. We do run BRU Server still, and have several Licensing and File services in use on the network.

[Maser]

Fully realizing that test environments aren't perfect, here's what I could compare:

 

 

Two folders on my computer -- one a 4G .mp3 folder (1416 items), one my "mail.app" folder -- /users//library/mail (mail.app *not* running) -- 1.9G (21890 items.)

 

 

If I drag-drop these folders via an AFP connection (both machines running 10.5.7), the total amount of time it takes to copy these folders over the network to my FW 400 external drive on my retro-engine-console-running MacBook: 5:31

 

 

So I set up scripts to backup these two favorite folders with 733.

 

I did this 4 times (all to "recycled" disk media sets each time -- *and* restarting the engine before doing each backup). Client machine was *not* rebooted at all.

 

Results follow:

 

All are with *no verification*. Media set has no password.

 

Test 1 -- no compression -- MD5 on: 754.3M/min (8:06 total)

 

Test 2 -- compression -- MD5 on: 713.5M/min (8:32 total -- 6% compression)

 

Test 3 -- no compression -- MD5 *off*: 1.1G/min (6:01 total)

 

Test 4 -- compression -- MD5 *off*: 821.1M/min (7:31 total -- 6% compression)

 

That's a *big* difference with MD5 off in this test!

 

Of note: based on the names of the folders, the .mp3 folder went first.

 

As I watched the backups, this MP3 folder was being backed up at about 1G/min near the end of the folder contents with MD5 on (compression on or off didn't matter, interestingly enough -- does the program know to not bother attempting to compress certain file extensions?)

 

With MD5 off, near the end of the .mp3 folder it had reported 1.2G/min. 20% increase alone (?!)

 

Once it started hitting the mail folder (larger number of smaller files), then the performance dropped noticeably to give the "total" performance reported above.

 

 

However, for what it's worth, on my production Mac (which is faster and has more RAM, but more proactive scripts/clients *but* backing up to a FW *800* drive), I turned MD5 off and I'm really not seeing any speed increases on my clients incremental backups (didn't really expect to...)

 

However, more factors are certainly at play there (matching *and* software compression...) But these are much smaller sets of incremental daily data (usually less than 500M) and would *primarily* be changes in Mail.app folder contents (though not exclusively).

 

 

The other thing of note: After Test 4, I did a "refresh" of my client source -- this reported 4180 kbps. Before I did any testing? It had reported 15515 kpbs -- this "speed" is all over the map, apparently.

 

 

[Maser]

So I then tried something that I know has tons of *really small* files -- I run an iCal server on my 2x2G Dual-Core intel Xeon xserve (which only has 2G of RAM...)

 

So I started with a virgin "config80.dat" file -- so there would be no proactive client scans going on, etc...

 

I added the /Library/CalendarServer folder as a favorite folder for this client.

 

This "calendarserver" folder has approx 37,800 files, but only takes about 63M of space. This would be a good example of what many of my clients would also see on an incremental daily backup basis (as they are big iCal/Mail.app users...)

 

I guarantee you the xServe is running at Gigabit speeds. I just can't (easily) do a finder copy of this folder because of permissions, etc...) And, yes, the xServe is running many different "services" and having users hit it for other purposes (but it's fairly quiet now while I'm doing these tests...)

 

Refresh speed (one time) reported 6659. There's no way that's true, but...

 

 

 

 

 

Repeated the tests above with only this folder:

 

Test 1 -- no compression -- MD5 on: 13.6M/Min (5:44)

 

Test 2 -- compression -- MD5 on: 14.2M/min (5:22 -- interesting reported 0% compression!) Odd that this was faster, but the "server" in use so there could be changes going on while these tests are running)

 

Test 3 -- no compression -- MD5 off: 14.5M/min (5:11) -- things keep getting faster?

 

Test 4 -- compression -- MD5 off: 14.0M/min (5:22 -- with 3% compression this time -- weird...)

 

 

I suppose I could put Retro 6 on this and back things up to a "file" for comparison, but would there be any value in that?

 

 

 

 

[Maser]

Oh, and why not -- here's one more data point.

 

On my client where I did the first tests? my 3G iMac?

 

I made my personal user "library" folder a favorite and only added a rule that said (basically) "use the "all files except cache files" rule and don't backup anything with "itunes" in it.

 

This was about 75,500 files -- about 5.9G

 

 

No compression. No verification. MD5 *off* Performance was: 536.7M/min

 

Not the 1G with just the .mp3 files.

 

Not the significant drop (but still around 800M/min if I do it alone) if I backup the mail folder.

 

Even less.

 

I think performance (as I watched this) seem to really, really suck when it started backing up my /Library/Pubsub folder -- 170M -- 23,021 items.

 

 

All those dinky, long-named, .xml files really made things start to drag. The performance actually was showing *27M/min* right before it finished to show the end result of 536M/Min

 

 

 

 

 

 

[Maser]

Oh, and why not...

 

Over lunch, I slapped Retro 6 on my test box (stopping the Retro 8 engine). Set up 3 test "file" backup sets.

 

Backed up the "calendarserver" folder above -- no verification, no compression.

 

Backup of *that* folder was slower than 8: 12M/min

 

 

So, I did my MP3 and "mail" folder like in the original test -- no compression, no verification, different "file" backup set:

 

Slightly different results:

 

Total backup of those two "subvolumes" was 968M/min -- *between* the MD5 on and MD5 off speeds of Retro 8 -- closer to the faster speed of Retro 8/no MD5, though.

 

 

And, finally, did my personal "library" folder (results would be slightly different because I used Retro 6's "all files except cache files" and added "folder name contains iTunes"), so this was 6.7G of stuff rather than 5.9G -- probably even *more* small cache files, actually...

 

This backup took: 498M/min -- slower than Retro 8.

 

 

Like Retro 8 -- speeds were about 828M/min until it hit that last 53M that was over 20000 files -- then things started to drop significantly.

 

So, in *these tests* -- Retro 8 is clearly faster than Retro 6 (at least with MD5 turned off on Retro 8). The last test isn't exactly the same, though, because there was an "extra" .8G of files somewhere. That'd be a lot of "cache" files between the the "selector" and the "rule", but that's quite possible...

 

 

 

Of course, none of this really explains anything -- particularly why a "refresh" of most of my Gigabit clients rarely say over 10000kbps -- at best I see 2-5000kbps when I spot-check random clients. They really *all* should be in that higher range. Rarely have I seen anything over 6000kbps with the exception of my personal iMac which showed 23814 once and now shows 913kbps after I just restarted the engine. I didn't do anything with my iMac in between those two refreshes...

 

 

 

My *first* backup to a new set in Retro 8 of my entire xServe (114G -- 438K files) -- ran only (?) at 425MB/min (compression on, MD5 on). Somehow, I think I'm still expecting this to be faster...

 

 

But maybe it really is all a function of "tiny files"... Clearly both 6 and 8 "performance" suffers when backing up those tiny files...

 

And maybe that's the difference people are seeing?

 

Who knows. YMMV...