performance question Xserve RAID slow backup

[derek500]

I apologize in advance for the long post. I have tried to include all relevant information about my problem here. I also searched the Dantz knowledgebases for anything similar to this problem we are seeing, please point out if I missed a KB article or forum post that addresses this issue.

 

I have two Xserves with RAIDs attached via Fibre channel. When I back up these machines to my backup server (PowerMac G4), I get very different backup speeds, depending on if it is the internal hard drive of the Xserve, or the Xserve RAID volume. Each Xserve RAID is connected with Apple Fibre channel cards, and have no problem moving large amounts of data at high speeds to network destinations via afp or smb.

 

The first Xserve (our email server) is a dual G4 and backs up at between 300 and 400 MB/min for the internal hard drive. Our last Full backup happened at 318.6 MB/Min (according to the report) for ~13GB. The last incremental ran 414 MB/Min. (1.5 GB)

 

It's RAID volume is the email store. It is configured as RAID level 5 and currently holds approximately 75 GB of data. The RAID volume is selected in the Xserve's retrospect client on the server. Data transfer during the last Full backup was 21.6 MB/Min taking over 10 hours. This seems like a lot of time for 75 GB.

 

The second Xserve (our file sharing server) is a single G5 and the internal hard drive backs up at 200 MB/Min for Full backups. This Xserve controls the file share of it's RAID and some authentication duties (only relative to the RAID access).

 

The G5 Xserve's RAID volume is also RAID 5 and stores about 200 GB of data. The backup transfer rate was 53.8 MB/Min on the last Full backup, taking 16 hours. Better than the first set, but still slow!

 

Mostly both servers are reasonably idle except for occasional file transfers, except that the mail server always has a light steady stream of activity. The full backups start every other Friday night to make use of the light weekends.

 

For confirmation, I copied a 1.6 GB file through finder from the local volume of the G5 Xserve and from that Xserve's RAID. Performance of both file copies was superb, taking 2.8 minutes from the Xserve's local drive and 2.77 min from the RAID. Needless to say, we were concerned to find out the discrepancy of data transfer with the retro client. At afp transfer rates, 200 GB should take under 120 minutes to copy, and that again to verify, that's 4 hours, not 16. Why does Retrospect take so long when afp does not? What can we do about it? I understand that the backup is more than just copying - I should allow some extra time for matching, comparing, spanning multiple tapes (performed by an auto-loading library), so even 5-6 hours seems reasonable here.

 

The backup server is a 400 MHz G4 PowerMac with 1.5 GB of RAM, AIT-2 tape drive and ATTO UL3S SCSI card. Our network is a mix of gigabit and 10/100 switches. The Xserves and the Retro server are all on the same gigabit switch (although the PowerMac does not have a Gig network card, it runs correctly at 100 MB with no errors). The best performance of the backup comes from the second drive internal local to the Retro server, it runs at 600 MB/Min to the tape drive, so I know the capacity of data flow to the tape drive is sufficient.

 

Backup Server is 10.3.6 client with Retrospect Server 6.0.204 with RDU 6.0.103.

Both Xserves run 10.3.4 Server with Retro Client 6.0.108.

 

Thanks for any ideas. We would really like to speed up the backups of the RAID volumes!

 

 

P.S. Please feel free to correct my math if I have made a mistake here - the 1.64 GB file copy took 2 minutes, 48 seconds, which I figured is 2.8 minutes. 1.64 GB / 2.8 minutes is .58 GB/min. At this rate, 200 GB should take 116 minutes?

[Zarian]

What are you backing up to? Which model tape drive? Also is the drive firewire or SCSI?

[derek500]

We are backing up to a SCSI AIT-2 Sony SDX-500C in a Qualstar 4212 Library.

[natew]

Hi

 

Are you using the 3.20 driver from Atto? The drivers that ship with OSX are slow as molasses.

 

Thanks

Nate

[derek500]

Hi Nate,

 

According to the ATTO config tool, it is driver version 3.1.0. Should we upgrade it?

 

It is also running flash version 1.6.6f0.

 

These are the settings I left it at after a long battle with -206 errors and a call to ATTO tech support. Other than the speed from the Xserve RAID, it seems to work very well.

 

Since we are seeing 600+ MB/Min from the internal hard drive, I thought I had ruled out SCSI communication as an issue...

 

Thanks

 

-Derek

[natew]

Hi

 

The 3.1 driver is probably new enough.

 

You say you copied a 1.6 GB single file as a benchmark for AFP. Did you try copying the same single file via Retrospect client? I'm curious to see what kind of speed you get.

 

Are there lots of small files on these server machines? Large files back up _much_ more quickly than many small ones.

 

Thanks

nate

[derek500]

Quote:

---

 

Are there lots of small files on these server machines? Large files back up _much_ more quickly than many small ones.

 

 

---

 

There are a lot of files on both machines.

 

The dual G4's RAID is 76 GB and 649,465 files.

The G5's RAID has 207 GB and 606,144 files.

 

The dual G4 has the most files and the smaller files, it is our email server. It makes sense that it has the slowest backup rate. The G5 acts as our file server with years of data and many users.

 

The log claims the backup rate of the G5 raid at 400 MB/Min average between copy and compare, but apparently this does not factor in the overhead time needed for multiple files.