Massive drop in backup speed over 6 months

[mlenny]

First off, my setup:

 

* Retrospect Server 7.6.123 (RDU 7.6.2.101)

* 2GHz Xeon 5130, 4GB RAM.

* WSS2003 SP2

* Infortrend external disk array, connected directly to backup server (also our NAS head) via fibre to QLogic QLA2460, RAID6, 6TB usuable space, 3.1TB free.

* Dell PV124T LTO3 autoloader (qualified for Retrospect) connected via Adaptec 39160 SCSI card.

 

When we migrated all our data to this external RAID in February 2008, our backup speed to LTO-3 was around 4300MB/min.

 

When I looked at new backup sets started in August, the backup speed was averaging around 2500MB/min. This is also the average we are getting with more recent full backups.

 

I'm now trying to determine how we got such high performance in Feb and now have dropped almost 50%.

 

All our testing so far has happened outside of business hours so there has been no competing I/O from users on the data being backed up.

 

First off, used HDTach to test the read performance of the RAID. It scored around 200MB/s across multiple samples so the raw performance of the hardware didn't seem an issue.

 

We then re-enabled the Windows device drivers for the autoloader and ran a test backup against 1TB of files using NT Backup. Approximately same results on tape speed.

 

My next thought was file fragmentation being so bad it affected read performance of existing data, as we generally have never defragmented our server drives. We ran a full defrag pass using PerfectDisk, consolidating free space, etc. We also ran it again prior to a test backup so we were backing up as "pristine" a set of files as possible, but no difference on speed.

 

Our anti-virus software on the server only does nightly scans and no active scanning, but to eliminate any bottleneck it might be causing, we uninstalled anti-virus software and ran the test again. No difference.

 

Racking our brains for differences between now and then, we weren't sure whether volume shadow-copies were active at the initial backups, so we disabled VSS on the drive and tried again. No difference.

 

So now I'm posting to the forum to seek further advice on how to proceed, or explanations for what we're seeing.

 

One further point: the LTO3 autoloader was brand new and used pretty much for the first time on those initial backups in Feb 2008. Anyone know whether device malfunction or abnormal wear could account for the speed drop? It has been regularly cleaned (in fact we have activated its auto-clean function and left a tape in one of the magazine slots).

 

Thanks.

[railes]

Two things you may want to check on:

 

1- My LTO-3 vendor told me that LTO devices rarely need to be cleaned. I have had a HP Ultrium 3 device for 3 years and it has given me perfect service and I have never applied a tape cleaner to it. You might be doing some damage to the tape heads by cleaning it so much.

 

2- Check your MaximumSGList registry setting which governs maximum transfer block size. Windows defaults to 64k, and your setting probably should be set at 256k. Go here:

 

http://forums.dantz.com/showtopic.php?tid/28212/post/112080/hl/MaximumSGList/fromsearch/1/#112080

 

or here:

 

http://searchoracle.techtarget.com/generic/0,295582,sid41_gci1049896,00.html

 

for info on that setting. The Dantz forum info, I believe is inaccurate on the value of the setting, but both URL's are accurate on the reg location.

 

I am also dealing with a speed drop off issue with my LTO3 and Retrospect Server 7.6.123 (RDU 7.6.2.101) of about 30%. After extensive testing I'm thinking the slow down my be the result of the newest EMC upgrade which I applied a few weeks ago but I can't be sure. So that is a 3rd possibility for your slowdown.