Increase Retrospect's Buffer Size?

[JBucanek]

Hello all,

 

 

 

I have a Mac OS X Server, running Retrospect Workgroup Server 5.0, backing up to a SCSI tape drive.

 

 

 

I've got a really annoying performance problem when backing up computers over the network. It appears that the network just isn't quite fast enough to feed the tape (2MB/sec) continuously. So after writing a bit, the tape stops, rewinds a little, repositions, then starts writing again.

 

 

 

This is as one would expect. But the problem is that not more than a second after the tape stops and begins backing up, Retrospect stops reading from the network. It's acting like it's internal buffer has filled up and it's not going to read anymore until it can empty some of the buffer by writing to the tape. The problem is that the buffer seems to be fairly small (a few MB), so when it does start writing to the tape again the buffer quickly empties and whole mess starts over.

 

 

 

The net result (no pun intended) is that backups from the local HD run at about 100+MB/min, while backups over the network run at less than 40MB/min[1]. Looking at the network traffic through the router, the network is idle almost 50% of the time. So, in theory, I should be able to backup my network computers at close to 80MB/min -- if only I could get Retrospect's buffer big enough to stop playing musical chairs with the network.

 

 

 

Is there any way of doing this? I have *gobs* of RAM.

 

 

 

James

 

 

 

[1] Not to mention the additional wear and tear on my tape drive.

[IrenaS]

At this time, no, there isn't a way to increase Retrospect's buffer. It's possible that we may be able to add such a feature in a future release.

 

 

 

Thanks,

 

 

 

Irena Solomon

 

Dantz Tech Support

[Sheppard]

I don't know how R's buffer system is designed. In other systems I've used, the buffer is constantly being filled from the network at the same time it is being drained to the media. If you're using a backup device that is much faster than the network/client combination, then the backup will always stall when the buffer empties because it can't be filled as fast as it's drained. There's often hysteresis applied to optimize both the reading and writing.

 

 

 

To minimize the stalls, the buffer needs to be increased in size. Many systems adapt dynamically and adjust the buffer size as required to both minimize memory consumption and frequency of stalls. Limits have to be applied, of course, to prevent the buffer from growing too large.

 

 

 

Data may also be read and written in blocks, and not as a byte stream, which adds another level of complexity. I would expect the numerous devices supported by Retrospect have quite a variation.

 

 

 

I certainly sympathize with the wear and tear on your tape drive. I used to use DAT and decided to move to DVD-RAM (which is fine in 4.3 but problematic in 5.0). I don't care about wear and tear anymore. The other big advantage for me is that restoring from a random access device is much faster than tape, particularly if you have a lot of incrementals. But choice of backup medium is another topic... :-)

[impala]

PLEASE PLEASE PLEASE!!!!

 

 

 

I've found retro 4.3 to be unusable with an AIT tape drive and network clients because the drive is so much faster than the clients. DLT handles the slow client problem much better by adapting it's own buffer, but still...

 

 

 

We have backup servers with gobs of RAM, gobs and gobs of fast free local disk space, but retrospect behaves like it's running on a Mac IIci with 8MB ram because it doesn't buffer network clients adequately. I was hoping retro 5 had improved this situation. Sounds like NOT!

[Sheppard]

impala: I've found retro 4.3 to be unusable with an AIT tape drive and network clients...

 

 

 

I'm aware of a company that had great success using 4.3 with AIT 1 autoloaders on over a hundred servers with 10,000 clients. Some servers could incrementally back up 120 to 140 clients a day including a mix of desktop and portables and still have some idle time. Full backups after a tape rotation took a couple of days to complete.

 

 

 

Of course I'd still like to see multiple streams from more than one client and NIC at a time so that the backup medium becomes the bottleneck and not the client and/or network. Perhaps in Retrospect 10?

 

:-)

 

 

 

...Tom

[impala]

In my opinion, AIT would probably be great if it is saturated with data. If it has to wait for data, look out!

 

 

 

The problem I have with the AIT drive is media errors. Yes, I tried different media. Yes, I cleaned it. It could happen on new media! The drive is like new because shortly after getting it I inherited an old DLT 4000 drive which I like much better. I think the media errors come from the frequent pauses the drive is forced to endure while retrospect downloads data from clients. I think the frequency of the pauses has more to do with it than the length of the pauses. I have no problem with the AIT drive as long as I'm doing a local server backup, or an immediate full backup of a faster net client. All notebook clients are in the slow category. I agree that a multi-threaded system would be the best, but increasing retro's buffer should be a quick and easy fix. Rewriting retro for multi-threaded operation would be a major task. Legato is multithreaded and costs big big bucks! I can't afford multithreaded backup. But I can feed cheap RAM to a retrospect buffer. The best situation would be to give a user-defined option for the size of a RAM cache, like Adaptec TOAST gives. I can use a 32MB ram buffer in TOAST and burn a cd-rom from an appleshare!

 

 

 

If I had known the AIT drive was a cousin to DDS I never would have gotten one.