Ramon88 Posted June 9, 2009 Report Share Posted June 9, 2009 Our setup is as follows: Remote location backup server: - Windows 2008 STD server on a Xeon Quadcore with 8GB RAM 2TB RAID storage - Retrospect Multi Server 7.6.123 Local storage server: - Windows 2003 Server on 'green' server platform (T7200 C2D 2GB RAM 4TB RAID storage) The remote server backs up 16 servers at that location to its 2TB RAID without performance issues. Afterwards it pushes two 'Duplicate' scripts of the stored backups to the local storage server. The network connection used is a fiber optic leased line, connecting the remote and local server with a speed of 100Mbit full duplex. When we manually copy (using RDP) the remote storage (desktop drag and drop) to the mounted share on the local storage server we get 100% network performance (about 12MB/sec). However when we let Retrospect do the job with 'Duplicate' (or ' Backup') network performance only peaks at 25% quadrupling the time needed. On both machines resources seem to not be a problem and there is enough free RAM. CPU load peaks at about 8% (mean about 4%) at the local storage server and 4% (mean about 2%) for the remote server. I'm suspecting the NIC at the local storage server, but than why is the performance good when doing a regular copy? Any ideas? Quote Link to comment Share on other sites More sharing options...
robvil Posted June 10, 2009 Report Share Posted June 10, 2009 it could be a MTU issue. Robert Quote Link to comment Share on other sites More sharing options...
rhwalker Posted June 10, 2009 Report Share Posted June 10, 2009 I suspect that the problem, instead, is because of windowing. I believe that Retrospect uses UDP datagram packets during backup chit-chat between the client and the server, rather than windowed TCP/IP packets. If true, then latency is what is doing the damage, as the next packet is not sent until the previous one is ACK'd, rather than having multiple outstanding packets in the window as for TCP/IP windowing. You probably could see this by using a packet sniffer such as Ethereal (now WireShark) on port 497. Retrospect may need some tweaking for efficient use in a non-LAN situation rather than the LAN environment for which it was designed. Russ Quote Link to comment Share on other sites More sharing options...
robvil Posted June 12, 2009 Report Share Posted June 12, 2009 UDP do not use ACK. and sequence. Only TCP does. Quote Link to comment Share on other sites More sharing options...
rhwalker Posted June 12, 2009 Report Share Posted June 12, 2009 UDP do not use ACK. and sequence. Only TCP does. Yes, of course. UDP leaves reliability and sequenced delivery to the application layer, rather than handling it in the protocol, likewise for any pipelined transmission (a la TCP windowing). But that still has to be done (acknowledgment, error checking/retransmission, and optimization such as pipelining ("windowing")), and Retrospect's handling of it may not be optimal. That's why I suggested use of a packet sniffer, which would also show whether the packets are being split/reassembled because of MTU issues, etc. Russ Quote Link to comment Share on other sites More sharing options...
robvil Posted June 12, 2009 Report Share Posted June 12, 2009 you stand correct. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 12, 2009 Author Report Share Posted August 12, 2009 Okay, we've now installed a client at the remote site and installed retrospect server directly on the receiving end. Between the machines there is only switches for conversion from copper > fiber and vice versa. Some test results: - Drag and drop copy (so not using Retrospect) in Windows explorer causes ±100% network utilization. - "Pull" backup via a share causes ±40% network utilization. - "Pull" backup via remote client causes ±18% network utilization. The performance graph is also very jumpy (al lot of higher/lower jumping fast. CPU is not the problem here. Readouts remain low. Next stop will be to test if performance is better through the other NIC and not direct from the fiber > copper conversion switch but through our main switch... Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 13, 2009 Author Report Share Posted August 13, 2009 Other NIC and routing through main switch yielded the same unsatisfactory results. It seems like the server itself is throttling back the data stream. As we can 'stack' streams performance improves. We're going to test running Retrospect from another dedicated machine which isn't a server but a regular Vista installation. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 13, 2009 Author Report Share Posted August 13, 2009 Running from the new Vista based dedicated machine improves performance a lot. It's now about two-thirds the speed we get when copying through Windows explorer (drag & drop) instead of one-thirds. Relatively speaking this has doubled the backup performance, though we need to do some more testing and fiddling to see if we can improve a bit more. Quote Link to comment Share on other sites More sharing options...
Petek Posted August 13, 2009 Report Share Posted August 13, 2009 The network connection used is a fiber optic leased line, connecting the remote and local server with a speed of 100Mbit full duplex. What is the latency between sites? It sounds like a socket window size issue to me.. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 13, 2009 Author Report Share Posted August 13, 2009 Latency is only ±3ms. But we think it's either a driver issue or just the way Retrospect handles it's traffic, like Russ suggested. We are going to look into optimising things like MTU. Either way, it looks like our Vista setup works a lot better. But the hardware used is much faster too. So it's a bit hard to judge exactly what caused the improvement. For now I'll settle for ±440 MB/min (in Retrospect) we can squeeze through. That leaves ample bandwidth for the devs when they access the remote infrastructure. Quote Link to comment Share on other sites More sharing options...
Petek Posted August 13, 2009 Report Share Posted August 13, 2009 And of course its actually UDP traffic isn't it so the TCP socket window is presumably irrelevant. Does beg the question why you need to open TCP and UDP 497 if it doesn't use TCP. Might be worth looking at the socket window sizes and giving it a try. For a customer taking a 100Mb/s connection over a circuit with 3ms round trip delay =100,000,000 * .003s =300,000 bits /8 socket buffer size = 3 * BDP =112,500 Therefore the TCP window would need to be 112,500 or greater XP's default is 64,512 bytes and I guess Server 2003 would be the same. I wonder if Vista is larger by default, therefore you are seeing the improvement? Quote Link to comment Share on other sites More sharing options...
rhwalker Posted August 13, 2009 Report Share Posted August 13, 2009 Latency is only ±3ms. But we think it's either a driver issue or just the way Retrospect handles it's traffic, like Russ suggested. Well, I've been corrected. Apparently Retrospect only uses UDP for client discovery, and all backup is done using TCP protocol. Could still be an MTU or windowing issue. Retrospect uses only TCP protocol for backup Packet capture on port 497 might reveal some insights into what is happening. Russ Quote Link to comment Share on other sites More sharing options...
Petek Posted August 13, 2009 Report Share Posted August 13, 2009 Right, well in that case, its almost certainly a TCP socket window issue, and indeed with a 3ms latency at 100meg, the TCP socket needs doubling to achieve full throughput from standard Windows stuff.. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 13, 2009 Author Report Share Posted August 13, 2009 Thanks Russ, that is useful information. I've asked one of the engineers to look at this, but he too suspected MTU issues. We need to test this, but the link is also used extensively by our developers for production, so we need to be careful a bit (especially because it sort of 'works' at the moment *lol*). As far as I know Vista/2008 and OSX handle MTU differently. And different hardware (ethernet controllers) can have different results with Retrospect as well. Quote Link to comment Share on other sites More sharing options...
robvil Posted August 14, 2009 Report Share Posted August 14, 2009 Vista/2008 has Window Auto-Tuning enabled by default and is a new feature. Robert Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 14, 2009 Author Report Share Posted August 14, 2009 I knew OSX had this for sure, but this would explain the better throughput (the source server is 2008 based)... Still, it's not as good as what Windows itself can do. However googling around seems to suggest people are still fiddling around with their Vista's MTU-settings. Quote Link to comment Share on other sites More sharing options...
Ramon88 Posted August 28, 2009 Author Report Share Posted August 28, 2009 Just tested using a Retrospect Client Agent for the remote 'source' server. I got a 'staggering' 20MB/minute out of it. Compared with the UNC share method that does about 440MB/minute that's a really big difference. Only scanning preparations are much quicker. As the Dutch saying goes: "It doesn't even dent a stick of butter". We'll settle for the UNC method for now. Maybe in the future Retrospect will improve its track record for this kind of connection. We'll see it when it happens. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.