Does anyone else backup over 10 Gbps?

[Xenomorph]

This is similar to my last thread:
http://forums.retrospect.com/topic/154613-optimizations-for-10-gbps-networking-or-is-this-just-the-limit-of-lto-5/

Some things have changed. Some haven't. I have even more systems connected at 10 Gbps, and I've moved to faster & more dense LTO-6 tapes and tape drives. I'm also now backing up close to 500TB of data. I find myself falling behind more and more because Retrospect itself seems to drag its feet.

Server: Windows Server 2008 R2 w/ 32 GB RAM, running Retrospect Multi Server, version 16.5.0.218.
Client: Windows Server 2008 R2 w/ 32 GB RAM, running Retrospect Client, version 16.5.0.218

The Server and all Clients are connected with 10 Gbps Ethernet.

Doing direct client to server file copies over SMB easily hits 600-900 MB/sec.

When doing a backup of the exact same files, Retrospect only copies at 60-90MB/sec (3704 to 5241 MB/min). I can have Retrospect back up to Tape, local Disk, or even a RAM drive. It never gets above 60-90MB/sec.

I can see in Task Manager that the Retrospect.exe  process uses minimal CPU or RAM.

For the latest test copy & backup, I selected four Windows ISOs, around 3.5GB each, 14GB total. I configured a 16GB RAM Disk on both Client and Server. This ensures that I'm NOT dealing with Disk I/O or Tape Drive performance issues. This is direct RAM to RAM file copies over a 10 Gbps network.

* Windows Explorer showed "0.99 GB/sec" when doing a "drag & drop" copy from Client to Server over the network. This is about what I'd expect.

* Retrospect Backup (compression off) got just ~90 MB/sec backing up from the same source (a RAM disk on the Client) to the same destination (a RAM disk on the Server) over the network. This is no better than when I back up to Tape or Disk or when backing up over a 1 Gbps network connection.

* Retrospect Backup (compression off) did get 176-226 MB/sec backing up on from one RAM Disk to another RAM Disk on the same system. Doing it like this eliminates the network completely.

Does anyone else do backups over 10 Gbps networks?

What is the bottleneck with the backup? Does the Retrospect client application hold things back? Does the Retrospect server application hold things back? Are there any tweaks or modifications that I could make to the Retrospect server or client application?

 

[DavidHertzberg]

12 hours ago, Xenomorph said:

This is similar to my last thread:
http://forums.retrospect.com/topic/154613-optimizations-for-10-gbps-networking-or-is-this-just-the-limit-of-lto-5/

....

 

So why not add this as an additional post to your last thread, and save other people's time answering the same question?  My answer, for instance, would be exactly the same.  A new post to an existing  topic makes that topic in the Forums list appear bolded, just like a new topic.

The cumulative Release Notes for Retrospect Mac do say under Engine for 16.5 "Improved: Client scanning 2x faster".  However that doesn't appear in the cumulative Release Note for Retrospect Windows, and may apply only to the scanning phase—for which there had been an Apple-created problem with Instant Scan for APFS-formatted Mac volumes which the Retrospect engineers said was going to be solved by switching to a 64-bit API to speed up non-Instant scan .  So that probably doesn't improve the actual backing up phase speed for Retrospect Windows.

Lennart_T suggested in a PM that the Engine should be made multi-threaded.  I responded that the Engine has been running each operation in a separate thread since Retrospect Windows 7.5 and Retrospect Mac 8, and I couldn't see how one would usefully multi-thread an individual backup operation.

[Nigel Smith]

On 10/8/2019 at 6:14 PM, Xenomorph said:

Does anyone else do backups over 10 Gbps networks?

Yes, but only with 1Gbps clients in the main. I use it more so I've the network capacity to parallelise operations rather than to speed up a single op like you. And I run on a Mac -- not a particularly well specced Mac, either... That said, my single-op speeds are comparative to yours and Retrospect transfers at less than half the speed of a Finder copy.

But...

If I set up a second share on the same NAS as another source, script that to back up to a different set, and run both that and the test above at the same time, the transfers are almost as fast on each as they were on the single (ie I'm now hitting constraints on the NAS-read and/or RS server).

My totally-pulled-out-of-a-hat theory is that each RS activity thread has a bottleneck which limits op speed to what you are seeing, probably server hardware dependant. Think something like "all of an activity thread's ops happen on a single processor core". So a server with a pair of 4-core processors would only be reporting maybe 20% usage, but that is 7 cores barely ticking away while the RS activity thread is running at 100%, and constrained, on the eighth. But it could equally involve the buffer (as I understand it, an RS backup is repeated "read data from client into buffer til full, write data to disk from buffer til empty, do housekeeping, repeat") or any number of things I'm not qualified to even guess at!

If you can, try splitting your multiple TBs into separate "clients" backed up to separate sets, and see if it makes a difference. Otherwise you may just have to accept that you've outgrown RS, at least in the way you're currently using it, and will have to think again.