Retrospect 13 - slow and continuous 100% CPU uitilisation

[jelockwood]

I have used Retrospect from version 1 literally and way back then we used TEAC data cassettes. I used it through to version 8 - aka. the version from hell. I tried a few updates for version 8 but then like many, many former Retrospect users abandoned ship.

 

Ignoring the initial significant reliability issues in version 8, the big, big disappointment was that it made zero improvement in performance. Remember this new version was released at the same time Apple switched from PowerPC to Intel so I had been using version 6 on a PowerMac G5 with 2GB of RAM and a SCSI AIT5 tape drive, and then switched to a Mac Pro with twelve CPU cores and 12GB of RAM and an eSATA connected hard disk as the backup medium. By all measures the Mac Pro should have been massively faster both for CPU and IO and have far more RAM to work in. Despite this there was as mentioned zero performance improvement. (Version 8 was I believe an Intel native and 64bit memory capable app.)

 

I am currently looking for a new backup solution so I am evaluating solutions and this includes looking at Retrospect version 13. While the user interface is mostly unchanged compared to version 8, it does appear more polished and the software in the so far very limited testing does seem more stable, however the performance is still god awful slow. It is also still as version 8 constantly running a single CPU core at around 100% utilisation.

 

So there are two problems.

 

1. Why on earth does a mere backup program thrash a CPU core at 100% all the time. This staggers the mind. A backup program like Retrospect is most of the time reading and writing and should not need to do vast amounts of computation for that, as I watch it right now it is doing a compare task and even this is thrashing it at 100% constantly for hours on end.

 

Note: Since the CPU utilisation is constantly around 100% it would seem the bottleneck is not disk IO, nor network IO but unbelievably CPU!

 

2. As above I previously tried Retrospect 8 on a 12 core Mac Pro and back then also found Retrospect would thrash a single CPU core at 100% constantly. So the second issue is that Retrospect 8 and clearly also still Retrospect 13 is obviously failing to take any advantage of the additional CPU cores. This is despite the fact that the backup process would on the face of it be one that lends itself well to splitting tasks across multiple CPU cores, for example the tasks of scanning, reading, de-duplication, compressing, encrypting, writing and cataloging would for the most part seem to lend themselves well to being split across multiple threads and therefore across multiple CPU cores.

 

Note: I am aware that for any single file the above tasks might have to for the main be run in serial, but when you are as always backing up multiple files, many of these tasks could be run in parallel with the next/previous files.

 

 

 

While Archiware P5 has already been evaluated and rejected it was massively faster than Retrospect 13 even when running on the same test system and backing up the same test data. I am not surprised many Mac sites switched to it after the debacle of version 8 of Retrospect. I did find the user interface of P5 disappointing but it did the job although for other reasons as mentioned we have rejected it.

 

 

I would appreciate comments/suggestions regarding Retrospect. I would even more appreciate it if Retrospect redesigned what still seems an appalling inefficient engine and delivered real improvement. 

[Lennart_T]

Have you turned on data compression in software?

 

I have used Retrospect 9 for years and now on version 12 and I don't see the CPU usage you see. I backup to Disk Media sets with data compression OFF.

 

Each execution thread uses one CPU core each.

[jelockwood]

Have you turned on data compression in software?

 

I have used Retrospect 9 for years and now on version 12 and I don't see the CPU usage you see. I backup to Disk Media sets with data compression OFF.

 

Each execution thread uses one CPU core each.

 

 

I did have compression turned on and have since turned it off I will double-check the CPU utilisation. OK, it has just done another incremental backup with compression turned off but with encryption turned on. I have also disabled de-duplication as much as possible. It is still hitting 100% CPU utilisation plus or minus 10% as it fluctuates. By the way AES encryption is the same encryption used by FileVault2 and is a function built-in to i5 and i7 CPUs which is why on Macs fitted with such CPUs the overhead of a FileVault2 drive is minimal and does not thrash the CPU at 100% constantly. So Retrospect should not be thrashing the CPU if this is down to the encryption I have enabled.

 

Note: I chose 128bit AES encryption the same as used by FileVault2.

 

I am using a Mac mini 2014 with a 3GHz i7. This is a single i7 CPU with two cores, plus it appears it supports hyper-threading giving the effect of four cores. It has 8GB of RAM and a 256GB SSD boot drive and is connected to the backup drive - a Promise Pegasus2 R4 via Thunderbolt 2.

 

I am aware that in Retrospect each 'task' can be assigned to its own CPU core so that if you are running two simultaneous backups each could have its own CPU core. This however I feel is still a pathetically inadequate solution.

1. It only works if you are doing more than one simultaneous backup
2. Even for that it requires an administrator to configure which thread to assign each backup task to
3. It does nothing to help spread the load of a single backup which is what I feel is the far more common workflow i.e. backing up a series of drives to a single tape/disk

As I detailed I feel there is plenty of opportunity to spread the individual sub-tasks across different threads which should then automatically pick their own CPU cores.

[jelockwood]

A further update. Upon reflection I wondered if the fact the initial backup having been compressed meant it was still having to do a lot of decompression to do file compares for subsequent incremental backups. I therefore have recycled the backup i.e. wiped it.

 

It is now running the equivalent of a first initial backup again. I can see that currently with compression turned off but encryption still turned on that the CPU utilisation of RetrospectEngine is fluctuating between 70 and 95%. So it looks like turning off compression has made a very small improvement.

 

Since no matter what Retrospect might think these days it is essential to use encryption to secure backups and since as I pointed out AES encryption should be possible with hardware assistance via the i7 CPU I would still say the performance of Retrospect is appallingly bad.

 

See https://software.intel.com/en-us/articles/intel-advanced-encryption-standard-instructions-aes-ni

 

When Retrospect 8 was launched and obviously more so for earlier versions Intel i series chips did not exist. So it would be reasonable to expect that originally AES encryption was having to be done purely in software. This of course has not been true for years, it would seem Retrospect have failed to update their software to implement even this basic and obvious improvement.

[Mayoff]

I would not expect compression or encryption to have this type of impact on CPU utilization.

 

1) If you check the OS X Activity Monitor, which exact Retrospect process is using 100% of the CPU?

 

2) Have you tried to disable Instant Scan in the Retrospect preferences?

 

3) Which verify type are you using? Through verification or media verification?

 

4) AES 256 may actually be faster then AES-128 due to optimization code being used.

[jelockwood]

I would not expect compression or encryption to have this type of impact on CPU utilization.

 

1) If you check the OS X Activity Monitor, which exact Retrospect process is using 100% of the CPU?

2) Have you tried to disable Instant Scan in the Retrospect preferences?

 

3) Which verify type are you using? Through verification or media verification?

 

4) AES 256 may actually be faster then AES-128 due to optimization code being used.

 

 

1) It is the RetrospectEngine process

2) It is still in the middle of the initial backup - remember I recycled the media to remove all traces of previous use of compression, I did find the Instant Scan option for the source network client and it was enabled, I have turned it off, it has made no difference yet but as mentioned it is already in the middle of doing a backup.

3) I currently have it set to NO verification

4) I would have to completely destroy and recreate the mediaset to switch to AES256 but both AES128 and AES256 should be able to be done via the Intel CPU routines and therefore neither should cause this level of CPU utilisation.

 

I strongly believe Retrospect is still doing AES purely in software which has not been necessary for years. Using FileVault encryption again as a comparison I believe Apple say that with Intel routines i.e. running on a suitable Mac then FileVault encryption only adds a 5% CPU overhead, whereas doing the same encryption on an older CPU without Intel routines to offload the work it would be about a 20% overhead - still a lot less than 100%.

[jelockwood]

Ok, I deleted and created a fresh new unencrypted media set. So now there is no compression, no encryption, no verification, client fast scan is turned off as suggested, deduplication is as much as possible turned off.

 

Even network client traffic encryption is turned off.

 

I am now running an initial backup and it (RetrospectEngine) is once again taking as near as damn it 100% CPU. This echoes my experience back with Retrospect 8.

 

Obviously being an initial backup there is no grooming happening.

 

 

"Something is wrong in the state of Denmark."

 

Shakespeare's Hamlet - Hamlet (1.4), Marcellus to Horatio