duplicate state information - turn that setting off?

[redleader]

'duplicate state information'

I cannot see where to turn this off.

I think I do have it off 😮

 

It adds soooo much time to a full backup run.

I'd rather live without it set especially over  remote backups.

[Nigel Smith]

2 hours ago, redleader said:

'duplicate state information'

Weird -- that's an option in the Windows client section. You could try turning it off there and seeing if it makes any difference.

Version of RS server, version of client, and OS version of both would be a help here. Also info about the destination -- you're not trying to "copy" to a Windows share, are you?

[redleader]

Thank you Nigel, and apologises for forgetting specs:

All Macs OSX Mojave 10.14.latest
Retrospect 17.0.latest
for both Client and Server

Copying from Mac HD Client to Mac HD Server.
Either on the LAN or over the WAN.

Nothing Windows involved.

[Nigel Smith]

Give that Windows client setting a go anyway -- IIRC, Lennart found some unexpected "leakage" between client settings a while back. Or it may just be mis-reporting its activity and is actually setting ACLs etc on the copied files.

Possibly a dumb question -- but why a copy operation rather than a backup? That would probably be a lot faster, especially since it wouldn't have to reset metadata on each of the thousands of small files in your Library folder (which, I suspect, is the problem).

[redleader]

18 minutes ago, Nigel Smith said:

Possibly a dumb question -- but why a copy operation rather than a backup? That would probably be a lot faster, especially since it wouldn't have to reset metadata on each of the thousands of small files in your Library folder (which, I suspect, is the problem).

We don't have the space for incremental backup ... 30% of the files change almost daily ... it would soon fill even large drives.

A backup Worm might work, but I;ve never figured that one out yet.

[Nigel Smith]

So are you doing a Copy that replaces everything with what's "currently" on the MacBook (and other clients), or just replacing the ~30% of files that have changed and leaving the unchanged files in place?

Is it 30% of files, or 30% of data? Are you applying any filters? Are you, perhaps, backing up huge numbers of small cache files that you don't really need to copy, or some big files which actually only change in small ways (databases, VM images) that could be handled better? Could you just add a couple of external USB drives to the server and increase your capacity? If you could get the stage where you can store all the weekday incrementals, groom the set down over the weekend, then start again with incrementals the next Monday, that would probably make your life easier.

Lots of very small operations, like one-by-one file metadata updates, can be ridiculously slow over a remote link. The more you can minimise them the happier you'll be.

[DavidHertzberg]

redleader,

I was going to ask the same question that Nigel Smith did in the second paragraph of his second post, but he beat me to it while I was out buying apples in my local farmers' market (the apple-growing farmers' employees, who are mostly Tibetan immigrants, bring the apples from upstate New York—rather than growing them in Manhattan).  And now you've provided the answer in your latest reply.

A few hours ago I started to create a test Copy script;  Back up System State—described on page 101 of the Retrospect Mac 17 User's Guide—is automatically checked under Source->Windows on the Options tab.  As Nigel Smith suggests in his first post, you might try unchecking that option.  I've done some Forums searches, but I can't find any post by Lennart_T that "found some unexpected 'leakage' between client settings a while back".  If—after swearing on the head of Her Majesty QE II (which I'm sure your fellow Brit would insist on 🤣) that Users on Michael's MacBook Pro doesn't contain any Windows files—you want to create a Support Case for a bug, here's why and how to do it.

Since  MB MICHAEL 2018 seems to be a dedicated "backup" folder on your "backup server" FS SERVER DATA 02, you might instead consider replacing the copied files and folders with a Media Set on that "backup" folder to be the daily Destination using the Recycle media action on a Backup script.  I actually do that every Saturday morning for my MacBook Pro "client", except that my Destination is inside the "Retrospect" folder on a portable HDD (which, after 6 subsequent incremental daily backups using No Media Action, I carry off to a week's holiday in my bank safe deposit box before bringing it back to sit for a week in my apartment before being used again). 

Nigel Smith has now suggested in his third post, which he just sneaked in while I was taking a long phone call, what I do.  The Saturday morning Recycle Backup of my MBP backs up 71.1GB over 3.3 hours, not including a 0.7 hour compare phase (which I do because I'm a worrywart who used to back up to tape).  That's 739K individual files, not your piddling 37K.  I don't mess with grooming, because having 2 complete backups of my MBP—one (offsite) less than a week old and one less than 2 weeks old—to supplement the portable drive cabled to my "backup server" is adequate for my needs.

 

[Nigel Smith]

1 hour ago, DavidHertzberg said:

I've done some Forums searches, but I can't find any post by Lennart_T that "found some unexpected 'leakage' between client settings a while back".

It was something about backing up (or to!) a NAS volume and seemed weird at first but obvious in (hah!) retrospect. Using Windows settings because it was an SMB mount? Using Mac settings because, although it was a Windows server, it was mounted on a Mac?

Whatever it was, red's posted status message is pretty Windows specific and while I don't hold out much hope changing that client setting will help, it won't hurt either -- so even a tiny chance of success makes it worth a try.

[DavidHertzberg]

4 hours ago, Nigel Smith said:

It was something about backing up (or to!) a NAS volume and seemed weird at first but obvious in (hah!) retrospect. Using Windows settings because it was an SMB mount? Using Mac settings because, although it was a Windows server, it was mounted on a Mac?

Whatever it was, red's posted status message is pretty Windows specific and while I don't hold out much hope changing that client setting will help, it won't hurt either -- so even a tiny chance of success makes it worth a try.

Was it the thread in which this post is the OP?  The only post in that thread that isn't by the OP or me is this one by mbennett, but he's from Fair Grove MO—not Helsingborg in Sweden—so he's not Lennart_T.  What the OP was reporting doing in that thread was indeed weird, though—and obsolete.

[redleader]

Hi NIgel, Thank you,

I tend to do a backup in the office on the much faster LAN as often as they are in the office, but with COVID most employees are shelereing at home, and so I am relying on 'copy'* [new or changed files only] as it's over some slower Broadband WAN, some staff are on 8Mb down and 23 Mb up 😮 !!

I am filtering our MacBook USER folders such as Music, Photos, Downloads, iMail, Caches...

 

We do have some external drives attached, for onsite and offsite [removable] use.
Just so we don't have One backup only 😮 🙂

And some in the OneDrive or DROPBOX , etc ... Cloud. 

 

We do have 500k to 2.5M files in these copy/Archives we are 'updating'* our backups to.

Yes, there are a lot of small files for local Server HD to External HD; 30Gb to 190Gb on a Wed, Sat, Sun.

 ...but not so? for the USERs MacBooks HD over the WAN to Server HD's; 1Gb to 4Gb bi/tri-weekdaily.

 

Hi David, Thank you,

I must look more and do some knowledge/training tests into 'using the Recycle media action on a Backup script'.

We do have Mac Sharing on this Mini-Mac Server running Retrospect Server.
I have that set to SMB [as AFP sems rather unstable] so maybe that is where the Windows processes are being recognised?

 

----------------------------------------------------------------------------

 

I guess we just have a lot of files!
And Im do a lot of copies on.off site... because I worry 🙂

But saving time on comparing file states or thorough verification is the best help or time-saving I can see at the moment?

 

[Nigel Smith]

10 hours ago, redleader said:

I tend to do a backup in the office on the much faster LAN as often as they are in the office, but with COVID most employees are shelereing at home, and so I am relying on 'copy'* [new or changed files only] as it's over some slower Broadband WAN, some staff are on 8Mb down and 23 Mb up 😮 !!

Sure -- what I'm trying to understand is why you've chosen Copy rather than Backup, where you'd do the first backup in the office then the much smaller incrementals over slow connection. It's faster to record the file metadata in the catalog than it is to recreate it on the server (if that, indeed, is what the bottleneck is).

There are reasons to choose Copy -- you want to replicate the user's data on your office server so others can use it, for example.

30% churn still seems awfully high, though (unless your situation is of people not having many files, but changing those they have frequently?). I'd take a good look at what is actually being backed up and check the filters are working as expected.

10 hours ago, redleader said:

I must look more and do some knowledge/training tests into 'using the Recycle media action on a Backup script'.

I'd avoid Recycle at the moment. That would mean starting all over again every so often, so all the data would have to be transferred again over the slow, remote, connection. The thing with Grooming is that it saves space on your target disk by throwing away the "unwanted" data (eg previous versions of a file) but keeps what's current, so the next backup is just another incremental, with the much smaller transfer that implies.

"Resetting" with a Recycle is great for David, with his high-speed LAN, but will probably hurt you more than it helps.

Start with defining what you are trying to achieve. (Backing up user data? Replicating it? Just "Users" or the whole volume? Etc, etc...). You can get similar effects with different methods, but there's usually one way that's more suitable than the others.

[Nigel Smith]

16 hours ago, DavidHertzberg said:

Was it the thread in which this post is the OP?

Nah, it was this one. Mis-remembered and not really relevant, I guess.

[DavidHertzberg]

redleader and Nigel Smith,

I just finished running a test, in which I did a Copy of the entire contents of my MacBook Pro's SSD to a folder—defined to Retrospect as a Favorite Folder so it can be used as a Destination—placed to use the spare space on the portable HDD I'm going to use as a Destination for my weekly Recycle Backup tomorrow.  The Copy script was defined as redleader's is—specifying Copy only missing files; however—since the destination folder is empty—it was in fact copying all files from my MBP's SSD.  I predicted the Copy script's copying phase would take almost exactly the 3 hours 21 minutes that the copying phase of my Recycle Backup of the same data took last Saturday morning.

The copying phase took 3 hours 40 minutes, followed by a Closing phase that went on for 6 minutes before I killed the run (so the MBP's Friday a.m. incremental Backup would run).  That may have been the  "duplicating state information" phase that redleader has been getting, but I'm running Retrospect Mac 16.6 instead of 17.0.  There were 165 "Map error: unknown Mac error 22" messages shown or summarized in the script's log.

I had expected that the speed of the Copy script's copying phase would be be approximately the same as the 371MB/min. of the equivalent Backup copying phase last Saturday.  In fact it was lower—325MB/min..  Why, you may well ask, is the speed of that copying phase so much lower than the speed of what Nigel Smith has described as my "high-speed LAN"?  After all, the net throughput of the MoCA 1.1 adapters at either end of my in-the-wall inter-room coax link is 175Mbit/sec—with all intra-room hardware rated at 1Gbit/sec.; assuming 10 bits per byte to allow for overhead, that should allow 1.05GB/min..  Why is my speed 35% or less of what it should be?

My hypothesis is described in this post in a 2017 thread.  It was validated by this later post in that thread, and by the next 2 posts that followed it.

 

[redleader]

My Closing phase   goes on for (I would guess here...) 40%-60% as long as the backup time, then it is followed by a duplicating state information process 10%-15% as long as the backup took.

If there are files on the destination that are not on the source it reports errors (in the case of my FileMaker incremental backups folder 'sync/copy' Script that's in the 000's of file errors.)

  

[Nigel Smith]

According to an old post by Lennart, the closing phase of a Duplicate/Copy script involves setting permissions on all the target files, not just the ones copied over in that run. If still true that could explain what you're seeing -- and be another push towards doing "proper" backups rather than using Copy.

But "Closing..." includes a bunch of other stuff too, including catalog updating, and doesn't let you know which "sub-task" is taking the time. You might learn more by turning up the log levels (long, and high variable, "Closing..." times have been a regular feature on the Forum over the years, and I don't think they've ever been satisfactorily explained/resolved).

If I ever get my codes for RS17 (grindingly slow procurement process here) I'll try a similar test to David's.

[DavidHertzberg]

redleader and Nigel Smith,

There surely can't be any Catalog updating in the run of a Copy script, because that script doesn't designate a Media Set as a destination—no Media Set means no designation of a Catalog.  That's why I thought my test might run faster than the equivalent Recycle run of my Backup script.  In fact its copying phase ran slower. Maybe cramming copies of multiple source files into a single .rdb file is faster than adding each copied file to the macOS HFS+ filesystem, but I'm not inclined to investigate it.

There weren't any files in the destination folder.  I had deleted all those that were copied there by a test run I killed after 5 minutes, when I noticed the script mistakenly specified Copy all files (which wouldn't require name comparison) instead of Copy only missing files—which redleader specifies.

In any case, my test proves that redleader could get his backing up done faster with Backup scripts.  He could also use the resultant Catalogs to do grooming as Nigel Smith suggested.  I don't bother with grooming,  since  I must—because I've experienced multiple cases of water leaks from an apartment two floors above mine—swap a portable HDD containing complete-as-of-Friday-morning backups of all my drives off-site once a week.

P.S.: On pages 120–121 of the Retrospect Mac 16 User's Guide, under item 6. there are 5 paragraphs following this single-sentence paragraph:

Quote

The Destinations tab has a pop-up menu with several copying options. Choose the option you want:

that describe not only the pop-up that redleader shows he chose in the screenshot in this up-thread post, but each of the other pop-up options.  Those 5 paragraphs have been deleted from page 110 of the Retrospect Mac 17 UG—evidently by the StorCentric Slasher (my name for him 🤣 ).

Edited November 9, 2020 by DavidHertzberg
P.S. Paragraphs describing each of the Destinations pop-up options that were in the Retrospect Mac 16 UG have been deleted from the Mac 17 UG.

[DavidHertzberg]

All backup administrators,

What I've described in my post directly above's P.S. is the second case I've found —here's the first—of option-description paragraphs that were in the Retrospect Mac 16 User's Guide being deleted from the Mac 17 UG.  Therefore I've now submitted Support Case #76445, and I've sent an e-mail to the Worldwide Director of Sales expanding on and analyzing the Support Case.  I don't want to commit the sin of double-posting, so you can see my analysis in the P.S. of my post about the first case.  That analysis isn't charitable to the corporate motivation behind  the StorCentric Slasher's acts.