Jump to content
JonHendry

Can't exclude build folders (in preview)

Recommended Posts

I'm trying to get Retrospect 8.2 set up on an OS X server.

 

I would like to exclude Xcode build folders, but it isn't working.

 

A rule consisting only of an Exclude rule for "folder named build" is apparently ignored. If I preview a backup, using only that rule, the contents of any build folder are marked for backup.

 

Edit: Apparently the problem is in Preview, not actual backups.

 

But I am wondering why Retrospect is taking 40 minutes (and counting...) to build a snapshot of a 27 MB backup that took under 8 minutes to complete.

 

 

Edited by Guest

Share this post


Link to post
Share on other sites

Which version of xcode do you have installed and what version of OSX server?

 

Somebody posted a while back that building a snapshot seemed to take forever and they indicated they had xcode installed, but never said what version.

Share this post


Link to post
Share on other sites

The client was a MacBook Pro running 10.6.4 and not busy doing anything else. The server is running 10.5.8. The server is a Mac Pro with 10GB RAM, and doesn't do much of anything but run Retrospect. It's backing up onto a RAID-5 array hooked up via eSATA.

 

The version of Xcode on that client is 3.2.3

 

I noticed that the engine didn't seem to be doing much while it was supposedly building the snapshot. CPU usage was low. Next time I see it in a similar state I'm going to run Shark or Instruments on it and see what it's doing.

 

Perhaps the snapshot is done but the Console isn't being told.

Edited by Guest

Share this post


Link to post
Share on other sites

Just a guees, see below.

Isn't the snapshot being built on the client? (So the engine merely waits for the client to send the finished data for the snapshot.)

 

On our Windows Retrospect server, there is practically no activity when it is "building snapshot". That's why I think (part of?) the snapshot is built on the client.

Share this post


Link to post
Share on other sites

No, snapshots are built on the engine using client information.

 

I'm thinking this must be related to having installed some specific version of xCode. (Or having upgraded xCodes over time)

 

I tried to replicate it with a 10.5.8 client and the current -- at the time -- version of the iOS xcode and could not.

 

My guess is that some version of xcode installs some hard links somewhere and that's the problem.

 

It would be nice to isolate this if you could spend the time on it by trying to isolate this to a specific "Favorite Folder".

 

 

Share this post


Link to post
Share on other sites

I'll try the Favorite Folder thing when I get a chance.

 

In the meantime, here's timing info for two machines. Both are running 10.6 with Xcode installed. The first is a Mac Pro, quad 2.66GHz Xeon with 3GB RAM . The second is a 2.4 GHz 2007 MacBook Pro with 4GB RAM.

 

+  Normal backup using MouseRigTestBackup - 10/1/10 6:20 AM at 10/1/10 (Activity Thread 2)
   To Media Set Rig Disk Backup Set...
   -  10/1/10 6:22:08 AM: Copying Snow Leopard on Mouse-Rig-Test
   10/1/10 6:56:43 AM: Snapshot stored, 225.2 MB
   10/1/10 6:56:57 AM: Comparing Snow Leopard on Mouse-Rig-Test
   10/1/10 7:06:36 AM: Execution completed successfully
   Completed: 90751 files, 5.2 GB, with 24% compression
   Performance: 496 MB/minute (449.1 copy, 553.8 compare)
   Duration: 00:44:28 (00:23:03 idle/loading/preparing)


10/1/10 4:15:04 AM: Connected to Hendry
   *  Resolved container Hendry to 1 volumes:
   Hendry HD on Hendry
   -  10/1/10 4:15:02 AM: Copying Hendry HD on Hendry
   10/1/10 5:08:55 AM: Snapshot stored, 207.4 MB
   10/1/10 5:09:13 AM: Comparing Hendry HD on Hendry
   10/1/10 5:09:40 AM: Execution completed successfully
   Completed: 15 files, 36.8 MB
   Performance: 28.6 MB/minute (17.2 copy, 88.3 compare)
   Duration: 00:54:37 (00:52:02 idle/loading/preparing)

Share this post


Link to post
Share on other sites

Which version of xCode?

 

And have you had other versions of xCode installed, but then upgraded?

 

Were the 10.6 machines with xCode originally *10.5* machines with xCode, but then upgraded to 10.6 and the 10.6 version of xCode installed?

 

It's information like that which will help reproduce this...

Share this post


Link to post
Share on other sites

Okay, I set up favorite folders for my Snow Leopard laptop: /Developer, /Users, and /Library

 

The test script is using a rule that should exclude /Developer, among other things, using an OS X Folder exclude rule.

 

Here are times for two runs, with nothing else happening on server or client.

 

The backup of /Developer, which copies no files, takes a huge amount of time to generate a snapshot. Is there any way to exclude /Developer, make it an un-Favorite Folder, and back up everything else?

 

   +  Normal backup using Snapshot Test at 10/14/10 (Activity Thread 1)
   To Media Set Rolling Backup...
   -  10/14/10 1:46:09 PM: Copying Developer on Hendry
   10/14/10 1:46:09 PM: No files need to be copied
   10/14/10 2:22:31 PM: Snapshot stored, 35.1 MB
   10/14/10 2:22:36 PM: Comparing Developer on Hendry
   10/14/10 2:22:39 PM: Execution completed successfully
   Duration: 00:36:30 (00:36:20 idle/loading/preparing)

   -  10/14/10 2:22:40 PM: Copying Users on Hendry
   10/14/10 2:29:59 PM: Snapshot stored, 37.2 MB
   10/14/10 2:30:04 PM: Comparing Users on Hendry
   10/14/10 2:30:20 PM: Execution completed successfully
   Completed: 3302 files, 19.3 MB, with 16% compression
   Performance: 67.9 MB/minute (64.1 copy, 77 compare)
   Duration: 00:07:40 (00:07:05 idle/loading/preparing)

   -  10/14/10 2:30:20 PM: Copying Library on Hendry
   10/14/10 2:32:53 PM: Snapshot stored, 24.6 MB
   10/14/10 2:32:58 PM: Comparing Library on Hendry
   10/14/10 2:33:01 PM: Execution completed successfully
   Completed: 40 files, 91 KB, with 0% compression
   Performance: 2.1 MB/minute (1.3 copy, 5.3 compare)
   Duration: 00:02:40 (00:02:34 idle/loading/preparing)


+  Normal backup using Snapshot Test at 10/14/10 (Activity Thread 1)
   To Media Set Rolling Backup...
   -  10/14/10 7:37:32 PM: Copying Developer on Hendry
   10/14/10 7:37:32 PM: No files need to be copied
   10/14/10 8:56:37 PM: Snapshot stored, 35.1 MB
   10/14/10 8:56:43 PM: Comparing Developer on Hendry
   10/14/10 8:56:51 PM: Execution completed successfully
   Duration: 01:19:19 (01:19:05 idle/loading/preparing)

   -  10/14/10 8:56:53 PM: Copying Users on Hendry
   10/14/10 9:05:01 PM: Snapshot stored, 37.2 MB
   10/14/10 9:05:05 PM: Comparing Users on Hendry
   10/14/10 9:05:16 PM: Execution completed successfully
   Completed: 4 files, 9 KB, with 0% compression
   Performance: 0.1 MB/minute (0.1 copy, 0.1 compare)
   Duration: 00:08:23 (00:08:05 idle/loading/preparing)

   -  10/14/10 9:05:16 PM: Copying Library on Hendry
   10/14/10 9:08:15 PM: Snapshot stored, 24.6 MB
   10/14/10 9:08:20 PM: Comparing Library on Hendry
   10/14/10 9:08:30 PM: Execution completed successfully
   Completed: 3 files, 52 KB, with 69% compression
   Performance: 0.4 MB/minute (0.6 copy, 0.3 compare)
   Duration: 00:03:14 (00:02:59 idle/loading/preparing)

   10/14/10 9:08:48 PM: Execution completed successfully
   Total performance: 0.1 MB/minute with 54% compression
   Total duration: 01:30:58 (01:30:10 idle/loading/preparing)

Edited by Guest

Share this post


Link to post
Share on other sites

Was this the *only* version of xCode you ever had installed on that machine? Or had you upgraded from previous versions in the past?

 

/Developer must have some hard links in some folder that cause the extra snapshot time.

 

I suppose the next step would be to make favorite folders *within* /Developer to figure out exactly what subfolder creates this problem and go from there.

 

Share this post


Link to post
Share on other sites

I'm doing that right now.

 

I suspect it is probably the DocSets in /Developer/Documentation. It's been chewing on the snapshot of /Developer/Documentation for 45 minutes so far. /Developer/Applications was quick.

 

The machine was upgraded to 10.6 from 10.5, and Xcode was updated through many versions.

 

It isn't running Xcode 4, though.

Share this post


Link to post
Share on other sites

Okay, here's the results for /Developer/ subdirectories:

+  Normal backup using Snapshot Test at 10/15/10 (Activity Thread 1)
   To Media Set Rolling Backup...
   -  10/15/10 12:18:20 AM: Copying Applications on Hendry
   10/15/10 12:18:20 AM: No files need to be copied
   10/15/10 12:19:45 AM: Snapshot stored, 3.1 MB
   10/15/10 12:19:48 AM: Comparing Applications on Hendry
   10/15/10 12:19:49 AM: Execution completed successfully
   Duration: 00:01:28 (00:01:23 idle/loading/preparing)

   -  10/15/10 12:19:49 AM: Copying Documentation on Hendry
   10/15/10 12:19:49 AM: No files need to be copied
   10/15/10 1:18:56 AM: Snapshot stored, 20.1 MB
   10/15/10 1:18:59 AM: Comparing Documentation on Hendry
   10/15/10 1:19:01 AM: Execution completed successfully
   Duration: 00:59:12 (00:59:07 idle/loading/preparing)

   -  10/15/10 1:19:01 AM: Copying Headers on Hendry
   10/15/10 1:19:01 AM: No files need to be copied
   10/15/10 1:19:24 AM: Snapshot stored, 164 KB
   10/15/10 1:19:26 AM: Comparing Headers on Hendry
   10/15/10 1:19:26 AM: Execution completed successfully
   Duration: 00:00:24 (00:00:21 idle/loading/preparing)

   -  10/15/10 1:19:26 AM: Copying Library on Hendry
   10/15/10 1:19:26 AM: No files need to be copied
   10/15/10 1:19:56 AM: Snapshot stored, 1.1 MB
   10/15/10 1:19:58 AM: Comparing Library on Hendry
   10/15/10 1:19:59 AM: Execution completed successfully
   Duration: 00:00:32 (00:00:29 idle/loading/preparing)

   -  10/15/10 1:19:59 AM: Copying Platforms on Hendry
   10/15/10 1:19:59 AM: No files need to be copied
   10/15/10 1:20:21 AM: Snapshot stored, 215 KB
   10/15/10 1:20:23 AM: Comparing Platforms on Hendry
   10/15/10 1:20:24 AM: Execution completed successfully
   Duration: 00:00:25 (00:00:22 idle/loading/preparing)

   -  10/15/10 1:20:24 AM: Copying SDKs on Hendry
   10/15/10 1:20:24 AM: No files need to be copied
   10/15/10 1:21:22 AM: Snapshot stored, 7.6 MB
   10/15/10 1:21:24 AM: Comparing SDKs on Hendry
   10/15/10 1:21:26 AM: Execution completed successfully
   Duration: 00:01:01 (00:00:57 idle/loading/preparing)

   10/15/10 1:21:33 AM: Execution completed successfully
   Total duration: 01:03:05 (01:02:42 idle/loading/preparing)

Share this post


Link to post
Share on other sites

/Developer/Documentation...

 

This narrows it down a bit more. The first DocSet takes up the most time building the snapshot: half an hour.

 

+  Normal backup using Snapshot Test at 10/15/10 (Activity Thread 1)
   To Media Set Rolling Backup...
   -  10/15/10 1:26:14 AM: Copying CHUD on Hendry
   10/15/10 1:26:14 AM: No files need to be copied
   10/15/10 1:26:33 AM: Snapshot stored, 93 KB
   10/15/10 1:26:35 AM: Comparing CHUD on Hendry
   10/15/10 1:26:35 AM: Execution completed successfully
   Duration: 00:00:21 (00:00:18 idle/loading/preparing)

   -  10/15/10 1:26:36 AM: Copying ]com.apple.adc.documentation.AppleSnowLeopard.CoreReference.docset... on Hendrycom.apple.adc.documentation.AppleSnowLeopard.CoreReference.docset...nowLeopard.CoreReference.docset...com.apple.adc.documentation.AppleSnowLeopard.CoreReference.docset/418/...nowLeopard.CoreReference.docsetcom.apple.adc.documentation.AppleSnowLeopard.CoreReference.docsetnowLeopard.CoreReference.docset on Hendrycom.apple.adc.documentation.AppleSnowLeopard.CoreReference.docset
   10/15/10 1:26:36 AM: No files need to be copied
   10/15/10 2:02:10 AM: Snapshot stored, 11.6 MB
   10/15/10 2:02:13 AM: Comparing ]com.apple.adc.documentation.AppleSnowLeopard.CoreReference.docset... on Hendrycom.apple.adc.documentation.AppleSnowLeopard.CoreReference.docset...nowLeopard.CoreReference.docset...com.apple.adc.documentation.AppleSnowLeopard.CoreReference.docset/418/...nowLeopard.CoreReference.docsetcom.apple.adc.documentation.AppleSnowLeopard.CoreReference.docsetnowLeopard.CoreReference.docset on Hendrycom.apple.adc.documentation.AppleSnowLeopard.CoreReference.docset
   10/15/10 2:02:14 AM: Execution completed successfully
   Duration: 00:35:37 (00:35:34 idle/loading/preparing)

   -  10/15/10 2:02:14 AM: Copying ]com.apple.ADC_Reference_Library.CoreReference.docset... on Hendrycom.apple.ADC_Reference_Library.CoreReference.docset...reReference.docset...com.apple.ADC_Reference_Library.CoreReference.docset/365/...reReference.docsetcom.apple.ADC_Reference_Library.CoreReference.docsetreReference.docset on Hendrycom.apple.ADC_Reference_Library.CoreReference.docset
   10/15/10 2:02:14 AM: No files need to be copied
   10/15/10 2:14:32 AM: Snapshot stored, 5.5 MB
   10/15/10 2:14:34 AM: Comparing ]com.apple.ADC_Reference_Library.CoreReference.docset... on Hendrycom.apple.ADC_Reference_Library.CoreReference.docset...reReference.docset...com.apple.ADC_Reference_Library.CoreReference.docset/365/...reReference.docsetcom.apple.ADC_Reference_Library.CoreReference.docsetreReference.docset on Hendrycom.apple.ADC_Reference_Library.CoreReference.docset
   10/15/10 2:14:35 AM: Execution completed successfully
   Duration: 00:12:21 (00:12:17 idle/loading/preparing)

   -  10/15/10 2:14:35 AM: Copying ]com.apple.ADC_Reference_Library.DeveloperTools.docset... on Hendrycom.apple.ADC_Reference_Library.DeveloperTools.docset...veloperTools.docset...com.apple.ADC_Reference_Library.DeveloperTools.docset/369/...veloperTools.docsetcom.apple.ADC_Reference_Library.DeveloperTools.docsetveloperTools.docset on Hendrycom.apple.ADC_Reference_Library.DeveloperTools.docset
   10/15/10 2:14:35 AM: No files need to be copied
   10/15/10 2:17:35 AM: Snapshot stored, 1.5 MB
   10/15/10 2:17:36 AM: Comparing ]com.apple.ADC_Reference_Library.DeveloperTools.docset... on Hendrycom.apple.ADC_Reference_Library.DeveloperTools.docset...veloperTools.docset...com.apple.ADC_Reference_Library.DeveloperTools.docset/369/...veloperTools.docsetcom.apple.ADC_Reference_Library.DeveloperTools.docsetveloperTools.docset on Hendrycom.apple.ADC_Reference_Library.DeveloperTools.docset
   10/15/10 2:17:37 AM: Execution completed successfully
   Duration: 00:03:01 (00:02:58 idle/loading/preparing)

   -  10/15/10 2:17:37 AM: Copying ]com.apple.ADC_Reference_Library.WebObjectsReference.docset... on Hendrycom.apple.ADC_Reference_Library.WebObjectsReference.docset...bObjectsReference.docset...com.apple.ADC_Reference_Library.WebObjectsReference.docset/389/...bObjectsReference.docsetcom.apple.ADC_Reference_Library.WebObjectsReference.docsetbObjectsReference.docset on Hendrycom.apple.ADC_Reference_Library.WebObjectsReference.docset
   10/15/10 2:17:37 AM: No files need to be copied
   10/15/10 2:19:03 AM: Snapshot stored, 1.1 MB
   10/15/10 2:19:06 AM: Comparing ]com.apple.ADC_Reference_Library.WebObjectsReference.docset... on Hendrycom.apple.ADC_Reference_Library.WebObjectsReference.docset...bObjectsReference.docset...com.apple.ADC_Reference_Library.WebObjectsReference.docset/389/...bObjectsReference.docsetcom.apple.ADC_Reference_Library.WebObjectsReference.docsetbObjectsReference.docset on Hendrycom.apple.ADC_Reference_Library.WebObjectsReference.docset
   10/15/10 2:19:06 AM: Execution completed successfully
   Duration: 00:01:29 (00:01:27 idle/loading/preparing)

   -  10/15/10 2:19:06 AM: Copying Perl on Hendry
   10/15/10 2:19:06 AM: No files need to be copied
   10/15/10 2:19:28 AM: Snapshot stored, 70 KB
   10/15/10 2:19:30 AM: Comparing Perl on Hendry
   10/15/10 2:19:31 AM: Execution completed successfully
   Duration: 00:00:24 (00:00:20 idle/loading/preparing)

   -  10/15/10 2:19:31 AM: Copying Python on Hendry
   10/15/10 2:19:31 AM: No files need to be copied
   10/15/10 2:19:48 AM: Snapshot stored, 93 KB
   10/15/10 2:19:50 AM: Comparing Python on Hendry
   10/15/10 2:19:50 AM: Execution completed successfully
   Duration: 00:00:18 (00:00:16 idle/loading/preparing)

   -  10/15/10 2:19:50 AM: Copying wxWidgets on Hendry
   10/15/10 2:19:50 AM: No files need to be copied
   10/15/10 2:20:12 AM: Snapshot stored, 404 KB
   10/15/10 2:20:14 AM: Comparing wxWidgets on Hendry
   10/15/10 2:20:14 AM: Execution completed successfully
   Duration: 00:00:23 (00:00:19 idle/loading/preparing)

   10/15/10 2:20:18 AM: Execution completed successfully
   Total duration: 00:54:00 (00:53:33 idle/loading/preparing)

Share this post


Link to post
Share on other sites

A dev tools person at Apple says DocSets don't use any hard links or anything tricky like that.

 

If it's a matter of # of files:

 

Hendry:DocSets jon$ ls -lR com.apple.adc.documentation.AppleSnowLeopard.CoreReference.docset/ | wc
  79452  444395 5115784

 

Other directories with more files don't exhibit the snapshot problem.

Share this post


Link to post
Share on other sites

I'm going to try moving the DocSets to a different disk and symlinking them back to /Developer. See if that helps.

 

...

 

Yep, that worked nicely.

 

Moved /Developer/Documentation/DocSets to a flash drive that isn't being backed up, symlinked them back to /Developer/Documentation, and then a no-files "backup" of /Developer flew to completion in 5 minutes, including snapshots.

 

Edited by Guest
Added result of moving DocSets

Share this post


Link to post
Share on other sites

Weird. I have xCode 3.2.4 on my machine (in all honesty, I can't remember if I had a version of xCode on my machine prior to moving from 10.5 to 10.6)

 

I have the engine running on my machine (3Ghz intel core 2 duo iMac). I set "Developer" up as a favorite folder. I made a new media set (stored on my internal hard disk).

 

I did a backup of the FF using the wizard. It went quickly as expected and did not have any issues with building a snapshot (I used the backup wizard, so this had a verification step I normally wouldn't use...):

 

+ Normal backup using Backup Assistant - 10/15/10 11:16 AM at 10/15/10 (Activity Thread 1)

To Media Set DEV...

- 10/15/10 11:16:15 AM: Copying Developer

10/15/10 11:21:09 AM: Snapshot stored, 13.4 MB

10/15/10 11:21:12 AM: Comparing Developer

10/15/10 11:23:47 AM: Execution completed successfully

Completed: 49802 files, 1.6 GB

Performance: 458.2 MB/minute (354.1 copy, 651.8 compare)

Duration: 00:07:31 (00:00:16 idle/loading/preparing)

 

 

 

Maybe it would be worth uninstalling xCode and reinstalling it with the current version (3.2.2 is on ADC, but Software Update brings this to 3.2.4) to see if that makes any difference?

 

 

 

 

 

Share this post


Link to post
Share on other sites

and here's a subsequent backup (nothing changed):

 

+ Normal backup using Backup Assistant - 10/15/10 11:16 AM at 10/15/10 (Activity Thread 1)

To Media Set DEV...

- 10/15/10 11:37:24 AM: Copying Developer

10/15/10 11:37:24 AM: No files need to be copied

10/15/10 11:37:37 AM: Snapshot stored, 13.4 MB

10/15/10 11:37:39 AM: Comparing Developer

10/15/10 11:37:39 AM: Execution completed successfully

Duration: 00:00:15 (00:00:13 idle/loading/preparing)

Share this post


Link to post
Share on other sites
Is there any way to exclude /Developer, make it an un-Favorite Folder, and back up everything else?

 

No.

 

This has been the number one feature suggestion I have made when I've had the opportunity, brining it up over the years to product managers and engineers.

 

I kinda got a blank look; to be honest, giving me the feeling that the power of such a "non-favorite" designation was not appreciated.

 

Oh well. It's really a good idea.

 

 

Dave

Share this post


Link to post
Share on other sites

I agree. The "favorites" (f/k/a/ "subvolume") feature probably was a result of Retrospect's extremely long scanning phase, and was easy to add using the unix "chroot" syscall.

 

Adding an "exclude" folder designation would require writing real code in the scan path, which would, um, require work. And it could also produce unexpected results if, for example, a symlink or hard link introduced a path into the middle of that (excluded) filesystem branch.

 

Perhaps the product managers and engineers to whom you have spoken, Dave, have never used Retrospect in a real production environment. Those of us who do use Retrospect in a real production environment have to deal with:

 

(a) sections of a volume that cause Retrospect to barf due to bugs, and which need to be excluded as a workaround or during bug testing - this thread is a perfect example of such a need; and

 

(B) the slow volume scan that necessarily occurs with the current design, necessitating a full filesystem traversal with filters applied at the single-file level; such scanning time is wasted for filesystem branches that are read-only (documentation) or for which daily backup is not warranted due to the backup policy.

 

This sure would be a very good feature to have.

 

Russ

Share this post


Link to post
Share on other sites

I've found that the following also works:

 

mv DocSets "•DocSets"

ln -s "•DocSets" DocSets

 

assuming private folders is enabled.

 

This will be much more convenient than moving the folder to a different volume.

Share this post


Link to post
Share on other sites

"Adding an "exclude" folder designation would require writing real code in the scan path, which would, um, require work."

 

I'd think it could piggyback on the private files/folders/volumes implementation that is already there, and seems to work.

 

The client would have to be able to keep a list of paths which would be treated as if they were bullet-prefixed. The server would tell the client which paths to treat that way.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×