Is There A Limit In Retrospect 6 On Folder Size?

[c8ae2f49-7ecf-4762-a7ea-b9d732d0466c]

I have a Mac OS X 10.5.8 server making several backups during the day and evening of local files and network clients using Retrospect 6.1.230. The many scripts are all working except one: On a FileMaker Pro server on the LAN, there are several folders I backup nightly. Most have copies of the FMP databases created by FMP's own backup -- they all work fine, including the large files (between 1G and 1.5G). There is another folder that stores files used by a FMP database and the SuperContainer plugin. That folder is in /Users/Shared/SuperContainer and it has four subfolders. If I try to back up the whole SuperContainer folder, I get a -116 error -- the script quits while scanning the folder. So I defined the four subfolders as separate volumes -- AR, CA, IN, RC -- and this worked for a while. But now I get the same problem on just the backup of the "AR" folder.

 

That folder includes nearly 40,000 subfolders, each with a document in it -- almost always a PDF. The subfolder names index to the FMP database. So the storage size of the folder, while large, isn't particularly large -- 11G for 40k folder/file pairs.

 

The behavior persists after any number of restarts; it occurs using the nightly script; it occurs when I create and scripts for just that folder as a Normal backup to my existing or new backup sets; it occurs if I create a Duplicate script and just try to copy the folder.

 

Question is, am I running into some limit in Retrospect for the number of subfolders it can process at one level (or several levels) of a directory? That's my guess -- that when the entire SuperContainer folder grew to some number, I got the -116 errors. Then, when the AR subfolder grew to that same number, the errors began again.

[Lennart_T]

-116 is a memory manager error. Are you running out of free memory (RAM)?

 

Have you checked the volume for errors?

http://support.apple.com/kb/ts1417

(The solution is OK for your problem, even if the error description doesn't really apply.)

[c8ae2f49-7ecf-4762-a7ea-b9d732d0466c]

-116 is a memory manager error. Are you running out of free memory (RAM)?

 

Have you checked the volume for errors?

http://support.apple.com/kb/ts1417

(The solution is OK for your problem, even if the error description doesn't really apply.)

 

 

Looking at Activity Monitor, you can see Retrospect grabbing RAM as you'd expect when it tries to run the script. The "CPU%" runs as high as 75% or so -- there is 3G of RAM in the G5 I'm using as the server. The server isn't otherwise busy -- though it is running OS X Server, it has been decommissioned as the site's file server. There are a couple of rarely used share points; its primary job is running the retrospect backups and, for an hour or two each day, some OCR scanning. So, other processes don't seem to be the issue.

 

When I've run Disk Utility in the past the disk has been fine. Haven't done it recently but that's a good guess -- I'll try again (that's a busy server with folks on it 12 hours a day, and I'm not on site -- good thing I have no social life and can run utilities in the evening).

 

Any thought on whether there is a hard limit on a number of folders in a directory for Retrospect? That's the guess my co-workers are making, but I can't find a documented answer to that.

 

Thanks for the feedback, Lennart.

[Mayoff]

Do these folders go 40,000 folders deep (nested) or are all folders at the same level at the root?

[c8ae2f49-7ecf-4762-a7ea-b9d732d0466c]

Do these folders go 40,000 folders deep (nested) or are all folders at the same level at the root?

 

The second case -- the "AR" directory contains about 35000 folders; each folder has one file in it. You look at the folder in Finder and it says "34,786 items, 219.22 GB available" (Each folder has as its name a serial number that the FMP database uses to access the files in the folders. That all works just fine, though my guess is that the AR folder just has too many items at this point. I am, of course, trying to avoid the database programming problem of reorganizing those 40000 records into subfolders.)

 

So the directory structure looks like this:

/Users/Shared/SuperContainer has four items: AR, CA, IN, RC -- each is a folder

/Users/Shared/SuperContainer/AR has 34,786 items (each one a folder; each folder has one pdf file).

/Users/Shared/SuperContainer/CA has 6,000 items

/Users/Shared/SuperContainer/IN has fewer than 100 items

/Users/Shared/SuperContainer/RC has about 1000 items

 

The backup fails when I try to get either the containing folder (/SuperContainer and its contents) or just the /AR folder. (That is, when I define either folder as the Source volume in retrospect)

The backup succeeds for the CA, IN, and RC folders.

The backup succeeds if I define as the source volume any of the subfolders in /Users/Shared/SuperContainer/AR (which is impractical as the long term solution -- can't define each of those as a Source volume -- but demonstrates that Retrospect can read the contents of the AR folder).

 

 

Thanks!

[twickland]

The backup succeeds if I define as the source volume any of the subfolders in /Users/Shared/SuperContainer/AR...

I'm wondering if that's actually true, since you obviously can't have tested them all. Perhaps there's something peculiar with one of the files or folders.

 

When one is backing up an entire boot volume, there are typically many times the total number of folders you're talking about. Although they aren't all at the same directory level, of course, I don't know why this should make any difference to Retrospect.

 

As I understand it, Retrospect 6 does have a 2 GB limit as to the real memory it can use, but we've never found that to be an issue. I just performed a quick scan of a HD volume that had just under a million files in over 140,000 folders, and Retrospect only used 280 MB of real memory. The matching process would, of course, take more memory, but you said the failure occurred during the scanning phase. What do you see for Retrospect's memory use during the scanning process for this folder?