'File "xyz" appears incomplete' when backup media is not corrupt

[gggwww]

A posting from Robin at EMC indicates that the "appears incomplete" error is a result of corrupt media. This is inconsistent with my current circumstances. Could there be another cause?

 

I have 8 removable hard drives used as two backup sets (A & for

backing up 4 source disks. Three source disks back up fine with

zero "appears incomplete" errors. Backups of the fourth source

disk produce "appears incomplete" messages when backing up

onto both set A and set B backup drives. I have seen over

200,000 instances of this message during one incremental

backup!

 

The hardware consists of two SANS Digital firewire enclosures connected to an Apple Xserve server. OS X 10.5.2.

 

It's possible that I have one bad drive bay in one of the drive enclosures but I pay no attention to the order I insert drives so It's unlikely I plugged the set A and set B backup drives into the same bay.

 

It seems more likely that the problem is on the source drive. It passes the verify check in DiskUtility and the files with errors are not zero length files. In fact the set of files labeled as incomplete changes from one backup to the next.

 

I guess to diagnose this I could restore one of the "incomplete" files ... I'd pick a text file so I can see any differences ... and manually compare then to see if the source or destination is incomplete. What if neither is incomplete?

[Mayoff]

Can you include a couple of complete lines from the log? It may help understand the failure.

 

Does it happen during copy or during compare?

 

What happens if you try to backup a few of these files to a different destination?

[CallMeDave]

> A posting from Robin at EMC indicates that the "appears incomplete" error is a result of corrupt media

 

Providing a link to the post in question would have been helpful here.

 

Just to be clear with your configuration:

 

- Removable Disk Backup Sets A & B

 

- 4 physical hard drives are used as Members of Backup Set A

- 4 physical hard drives are used as Members of Backup Set B

 

- 4 unique Source volumes; each volume is a single partition on a physical hard drive (?)

 

- 3 Sources backup successfully to Backup Set A and Backup Set B, no matter which Member of Media is requested.

 

- 1 Source fails to backup successfully to either Backup Set A or Backup Set B. Failure occurs on any/all Member of the Media in the set (fails on 1-FooA, 2-FooA, 3-FooA, 4-FooA, 1-FooB, 2-FooB, 3-FooB and/or 4-FooB).

 

- To add complexity to the troubleshooting requirements, any of the Member hard drives might find themselves physically connected to the host computer via n (?) different physical bus connections.

 

As Robin asks, what happens if you attempt to backup the problem Source to a different Backup Set, one that uses different physical media (such as a File Backup Set to a different hard drive)?

 

 

Dave

[gggwww]

CallMeDave says:

 

> A posting from Robin at EMC indicates that the "appears incomplete" error is a result of corrupt media[/i]

 

> Providing a link to the post in question would have been helpful here.

 

http://forums.dantz.com/showpost.php?post/97452/

 

> Just to be clear with your configuration:

 

> - Removable Disk Backup Sets A & B

 

> - 4 physical hard drives are used as Members of Backup Set A

> - 4 physical hard drives are used as Members of Backup Set B

 

I think in Retrospect lingo I have 8 Backup Sets. 4 Backup Sets are A sets and 4 are B sets. There is one Backup Set per removable drive. And two Backup Sets per Source Volume, i.e. an A set and a B set.

 

> - 4 unique Source volumes; each volume is a single partition on a physical hard drive (?)

 

yes

 

> - 3 Sources backup successfully to Backup Set A and Backup Set B, no matter which Member of Media is requested.

 

> - 1 Source fails to backup successfully to either Backup Set A or Backup Set B. Failure occurs on any/all Member of the Media in the set (fails on 1-FooA, 2-FooA, 3-FooA, 4-FooA, 1-FooB, 2-FooB, 3-FooB and/or 4-FooB).

 

There are only two backup drives with these errors: 1-FooA and 1-FooB. My backup drives are large enough that backups never roll onto a second drive. They are recycled before they fill with incremental backups.

 

It might also be useful to note that the source drive/partition with errors is the largest of the four. It may have 500,000 files while the next largest may have 250,000 files. Also, because of the _number_ of errors generated the log file is truncated so I never see the first of the errors in case some other type of error precedes the "appears incomplete" spew. Furthermore either the line count at the top of the log window is incorrect or the error count in the summary of the backup operation is incorrect. I'll see the log window say it is showing 60,000 lines and the summary saying there are 40,000 errors but I can't scroll back to a point before those errors start.

 

> - To add complexity to the troubleshooting requirements, any of the Member hard drives might find themselves physically connected to the host computer via n (?) different physical bus connections.

 

I don't think in terms of busses. There are two, two-bay firewire enclosures so there are 4 possible bays that a drive can plug into.

 

> As Robin asks, what happens if you attempt to backup the problem Source to a different Backup Set, one that uses different physical media (such as a File Backup Set to a different hard drive)?

 

I don't have space for a full backup to an alternate source. I did backup two of the offending files to a USB jump drive. The full backup had no errors. Immediate backups don't seem to support incremental mode so I had to start over defining files for an incremental so this is in progress.

 

Picking a not-too-large subset of files for a test backup is challenging. It takes a lot of messing with include/exclude queries and waiting 45 minutes for the file system scan each time I run a test.

 

Anyway, if testing with a larger set of files and using a File Backup Set is important I can run some more tests.

[gggwww]

All the backup drives occasionally fail the Verify check in DiskUtility with a "Directory Locked" error. I always repair them when I see this error. It just happened to the USB flash drive after a recycle backup during my tests. I did check this drive _before_ the backup so the problem occurred as a result of the backup.

-Glenn

[CallMeDave]

I think in Retrospect lingo I have 8 Backup Sets...

 

I'm disinclined to try and read much past this point. You need to describe, using Retrospect lingo, exactly what you are doing and what you see when you do it. Sentences and paragraphs are not a good method to use when describing hardware setup and steps; lists work much better.

 

 

All the backup drives occasionally fail the Verify check in DiskUtility with a "Directory Locked" error.

 

Game over. You have hardware problems.

[gggwww]

> Game over. You have hardware problems.

 

No, it's not over because I wasn't clear.

 

The USB flash drive _and_ the firewire hard drives both exhibit the problem of failing a disk verification after Restrospect erases and backs up to them. USB flash drives and firewire hard drives use completely different hardware and interfaces so the problem must be upstream.

 

As per your request here is a blow-by-blow of what I did.

 

- Use DiskUtility to erase flash drive. Named it "1-test".

- Use DiskUtility to verify "1-test". No errors.

- In Retrospect create a backup set called "test". Type is removable media.

- Use Immediate backup feature and query style file selectors to choose a set of files to backup from my single source hard drive to backup set, test.

- Run backup.

- files in my list were too large so the backup hung up prompting for a second backup media called "2-test".

- cancelled backup operation.

- went through same steps setting up immediate backup except I improved the file selection query to exclude the large files.

- Retrospect prompts for media. I chose "1-test" and answered yes to the prompt asking to erase the existing data on the drive.

- Backup completes successfully. Flash drive is now named "2-test".

- Run disk verification in DiskUtility. Verification reports a "Locked Directory" error.

 

I've seen pretty much the same problems on all the destination hard drives.

 

Note that directory errors are software errors. They _can_ be caused by underlying hardware problems but can also be caused by some piece of software not writing the expected data to the hardware. To me this doesn't look like a hardware problem. How could the exact same hardware problem occur on two completely different devices. I've used the flash drive a great deal and it has never exhibited errors before this.

[CallMeDave]

Picking a not-too-large subset of files for a test backup is challenging. It takes a lot of messing with include/exclude queries and waiting 45 minutes for the file system scan each time I run a test

.

 

Uh, Subvolumes? Just define a folder that contains the amount of data that is appropriate for your tests as a Retrospect Subvolume, and use that as your Source. No need to scan more then you need to use.

 

I repeated your tests (without the steps related to a too-large Source selection), using both an iPod Nano and a USB flash drive as Members of a Removable Disk Backup Set (my source was an always reliable /bin/). Disk Utility reported no problems before or after the Backup ran.

 

Either this is a red herring for your problem, or it's exactly what your problem is.

 

The top level directories present on my volumes are:

.Spotlight-V100

.Trashes

.fseventsd

 

Which include some their own sub-directories. These are all created by Mac OS X, and have nothing to do with Retrospect.

 

Have you tried testing while booted from some other OS X volume?

 

 

Dave

[gggwww]

The failed disk verification is definitely a red herring. I figured out exactly what it's all about. I had abbreviated disk verification the error as "directory locked" but it's actually "Directory name locked". Once I started googling "name locked" I got some clues. It goes like this:

 

The mac HFS file system supports a dozen or so one bit attributes which can be set on each folder (aka directory). For the most part these attributes are hidden from OS X users. (The "locked" attribute may be an exception). You may be familiar with the attributes which make a file hidden or a "system" file. The OS X developer's kit includes a set of utilities called GetFileInfo and SetFile which get and set these attributes.

 

So it turns out that the "system" attribute is also known as "name locked". When I ran GetFileInfo on my USB flash drive, the one that was failing the disk verification with the "Directory name locked" error, I got this result:

 

anyone$ GetFileInfo /Volumes/1-test/

directory: "/Volumes/1-test"

attributes: avbStclinmedz

created: 04/14/2008 13:50:09

modified: 04/15/2008 16:52:38

 

The upper case "S" means the "system (name locked)" attribute is on. All other attributes are off.

 

Then I ran "SetFile -a s /Volumes/1-test/" and reran the disk verify. The verification was successful ... no errors.

 

In the process of doing the testing I had plugged the flash drive into a box running Leopard. On Leopard it passes disk verification when the name locked attribute is on. Apparently when Apple reworked diskUtility for Leopard they decided not to flag the "name locked" error. (My system which hosts Restrospect runs Tiger).

 

I'm guessing that under certain conditions Retrospect sets the "system (name locked)" attribute. I've seen mention of special treatment of "system" files in the Retrospect documentation so the code is at least aware of these attributes. Maybe windoze has a similar attribute and setting the "system' attribute of the top level directory on the backup drive makes sense for windoze and they just left the code in place for the mac version?

 

Anyway, the value of the "system" attribute on my backup media seems totally irrelevant to the "appears incomplete" errors I get from the Retrospect "Compare" operation.

 

I was not able to reproduce the "appears incomplete" errors on my flash drive. I restored a couple of the offending files from the exact backup set with the errors and the matched the source files exactly. Now, hearing this, you might say, "Big deal. Just ignore the errors." However, the errors aren't the real problem. The problem is that as far as I can tell Retrospect is adding every file with said error to the incremental (Normal) backup every night. So a backup which should contain 7,000 files each night can contain 250,000 files, most of which have not changed for months.

 

Robin,

The "appears incomplete" error is not documented in the manual. What conditions trigger the error? Can it be referring to the source file or only the backed up copy of the file?

 

BTW, after seeing thousands of these errors in the Retrospect log I've used the Retrospect "Verify" command found in the "Directory" menus. The Backup Set has always passed this verification.

 

-Glenn

[CallMeDave]

...hearing this, you might say, "Big deal. Just ignore the errors.

 

I'd never say that. Ignore Retrospect errors at your (and your data's) peril.

 

You are using Removable Disk Backup sets even though you are not utilizing their main advantage, the ability to span across multiple volumes (there are some advantages of speed and memory, too).

 

You might try using File Backup Sets stored on your external hard drives; perhaps you'll have better results.

 

Dave

[gggwww]

Here's the smoking gun:

 

I went through the restore process again for one of the files that had an "appears incomplete" error. The file was last modified on April 3:

 

$ls -l Database3

-rw-r--r-- + 1 user1 admin 140288 Apr 3 13:46 Database3

 

This time I happened to notice that the size listed in the Retrospect Restore Browser was 137KB!

 

I restored the file and it's restored size was 140KB. It matched the original file exactly:

 

$ sum Database3 /Volumes/DataHD/back1\ Multi-Disk\ B/Database3

24812 137 Database3

24812 137 /Volumes/DataHD/back1 Multi-Disk B/Database3

 

Now, perhaps the size displayed in the Restore Browser is supposed to be the compressed size and that would explain this discrepancy. However, the compressed size isn't very useful to a user so even this would be a usability deficiency.

 

I'm assuming the size displayed in the Restore Browser comes from the catalog. If it's supposed to be the actual file's size then I think we have a bug were the code used to determine the size recorded in the Catalog is different from the code which determines the size of the original file on the source drive.

 

Maybe the code that determines the size recorded in the catalog counts bytes and the other code uses the size reported by the HFS file system. And maybe the HFS file system allows some kind of hole in the file which is counted in the size??

 

-Glenn

 

[gggwww]

Is it really true that Mac OS X defines 1KB of file size as 1,000 bytes? I've always thought 1KB == 1,024 bytes. In the shell:

 

$ ls -l

-rwxrwxr-x 1 admin admin 140288 Apr 3 13:46 Database3

$ ls -lh

-rwxrwxr-x 1 admin admin 137k Apr 3 13:46 Database3

 

Selecting the file then issuing the File -> Get Info command pops up and info window that says:

 

Size: 140KB on disk (140,288 bytes)

 

Even the Finder windows show the file size as 140KB. So, it seems there is no smoking gun. Just strangeness from Apple.

 

Arghh. Why do complete files "appear incomplete" to Retrospect??

 

-Glenn

 

 

 

[CallMeDave]

From the Original Post:

 

> Three source disks back up fine with zero "appears incomplete" errors.

> Backups of the fourth source disk produce "appears incomplete" messages

 

{snip}

 

> It seems more likely that the problem is on the source drive

I agree. If you were to take my most recent suggestion (using a File Backup Set instead of the Removable Disk Backup Set) and got the same error(s), I'd say that it's pretty conclusive that it's an issue with the Source volume.

 

So, looking back upthread, I find virtually _no_ description of this suspicious volume at all!

 

So it all goes back to my original contributions to this discussion; having a complete, thorough description of the hardware and software configuration in play is the _first_ step in getting help from online users.

 

 

Dave

[gggwww]

Dave,

 

Thanks for staying with me on this ... much appreciated!

 

Regarding the source Drive info:

 

I've got an Apple Xserve server running OS X 10.5.2. It has

two file systems. One is the boot drive which backs up

without errors. The second is the "data" drive which holds

users' home directories. The "data" drive is the one with

the "appears incomplete" errors in its backup. The

data drive is a RAID mirrored pair of 750GB SATA drives

with one HFS+ Journaled file system. There are 785,000

files and 386GB of space used. This box hosts the

Retrospect server v 6.1.126. Note that there is a second

xserve with the same configuration except that less disk space

is used and the file count is smaller on the "data" file

system. The second xserve is a Retrospect client. The

client xserve has no "appears incomplete" errors in

its backup logs.

 

WRT the file backup experiment you suggested, that will have to wait until next weekend since the backup & compare takes 20 hours. This weekend I performed a different experiment. I swapped the media which hold the two data backups so the file system on the Retrospect server was backed up to the media previously used for the Retrospect client backup. The result was that the backup of the Retrospect server's data file system produced 235,701 "appears incomplete errors." The other backups had no such errors.

 

Again, this points fingers at the source file system.

 

I hope to take the system down some time this week to run a DiskUtility verify on the suspicious source volume. I remember doing this weeks ago after seeing some of these backup errors but perhaps my memory is failing.

 

-Glenn

 

 

[CallMeDave]

The data drive is a RAID mirrored pair of 750GB SATA drives with one HFS+ Journaled file system.

 

- What are the physical attributes of the drives that make up the RAID? 3rd party enclosure? Make/model?

- What software or hardware controls the RAID?

[gggwww]

I'm not sure what you mean by physical attributes. This is Apple Xserve out-of-the-box. Two of the internal drives which came with the Xserve are RAIDed using OS X. I'm not familiar enough with this config to know if this is software or hardware RAID. The Xserve uses an SAS storage bus.

 

Also, I have to apologize for a brain fart. The OS is tiger, not leopard. It's v10.4.11.

 

I was able to run DiskUtility last night: No errors in the verification.

 

And I've probably said this before but I'll repeat it since it is baffling: The set of files with "appears incomplete" errors changes from one backup to the next. I see a lot of overlap but it isn't always the same files. And the difference is not explained by files being recently updated. This happens for files that haven't changed for months.

 

wrt to testing the file set backup maybe I could create a subvolume, add the subvolume backup to one of my existing backup sets (if you can do that), verify "appears incomplete" errors, then switch it to be file backup to see if the errors go away. Subvolume would make the backup set smaller so it would complete overnight along with the other essential backups.

[CallMeDave]

It's multiple messages and eleven days and we're just getting to the most basic information:

 

- Apple XServe (what model? Processor?)

- OS X Server 10.4.11

- Retrospect 6.1.126 (RDU version?)

- 3 internal hard drives, one used as boot volume, other two 750 GB configured as RAID 0

 

> The Xserve uses an SAS storage bus.

Not with 750 GB drives it doesn't. More likely it's SATA.

 

- RAID controller unknown; either Xserve RAID Card or built-in SATA/SAS controller board

 

- Local backups to Removable Disk Backup Sets exhibit file errors during the Verify pass when the Source is the RAID 0 volume. Other Sources work as expected

 

- Backing up to other Types of Backup Sets has not been tried

 

> I was able to run DiskUtility last night: No errors in the verification.

 

- I don't know enough about the RAID configuration to be willing to offer suggestions that might impact your data security/integrity. But it would be interesting to know if splitting the mirror had an effect on the behavior here. I know that Russ splits his server mirror regularly, so perhaps he will contribute.

 

- It would also be helpful to confirm that you have disabled backing up ACLs; the reported problems are different then what you're seeing, but it would be worth a try.

[gggwww]

Intel processor.

As I understand it an SAS controller supports SATA drives and that's what the drives are.

 

I'll have to do some research to answer the other questions since I'm not familiar with these topics.

 

Splitting RAID seems scary to me since it'll need to rebuild when I un-split and I don't know if the rebuild would take minutes or days.

 

-Glenn

 

 

[rhwalker]

Splitting RAID seems scary to me since it'll need to rebuild when I un-split and I don't know if the rebuild would take minutes or days.

I would only reliably do a RAID 1 mirror split using SoftRAID; I wouldn't do that with Disk Utility, which is the standard Apple RAID. And don't ever do a RAID split unless it's RAID 1 (mirror); anything else will be a bad thing. I don't see any information here that indicates the type of RAID on this machine.

 

As for time to rebuild, we do a RAID 1 mirror split only on our 50 GB OS (boot) volume, and it takes less than 2 hours. Time to rebuild will depend on the speed of your disks, because the RAID software has to read and write every sector on the volume to do the rebuild. It takes a bit longer than just a straight pass through the volume if the volume is live (since there is a bunch of seeking going on at the same time).

 

As a comparison, our Apple Hardware RAID takes about a day to rebuild a 1/2 TB RAID 5 when a member is replaced (been there, done that). We have the write cache disabled on our Apple Hardware RAID because of a bug in the Apple Hardware RAID for which I've had an open RADAR report for three years; I don't ever expect that bug to be fixed. Better to have reliability than speed. So much for the Apple Xserve Hardware Maintenance Agreement.

 

Russ

Edited April 21, 2008 by Guest

[gggwww]

RAID config:

 

There is no evidence of a hardware RAID controller so it looks to me like the RAID is software RAID implemented in OS X Server. The RAID config is mirrored. It's called "Apple RAID" in disk utility and the profiler shows no PCI cards or RAID controller. Instead, it shows an LSI-Logic "built-in" SAS/SATA controller. Here is what you see in profiler:

 

Hardware Overview:

 

Model Name: Xserve

Model Identifier: Xserve1,1

Processor Name: Dual-Core Intel Xeon

Processor Speed: 2 GHz

Number Of Processors: 2

Total Number Of Cores: 4

L2 Cache (per processor): 4 MB

Memory: 4 GB

Bus Speed: 1.33 GHz

Boot ROM Version: XS11.0080.B01

SMC Version: 1.11f5

LOM Revision: 1.2.8

 

SAS Domain 0:

 

Vendor: LSILogic

Product: Built In SAS

Revision: Firmware 1.17.0.0, NVData version 8

Initiator Identifier: 169

 

SCSI Target Device @ 3:

 

SCSI Target Identifier: 3

SCSI Peripheral Device Type: 0

Manufacturer: ATA

Model: ST3750640NS P

Revision: BUF

 

SCSI Logical Unit @ 0:

 

Capacity: 698.64 GB

SCSI Logical Unit Number: 0

Model: ST3750640NS P

Revision: 3.BUF

SATA Device: Yes

Removable Media: No

Detachable Drive: No

BSD Name: disk5

OS9 Drivers: No

Bay Name: "Bay 1"

S.M.A.R.T. status: Verified

Volumes:

ServerHD:

Capacity: 698.32 GB

Available: 673.63 GB

Writable: Yes

File System: Journaled HFS+

BSD Name: disk5s2

Mount Point: /

 

SCSI Target Device @ 4:

 

SCSI Target Identifier: 4

SCSI Peripheral Device Type: 0

Manufacturer: ATA

Model: ST3750640NS P

Revision: BUF

 

SCSI Logical Unit @ 0:

 

Capacity: 698.64 GB

SCSI Logical Unit Number: 0

Model: ST3750640NS P

Revision: 3.BUF

SATA Device: Yes

Removable Media: No

Detachable Drive: No

BSD Name: disk6

OS9 Drivers: No

Bay Name: "Bay 2"

S.M.A.R.T. status: Verified

Volumes:

Boot OSX:

Capacity: 128 MB

Writable: Yes

BSD Name: disk6s3

Mount Point:

 

SCSI Target Device @ 5:

 

SCSI Target Identifier: 5

SCSI Peripheral Device Type: 0

Manufacturer: ATA

Model: ST3750640NS P

Revision: BUF

 

SCSI Logical Unit @ 0:

 

Capacity: 698.64 GB

SCSI Logical Unit Number: 0

Model: ST3750640NS P

Revision: 3.BUF

Serial Number: 3QD0ZNSC

SATA Device: Yes

Removable Media: No

Detachable Drive: No

BSD Name: disk8

OS9 Drivers: No

Bay Name: "Bay 3"

S.M.A.R.T. status: Verified

Volumes:

Boot OSX:

Capacity: 128 MB

Writable: Yes

BSD Name: disk8s3

Mount Point:

[rhwalker]

Ok, then, you have a software RAID 1 mirror created by Apple's Disk Utility (or by the appropriate command line item). Splitting the mirror is not advised.

 

Russ

[gggwww]

Dave,

 

I can't see any option for controlling whether ACLs are

backed up. I looked through all the options for backup

scripts and all the Retrospect Preferences.

 

I did notice an option, on by default, which checks for

changes to ACLs and HFS attributes and causes a file to

be added to an incremental if only these attributes were

changed. It's called "Use attribute modification date when matching." I wonder if I should try turning this off. Attribute

modifications could explain why month-old files are being

backed up during incrementals.

 

-Glenn

 

 

[rhwalker]

I can't see any option for controlling whether ACLs are backed up. I looked through all the options for backup scripts and all the Retrospect Preferences.

Here's the KB article:

Workaround Hack for ACL bug

 

Note that the article is wrong when it says this only happens on certain Tiger releases, and that it requires the Intel platform. The bug occurs on any "Universal Binary" version of Tiger or Leopard, on PPC or Intel platforms, if ACLs are turned on. Note that Universal Binary Tiger was never delivered for PPC, but you can install it on PPC and turn on ACLs for Tiger non-server if you try hard using the command line.

 

Russ

[CallMeDave]

Just a couple of things.

 

- Apologies for typing RAID 0 above when trying to lists our "known" configurations. Brain fart.

- I don't think that this is related to ACLs, but it's an easy test to try (assuming that a small defined Subvolume used as a Source will always result in file compare errors).

- Although the RAID volume is named "Boot OSX" can we infer that it is not, in fact, booting anything?

- System Profiler lists Capacity for the RAID volume, but not Available space; how much data are we talking about?

 

This is some serious hardware, and Retrospect is reporting some serious errors when you try and use it. I've seen enough times that Retrospect ended up being the digital canary in the virtual coal mine, so I'd advise taking this seriously.

 

At this point, I'd also advise opening a Support Incident with EMC; other then testing with a File Backup Set instead of the Removable Disk Backup Set you've been using, I'm out of ideas for what to try.

[rhwalker]

The Mac (OS X) can boot from a RAID 1 (mirror) because, during the initial boot phase, it just boots from one of the drives (both should be the same), and the OS is in read-only mode at that time. As the drivers get loaded, and before the switch to single-user mode to bring things up, the RAID driver realizes what is happening and goes into RAID 1 (mirroring) mode. That's why you can't boot from a RAID 0 (striping) unless the RAID 0 is done in hardware, because you can't get everything off of one disk while trying to get the kernel loaded.

 

It's a real wonder when you see how the kernel loads and then brings itself up. Been there, done that.

 

Russ