NTFS Mount Point destination

[dkruger]

Good morning,

 

Currently using Retrospect multi-server 9.5.x.  Last week had setup backup jobs that backup to individual disks with NTFS mount points.  the backups worked great last week for these backup jobs, however this week, now that the backup process had moved to a second disk in the backup volumes media (I think is what caused the issue to start), it is getting an error:

 

Trouble writing: "2-Monday" (2667610112), error -102 ( trouble communicating)

Trouble writing media:

"2-Monday"

error -102 ( trouble communicating)

Additional error information for Disk Backup Set member "2-Monday",

Can't write to file C:\BVOLs\02\Retrospect\Monday\2-Monday\AA005739.rdb, error -1017 ( insufficient permissions)

 

I have setup Retrospect to login with a user that does have permissions to access these data locations, through preferences->security.  I also tested creating a file and editing it with the same user, in the locations with the issue occurring.  I was suspecting that the hard disk maybe was starting to fail, but when proceeding to the next days backups the same error occurred, along with other days volumes that I have tried.  The reason that the backup disks are setup to use NTFS mount points, is primarily because there are 45 disks, and as such not enough drive letters to assign each individually.  The 2-Monday volume is only about 1/3 full from the previous weeks backup data.

 

I am out of ideas to try with this, and hoping someone can give me some assistance.  Worse case scenario would be that I would have to re-create the disks in software raid volumes in windows, and assign them a drive letter, but am not sure why I can't seem to get the NTFS mount point volumes to work.  I saw in the list of fixes for previous patches to Retrospect 9, that it is supposed to correctly calculate the free space on NTFS mount point volumes now, so am guessing that this setup should be able to work, I am just not seeing what the issue is caused by.

 

Thanks,

David

[Lennart_T]

Trouble communication is just that: trouble communicating. It is not a permissions error, it is not a setup error (in which case you could not set up that volume as a member of the backup set.) 

 

I say the permissions error is just the resulting effect of the communications error.

 

You don't tell us anything about how that NTFS mount point is actually connected and nothing in general about the hardware you are using.

 

I assume from the error message writing to file number 5739 that the backup has been running a good while before getting the error. But I can be wrong.

Your log excerpt isn't informative enough as you don't show what happens before and after the error message(s).

[dkruger]

I understand that, but as retrospect is attempting to write content to the disk, I can go in through windows explorer and create a new file in the same location that retrospect is attempting to write data to and do not get an error.  If there were truly a communication error causing it, I would think I would have the same issue when trying to do what I mentioned, the the file could not be created/modified etc.  The user that retrospect is configured to use is the same one as the user I was creating the file with, then modifying the file etc.

 

The NTFS mount point is an SATA hard drive (multiple drives) in a JBOD enclosure.  The individual drives in the encosure are Seagate Barracuda 3 TB drives mostly (ST3000DM0011CH1), and a few HGST (HGSTHDN724030AL).  It is using NTFS mount points since the number of drives available exceeds the number of drive letters than can be assigned for each disk.  the drive enclosure and adapter has been and seems to be working properly, user permissions seem to be set properly, since I am able to write files to the volumes having issues with the user retrospect is using.  what else about the hardware are you wanting to know?

 

It has only been configured this way for about the past week, and worked throughout last week.  The disk enclosure has been in place for a little over 2 years, however, recently due to a significant number of the seagate drives failing shortly outside of warranty, we had changed the disk enclosure drives to not be spanned volumes, but to individual disks with NTFS mount points, in order to make it hopefully less of a headache when more of the drives fail in the future.  Is there a more detailed log in retrospect that will have the more informative information you are hoping for?  The log excerpt I included previously was from the operations log.

[Lennart_T]

I understand that, but as retrospect is attempting to write content to the disk, I can go in through windows explorer and create a new file in the same location that retrospect is attempting to write data to and do not get an error.  If there were truly a communication error causing it, I would think I would have the same issue when trying to do what I mentioned, the the file could not be created/modified etc.  The user that retrospect is configured to use is the same one as the user I was creating the file with, then modifying the file etc.

 

That  is simply not the same thing. Retrospect puts high demand on speed, which Explorer (normally) doesn't.

 

What if you used Explorer to copy a couple of dozen GB worth of large files from the same source to the same destination as Retrospect does.? That would be a better test.

 

How is the JBOD enclosure connected to the server? Fibre channel? SCSI? Via a PCI card?

All firmware fully updated? All drivers fully updated?

[Scillonian]

What interface is used to connect the enclosure to the server (eSATA, USB, etc)? What is the make and model of the enclosure?

 

 

(My sympathies on the Seagate drives — I suffered with their infamous predecessors the ST2000DL003.)

[dkruger]

That  is simply not the same thing. Retrospect puts high demand on speed, which Explorer (normally) doesn't.

 

What if you used Explorer to copy a couple of dozen GB worth of large files from the same source to the same destination as Retrospect does.? That would be a better test.

 

How is the JBOD enclosure connected to the server? Fibre channel? SCSI? Via a PCI card?

All firmware fully updated? All drivers fully updated?

Understood, copied 35 GB of data to the same location with no problems.  Obvious normal slowdowns occurred when getting to the thousands of tiny files that are there, but no errors.

 

JBOD enclosure is connected through an SAS.  It is a PCI LSI SAS controller.  it is a super micro enclosure and LSI SAS2 2308 controller, with up to date drivers and firmware.  I have not done any firmware updates for the hard disks.

 

Yeah, these Seagate drives are pretty infamous/poorly rated drives as well.  Hopefully/should have better luck with the HGST from what I have read.

[dkruger]

Any ideas?  If I delete the mount point partitions and recreate them, the backup job will work once, or as many times as I want until the system is restarted.  After that it consistently receives the errors as described previously.  We have upgraded to Retrospect 10 as well...any help would be appreciated, I am certain that this is not a hardware issue, or permissions issue, since the same thing occurs with Everyone granted full access to both the mounts folder, and all of the drives used in the backup set below it.  Detailed operations log following:

 

--Script Summary--

* Script: test

* Date: 4/20/2015 8:09 AM

* Errors: 42

* Warnings: 0

* Performance: 0.0 MB/minute

* Duration: 00:00:59

* Server: STORSTER

 

--Log--

??+?Normal backup using test at 4/20/2015 8:08 AM (Execution unit 1)

Adding to 3,420.9 GB of backup set data, starting at file index 5904

To Backup Set MTHRUF...

VssWEnumWriterMetadata: Excluding writer "System Writer" from the backup.

VssWEnumWriterMetadata: Excluding writer "DFS Replication service writer" from the backup.

VssWGetDevice: volPath =?\Volume{f2c95649-11e3-11e0-ae3b-0025900ca40f}\, devString =?\GLOBALROOT\Device\HarddiskVolumeShadowCopy127

 

- 4/20/2015 8:08:10 AM: Copying Pin-IT on PinOther (P:)

ScanFile::ScanFileOpen: Can't FOpen Scan file, will now try read only access, error -1020 (sharing violation)

ScanFile::ScanFileOpen: Scan file "C:\ProgramData\RetroISA\RetroISAScans\RetroISAScan-f2c95649-11e3-11e0-ae3b-0025900ca40f" successfully read

Using Instant Scan

VssWEnumWriterMetadata: Excluding writer "System Writer" from the backup.

VssWEnumWriterMetadata: Excluding writer "DFS Replication service writer" from the backup.

4/20/2015 8:08:45 AM: Found: 8,606 files, 1,641 folders, 12.9 GB

4/20/2015 8:08:50 AM: Finished matching

4/20/2015 8:08:50 AM: Copying: 30 files (2.1 GB) and 0 hard links

runDoRequest: run 0x64f0000 rsrvRef 0xaf61

Additional error information for Disk Backup Set member "2-MTHRUF",

Can't write to file C:\Mounts\02\Retrospect\MTHRUF\2-MTHRUF\AA005904.rdb, error -1017 (insufficient permissions)

soccRecv: connection unexpectedly closed, error 0

Trouble writing: "2-MTHRUF" (276475904), error -102 (trouble communicating)

Trouble writing media:

"2-MTHRUF"

error -102 (trouble communicating)

runDoRequest: run 0x12cb70c0 rsrvRef 0xaf61

ddskUpdateVol: (vref|vdex) [0x9c28|0x1], C:\Mounts\02, file AA005904.rdb, use disk space =000,034,656,256, used in catalog e8,413,383,680, free on volume =341,848,129,536

Additional error information for Disk Backup Set member "2-MTHRUF",

Can't write to file C:\Mounts\02\Retrospect\MTHRUF\2-MTHRUF\AA005904.rdb, error -1017 (insufficient permissions)

Trouble writing: "2-MTHRUF" (276475904), error -102 (trouble communicating)

Trouble writing media:

"2-MTHRUF"

error -102 (trouble communicating)

runDoRequest: run 0x12cb70c0 rsrvRef 0xaf61

ddskUpdateVol: (vref|vdex) [0x9c28|0x1], C:\Mounts\02, file AA005904.rdb, use disk space =000,034,656,256, used in catalog e8,413,383,680, free on volume =341,848,129,536

Additional error information for Disk Backup Set member "2-MTHRUF",

Can't write to file C:\Mounts\02\Retrospect\MTHRUF\2-MTHRUF\AA005904.rdb, error -1017 (insufficient permissions)

Trouble writing: "2-MTHRUF" (276475904), error -102 (trouble communicating)

Trouble writing media:

"2-MTHRUF"

error -102 (trouble communicating)

runDoRequest: run 0x12cb70c0 rsrvRef 0xaf61

ddskUpdateVol: (vref|vdex) [0x9c28|0x1], C:\Mounts\02, file AA005904.rdb, use disk space =000,034,656,256, used in catalog e8,413,383,680, free on volume =341,848,129,536

Additional error information for Disk Backup Set member "2-MTHRUF",

Can't write to file C:\Mounts\02\Retrospect\MTHRUF\2-MTHRUF\AA005904.rdb, error -1017 (insufficient permissions)

Trouble writing: "2-MTHRUF" (276475904), error -102 (trouble communicating)

Trouble writing media:

"2-MTHRUF"

[Lennart_T]

I suggest you contact Retrospect support.

Support

 

(This is mainly a user-to-user discussion)

[dkruger]

Yes, I have contacted support prior to posting this in the forum about it already as well, just thought that maybe someone else had run into a similar issue that might spark an idea.  Unfortunately so far nothing they have had me try is or has resolved the issue.

 

For anyone that may be following this post, or encounter a similar issue in the future...I may have found something further.  Reading from a microsoft post about it: https://support.microsoft.com/en-us/kb/280297 it mentions to use a root volume exclusively for the mount points, so for example create a small volume with a drive letter, and then mount the NTFS mount point directly under that.  I changed my mount points to do so, and after moving those mount points over to the new host volume, so far the test backup has worked.  I don't know if it will continue to do so or not, but at least it is for the time being.