Error -557 (transaction already complete)

[misleb]

Hi,

 

I recently upgradated to Retrospect 7.5 from 7.0 and I just started getting this -557 error half-way through backup up of a Netware volume. The Netware server itself was also recently upgraded from 6.0 to 6.5. It is running all the latest SMS modules. I have gotten a full backup of this server since the upgrades. It just started happening. It happens about the same place on the same volume. The backup of other volumes on the same server continues, but this one stops.

 

What can I do to troubleshoot this?

 

-matthew

[natew]

Hi

 

If it always fails at the same place there may be a problem with a certain file or directory on disk.

 

Do you have an idea of where the problem file might be? To narrow it down you can create subvolumes in Retrospect and back up the volume in pieces. When you backup the subvolume containing the bad file the error will occur. You can then create subvolumes within that folder until you can narrow down the cause.

 

Thanks

Nate

[misleb]

Ok, after narrowing the problem down using subvolumes, I now get an Assert error any time I try to backup subvolumes under a certain directory. It doesn't seem to be a particular file. Sometimes it is -557 error and Retrospect doesn't crash, but sometimes it is this Assert error which crashes Retrospect:

 

Code:

---

OS: Windows XP version 5.2 (build 3790), Service Pack 1, (32 bit)

Application: C:\Program Files\Retrospect\Retrospect 7.5\Retrospect.exe, version 7.5.251

Driver Update and Hot Fix version 7.5.1.105

Exception occurred on 4/6/2006 at 9:22:31 AM

Error info: Assertion failure at "tyce.cpp-158", tyceCheckDL: DL 7 key 'CTr' bad

Exception code: E0000000 ASSERTION

Fault address: 77E55DEA 0001:00014DEA kernel32.dll

---

 

I can send the full report if you'd like.

 

-matthew

[misleb]

*bump*

 

Any help with this? Now that I have found the problem directory, what can I do? I tried excluding the directory and its contents from the backup, but it still fails.

 

-matthew

[misleb]

Actually, I was mistaken. The error happens all over the volume. Not just the directory that I narrowed down.

 

-matthew

[misleb]

Bump!

 

I still have not resolved this issue. I am unable to narrow down a single location where this error occurs using subvolume. It happens all over the place. Since my last post, I've updated the Netware client to 1.0.0.141, applied the latest Hot Fix, and rebooted the Netware server. No luck.

 

Here is some more info from the client log:

 

Code:

---

...

1149638062: smsfetFileSpec: smsGetDataOffsetAndLen failed, myName = VOL1:USER/Becca/Marketing Media/final draft 91010, ccode = -1

1149638062: smsfetFileSpec: smsGetDataOffsetAndLen failed, myName = VOL1:USER/Becca/Media Letters/Icon0, ccode = -1

1149638063: smsfetFileSpec: smsGetDataOffsetAndLen failed, myName = VOL1:USER/Becca/press releases/general exhibitions/MissionPurpose - Swigert Commo0, ccode = -1

1149638066: smsfetFileSpec: Big Problem! No more data sets

1149638796: smsfetFileSpec: smsGetDataOffsetAndLen failed, myName = VOL3:Back Ups/Vol 1 Backup/Development/Sarah/New Microsoft Word Document.doc, ccode = -1

1149638801: smsfetFileSpec: smsGetDataOffsetAndLen failed, myName = VOL3:Marketing & Communications/BFA/2006/2006_Commencement_Invitation.pdf, ccode = -1

1149641544: smsfetFileSpec: smsGetDataOffsetAndLen failed, myName = VOL3:Marketing & Communications/Transfer/for sarah n/Template Gothic/.typeAttributes.dict, ccode = -1

1149642954: smsfetFileSpec: smsGetDataOffsetAndLen failed, myName = VOL3:Marketing & Communications/Transfer/for work study/russo fonts/Template Gothic/.typeAttributes.dict, ccode = -1

1149642954: smsfetFileSpec: smsGetDataOffsetAndLen failed, myName = VOL3:Marketing & Communications/Transfer/for work study/Template Gothic/.typeAttributes.dict, ccode = -1

1149643235: ConnReadData: Connection with 192.168.3.5:2058 closed

1149646500: Connection established by 192.168.3.5:2086

1149648056: smsfetFileSpec: Big Problem! No more data sets

1149648326: connTcpConnection: invalid code found: 111

1149648327: ConnReadData: Connection with 192.168.3.5:2086 closed

---

 

There are a lot of "smsGetDataOffsetAndLen failed" errors, but they don't seem to be fatal. The fatal error seems to be "Big Problem! No more data sets". VOL1 is the volume that is failing. SYS and VOL3 backup OK.

 

Any ideas?

 

-matthew

[nekr0phage]

Hi Matthew,

 

There are two instances of the 'Big Problem!' message listed in the log, so perhaps that is not fatal. It would seem more likely that the message'connTcpConnection: invalid code found: 111' is where the failure occurs.

 

Have you been able to find out what ccode -1 means? Is volume1 on it's own physical disk, or is it a partition on a disk with the other (succesful) volumes?

[misleb]

The reason I suspect taht "Big Problem!" is fatal (aside from the exclamation mark) is that it occurs right after the last VOL1 error and it doesn't show up until after the backup server finishes VOL3 and disconnects. That 111 error also doesn't occur until after the backup server disconnect.

 

I don't know what the ccode -1 means. That is one of the reasons why I am posting it here. VOL1 and VOL3 are in the same NSS pool on the same RAID volume.

 

-matthew

[nekr0phage]

Hi Matthew,

 

The errors listed in your operations log appear to be errors echoed from the operating system.

 

I googled the line 'connTcpConnection: invalid code found: 111' and came up with a few returns - mostly having to do with Red Hat systems. I've been trying to come up with an error code descriptor list for Netware, but so far have had no luck. Can you check the logs on the netware machine to see if there are any read/communication errors related to the backup?

[misleb]

I actually recently came across the connTcpConnection: invalid code found: 111' error in the logs on one of our FreeBSD servers (running the Retrospect client through Linux emulation). That server isn't have a problem.

 

I couldn't find any Netware error messages related to the backup. As far as Netware is concerned, everything is AOK.

 

-matthew

[Nyle]

I hate to sound like a me too person but I just purchased Retrospect 7.5 and installed it fresh on a Dell Precision workstation. (Latest patches). I then downloaded the latest client and installed it onto our multiple Novell Netware 6.5sp5 servers with no known issues.

 

I receive the same error from two of my servers and Restrospect ultimately crashes receiving some kind og a network error. The crash reports that a power failure may have occured which has not.

 

 

Have you had any resolution to this issue. I'd say it's confirmed if two different environments are having the same issue.

[nekr0phage]

Quote:

---

I'd say it's confirmed if two different environments are having the same issue.

 

---

 

You'd say what is confirmed? What issue are you having that is the same?

 

Matthew and Nyle, could you please confirm that your Netware 6.5 systems meet these minimums, since we haven't brought it up yet?

 

NetWare 6.5 client computers require:

 

* Service Pack 3

* eDirectory 8.7.3 and above

* Smdr 6.54a

* Smsut 1.01

* TSAFS 6.50.11

* Dsbk 10510.81.05

* Backupcr 10510.81.05

* Libcdebg.nlm 1.00.00

 

And Nyle, could you be more specific in what you are seeing? Perhaps post some log excerpts to show that you're getting the same thing as Matthew?

[Nyle]

I would be happy to give more information. When I say confirmed, what I mean is that we are getting the same Error -557 errors described in the thread so it's not just one environment seeing them. My logs looks similar to what he's seeing and I'm running the latest patches for Retrospect 7.5 and Retro Novell client.

 

This is the notice I get sent ->

 

Script: HD BOCES Backup Schedule

Client: JEFFMAIL

Date: 6/12/2006

Trouble reading files, error -557 (transaction already complete)

 

Snippet from log -

1150144750: Main: Retrospect Client for Netware, 1.00.141

..

..

1150154255: smsfetFileFetch: BackupNDS failed with error ffffffff

1150154276: smsfetFileSpec: Can't open data set "Server Specific Info", error FFFDFFCC

1150160320: smsfetFileSpec: smsGetDataOffsetAndLen failed, myName = STORAGE:Apps/ProgramKiller/progkill.dll, ccode = -1

...

...

1150161408: smsfetFileSpec: Big Problem! No more data sets

 

Novell 6.5 Service Pack 5 (As per my post)

Yes, 8.7.3 latest patch

smdr 6.54.02

smsut 1.01

TSAFS 6.51

DSBK 0553.87.00 (I have eDirectory backup turned off currently because it seemed like it might be causing the crash - However, it didn't help and seems to be something else in the client.)

Backupcr 10533.87

Libcdebg.nlm??? - The only reference to this NLM I find using Google, EMC Knowledgebase and Novell's Knowledgebase is in the readme for Retrospect 7.0 listing Libcdebg.nlm 1.0 as a requirement. Where can this file be found as it's not on any of my servers, my workstation and I can't seem to find it on the Internet. (I may have missed it but where?) Seems to be a debugging NLM for the libc routines. I'd assume it'd only be called to do debugging of libc.

 

3 - Dell PowerEdge 2650 servers with dual processors, 2GB RAM, SCSI RAID 5 137GB arrarys.

X - Mix of other older Dell PowerEdge servers that all exceed the minimum specifications for what they are running. Primarily file and printers sharing(NDPS).

 

Backup Workstation - Dell Precision 380 1GB Ram, Windows XP Pro sp2(firewall off) backing up to SATA RAID 5 2TB drive array(JBOD) and hopefully to Sectra Logic Tree Frog 1.5TB tape library for offsite storage.

 

Any other information that would be helpful, I'll gladly provide. I'm really not just here to complain I want to fix the problem and move on. I just spent over $1K on Retrospect 7.5 to migrate from Symantec Backup Exec 9.1 to it. I need it up now for reliable backups.

 

Any help would be grealtly appreciated and I applogize if my post seemed otherwise.

 

Thanks. -Nyle

[Nyle]

Sorry to second post but I wanted to mention that all my servers meet the specifications listed they are all running the same patch levels of Novell 6.5 with SP5 and one of the latest SMS patches beyond SP5.

 

I also get slower backup performance using the client then going to a UNC path but I'll save that for another thread after this is stable.

 

Again, I greatly appreciate any help.

[misleb]

I have the same NLM versions except for SMDR. I have version 6.54.04.

 

And yes, help *would* be appreciated. ;-)

 

I was considering opening up regular (paid?) tech support incident, but since it is now more than me with the problem, perhaps it is best resolved here.

 

-matthew

[Nyle]

I've upgraded my SMS to match yours but the problem didn't change. I've applied all current Novell patches. My servers are running fine, I have no health check issues with DS and none of my volumes are reporting issues.

 

I have paid support contract and I've emailed them about it. Maybe I should call for them to concentrate on the issue more.

 

-Nyle

[Nyle]

Anyone have any new ideas or is this just a major bug in the Netware Client? With EMC, I've sent logs, checked files to see if they exist, completely redid my Retrospect configuration, checked all the versions of files on the server. turned off eDir backup and just did volumes.

 

The only work around that I have that is really not acceptable long term is to back up the volumes directly using UNC paths.

 

The client should make scanning volumes for what to back up faster, it should make sending the data faster and it should support backing up the current state of eDirectory using Novell's built in program.

 

So far it's made scanning volumes for what to back up slower, made sending the data slower and failed backing up edirectory. (Trouble reading files, error -557 (transaction already complete)) This same error occurs when backing up volumes.

 

There isn't any details that I can find on whata -557 error really is but when I see a "transaction already complete" in programming, it sounds like the program is trying to do something again that it already did. Either because it didn't receive a response to a request or it lost the response to the request. Without having access to the source code I'm at a loss.

 

Anyone else out there using Retrospect 7.5 with the Netware client care to comment. Is it working for you? Maybe you can help find what is different in my environment. I'd hate to run poolverify again on all my servers, I'm not seeing any inconsitencies or errors from other software. My old Symantec Backup Exec 9.1 doesn't have any problems backing up all the files using it's clients.

 

HELP, please.

[Nyle]

Any updates to this problem? I'm still working with EMC on it. I've blown away my configuration, updated to the latest version just released, done detailed logging and even installed restrospect onto another computer and still received the -557 transaction complete errors.

 

Anyone else get any further with this? Really getting frustrated as a new customer I really need a reliable backup solution and this problem is making me nervous.

 

Thanks.

[misleb]

*bump*

 

Any update on this? I still can't fully backup one of my Netware servers.

[RonaldL]

Hi misleb,

 

I'm currently working with Nyle on this issue and we have a bug opened for it. I will PM you my email address so that I can collect some logs from you for this error. Thanks.

[zakrz1]

I'm not seeing the error code... Retrospect just hangs up while backing up Netware (6.5 SP5).

I can't find anything on LIBCDEBG either....

Zak

[Nyle]

*bump*

 

Now Retropect doesn't -557 it just asserts when trying to backup my Novell EMC Client servers. I've sent all kinds of debugging information in multiple times using debug builds. My trouble ticket has switched hands twice now. Still no resolution to the original problem but I guess they technically got rid of the -557.

 

Now it asserts instead. Anyone have any tips to getting 7.5 to work with Novell 6.5sp5?

 

-Nyle

[Nyle]

Working on it with me? Well they were. I don't hear back on my trouble ticket anymore and that's after sending a lot of logging information. We still can't back up our Novell servers using the client.

 

I am not impressed at all with EMC Retrospect support.

[cfweb]

Was there ever resolution to this problem? We have recently started seeing error -557's being returned on several of our clients.

 

Running retrospect 7.5 on a WinXP box (fully patched), backing up Win2k, WinXP, and Red Hat clients.

[usernamething]

Hello I have a similar issue. I have recently upgraded everything to 7.5 from 6.5 because of stupid vista. I have found 7.5 to be a buggy pos with frequent crashes. Todays problem is similar to yours. I am getting the following error when trying to view a clients "volumes" under the properties tab: "Can't track volumes, error -557 (transaction already complete)

 

The only thing different about this client is that I ghosted the hard drive like last week, so its not the original drive. I tried deleting and re adding the client in retrospect but no dice. Client machine in winxp_sp2. Previously (6.5), when ghosting a hard drive, I noticed that you had to re add the new drive to the backup. Perhaps it has something to do with this?

 

I am not running novel. I was going to create a new thread but the frontpage seems to be crowded with threads with no replies. Dantz Retrospect used to be a really stable product, what happened?