Assertion failure at "tmemory.cpp-1709"

[cummings]

can anybody give some insight on how to prevent the following error?

 

 

 

 

 

 

it was followed by these two errors:

 

 

 

 

 

 

 

 

 

note: there was no power failure, and windows did not crash.

 

 

 

i am currently running:

 

 

 

 

 

 

thank you.

[Lennart_T]

The second dialog says "not enough memory". The first one also suggest lack of memory.

 

 

 

Install more RAM.

 

 

 

How much RAM do you have? How many files are you trying to backup? Running anything else but Retrospect at the same time?

[cummings]

the retrospect server has 2GB of RAM. the individual servers being backed up range from 2GB to 4GB each. i hope that would be enough. =)

 

we are backing up a lot of files, it hits 5 servers on disaster recovery and they range from about 20GB to 600GB of files per server.

 

nothing except retrospect is running on the server. on the client machines, other apps are running.

 

another message i have gotten since (on about 2,000 files in a row) since then is.....

 

File "E:\TrueflowWork\job\JobRoot\1593_YRB\.db\Contents\YRB052_R1.pdf\Pages\YRB052_R1.pdf.1.tif": can't read, error -1019 (not enough resources)

 

any suggestions?

[Lennart_T]

How many files do you have in the storageset? I'm not talking about their size here, just the number of files.

[waltr]

hi c,

 

how much hard drive space is left on the computer? check your page outs, how does that look? also, what is the configuration of RAM? could you be hitting a bad stick? you may want to replace some RAM to test it. it sounds like nothing else on the computer will use the higher addresses of RAM so this would be a problem you'd only see in Retrospect if you had some bad RAM. Retrospect is a resource HOG, as well it should be.

 

what does the event viewer say when you get these errors? does a reboot help the situation?

[cummings]

1. the storageset of our disaster recovery (going to LTO3) contained 62,870 files on its last 'normal' backup.

 

2. on the retrospect server, c: 28GB free (6 used), d: 1.5TB free (for d2d), f: 715GB free (for b2d), g: 1TB free (for b2d). LTO3: using 6th tape of 11.

 

3. how do i check page outs? are you referring to performance monitor: pages output/sec? what is a good value for this vs. a bad value?

 

4. does this look like a good program to use to determine if the RAM in the retrospect server is ok? http://hcidesign.com/memtest/

 

5. this is the error created by event viewer at the crash. there were several of them similar to this with different computer names in the description. since reboot, i haven't had this error again (yet).

 

Event Type: Error

Event Source: Retrospect

Event Category: None

Event ID: 1

Date: 4/21/2006

Time: 5:01:10 PM

User: D2DSERVER\retrospect

Computer: D2DSERVER

Description:

Trouble matching TRUEFLOW_SYSTEM (D:) on TF038 to TF38, error -625 (not enough memory)

 

 

thanks for everyone's help so far.

[waltr]

hi c,

 

there's something weird about your description of the setup.

 

Quote:

---

nothing except retrospect is running on the server.

 

---

 

but you get this error:

 

Quote:

---

File "E:\TrueflowWork\job\JobRoot\1593_YRB\.db\Contents\YRB052_R1.pdf\Pages\YRB052_R1.pdf.1.tif": can't read, error -1019 (not enough resources)

 

---

 

which i thought originally was on a client, but then in the Event Viewer you saw:

 

Quote:

---

Event Type: Error

Event Source: Retrospect

Event Category: None

Event ID: 1

Date: 4/21/2006

Time: 5:01:10 PM

User: D2DSERVER\retrospect

Computer: D2DSERVER

Description:

Trouble matching TRUEFLOW_SYSTEM (D:) on TF038 to TF38, error -625 (not enough memory)

 

---

 

which indicates that this "Trueflow" application is running on the local machine. is this correct? i looked up Trueflow:

 

http://www.screen.co.jp/ga_dtp/product_e/trueflow/trueflowE.html

 

and it looks like it could use a lot of RAM, but there are several versions and i don't know which you might be using. could it be that this program and Retrospect are competing for resources?

[cummings]

trueflow is running on the machine using the retrospect client, not retrospect server. the event viewer error is reported on the retrospect server. there are no retrospect errors in the app log on the machine running retrospect client.

 

do you think the "not enough memory" error could be referring to the source (machine running the client) and not on the retrospect server? i assumed it was referring to the retrospect server (which only runs retrospect server 7.5).

[waltr]

hi c,

 

interesting. that's what i originally thought, but the Event Viewer message threw me.

 

i've never seen "not enough memory" errors that don't refer to the server machine, and i don't think it's refering to the client computer, but that is one i haven't seen before.

 

does the assert only happen when backing up this client? do you think you could run a test and _only_ backup this client and see if that's the case? also, when the assert happens do you see any odd behavior on the client or do your users report anything unusual? if we've narrowed it down to just this client computer, what's the OS? do you use the "open file" feature to back this computer up?

[cummings]

here is the app log (warnings and errors only) leading up to the crash. the memory error was happening on all machines. we don['t use the 'open file' feature and users did not complain of any odd behaviour on the machine (to my knowledge).

 

side note: of course i took last friday off, and the errors started last thursday at 5:11pm *sigh* mr. murphy, you are cruel. =)

 

4/21/2006 5:01:10 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching TRUEFLOW_SYSTEM (D:) on TF038 to TF38, error -625 (not enough memory)

4/21/2006 5:00:49 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching Local Disk (C:) on TF038 to TF38, error -625 (not enough memory)

4/21/2006 12:32:32 PM Retrospect Information None 2 D2DSERVER\retrospect D2DSERVER "Script ""RITE_ONLINE"" completed successfully"

4/21/2006 11:32:45 AM Retrospect Information None 2 D2DSERVER\retrospect D2DSERVER "Script ""FSPP"" completed successfully"

4/21/2006 11:13:13 AM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER "Script ""TF37"" incomplete"

4/21/2006 11:13:09 AM Retrospect Information None 2 D2DSERVER\retrospect D2DSERVER "From Retrospect: Script ""TF37"" failed during automatic execution, error -625 (not enough memory). Please launch Retrospect and check the log for details."

4/21/2006 11:12:29 AM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching TRUEFLOW_WORK (E:) on TF037 to TF37, error -625 (not enough memory)

4/21/2006 11:10:04 AM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER "Script ""TF38"" incomplete"

4/21/2006 11:09:59 AM Retrospect Information None 2 D2DSERVER\retrospect D2DSERVER "From Retrospect: Script ""TF38"" failed during automatic execution, error -625 (not enough memory). Please launch Retrospect and check the log for details."

4/21/2006 11:09:53 AM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching Xtra_Space (X:) on TF038 to TF38, error -625 (not enough memory)

4/21/2006 11:01:15 AM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching TRUEFLOW_SYSTEM (D:) on TF038 to TF38, error -625 (not enough memory)

4/21/2006 11:00:54 AM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching Local Disk (C:) on TF038 to TF38, error -625 (not enough memory)

4/20/2006 10:19:15 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER "Script ""Disaster Recovery"" incomplete"

4/20/2006 10:19:10 PM Retrospect Information None 2 D2DSERVER\retrospect D2DSERVER "From Retrospect: Script ""Disaster Recovery"" failed during automatic execution, error -625 (not enough memory). Please launch Retrospect and check the log for details."

4/20/2006 10:19:03 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching Xtra_Space (X:) on TF038 to Offsite Set B, error -625 (not enough memory)

4/20/2006 10:11:13 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching TRUEFLOW_SYSTEM (D:) on TF038 to Offsite Set B, error -625 (not enough memory)

4/20/2006 10:10:52 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching Local Disk (C:) on TF038 to Offsite Set B, error -625 (not enough memory)

4/20/2006 10:10:15 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching Xtra_Space (X:) on TF037 to Offsite Set B, error -625 (not enough memory)

4/20/2006 10:09:58 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching TRUEFLOW_WORK (E:) on TF037 to Offsite Set B, error -625 (not enough memory)

4/20/2006 10:05:28 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching TRUEFLOW_SYSTEM (D:) on TF037 to Offsite Set B, error -625 (not enough memory)

4/20/2006 10:05:09 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching Local Disk (C:) on TF037 to Offsite Set B, error -625 (not enough memory)

4/20/2006 10:04:27 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching Local Disk (C:) on SANEngine1 to Offsite Set B, error -625 (not enough memory)

4/20/2006 10:04:01 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching SAN_Storage (D:) on riteonline to Offsite Set B, error -625 (not enough memory)

4/20/2006 10:03:44 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching Local Disk (C:) on riteonline to Offsite Set B, error -625 (not enough memory)

4/20/2006 10:01:19 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching SAN_Storage (D:) on FSPP to Offsite Set B, error -625 (not enough memory)

4/20/2006 10:00:27 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching Local Disk (C:) on FSPP to Offsite Set B, error -625 (not enough memory)

4/20/2006 6:32:34 PM Retrospect Information None 2 D2DSERVER\retrospect D2DSERVER "Script ""RITE_ONLINE"" completed successfully"

4/20/2006 5:32:34 PM Retrospect Information None 2 D2DSERVER\retrospect D2DSERVER "Script ""FSPP"" completed successfully"

4/20/2006 5:13:53 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER "Script ""TF38"" incomplete"

4/20/2006 5:13:47 PM Retrospect Information None 2 D2DSERVER\retrospect D2DSERVER "From Retrospect: Script ""TF38"" failed during automatic execution, error -625 (not enough memory). Please launch Retrospect and check the log for details."

4/20/2006 5:13:40 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching Xtra_Space (X:) on TF038 to TF38, error -625 (not enough memory)

4/20/2006 5:11:56 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER "Script ""TF37"" incomplete"

4/20/2006 5:11:51 PM Retrospect Information None 2 D2DSERVER\retrospect D2DSERVER "From Retrospect: Script ""TF37"" failed during automatic execution, error -625 (not enough memory). Please launch Retrospect and check the log for details."

4/20/2006 5:11:11 PM Retrospect Error None 1 D2DSERVER\retrospect D2DSERVER Trouble matching TRUEFLOW_WORK (E:) on TF037 to TF37, error -625 (not enough memory)

[waltr]

hi c,

 

so all of this was working before, then the problem happened last friday and you haven't seen it since? is friday's backup a "full" or is there anything different about it?

 

1) so only 67,000 files in a backup of 5 servers. this seems a little low, is that from an incremental backup?

2) that looks good

3) here are some URL's to check out. i'm listing them easy to hard:

http://www.hal-pc.org/journal/2004/04_june/babystep.html

http://www.rojakpot.com/showarticle.aspx?artno=143&pgno=0

http://www.microsoft.com/technet/prodtechnol/windows2000serv/tips/w2mem.mspx

http://www.edbott.com/weblog/?p=687

4) i haven't used that program, but it looks fine

5) since a reboot has "fixed" the error for now, you may want to check out the above and monitor the system for this type of problem. if it is recurring, you may want to give a call to EMC Insignia Tech Support and report it.

[cummings]

1. 67,000 is a 'normal' (incremental) backup. on friday's the 10pm disaster recovery script becomes a 'recycle' (full) backup. the memory problems started thursday at 511pm, on a typical day of backing up.

2. thanks

3. thanks

4. thanks

5. i will not be taking this friday off so we'll see if tonight (thursday) the same thing happens again or not.

 

one thing i just noticed is that the B2Ds and the tape (disaster recovery) gave memory errors, but the D2D (FSPP script) never failed. just a point of interest....

[cummings]

this week's recycle backup gave several hundred "not enough resources" errors.

 

one example from our "TF37" machine backup...

 

File E:\TrueflowWork\job\JobRoot\PTwGATF\.db\Contents\wGATF_R4.pdf\Pages\wGATF_R4.pdf.9.tif": can't read, error -1019 (not enough resources)

4/29/2006 11:34:56 AM: Snapshot stored, 71.2 MB

4/29/2006 11:35:04 AM: 357 execution errors

Remaining: 357 files, 16.2 GB

Completed: 284271 files, 203.3 GB

 

one example from our "TF38" machine backup...

 

File "E:\TrueflowWork\job\JobRoot\1722_MCH\.db\Contents\MCH040[40].pdf\Pages\MCH040[40].pdf.1.tif": can't read, error -1019 (not enough resources)

4/29/2006 7:40:07 PM: Snapshot stored, 129.4 MB

4/29/2006 7:40:22 PM: 205 execution errors

Remaining: 205 files, 6.7 GB

Completed: 549528 files, 191.4 GB

 

 

here is the final entry in the log showing 562 errors (due to 'not enough resources').

 

- 4/29/2006 7:41:05 PM: Verifying Offsite Set B

4/29/2006 8:07:25 PM: Execution completed successfully

Completed: 974889 files, 1,012.7 GB

Performance: 3996.7 MB/minute

Duration: 04:19:59 (00:00:32 idle/loading/preparing)

 

4/29/2006 8:07:36 PM: 562 execution errors

Total duration: 22:07:24 (00:37:25 idle/loading/preparing)

 

any suggestions on why i get these 'not enough resources' errors each recycle backup on friday night? it only happens when backing up these two servers, the other machines in the script are doing fine.

 

thanks.

[Lennart_T]

Quote:

---

 

it only happens when backing up these two servers, the other machines in the script are doing fine.

 

 

---

Hi.

 

What if you change the order of the machines, does the errors move, too?

[cummings]

i moved the machines from 4th and 5th to 2nd and 3rd. i'll let you know if it makes a difference. thanks.

[cummings]

i changed the order of machines and i still get the same errors on the same machines.

 

i got 536 of these errors...

 

File "E:\TrueflowWork\job\JobRoot\1593_YRB\.db\Contents\YRB104_111[104-111].pdf\Pages\YRB104_111[104-111].pdf.1.tr1.pdf": can't read, error -1019 (not enough resources)

 

here are the summaries from those two machines.

 

5/6/2006 7:20:23 PM: Snapshot stored, 124.5 MB

5/6/2006 7:20:38 PM: 305 execution errors

Remaining: 305 files, 10.2 GB

Completed: 527865 files, 197.0 GB

Performance: 628.4 MB/minute

Duration: 05:31:50 (00:10:52 idle/loading/preparing)

 

5/6/2006 1:39:59 PM: Snapshot stored, 73.1 MB

5/6/2006 1:40:16 PM: 231 execution errors

Remaining: 231 files, 11.6 GB

Completed: 279694 files, 217.0 GB

Performance: 934.1 MB/minute

Duration: 04:06:55 (00:09:09 idle/loading/preparing)

 

any ideas on what i can do so over 500 files aren't getting skipped on my weekly recycle disaster recovery? TIA

[Lennart_T]

Quote:

---

File "E:\TrueflowWork\job\JobRoot\1593_YRB\.db\Contents\YRB104_111[104-111].pdf\Pages\YRB104_111[104-111].pdf.1.tr1.pdf": can't read, error -1019 (not enough resources)

 

 

---

Hi.

 

I thought that [ and ] was forbidden in a Windows file name.

I thought a Windows file/folder name couldn't start with a period.

 

http://www.xvsxp.com/files/forbidden.php

(Scroll down a bit to the Windows XP section)

 

Is there any error logs on the clients that can shed some light on the issue?

 

I looked through old threads for the -1019 error. One occured in version 6.5 and was fixed in 7.0. The other I found was due to an almost full system disk. So, how much free space are there on the client's C: disk?

 

Regards

Lennart

[cummings]

interesting results.........

 

if i use windows explorer, i cannot create a folder named ".test", however if i use total commander (a third-party file manager for windows), i CAN create a ".test" directory. i can also copy a folder from another machine which starts with a period with no problem. very bizarre. i don't think this is the problem though because there are a lot of successful backups of folders that start with ".db". it is odd, though, that these ".db" folders are where all of my errors are coming from.

 

i am able to create files with brackets "this is a [test] file.txt".

 

there is plenty of free space on the C drives of both client machines.

 

i am going to contact the software support and ask them what all these ".db" folders are and see if they can help me figure out why i get so many errors trying to back them up. hopefully they tell me that these folders don't need to be backed up (some kind of temp folders). that would be nice.

 

thanks for the help thus far. i will report back.