Retrosepct Server packed in!

[Richy_Boy]

Hi Jeff;

 

Yeah, I know where the catalogue files are (in my case c:\retrospect catalog files\).

 

Deleted BOTH the back and dat files wiped out all the client lists, filters etc - which I don't have time to rebuild right now.

 

It's strange how deleting (renaming) the dat file only removed the proactive backup settings though when I re-launched retrospect. All my client list and server backups remained untouched.

 

My server backups have kicked off now, which seem to be working fine, so I can't tamper with the server too much now. It's still not allowing me to stop/pause the proactive backups though, just ignored my request and refuses to close.

 

Rich

[eppinizer]

Are you sure you don't have config files from previous versions of retrospect that are generating new 7.5's?

[Richy_Boy]

There is nothing in the operations log since the last two completed backups happend. No errors, nothing.

 

I renamed the 7.0 config files after twigging that's where it was getting the data from. Once I deleted these it opened Restrospect completely blank (i.e. no client list, nothing).

 

So I then put the 75.bak file back in place and launched Retrospect and manually rebuilt the proactive scripts and it worked for two clients then did the same thing.

 

Each of these clients was backup up to each of the two drives (Wednesday and online).

 

Rich

Edited April 9, 2008 by Guest

[eppinizer]

Rich, I hadn't completely the beginning of this thread so I don't think I understood as well as I thought I did.

 

I am not Tech Support, and I am at work right now. I Like helping on the forums but I have a system I have to setup at the moment. If no one else from TS gets to this I should be back in 20-30 mins or so.

 

This seems a bit unusual, Was everything working fine previously?

-Jeff

[Richy_Boy]

Everything was working fine before this... thanks for your help.

 

I really should get back to bed. My 'sick day' has now turned into a 10 hour working day - yippee!

 

Rich

[eppinizer]

Rich,

There are a couple of things that lead me to think that the problem is generated from your Friday disk/backup set.

 

It's odd as both the catalog and destination files/drives are 100% clean.. Retrospect is also refusing to close, it hangs on 'Exiting'.. so I have to crash the app each time. Grr!

 

Hmm,

 

Ah! I think I've spotted the culprit!

 

by going into the Proactive Activity Monitor and selecting 'Backup Sets' view.. it shows Fridays backup is waiting for the Media!

 

{

Friday - Media

Online - Ready

}

Monday - Inactive

Tuesday - Inactive

Wednesday - Active

Thursday - Inactive

Friday - Inactive

 

I have no idea why it's trying to do something with the Friday disk! (Triple checked the schedule)

 

It seems to me that retrospect is trying to backup your Friday script which is making it look for its media. However, this is unusual because your Friday script appears to be inactive. You also deleted the Friday proactive script. Hanging on exit is an almost typical reaction that I will get when I try to abort a proactive script that is searching for something that is no longer there, or has changed.

 

I noticed a pro-active backup wasn't responding, so went to check the backup sets where it notified me that it was corrupt or seriously damaged. So I spent most of today repairing this thing (we back up ~400GB to a 500GB SATA drives).

 

How long ago did this happen, and how long was retrospect failing to pro-actively back up before you noticed?

 

Did you wipe all of the drives or just a few?

 

I see that you have checked the scripts, and that they are all correct. You also mentioned that your normal scripts were working.

 

Could you try and create a normal script that is identical to the proactive script that runs on Friday?

 

You also might want to try setting your Friday backup to a different disk and see if that fixes your problem. There is something about that disk/ backup set that is causing Retrospect to freak out a bit.

 

I think in the end you may find that (if I am correct in assuming that your Friday disk is not connected) when you connect the Friday disk your problem may solve itself. If it doesn’t, than the result would still give a better clue as to what’s really going on. I would be interested what would the “Backup Sets†dropdown would say after the media is found.

 

Well, good luck and get well!

-Jeff

[Richy_Boy]

Hi Jeff, thanks for your help again.

 

Thye server backups ran fine last night, so that's a relief! I have setup a normal scripted backup for all our clients today - although many are on laptops so will be on and off the network all day long.

 

In regard to the drop down menu in the proactive tab.. it's doing weird stuff again, although since all my hacking around yesterday it no longer seems to need the Friday disk.

 

Now (and this is equally odd) it's saying:

 

Backup Sets selected:

 

Online - Ready

Monday - [nothing]

Tuesday - [nothing]

Wednesday - [nothing]

Thursday - [nothing]

 

Note, Friday is missing entirely.

 

On the scripts tab:

 

Monday - InActive

Tuesday - InActive

Wednesday - Active

Thursday - InActive

Friday - InActive

 

Why would it say Wednesday is still active (double-triple checked the schedule) and webnesdays scripts should have stopped at 11pm last night - Thursday should have kicked off at 1am this morning (to back up any desktops left switched on)

 

Hmm

 

Clients seem to be backing up fine with the script right now - phew. So, in looks like somehow the proactive have been corrupt in a big way - so far even wiping them and starting again didn't work.

 

Except for wiping the entire system and rebuilding all the groups, scripts etc - what else can I try? (I really don't want to rebuild this whole system!!)

 

Rich

Edited April 10, 2008 by Guest

[Richy_Boy]

Getting this in assert_log.utx

 

Thread ID: 108, Name: Main thread

 

EAX:00000000 CS:001b EIP:7c90eb94 EFlags:00000246

EBX:7ffda000 SS:0023 ESP:0012d5f8 EBP: 0012d614

ECX:0012d5c0 DS:0023 ESI:0012d668 FS: 003b

EDX:7c90eb94 ES:0023 EDI:0012d6b4 GS: 0000

 

 

Module Fn (symbol or seg:offset in DLL) Args

=============== ================================= =======================================================================

ntdll.dll 0001:0000db94 12d668 0 0 0 16e9e4 12d684 606116b8 12d668

bdrock20.dll UGetMessage +25 12d668 0 0 0 0 12d918 0 0

meson.dll msgHelper +248 16e9e4 1 12dbe8 12e0d0 7ffda000 0 16e9e4 0

meson.dll doTask +21E 'Msg ' 60611470 0 0 12db74 12db74 1 12dbac

meson.dll TThread::TaskCall +21 16e9e4 'Msg ' 60611470 16e9e4 1 12dbe8 12e0d0 7ffda000

meson.dll TThread::MsgBlock +96 'Msg ' 12e070 6063862e 12e560 6063862e 'LopT' 16e9e4 0

meson.dll msgInTask +1F 12e560 6063862e 'LopT' 16e9e4 0 0 12e5b8 12ea70

meson.dll doTask +21E 'MsgT' 60612700 0 0 12e0a8 12e0a8 16e9e4 12e0a8

meson.dll TThread::TaskCall +21 16e9e4 'MsgT' 60612700 12e560 6063862e 'LopT' 16e9e4 0

meson.dll loopInTask +27 'LopT' 16e9e4 0 0 12e5b8 12ea70 60641132 0

meson.dll doTask +21E 'LopT' 606126d0 0 0 12e598 12e598 28 12e5b8

meson.dll TThread::TaskCall +21 16e9e4 'LopT' 606126d0 'LopT' 16e9e4 0 0 12e5b8

meson.dll TThread::MsgLoop +45 'LopT' 12ea7c 6063862e 0 b 12ef74 6063862e 0

meson.dll mesonGoLoop +14 0 b 12ef74 6063862e 0 12ff14 6111dfdd 0

meson.dll doTask +21E 'Task' 6060a2b0 0 0 12eab4 12eab4 60640fb7 12eabc

meson.dll TThread::TaskCall +21 16e9e4 'Task' 6060a2b0 0 b 12ef74 6063862e 0

meson.dll mesonGo +129 0 12ff14 6111dfdd 0 0 0 12f070 7c91ce16

meson.dll doTask +21E 'Prog' 6060a170 0 0 12efac 12efac 6006722b 12efb0

meson.dll TThread::TaskCall +21 16e9e4 'Prog' 6060a170 0 12ff14 6111dfdd 0 0

meson.dll Meson +2C 0 0 0 12f070 7c91ce16 152fa0 9 0

Retrospect.exe _WinMain +4DD 61100000 0 15231a a 166920 bfcb4c 7ffda000 44

Retrospect.exe _WinMainCRTStartup +157 166920 bfcb4c 7ffda000 8054a6ed 12ffc8 85869910 -1 7c839aa8

kernel32.dll 0001:00015fd7

 

[Richy_Boy]

At the risk of repeating myself... yay! It seems to be working now.

 

I noticed that Retrospect was only hanging once I allowed Proactive to start up. So, I stopped all scripts and made a change to ONE of the proactive scripts, assuming that it will rewrite the config file on exit (?).

 

So, I exited (didn't hang) and then re-launched and it appears to be fine again now.

 

What a palava! I've tried so many things I have no idea now if that was the 'solution' or what, but it seems to be jumping through my clients quite happily backing up to both destinations.

 

Rich

[eppinizer]

Thats good news. I hope it stays good.

-Jeff

[Richy_Boy]

Well it's broken again now - joy.

 

For what it's worth it seem if I'm not in the office to change the backup tapes and a groom is scheduled to kick off I seem to get the problem.

 

For the life of me I CANNOT cancel this scheduled groom the following morning so my only option is to crash Retrospect and relaunch. After relaunch it when complains of corrupt catalogs and my proactive backup scripts seem corrupt again. THis then takes half a day of baby sitting to get it back online, which from a business point of view is very costly.

 

This is very annoying and in my opinion, there is something wrong with the software which is stopping me from cancelling the missed groom - which causes this mess.

 

Has anyone else seen this?

 

Rich

Edited May 8, 2008 by Guest

[Richy_Boy]

Just to add to this...

 

Retrospect is adement that it is waiting for the Friday and Monday drive to be inserted?! There is nothing scheduled or missed for these days, so having to crash the app seems to have got its knickers in a twist.

 

I'm going to have to drive into the office and play with drives now, which will cost me more time.

 

Proactive > Backup sets is showing:

 

Online_backup - Ready

Friday - Media

Monday - Media

Thursday - Ready

Tuesday -

 

The Action: is displaying "Checking media"

 

Wednesday and Thursdays drives are in the server right now... I'm having to rebuild the Wednesday drive, hence why it's currently not showing up. The online_backup is always online as it's used for user recovery (external USB drive).

 

Rich

[Richy_Boy]

Ok, put the Friday and Monday drives in... nothing.

 

Joy.

 

Rich

[eppinizer]

Hey Rich,

 

Have you ever actually had a span of time when this setup was working properly? Have you been receiving reoccurring errors?

 

if I'm not in the office to change the backup tapes and a groom is scheduled to kick off I seem to get the problem.

 

When you said "tape" here did you mean disk, or is there a tape drive involved with your setup as well?

 

Try playing around with the polling option in the proactive backup options (more choices). See if you can set it in a way where you wont get this hanging behavior.

 

You can also try setting up a media timeout in preferences > media > request >media request timeout. The default is set to never but it might be beneficial for you for the request to timeout so it doesn't cause problems with your other scripts.

 

there is something wrong with the software which is stopping me from canceling the missed groom - which causes this mess.

 

I'll look into this and see If I can reproduce similar behavior.

 

What is the exact version of Retrospect are you using?

 

Good luck,

-Jeff

[Richy_Boy]

Hey Rich,

 

Have you ever actually had a span of time when this setup was working properly? Have you been receiving reoccurring errors?

 

Sorry, that was me getting carried away. We only use hot-swap SATA backup drives. It works perfectly for several months and only when I'm not able to get into the office to change the drives does it fall over and force me to quit the application as the [missed drive] groom will not cancel.

 

Try playing around with the polling option in the proactive backup options (more choices). See if you can set it in a way where you wont get this hanging behavior.

 

The proactive backups work perfectly until this problem occurs, it appears to be in direct relation to the groom hanging and a forced quit corrupting everything.

 

You can also try setting up a media timeout in preferences > media > request >media request timeout. The default is set to never but it might be beneficial for you for the request to timeout so it doesn't cause problems with your other scripts.

 

The timeout is already set to 4 hours...

 

I'll look into this and see If I can reproduce similar behavior.

 

What is the exact version of Retrospect are you using?

 

Thanks mate, I'm looking at it now trying to get it working for tomorrow.

 

My version is:

Retrospect Multi Server with the addon pack

Version: 7.5.508

 

Our system is pretty simply setup. I have a SATA drive for each day... so I have todays + tomorrows drive in the server ready to roll.

 

The proactive script soley backs up the laptop and desktop machines and copies selected files to both the daily backup drive and a second user recovery external SATA drive.

In the evenings the evening backup scripts kick-off, which backups our domain controllers, exchange, SQL server and anything else useful.

 

 

If all goes wrong if I forget to put the next days drive in the server and then have a morning meeting off-site. A groom script runs every night and grooms the follow days destination to leave plenty of space for the impeding backup.

 

The groom script seems to kick off, waits for the drive, yet then when I get back in the office refuses to quit/cancel (no matter what I press or do). I then have no other choice other than to crash Retrospect, which then relaunches with a shafted proactive script (which seems to forget what day it is?!) and most of the time corrupt catalogs.

The normal scripts never seem to corrupt, only the proactive backups stop working.

 

Hope that helps - and thanks very much for trying to replicate this.

 

Rich

[Richy_Boy]

I should also add, once it's back up after the proactive corruption it seems to be waiting for random drives from a few days back (i.e. shows within the proactive backup sets tab "waiting media").

 

You can put in the requested drives and it'll still not do anything..

 

Rich

Edited May 8, 2008 by Guest

[eppinizer]

Ok, well I just got the same behavior, besides proactive backup becoming corrupt, however I set up a much simpler environment.

 

There is a good chance this is a bug, I'll experiment a little more and get back to you on it.

 

I have to say, I wasn't expecting to be able to reproduce it so easily...

 

-Jeff

[Richy_Boy]

Jeff, you're my hero - I'm not going mad after all.

 

If someone at Retrospect would like to fix my config file I'd be very grateful as proactive is still not working.

 

I'm relying on a normal script right now for client machines

 

Thanks again!

 

Rich

[Richy_Boy]

Weird.

 

I can delete all the proactive scripts.. shut down (which I'm assuming will save the new script file)

 

I then relaunch Retrospect, pause the scripts/proacitve service, create 5 new proactive scripts for each day and then shutdown again.

 

I finally launch again and turn on the scripts and it's still reports 'Checking Media'?!

 

So far I have done the above, but just created Monday and Tuesday scripts and it seems to be working OK. I'll leave it as it is until tomorrow and then create a Wednesday script to see how that goes.

 

Very odd.

 

Rich

[eppinizer]

Rich,

 

The grooming bug has been logged, engineers will start working on it soon. No ETA for the fix at the moment

 

Regarding your scripts. Its possible that it isn't your scripts that are messed up now, but your backup-sets. Have you rebuilt the backup-sets since you had one of those grooming issues? I found that if I had proactive backups running when the groom media request bug had forced me to close retrospect, my backup-sets occasionally would be come corrupt.

 

Try some simples test to check if its your scripts or backup-sets that are giving you problems.

 

Verify your media. Sometimes after I revived the grooming media request I got an error along the lines of 'incomplete grooming'. When I tried to run scripts or verify the media it would not see where the backup-sets media was located. If you have this problem you will need to rebuild the backup-set.

 

Try running immediate backups, or scripting non proactive backups to sets that are reporting 'Checking Media' and see if they run. If these also receive Checking Media status, your Backup-set may be corrupt. Try rebuilding the backup-set

 

Try changing the destination of your current scripts to a test backup-set and see if it runs properly. If you find that retrospect is still checking media than your scripts may be corrupt.

 

If you find your scripts are corrupt try this.

Make some backups of your current config files(config75.dat/.bak) and move them into a safe folder. After those backups are made remove both of your config files out of the folder and start Retrospect with no config files. This will create new clean conifg files that will have no memory of any previous backup-sets, catalog files, scripts, licensee, or preferences. Go ahead and try scripting a proactive backup to one of your old backup-sets that was not working previously (you will have to point retrospect to the catalog file). If this proactive completes with no error, you know that with these config files you shouldn't have issues with proactive backup.

 

I would try not to schedule any grooms that might run without the media inserted until we get the fix for that issue or after you fix yourself, you might start getting the same issues again.

 

Anyways, thats my 2 cents.

GL

-jeff

 

 

[MRIS]

an informal but ultra quick way of checking a disk based backup set is to browse the .rdb files simply with Windows Explorer.

 

Make sure you switch to details view, then switch on the column for DOS Attributes (RSHA). You can then sort on this column to bring the odd-ones out to the top/bottom.

 

All .rdb files should be read-only and have only the R attribute set.

None of the .rdb files should have their archive attribute A set.

 

If either of these above statements does not apply to the files you see, then a media-verify or catalog rebuild will be required.

Edited May 15, 2008 by Guest
mentioned sorting on attribute column

[baxsie]

I have a similar situation.

 

I use several removable 1TB eSATA drives.

 

It has been fairly reliable. I "upgraded" to to:

 

EN_7_5_508 with ru7514102

 

And now when I start up Retrospect, it seems to start OK, saying "Checking media". But if you do anything (change the scheduled date of a proactive backup, for instance) the Retrospect UI hangs.

 

Basically unusable at this point.

 

I have tried deleting:

 

C:\Documents and Settings\All Users\Application Data\Retrospect\Config75.bak

 

this makes no difference. Deleting:

 

C:\Documents and Settings\All Users\Application Data\Retrospect\Config75.dat

 

of course wipes all my proactive scripts.

 

The really weird thing is if I leave it for a long time (more than 5 minutes), it seems to finally get out of its stupor and take off.

 

What in the world is it doing when it goes to "Checking Media" and why in the world would it take so long to time out?

 

So just now, it seemed OK, then I went to change the dates of some proactive scripts to force them to go now. I got a couple switched, then it is back to "Checking Media" with some super long timeout.

 

I can do a directory of the external eSATA backup drive with a different command window, so that hardware is still responsive.

 

Right now the Retrospect window says "Checking media", gives the hour glass cursor, but is otherwise totally unresponsive. I can tell that at least one of the execution threads is going, because the I/O bytes in the task manager are climbing for Retrospect.

 

Just the UI seems frozen with that damn "Checking media" noise.

 

Any suggestions? (Other than wiping 24 proactive scripts and the 120 backup sets that go with them--long story.)

[baxsie]

I found that the super-long "Checking Media" timeouts on my machine were caused by having a mapped drive.

 

Apparently Retrospect tries to "Check Media" on the mapped network drive, and it takes forever.

 

Can anyone else having the "Checking Media" Retrospect timeout problem try disconnecting all the network drives and see if Retrospect gets better?

[Richy_Boy]

I upgraded to Retrospect 7.6 and I'm still seeing these problems. Any news on a fix, this is becoming very frustrating!?

 

I'm now trying to patch this system together again after going on holiday for a week, leaving developers in charge of the backups - bad decision!

 

Annnnnnnyway, a fix would be great.

 

Rich