Jump to content

Retro8 Log Analyser


alche

Recommended Posts

I've spent the last week developing a Filemaker 10 based log analysis program for Retro8. It works pretty well, and I've prettified it and made it suitable for public consumption.

 

Can be downloaded from here:

my website (lintburger.com)

 

The idea was to get a more systematic idea of what's going on with my backups, so summaries, running totals, and counts of errors and throughput are the goal here. There's even a chart function.

 

Chart.sm.png

 

Free to use, free to modify. Screenshots and more details are on software page. Hope you find it useful.

 

Cheers,

Paul Fabris

Link to comment
Share on other sites

I've updated Retro8LogAnalyser to version 1.1

 

The new version allows you to mindlessly paste data from the Retrospect Log window without having to worry if it overlaps data you previously pasted in. Should make life a little easier. It will automatically keep track of the last timestamp in the log data. The next time you paste in data, it will clear the pasted data up to that last timestamp, leaving only the new log lines to process.

 

I've also added an update engine that allows you to upgrade to future versions without having to reprocess all your data again. This won't work with the data in 1.0, but will work when I release future versions.

 

A few other bug fixes.

 

Can be downloaded from the Lintburger Software Page

 

Cheers,

Paul

Edited by Guest
Link to comment
Share on other sites

New in Version 1.2 (Dec 4 2009)

 

  • Charting: Added Machine Charting. Allows you to chart the growth of backups over time for each machine, by Media Set and Date Range.
  • Charting: Option-Click "Generate" button to automatically update the chart to the current date and move the ceiling if necessary. Shift-Click to Generate charts for a found set of records with the same parameters. Option-Shift-Click to update a Found Set of Charts.
  • Charting: Improvements allow for proper printing of charts
  • Added List Views for Exec Events and Charts
  • Improvements in automatic log line deduplication; a dialog now shows you where the log line processing will start after deduplicating
  • Other minor bug fixes and enhancements

 

You can import data from Version 1.1. Download the new version, launch it, pull down the "Scripts" menu and choose the "Import from an Older Copy of Retro8LogAnalyser".

 

Download from Lintburger Software Page

 

Machine Charting Screenshot:

Machines.sm.png

Link to comment
Share on other sites

Thanks!

 

As for taking directly from the server log file, I did try, but the operations_log.utx is much less readable to the layperson. It contains what appears to be variable placeholders and some raw code that I can't quite grasp.

 

If a future version of Retro8 wrote the readable log file out to a directory somewhere (which would be specifiable in a preference setting), then I could easily just suck in the new data into the Analyser without forcing anyone to copy/paste, and without needing to do any date parsing to figure out where the log overlap was. I would simply keep track of the last line processed and only import data from there.

 

The copy paste method isn't the end of the world, but it can introduce problems. A few weeks ago I deleted several SVN directories from my web server, apparently mid-backup. The Retro Log promptly filled up with around 20,000 lines of file errors. The next morning, i did my usual copy paste and it buggered up my log processor. I have my Log Display set to 2000 lines and so it wasn't making it anywere near the start of the previous nights backups, which kinda screwed everything up. (Luckily, there's the "Delete from last Import" script to deal with these problems). I had to change my Retro Log pref to view 30,000 lines, copy/paste, then swich my pref back to the usual 2000 lines.

 

If I we were looking at a static file instead, I wouldn't have had this problem, and I could make the Analyser process the logs unattended on a timed basis, and even send emails. (Of course, this would introduce the problem of having a log file of unlimited size eating up drive space. Mind you, I haven't seen operations_log.utx rotate, or clip early entries, so this may already be the case.)

 

Until then it's CMD-L, CMD-A, CMD-C, CMD-TAB, CMD-1, CMD-V.

 

Actually, a bigger problem is data inaccuracy in the log file. The CMD-L reports data sizes in GB/MB/KB as needed. This introduces rounding errors when compounding lots of log lines and trying to add them up--not a huge problem but there might be up to a GB of error after a lot of executions. Worse than that is when Retrospect crashes mid-execution. I had this happen over the weekend. Nothing from the execution made it into the operations_log.utx file at all. But my backup disk had 15GB more than the day before. The catalog had the data, but the log buffer didn't get written out before the crash occurred.

 

So what would be really nice would be a way to extract raw columnar data from the catalogue(s), on a timed basis. Or if Retro wrote out the columnar data for each catalogue as it was updated.

 

Anyway, I'm working on making the charts more flexible for version 1.3, plus a few other odds and ends.

Link to comment
Share on other sites

Absolutly lovely!

 

Great work. Only wish would be to make an export to file instead of copy paste from Retro. Or to take directly from server-log.

 

But really nice work MCI - impressive.

Ditto. I have yet to add more log data (cutting and pasting doesn't seem too bad for me), even though it would be nice to simply have it pick up the new data.

 

For some reason the machine list doesn't populate like the Media Set do (drop down) and must enter the machine name to chart by new record as warning indicates. But after that, it great!

Link to comment
Share on other sites

Try the "Generate Machine Records" script under the Scripts menu. Will automatically generate Chart records for all the Machines it can find in the Exec Events table.

 

Otherwise, yes, you have to create records manually for Machines. Figured some people with large lists of machines might not want automatic here.

 

And, agreed about the copy/paste, but sadly can't do anything about it at this point (see my post above).

 

Cheers,

PF

Link to comment
Share on other sites

  • 2 weeks later...

I have the same issue with dates...and no FMpro to amend...International Format is set to Canada...I changed to US to see if it changes something to the log but it doesn't seem to.

 

edit : the change to US M/D/Y actually worked.

 

Nevertheless, thank you for the nice piece of work!

Edited by Guest
Link to comment
Share on other sites

Version 1.2.1 is out. It addresses the issue of when the International System Preference is not set to use US formats by embedding the US format in the Filemaker Pro database file (effectively ignoring your International setting). This is necessary due to the fact that Retro Log Data uses dates that are always mm/dd/yyyy. I believe this should address the issues posted above without requiring user intervention. Please let me know.

 

There's a couple of other minor tweaks and fixes as well.

 

Download from Lintburger Software Page

 

Cheers,

Paul

Link to comment
Share on other sites

  • 1 month later...
I've updated Retro8LogAnalyser to version 1.1

 

The new version allows you to mindlessly paste data from the Retrospect Log window without having to worry if it overlaps data you previously pasted in. Should make life a little easier. It will automatically keep track of the last timestamp in the log data. The next time you paste in data, it will clear the pasted data up to that last timestamp, leaving only the new log lines to process.

 

I've also added an update engine that allows you to upgrade to future versions without having to reprocess all your data again. This won't work with the data in 1.0, but will work when I release future versions.

 

A few other bug fixes.

 

Can be downloaded from the Lintburger Software Page

 

Cheers,

Paul

FileMaker 11 (now out) adds a new repeating import facility. Would this be useful for your database so it can automatically import new log data from Retrospect? (I have Retrospect 11.)

Link to comment
Share on other sites

FileMaker 11 (now out) adds a new repeating import facility. Would this be useful for your database so it can automatically import new log data from Retrospect? (I have Retrospect 11.)

 

Hi John,

 

The problem isn't FileMaker's ability to import on a recurring basis (that can be done in a number of different ways). The problem is that the RetroLog.utx file, where all the log data is stored, is incomprehensible to me. It's filled with what appear to be variables that only make sense to the RetroConsole App, in conjunction with a Server configuration. I can't parse this file. I have to rely on the "interpreted" output of this file, which is currently only available by using the Show Log command from within the RetroConsole app.

 

2 ways around this:

1. Retro Engineers do something that allows RetroServer to output a plain text log file using the built in scheduling engine, preferably in a coherent delimited format. (Currently, I have to read the log data almost as if it were a story.)

2. You could write an Applescript that does the following:

  • Launch Retro Console
  • wait
  • issue a CMD-L
  • wait
  • CMD-A
  • CMD-C
  • Launch RetroLogAnalyser
  • wait
  • issue a CMD-1
  • wait
  • CMD-V
  • ENTER
  • wait
  • ENTER

Tie it to a crontab on a computer that's running all the time (without screensaver) and you've got Automation.

Link to comment
Share on other sites

  • 1 month later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...