Jump to content
alche

Retro8 Log Analyser

Recommended Posts

I've spent the last week developing a Filemaker 10 based log analysis program for Retro8. It works pretty well, and I've prettified it and made it suitable for public consumption.

 

Can be downloaded from here:

my website (lintburger.com)

 

The idea was to get a more systematic idea of what's going on with my backups, so summaries, running totals, and counts of errors and throughput are the goal here. There's even a chart function.

 

Chart.sm.png

 

Free to use, free to modify. Screenshots and more details are on software page. Hope you find it useful.

 

Cheers,

Paul Fabris

Share this post


Link to post
Share on other sites

If you made the post / topic, you should be able to delete it. Just click the Edit icon, then click Delete Post near the top navigation bar.

 

Russ

Share this post


Link to post
Share on other sites

I've updated Retro8LogAnalyser to version 1.1

 

The new version allows you to mindlessly paste data from the Retrospect Log window without having to worry if it overlaps data you previously pasted in. Should make life a little easier. It will automatically keep track of the last timestamp in the log data. The next time you paste in data, it will clear the pasted data up to that last timestamp, leaving only the new log lines to process.

 

I've also added an update engine that allows you to upgrade to future versions without having to reprocess all your data again. This won't work with the data in 1.0, but will work when I release future versions.

 

A few other bug fixes.

 

Can be downloaded from the Lintburger Software Page

 

Cheers,

Paul

Edited by Guest

Share this post


Link to post
Share on other sites

New in Version 1.2 (Dec 4 2009)

 

  • Charting: Added Machine Charting. Allows you to chart the growth of backups over time for each machine, by Media Set and Date Range.
  • Charting: Option-Click "Generate" button to automatically update the chart to the current date and move the ceiling if necessary. Shift-Click to Generate charts for a found set of records with the same parameters. Option-Shift-Click to update a Found Set of Charts.
  • Charting: Improvements allow for proper printing of charts
  • Added List Views for Exec Events and Charts
  • Improvements in automatic log line deduplication; a dialog now shows you where the log line processing will start after deduplicating
  • Other minor bug fixes and enhancements

 

You can import data from Version 1.1. Download the new version, launch it, pull down the "Scripts" menu and choose the "Import from an Older Copy of Retro8LogAnalyser".

 

Download from Lintburger Software Page

 

Machine Charting Screenshot:

Machines.sm.png

Share this post


Link to post
Share on other sites

Absolutly lovely!

 

Great work. Only wish would be to make an export to file instead of copy paste from Retro. Or to take directly from server-log.

 

But really nice work MCI - impressive.

Share this post


Link to post
Share on other sites

Thanks!

 

As for taking directly from the server log file, I did try, but the operations_log.utx is much less readable to the layperson. It contains what appears to be variable placeholders and some raw code that I can't quite grasp.

 

If a future version of Retro8 wrote the readable log file out to a directory somewhere (which would be specifiable in a preference setting), then I could easily just suck in the new data into the Analyser without forcing anyone to copy/paste, and without needing to do any date parsing to figure out where the log overlap was. I would simply keep track of the last line processed and only import data from there.

 

The copy paste method isn't the end of the world, but it can introduce problems. A few weeks ago I deleted several SVN directories from my web server, apparently mid-backup. The Retro Log promptly filled up with around 20,000 lines of file errors. The next morning, i did my usual copy paste and it buggered up my log processor. I have my Log Display set to 2000 lines and so it wasn't making it anywere near the start of the previous nights backups, which kinda screwed everything up. (Luckily, there's the "Delete from last Import" script to deal with these problems). I had to change my Retro Log pref to view 30,000 lines, copy/paste, then swich my pref back to the usual 2000 lines.

 

If I we were looking at a static file instead, I wouldn't have had this problem, and I could make the Analyser process the logs unattended on a timed basis, and even send emails. (Of course, this would introduce the problem of having a log file of unlimited size eating up drive space. Mind you, I haven't seen operations_log.utx rotate, or clip early entries, so this may already be the case.)

 

Until then it's CMD-L, CMD-A, CMD-C, CMD-TAB, CMD-1, CMD-V.

 

Actually, a bigger problem is data inaccuracy in the log file. The CMD-L reports data sizes in GB/MB/KB as needed. This introduces rounding errors when compounding lots of log lines and trying to add them up--not a huge problem but there might be up to a GB of error after a lot of executions. Worse than that is when Retrospect crashes mid-execution. I had this happen over the weekend. Nothing from the execution made it into the operations_log.utx file at all. But my backup disk had 15GB more than the day before. The catalog had the data, but the log buffer didn't get written out before the crash occurred.

 

So what would be really nice would be a way to extract raw columnar data from the catalogue(s), on a timed basis. Or if Retro wrote out the columnar data for each catalogue as it was updated.

 

Anyway, I'm working on making the charts more flexible for version 1.3, plus a few other odds and ends.

Share this post


Link to post
Share on other sites
Absolutly lovely!

 

Great work. Only wish would be to make an export to file instead of copy paste from Retro. Or to take directly from server-log.

 

But really nice work MCI - impressive.

Ditto. I have yet to add more log data (cutting and pasting doesn't seem too bad for me), even though it would be nice to simply have it pick up the new data.

 

For some reason the machine list doesn't populate like the Media Set do (drop down) and must enter the machine name to chart by new record as warning indicates. But after that, it great!

Share this post


Link to post
Share on other sites

Try the "Generate Machine Records" script under the Scripts menu. Will automatically generate Chart records for all the Machines it can find in the Exec Events table.

 

Otherwise, yes, you have to create records manually for Machines. Figured some people with large lists of machines might not want automatic here.

 

And, agreed about the copy/paste, but sadly can't do anything about it at this point (see my post above).

 

Cheers,

PF

Share this post


Link to post
Share on other sites

Hi Paul,

 

 

Like the DB. Very useful. I've notced that it is reading my log file for 5th Janaury 2010 (1/5/2010) as May 1, likewise 3 January as March.

 

I've got FMProAdv so can go and amend myself if it is an easy change that you'd recommend.

 

Cheers,

 

Frank

Share this post


Link to post
Share on other sites

That's strange. Can you tell me which screen you are seeing this in? My logs are being correctly interpreted, as far as I can see. Might also be useful to know what Date & Time settings you are using in System Preferences>>International>>Formats.

Share this post


Link to post
Share on other sites

I have the same issue with dates...and no FMpro to amend...International Format is set to Canada...I changed to US to see if it changes something to the log but it doesn't seem to.

 

edit : the change to US M/D/Y actually worked.

 

Nevertheless, thank you for the nice piece of work!

Edited by Guest

Share this post


Link to post
Share on other sites

Version 1.2.1 is out. It addresses the issue of when the International System Preference is not set to use US formats by embedding the US format in the Filemaker Pro database file (effectively ignoring your International setting). This is necessary due to the fact that Retro Log Data uses dates that are always mm/dd/yyyy. I believe this should address the issues posted above without requiring user intervention. Please let me know.

 

There's a couple of other minor tweaks and fixes as well.

 

Download from Lintburger Software Page

 

Cheers,

Paul

Share this post


Link to post
Share on other sites
I've updated Retro8LogAnalyser to version 1.1

 

The new version allows you to mindlessly paste data from the Retrospect Log window without having to worry if it overlaps data you previously pasted in. Should make life a little easier. It will automatically keep track of the last timestamp in the log data. The next time you paste in data, it will clear the pasted data up to that last timestamp, leaving only the new log lines to process.

 

I've also added an update engine that allows you to upgrade to future versions without having to reprocess all your data again. This won't work with the data in 1.0, but will work when I release future versions.

 

A few other bug fixes.

 

Can be downloaded from the Lintburger Software Page

 

Cheers,

Paul

FileMaker 11 (now out) adds a new repeating import facility. Would this be useful for your database so it can automatically import new log data from Retrospect? (I have Retrospect 11.)

Share this post


Link to post
Share on other sites
FileMaker 11 (now out) adds a new repeating import facility. Would this be useful for your database so it can automatically import new log data from Retrospect? (I have Retrospect 11.)

 

Hi John,

 

The problem isn't FileMaker's ability to import on a recurring basis (that can be done in a number of different ways). The problem is that the RetroLog.utx file, where all the log data is stored, is incomprehensible to me. It's filled with what appear to be variables that only make sense to the RetroConsole App, in conjunction with a Server configuration. I can't parse this file. I have to rely on the "interpreted" output of this file, which is currently only available by using the Show Log command from within the RetroConsole app.

 

2 ways around this:

1. Retro Engineers do something that allows RetroServer to output a plain text log file using the built in scheduling engine, preferably in a coherent delimited format. (Currently, I have to read the log data almost as if it were a story.)

2. You could write an Applescript that does the following:

  • Launch Retro Console
  • wait
  • issue a CMD-L
  • wait
  • CMD-A
  • CMD-C
  • Launch RetroLogAnalyser
  • wait
  • issue a CMD-1
  • wait
  • CMD-V
  • ENTER
  • wait
  • ENTER

Tie it to a crontab on a computer that's running all the time (without screensaver) and you've got Automation.

Share this post


Link to post
Share on other sites

I just wanted to say that this is a very useful utility and just say thank you for the work you obviously put into it.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×