Random Bits of Open Code



NOTE: Logpro code has been moved to it's own repository: logpro


With logpro you can write command files to rigorously analzye log files. For example you may have a complex simulation with many steps, compilation, simulation, analysis, clean up etc. You want to know in an automated way if the simulation ran. Manually reading the log file(s) once or twice seaching for errors, warnings and messages is fine but by the third or forth time you want to automate it. Logpro can help.

Running logpro. EXAMPLE: Run the program "runsim" passing the output through logpro using the cmdfile.logpro log analysis command file. The anotated log file is written to "run.html" and run-logro.log.

% runsim | logpro cmdfile.logpro run.html > run-logpro.log

Command file details

Your command file is written in scheme but don't let that scare you. Only a handful of single line stanzas are needed to write a rock solid log analyzer. Here is an example command file and the logpro output from running against this example log file


( trigger "trigger name" #/trigger pattern/ )
( section "start trigger" "end trigger" )
( expect [in|not-in] "section name" [> or < or =] integer  #/regular expression/ )


You can break your log file into sections to make applying the rules very specific. For example you may want to only apply a set of rules in the compile section of the log file related to compiler failures.

Example: Create a section called "Init" that only applies between a line containing the string "This is a header" and a blank line. The lines match on regular expressions delimited by #/ and /.

(trigger "Init"     #/This is a header/)
(trigger "InitEnd"  #/^\s*$/)
(section "Init" "Init" "InitEnd")

You almost certainly want a section which covers the entire log file, here we call it "body". We cheat and do not create a trigger for the end of the section since we want that section valid for the entire log file:

(trigger "Body"     #/^.*$/) ;; anything starts the body
(section "Body"     "Body" "EndBody")

Required Expects

There will be some lines in your log file that, if missing, indicate a problem. Create required expects for those:

;; If we don't get a header then we know something went badly wrong
(expect:required in "Init"  = 1 "Header"      #/This is a header/)

Catch and ignore errors

The rules are applied in the order they are specified in the command file. Once a rule is hit in the log file no other rules are tested. You can use this to create a catch all and then add exceptions. In this example ERROR 244 is false and should be ignored:

(expect:ignore   in "Body" = 1 "FALSE ERROR" #/ERROR 244/)
(expect:error    in "Body"  = 0 "ERROR BLAH"  #/error/i) ;; disallow any errors

Expire rules

Any rule can have a date set beyond which it will no longer be applied. This is intended to be used on ignore rules that are only intended to be temporarily applied. The date format is MM/DD/YYYY:

(expect:ignore in "Body" < 99 "These are real but I want to ignore them for now" #/ERROR 1234/ expires: "02/04/2011")
This kind of rule is best written as "< N" where N is more than you are ever likely to get of these that you want to ignore.