Logprocessor

Purpose

With logpro you can write command files to rigorously analyze log files. For example you may have a complex simulation with many steps, compilation, simulation, analysis, clean up etc. You want to know in an automated way if the simulation ran. Manually reading the log file(s) once or twice searching for errors, warnings and messages is fine but by the third or forth time you want to automate it. Logpro can help.

Download

Usage

Running logpro. EXAMPLE: Run the program "runsim" passing the output through logpro using the cmdfile.logpro log analysis command file. The annotated log file is written to "run.html" and run-logro.log.

% runsim | logpro cmdfile.logpro run.html > run-logpro.log

Or put the log into a file and postprocess:

% runsim > runsim.log
% logpro cmdfile.logpro run.html < runsim.log > run-logpro.log

In a Makefile:

simlog.html : cpu.v
       verilog cpu.v | logpro cpu.logpro simlog.html

The benefit in using logpro calls in a Makefile is that the exit code can be trusted to reflect the pass/fail status correctly. See the ./doc/tip/example.logpro (which operates on ./doc/tip/example.log and produces ./doc/tip/example.html) in the source repository)

Exit Codes

Expect Type	Exit code	Description
All ok	0	No "errors" or "warnings" found and all "requireds" found.
ERROR	1	One or more expect errors matched or a required or waive check did NOT get hit
WARNING	2	One or more warn rules matched
CHECK	3	A check rule matched
WAIVE	4	A waive rule matched
ABORT	5	An abort rule matched
SKIP	6	A skip rule matched
Reserved for future use	> 7

Installation

Command file details

Your command file is written in scheme but don't let that scare you. Only a handful of single line rules are needed to write a rock solid log analyzer. Here is an example command file and the logpro output from running against this example log file

More examples can be seen in the example.logpro file in the source directory ./doc/tip/example.logpro and examples

Syntax

Triggers:

( trigger "trigger name" #/trigger pattern/ )

The "trigger name" can be any string
The trigger pattern is a perl compatible regular expression
Triggers are required once defined. Not hitting the trigger is an error. Use trigger:non-required for triggers that may or may not be hit

Sections:

( section "section name" "start trigger" "end trigger" )

The "section name" is a string.
The "start trigger" and "end trigger" are strings that match the name of a trigger

Expect rules:

( expect:required [in|not-in] "section name" [> or < or =] integer  #/regular expression/ )

Types of expect include: expect:required, expect:ignore, expect:warn, expect:error and expect:waive

Sections

You can break your log file into sections to make applying the rules very specific. For example you may want to only apply a set of rules in the compile section of the log file related to compiler failures.

Example: Create a section called "Init" that only applies between a line containing the string "This is a header" and a blank line. The lines match on regular expressions delimited by #/ and /.

(trigger "Init"     #/This is a header/)
(trigger "InitEnd"  #/^\s*$/)
(section "Init" "Init" "InitEnd")

By default Logpro provides a section "LogFileBody" that covers the entire log file. This example illustrates how you can create sections that start at the beginning or end of a log file. To match the beginning we use a regex that matches anything. To match the end simply use a trigger name that has no specification.

(trigger "Body"     #/^.*$/)           ;; anything starts the body
(section "Body"     "Body" "EndBody")  ;; EndBody trigger is not defined

Some sections may occur multiple times in a file. Use trigger-with-limit for those. Use a large number if you don't know how many times the trigger may be hit.

(trigger-with-limit "Blah5"    100 #/^begin Blah5/)
(trigger-with-limit "Blah5End" 100 #/^end Blah5/)
(section "Blah5" "Blah5" "Blah5End")

Required Expects

There will be some lines in your log file that, if missing, indicate a problem. Create required expects for those:

;; If we don't get a header then we know something went badly wrong
(expect:required in "Init"  = 1 "Header"      #/This is a header/)

Sometimes you want to use required as a warning.

;; warn if something is missing
(expect:required-warn in "SomeSection" = 1 "Warn if not found" #/this is expected but if not found it is only a warning/)

NOTE: expect:desired is an alias for expect:required-warn.

Catch and ignore errors

The rules are applied in the order they are specified in the command file. Once a rule is hit in the log file no other rules are tested. You can use this to create a catch all and then add exceptions. In this example ERROR 244 is false and should not be flagged as an error. The expect:ignore rule will match and so the expect:error rule never gets triggered.

(expect:ignore   in "Body" = 1 "FALSE ERROR" #/ERROR 244/)
(expect:error    in "Body"  = 0 "ERROR BLAH"  #/error/i)   ;; disallow errors

Expire rules

Any rule can have a date set beyond which it will no longer be applied. This is intended to be used on ignore rules that are only intended to be temporarily applied. The date format is MM/DD/YYYY:

(expect:ignore in "Body" < 99 "These are real but I want to ignore them for now" #/ERROR 1234/ expires: "02/04/2011")

This kind of rule is often best written as "< N" where N is more than you are ever likely to get of these that you want to ignore.

Values

Extract values from your logfile and apply simple rules for pass/fail checks.

Syntax is:

(expect:value   in section  value  tol  comment_string regex [hook: hookname])

Example, extract a number following a string "Measured voltage output: " and compare it with 1.9 +/- 0.1. Call the hook "freq"

(expect:value    in "LogFileBody" 1.9 0.1 "Output voltage" #/Measured voltage output:\s*([\d\.\+\-e]+)v/ hook: "freq")

Example, extract two numbers from a string, the second number must be greater than 1.9:

(expect:value    in "LogFileBody" 1.9 >   "Time voltage" #/out: (\d+)\s+(\d+)/ matchnum: 2)

More values handling techniques here

Hooks

Create a hook to handle the information gathered on a frequency measurement:

(hook:add "freq"    "echo \"Value hook activated: expected=#{expected}, measured=#{measured}, tolerance=#{tolerance}, message=#{message}\"")

BACK: Back to index opensrc

LogProcessor