Rules File

Rule files

Rule files are the core of OmniCheck: they provide the patterns to use, and the actions to take when a pattern matches against the data being monitored. Each rule in the file must be separated by some amount of blank lines, and is comprised of two parts: the pattern and the actions.

Patterns

The pattern follows Perl's regular expression syntax, with some additional features. The following are in order of precedence.

• Rule labeling (mandatory for thresholding)

a tag to be applied to a rule to allow the number of matches for its pattern to be tracked. See Threshold use for more information.

Abra##pattern
actions

• Pattern expiration

a rule with an expiration date will only be used by OmniCheck until the specified date is reached. The following pattern will be used by the system until January 24, 2003:

20030124##pattern
actions

• Negated pattern

a rule with a negated pattern will activate when the pattern does not match any log entries within the block of content from the monitored file/program. If there are log entries matching the pattern, then the rule will not activate its actions.

! detected co-location site
actions

NOTE: not available in persistent mode.

• Ordered pattern

a rule with an ordered pattern will activate when a line matching one pattern occurs within a specific distance from another pattern:

pattern-a +3 pattern-b
actions

pattern-a 
+3 pattern-b
actions

• Block pattern

a rule with a block pattern will activate when two boundary patterns match. The actions for a rule with a block pattern will act upon the log entries between the entries matching the patterns, inclusive. The match is, in regular-expression speak, non-greedy.

pattern-a ... pattern-b
actions

pattern-a 
... pattern-b
actions

NOTE: not available in persistent mode.

• Multiple pattern

a rule with a multiple pattern will activate when all of the patterns (2 or more) match within the same block of content from the monitored file/program:

pattern-a && pattern-b
actions

pattern-a 
&& pattern-b
actions

• Alternation pattern

a rule with multiple possible patterns will activate when any of the patterns match within the same block of content from the monitored file/program:

pattern-a|pattern-b
actions

pattern-a 
|| pattern-b
actions

The patterns must follow proper Perl regular expression syntax. Any occurance of the these special characters in the data to monitor must be escaped with a backslash \ in the pattern:

( (left parenthesis)
) (right parenthesis)
[ (left bracket)
] (right bracket)
/ (slash)
\ (backslash)
. (period)
? (question mark)
* (asterisk)
+ (plus sign)

Any pattern with non-escaped special characters will be considered corrupted, will not be used by OmniCheck, and will be noted in the report file (if enabled) and/or in debug output (if enabled).

Actions

The actions are the list what to do when a pattern matches. The available actions are:

• mail

send an email to one or more recipients. The body of the email will be the lines that matched the pattern, and the subject will be the content after the semi-color (see example below) or the word "problem" if no subject is present:

mail admin ; test message
mail --recipients "admin" --subject "test message" --body "/file/to/mail"

The word admin will be translated to the value of the admin in the configuration file.

• page

send an email to one or more email-to-pager recipients. The body of the page email will be simply "/", and the subject will be the content after the semi-colon (see example below) or the word "problem" if no subject is present. A follow-up mail will be sent to the holder of the pager (either by interpretting the email-to-pager recipient or by reading a divided recipient).

page oncall ; system reboot

The word oncall will be translated to the value of the oncall in the configuration file.

By default, the account name used to form both the email-to-pager and follow-up email are the same. If this is undesirable, you can divide the two addresses in the following manner:

page pager@pager.foo.com/mail@foo.net ; split addresses

Here, any page generated by this action will go to pager@pager.foo.com, where the accompanying mail will go to mail@foo.net. This value can also be set in the oncall configuration file entry.

• file

file the lines that matched the pattern into another file:

file /usr/adm/logs/separate.log
file /usr/adm/logs/file_$1.log
file /usr/adm/logs/file_@1.log

The 'file' action has the ability to interpret the values of the parenthesized data within each matched log entry, and use that data to alter the filename being opened for appending.

• exec

run an external program that will use the lines that matched the pattern as input (STDIN):

exec /usr/local/bin/process_data.sh
exec /usr/local/bin/new_output.sh -d @1 -m @2
exec --ignore /usr/local/bin/new_output.sh -d @1 -m @2

The 'exec' action has the ability to interpret the values of the parenthesized data within each matched log entry, and use that data to alter the script name and/or the parameters passed to the script. In these situations, the script will be invoked one time per matched log entry, whereas the default behavior is to pass all matching log entries to a single invocation of the script.

When the --ignore option is used, the script does not receive the matched log entries as STDIN. This is to allow the external script/program to run without needing to manage the matching log entries if it is not designed to do so.

• modify

modify the lines that match the pattern, altering the data for subsequent actions within the rule.

modify --prepend "this" ;
modify --prepend "#!/output/of/script args" ;

modify --append "that" ;
modify --append "#!/output/of/script args" ;

modify --replace "this" "that" ;
modify --replace "this" "#!/output/of/script args" ;
modify --replace "regex" "that" ;
modify --replace "regex" "#!/output/of/script args" ;

# Using --replace to simulate --prepend:
modify --replace "^" "that" ;
modify --replace "^" "#!/output/of/script args" ;

# Using --replace to simulate --append:
modify --replace "$" "that" ;
modify --replace "$" "#!/output/of/script args"  ;

Instances of "this" and "that" represent simple text strings; "regex" represents a Perl regular expression; and /output/to/script represents some external program. The args of the script can contain the same $1, $2 variables as other actions: see Pattern-Action Interaction.

• ignore

ignore the lines that match the pattern

ignore juser ; junk messages

Altering when actions act

OmniCheck can be instructed to take a specified action only if a specific number of lines match the pattern. Known as a threshhold within OmniCheck, its syntax is this:

if >= 10 mail admin ; test messages

The valid relations are:

< (less than)
> (greater than)
<= (less than or equal)
>= (greater than or equal)
== (equal to)
!= (not equal to)

There must be a space after the word if and after the numeric value. Space between the relation and numeric value is optional.

Actions can be coded to only activate when a specific organization is using the rule file. This feature reads the 'organization' configuration file entry to test against the login in the rule file. In the following example, the FOO organization will get "host issue" mail, the BAR team will get a "fix me" page to their oncall, and everyone else will ignore the pattern:

<pattern>
if org eq "FOO" mail admin-team ; host issue
elsif org eq 'BAR' page oncall ; fix me
else ignore admin ; not important
endif

Use either single (') or double (") quotes to surround the organization value within the rule.
If there are actions outside, or after, an if block, they will always take effect. In this example, only instances used by the FOO organization will get the "host issue" mail, but all instances will send the "fix tomorrow" mail:

<pattern>
if org eq "FOO" mail admin-team ; host issue
endif
mail admin ; fix tomorrow

NOTE: Organization-sensitive actions and threshholded actions are current mutually exclusive, i.e., you cannot do this:

if (org eq "FOO" && >= 10) mail admin ; FOO and over 10 
if (>= 10 || org eq "BAR") mail admin ; over 10 or BAR

If this is a feature that is requested, the effort will be applied to work out the parsing logic. For now, however, you can do one or the other.

Pattern-action interaction

OmniCheck can capture sections of the lines that match the pattern and use them in the actions as pieces of the subject of a mail. Parentheses are used to surround the section of pattern to capture, then number variables ($1, $2, etc.) are used to insert the captured values into the action:

ftpd\[\d+\]: FTP LOGIN FROM (\d+\.\d+\.\d+\.\d+) as (\w+)
mail admin ; FTP from $1 as $2

Note: pattern-action interactions are now functional, so that patterns like this:

Error: (\w+) ... Description: "([^"]+)"

will now capture the data within two parentheses and provide it as expected.

Thresholding Use

To use the thresholding feature, you need to tag the pattern with a label. This label needs to be two or more alphanueric characters long, the first being alphabetic (think variable name), followed by a double pound (similar to the pattern expiration feature).
Then, you need to add a threshold to your actions:

Alpha##foobar
if 30/day mail admin ; lots of foobar
if over 30/day mail admin ; lots of foobar
if under 30/day mail admin ; not enough foobar

On each iteration of OmniCheck, the number of matches for all patterns that have been tagged will be stored in a .thresh file in the tmpdir directory, timestamped to when the match occurred. Also, the .thresh file is kept manageable by trimming off data entries that exceed by 2 times the maximum threshold time value within the rule files. If the number of matches for a particular pattern, including what is currently matching the pattern within the current iteration, and if the preface control word 'over' is used, and the number is greater than or equal to the quantity per time unit specified in the action, then the action is invoked; otherwise, it is not.

If the number of matches for a particular pattern, including what is currently matching the pattern within the current iteration, and if the preface control word 'under' is used, and the number is less than or equal to the quantity per time unit specified in the action, then the action is invoked; otherwise, it is not.

The default control is 'over'.

Labels can contain varibles that are filled in with captured sections from the pattern, just as in actions.

Alpha$1##foo: (\w+)
if 30/day mail admin ; lots of foobar
if over 30/day mail admin ; lots of foobar
if under 30/day mail admin ; lots of foobar

The available time units are:
Hours: stated as h, hr, hour, hrs, or hours
Days: stated as d, day, or days
Weeks: stated as w, week, wk, wks, or weeks