CAP_FILTERS*

Return to Home page

Syslogd2 Filters

Input Filters vs Output Filters

Filtering in Syslogd2 is divided into two CAP_* symbols: CAP_FILTESIN and CAP_FILTERSOUT. Either or both may be optionally enabled for any given binary. The only difference between the two filter types is when (and to what data) they are applied:

Input Filters: These filters are enabled by CAP_FILTERSIN and are configured as parameters to any of the command-line options that define data-input sources.

-- The term "input filter" comes from the data they act on. Input filters are applied to data that enters Syslogd2 via the input source they are declared by.
-- If CAP_OUTPUTTHREADS is defined, Input Filters are applied by either worker-threads or reader-threads (depending on whether CAP_WORKERTHREADS is also defined) after all message break-down and resolution steps are complete and as the last step before delivery to output queues.
-- If CAP_OUTPUTTHREADS is not defined, Input Filters are applied at the equivalent point (after all message break-down and resolution steps are complete and as the last step before searching destinations for matching selector-masks).

Output Filters: These filters are enabled by CAP_FILTERSOUT and are configured as options to output declarations in the configuration file. Output filters are actually assoicated with selector-strings (and selector-masks), but configured with the Destination-options at the end of the output-line.

-- As Syslogd2 walks the list of destinations (either those associated with an output-threadpool or the entire list), it checks each selector-mask associated with each destination in line-number sequence.
-- If the selector-mask being compared matches the message's facility and priority and if an output-filter is assigned to the selector-string that defined that selector-mask, the output filter is applied to the message before transmission.
-- Processing then skips directly to the next destination where the search for matching selector-masks is restarted.

Filter Concepts

The term "filter" and "filtering" means different things to different people. It can be used in the sense of "to filter out" or "to reduce in volume" or it can used to mean "a process that modifies data as it passes through" in the same way that the "grep" command is considered a filter.

In Syslogd2, both definitions of "filter" apply. Syslogd2 stores filters in individual files located in the ConfigDir directory. Using separate filter-files allows changed files (filters) to be easily detected and re-parsed without needing to restart the entire Syslogd2 daemon or to re-parse the primary configuration file:

-- Any source- or destination-declaration may refernece at most one filter-file, but any filter-file may be referenced by as many source- or destination-declarations as desired.
-- Each filter-file contains zero or more filter-lines, comments or changes to the active "sense" of the file.
-- Each filter-line may be either a "match" filter (that drops (discards) or accepts the message being processed) or a "transformation filter" that modifies the message it is applied to.
-- Each filter-line in a filter-file consists of two components: (1) a "keystring" and (2) a "field-list".

Each filter(-file) is characterized by a "default action" or "sense" which is either "default pass" or "default discard". This "sense" defaults to "pass", but can be changed at any one or more points throughout the file as desired. The "sense" of the file specifies the action to be taken should processing "walk-off" the end of the file and the implicit action to take when a line is "matched". Manipulating the "sense" of a filter-file allows any filter to act in "cherry-pick" mode (where only the messages selected are allwoed to pass) or in "ash-and-trash" mode (where selected messages are blocked and all other (including unknown) messages are allowed to pass).

-- Referencing an empty file is the same as not having defined a filter at all.
-- Referencing a missing file is the same as not having defned a filter at all.
-- Use the CheckReconfig() or CheckFilters() interrupt routines to check filter-file timestamps and dynamically update modified filters without restarting Syslogd2.

Filter Application

When a filter is applied to a message, each line of the file is evaluated as it is encountered:

-- The length of the keystring determines the number of "match fields" in the field-list.
-- The first non-whitespace character after the keystring (which is terminated by whitepsace) becomes the field-delimiter character for that line. This allows the selection of any (printable) character not otherwise found in the field-list-string as the field-delimiter character.
-- Any fields in the field-list in excess of those designated as "match fields" above are considered "transformation fields".
-- Each line is checked to see if a "match" has occurred. A "match" occurs when each component or location of the message designated by a given keystring letter "matches" the contents of the corresponding "match field". All keystring conditions must be satisfied for a "match" to occur. At the first failure of a "keystring" condition, processing of that filter-line is aborted and the next filter-line is processed.
-- If a filter-line "match" occurs:

-- If one or more "transformation-fields" exist, the keystring counter is reset and each transformation field is used as a "replacement" value for the string or value of the corresponding "match" field. Upon completion of the "transformation" process for any filter-line, processing of the entire filter-file is immediately terminated and the modified message is passed on (kept) for further processing.
-- If no "transformation-fields" exist, processing of the entire filter-file is immediately terminated and the action opposite to the currently active "sense" of the file is taken.

-- If no match occurs against any filter-line in the filter-file, processing is said to have "walked-off" the end of the file and the current "sense' of the file determines the action taken for that message (keep or discard).
-- The "sense" of a Filter-file defaults to "pass" ("keep"). The "sense" of a file can be set at any time with special filter-lines consisting only of either "dp" ("default-pass") or "dd" ("default-discard").

A Sample Filter File

A dummy filter file showing both pass/drop ("match") filter syntax and transformation filter syntax can be found here .
(It should be noted that this sample format is a mish-mash of ideas with no clear goal or purpose since it is totally out of (and without) context.)
Return to top

Return to Home page