Syslogd2 Wiki

High capacity syslog data collection, filtering, and management.

Brought to you by: efreesmeyer

ConceptsTermsDefinitions

Return to top
Selected features of Syslogd2 (Filters, Name-Cache and Spooling to be specific) make use of external files that are (by default) located in one of two directories.

The ConfigDir need only be root-readable (assuming Syslogd2 is run by root), but may at the host administrator's option be set to other permissions that allow selected user-groups to edit the files within. This directory is used by Syslogd2 to hold auxiliary user-defined input files. Specifically, this directory holds all input and output filter files and the designated cache-file if the cache-file is not specified by an absolute filename.

The SpoolDir needs to be root-writable but no one other than the Syslog daemon itself (and any curious system admins) needs any access at all to this directory. This directory is used only to hold "spool files" that contain data that could not be written to destination hosts or local applications due to connection failures -- files that will be read and removed once the connections to the applicable destinations are restored. The default location for this directory is /var/spool/syslog

# ~ --defaults configdir=/etc/syslog.d, spooldir=/var/spool/syslog

Recommended deployment for network management purposes would include setting the ConfigDir to 770 permissions with ownership of root and group-ownership set to some <networkGroup> where <networkGroup> is a user-group defined to include both network-management staff (members from the network team to manage network-related filters and cache-file entries) and host-management staff members to manage host-related filters and cache-file entries. It is the network- and host- management groups that have the most interest in insuring that the Syslogd2 filters and cache-file entries remain current and accurate.
Return to top

ConnectionSpec

Return to top
A "ConnectionSpec is any input source or destination target plus all the options specified for that source or target.

Variations of the term "<Connection-Spec>" include "<File-Spec>", "<Log-Spec>", "<Destination-Spec>", <TailFile>, <KernelSpec>, <InputSpec>, etc.

<File-Spec>::= An absolute filepath combined with optional uid, gid, mode and other options.
<Log-Spec>::= An absolute filepath to a log file (stderr or stdout) with optional uid, gid, mode and level values.
<Log-Spec> levels are defined to be the same as syslog priorities.
<Input-Spec>::= A generic term that includes the complete parameters (and options) for any of the --input, --tailfile, or --kernel command-line keywords.
<Output-Spec>::= A generic term that includes the complete contents (excluding comments and selector-string) of an output-line from the configuration file.
<TailFile>::= An input specification that defines a text-file input (including all options). The name comes from the --TailFile= command-line option keyword that defines it.
<KernelSpec>::= An input specification that defines how kernel input is to be read (including all defined options). The name comes from the --Kernel= command-line option keyword that defines it.
<InputSpec>::= An input specification that defines a socket input source plus all included options. The name comes from the --Input= command-line option keyword that defines it. The term "InputSpec may also be used generically to include TailFile and Kernel input types depending on context.
Return to top

ThreadPool

Return to top
A Syslogd2 ThreadPool is defined as a FIFO (First In / First Out) data-buffer of some number of messages, a number of processing threads that process data from that buffer, and a subroutine that is run by those threads to accomplish the work of that threadpool.
Reader ThreadPools deviate slightly from the above definition in that they have no explicit input buffer, instead utilizing their respective input sources as their "buffer" of input data.
Configuration parametes for a reader threadpool consist of a non-negative integer id, and some number of reader threads.

Defining Threadpools in Syslogd2 is done either directly via the --ThreadMaps= command-line option or indirectly by defining an input source and declaring that source to be member of a specific threadpool or by some combination of both. The input-source declaration can specify threadpool options in addition to simple threadpool membership. Multiple input sources assigned to the same threadpool can each assign threadpool options. The final values used for the number of threads in that threadpool will be the maximum of the global default value, the --ThreadMaps= value (if any) and the values specified by the assigned input sources.

The default Socket Reader Threadpool is used for reading either IP or Linux sockets or a combination of the two. This threadpool-type is configured using the "--input=" command-line option and takes one of two forms:
If there is only one socket defined in the threadpool and if that socket is a datagram socket, a high-performance read-algorithm will be automatically selected that is based on the recvfrom() system call.
Otherwise, the generic read-algorithm (based on the pselect() system call) will be selected.

The Text-file Reader Threadpool is only available when the CAP_TAILFILES declaration is selected at compile-time. If included, a text-file input source can be declared with the "--tailfile=" command-line option.

The Kernel-Reader Threadpool is an optional (special-purpose) threadpool designed to provide either direct (system-call) access to the kernel's syslog ring-buffer or to read the kernel file at /proc/kmsg without needing to invoke CAP_TAILFILES to read text-file-input. This threadpool is only created if CAP_KERNELTHREADS is defined at compile-time and if "--enable kernelthreads" has been specified on the command-line. If either of these requirements is not met, kernel input can still be obtained via the "--kernel=" command-line option that will attempt to use the inotify() feature of Linux file-systems or the CAP_TAILFILE feature to monitor the /proc/kmsg "text"-file for input. (Current code does not yet support either the CAP_KERNELTHREADS option or the inotify() capabilities of Linux, so (at least in initial release(s)), the workaround is to use the CAP_TAILFILES method to read kernel input.

The HouseKeeping threadpool has no explicit buffer of tasks to perform, but utilizes the virtual stream of "housekeeping requests" as a source of input tasks to perform.

All Threads (and therefore all threadpools) in Syslogd2 are fully blocking. This means that when no actual work is being performed, all threads are designed to be blocked (suspended) by the operating system. At no time does any thread "spin" in an infinite loop for the purpose of waiting on incoming work.

For higher performance, Syslogd2 can split the processing normally done by each thread into as many as 3 component phases (each with associated threadpools: reader-threadpools, worker-threadpools and output-threadpools:

Worker Threadpools: If the CAP_WORKERTHREADS symbol is declared at compile-time, Syslogd2 will split off the read-processing from the rest of the processing of each message by creating "Worker Threadpools". Syslogd2 is able to empty incoming network-buffers at a much higher rate of speed by focusing the Reader Threadpools on moving data out of network buffers into Worker-Threadpool input (FIFO) queues before the data is overwritten by more incoming data.
     Reader threadpools are then dedicated to reading input sockets/files/kernel, capturing envelope information (peer address, time-of-receipt, etc), and queueing that data for other (worker-) threads to actually process. The "new" "queueid=" option to the input-specifications and to the --ThreadMaps= option is used to identify which Worker-Threadpool's input queue should receive the incoming data.
     Worker Threadpool configuration values (lines= and workers=) become available on input-specifications and the --ThreadMaps= option gains a new ("worker") threadpool type.

Output Threadpools: If the CAP_OUTPUTTHREADS symbol is declared at compile-time, Syslogd2 will split off the output (terminal/device/socket/file/pipe I/O) portion of processing to one or more output threadpools. After a message is processed and the optional input-filter has been applied by a processing thread, the processing thread need only check the (smaller number of) output threadpools instead of individual output devices. Instead of actually writing to each device (and handling errors an contention), the processing thread can queue the message for other (output) threads to process. Queueing the data to output threadpools is much-faster than writing to disk, the network or user terminals and is not subject to timeouts or device contention. This allows the processing thread to return and start processing the next message, while multiple output threads in each of multiple output-threadpools handle the delivery of the message to various required destinations.
     When the CAP_OUTPUTTHREADS symbol is defined, each output specification in the configuration file gains additional options: (1) a non-negative output-threadpool identier (id=), a number of worker threads (workers=), and a minimum number of lines in the specified threadpool's input buffer (lines=). The --ThreadMaps= option also gains the "output" threadpool type.

Specifying both CAP_WORKERTHREADS and CAP_OUTPUTTHREADS provides the 3-phase maximum-performance configuration unique to Syslogd2.
Return to top

Internal Name Cache

Return to top
Corporate policies concerning DNS can severely limit or even prevent effective Syslog monitor deployments. Some companies refuse to put router & switch IP address/names in their internal DNS systems. Others don't want syslog systems deployed because of the extra load they place on (usually Microsoft) DNS servers -- Syslog servers have even been known to crash MSWindows-based DNS servers with frequent, repeated requests for hostname-resolution.

Syslogd2's internal Name Cache was designed to address both of the above issues as well as a response to the issue of identifying hostnames when that host is on the other side of a Netword-Address-Translatiing (NAT'ing) device (usually a firewall) from the DNS server where the DNS address does not match the actual pre-NAT'd address. Additionally activation of the Syslogd2 name-cache (and provision of a cache-file containing device-name and address information can replace the need for DNS access with internal look-ups altogether. This can greatly reduce Syslogd2 message processing times if Syslogd2 no longer has to wait for a network-query and a remote DNS server to respond for each message being processed.

When the CAP_CACHE symbol is defined in Syslogd2, the additional global parameters --cachefile= and --configdir= are activated and the global boolean value "NameCache becomes defined (and disabled). Whether or not the NameCache boolean value is enabled at runtime, Syslogd2 will build and use its internal cache to resolve IP output destinations as specified in the configuration file. Syslogd2 will not use the internal name-cache to resolve incoming message hostname sources unless the NameCache boolean value is enabled.

The Syslogd2 name-cache is built from two components. After reading and incorporating the /etc/hosts file, Syslogd2 looks for and reads the cachefile= if one was specified (The default setting for the cache-file is NULL). The purpose behind this is that Linux (and most operating systems) cannot / will not do a name-lookup in the /etc/hosts file without automatically resorting to a DNS query in the event that the file-lookup fails. In Syslog collectors, this means that unless scrupulous attention is given to maintaining (possibly huge) /etc/hosts files, some DNS "leakage" is bound to occur.

Because the NameCache parameter can be separately enabled from the DNS option, it is now possible to use either one without the other. If both are enabled, Syslogd2 will query the internal name-cache first and (if that query fails) will then query the DNS server. If the DNS server succeeds, the internal name-cache is updated with the DNS information and processing continues. If both queries fail, the incoming IP address is used as the hostname (as is current practice).

The /etc/hosts file is always a component of the Syslogd2 cache because many corporations use the /etc/hosts file as a way to reduce frequent DNS calls that can place excessive load on DNS servers. By building a cache-file instead of (or in addition to) using the /etc/hosts file, Syslogd2 allows for a common file to be deployed to all syslog collectors that can be accessed and maintained by someone other than the host administrator. The additional value of including the /etc/hosts file is that individualized host-names and aliases can be identified and used for locally-generated IP traffic (that traffic directed to "localhost".

Syslogd2 Name-Cache and Output Destination-Host Resolution

The Syslogd2 cache-file parameter can be either an absolute filename or a relative filename. If a relative filename is used, it will be relative to the ConfigDir directory that defaults to /etc/syslog.d. The format of this file is defined and discussed here and an example file can be found here.

# ~ --defaults= confdir=/etc/syslog.d, cachefile=cachefile.dat --enable cache

Return to top

Intervals and Schedules

Return to top
Many tasks in Syslogd2 are run at periodic intervals. Because there is no way to know how long a particular task will run it is pointless to attempt to implement any scheme that attempts to guess intervals between the start of one "pass" and the start of the next "pass". Instead, Syslogd2 intervals define the time-delay between passes of various routines. (a time interval of 90 seconds means 90 seconds after a routine ends, it becomes eligible to run again. This (re-)definition of "interval" may seem non-intuitive at first, but actually makes sense given that the alternative is to accept that a routine may otherwise be started before another copy of the same routine has finished execution due to overly-aggressive scheduling.

Because Syslogd2 is a multi-threaded environment, it is entirely possible that one housekeeping task may wish to change or delete structures that are in use by another housekeeping routine. (All housekeeping routines are designed to be non-intrusive with regards to the primary operating threads.) One of the major roles of the parent thread is to deconflict the execution of housekeeping routines to insure that no resource contention occurs. This deconfliction process may (in turn) result in additional time passing between scheduled run-times for a given housekeeping task due to the parent thread waiting for conflicting tasks to finish.

An Interval Schedule or Schedule is simply an array of integers that function as a series of time-intervals (as opposed to a single, fixed interval) to control the scheduling of certain HouseKeeping (background) tasks.

The idea behind the Interval Schedule is that the less recently an event has occurred, the less likely it is to occur and the less frequently it needs to be tested. It is not uncommon for the syslog system to be started before the network is initialized. Syslogd2 schedules allow frequent checks until the network would normally be available, then (if the network stays down) gradually less frequent checks until the network DOES come up.

Each schedule has an index that is either globally-maintained or that is contained within individual data-structures depending on the purpose of the schedule.
For example, the SourceCheckIntervals and DestinationCheckIntervals schedules control how often disabled input and output sockets are checked to see if they can be reopened. (Remember that for a connection to be declared "disabled" it must already have failed at least one attempt to re-open it).

-- The schedule indices and a "next-check-time" value for these schedules are stored within each individual input or output record.
-- The scheduling of each check-routine (a "global" actions) is based on the first entry in the schedule array (which is assumed to be the smallest interval).
-- Each time the check-routine executes it checks the individual connection's "next-check-time" value and if this time has not yet passed the entry is skipped. This results in each input/output connection effectively maintaining an individualized "check-interval" as long as the first schedule entry is reasonably small.
-- Each time an attempt to open a disabled socket fails, the "next-check-time" is set to "now + the current schedule[index] value", then the index is incremented.
-- If the end of the schedule array is reached, the system will loop on the last entry. If any entry is negative, the schedule is turned off and that connection will remain down.
-- Each time the connection is successfully restored, both the index for that connection and the "next-check-time" are reset to zero.

Any routine that uses a schedule of intervals can be converted to a fixed-increment-interval by defining a schedule with only a single entry.

If the first entry of any schedule is defined as zero, the entire schedule is disabled and the associated check-routine (if any) will not be run.
In addition to the SourceCheckIntervals and DestinationCheckIntervals, Syslogd2 supports a FlushIntervals schedule and a ReconfigCheckIntervals schedule.

The FlushIntervals schedule is the frequency with which duplicate messages are written to files with the (infamous) "Last message repeated <n> times" notification. What Syslogd2 refers to as the FlushIntervals schedule, Sysklogd-v1.4 hard-coded with values of "30 60 90".

The ReconfigCheckIntervals use a global index value and next-check-time value. This schedule determines how often the "CheckReconfig()" routine will be run. The CheckReconfig() routine has multiple purposes:

(1): CheckReconfig() implements the Delayed Resolution functionality.
(2): CheckReconfig() is run at the beginning of either CheckSources() or CheckDestinations() whenever its global next-run-time has expired.
(3): Each time CheckReconfig() runs, it:

(a): Re-acquires the current network state. If the state has changed, it calls a pair of routines to attempt to re-resolve IP references. (when it exits, either CheckSources or CheckDestinations will continue and will walk the connection lists, attempting to open closed (including newly-resolved) connections).
(b): Checks the cache-file's (if any) last-modification date for a change (if CAP_CACHE is defined). If the file has been updated, CheckReconfig() will call an internal routine to re-read the cach-file and install the new name-cache for active use.
(c): Checks each defined filter file's last-modification date for changes (if CAP_FILTERSIN or CAP_FILTERSOUT is defined). If a change is detected, CheckReconfig will cause the updated file to be reparsed and the new filters to be installed for the active threads to use.
(d): If any change was found (network state, cache-file or any filter-file), the index and next-check-time are reset. Otherwise the next-check-time is updated and the index is incremented.

If CAP_RECONFIG is defined and the Reconfig flag has been enabled:

CheckReconfig() will also check the last-modification date on the primary configuration file. If the primary configuration file has been updated, CheckReconfig will schedule a dynamic re-initialization sequence (run by housekeeping threads) starting with the Parse phase to re-parse the configuration file and update the entire configuration.

If the main configuration file is unchanged, but the network has changed state, CheckReconfig will schedule a dynamic re-configuration sequence (run by housekeeping threads) starting with the Build phase to use the pre-parsed structures and re-build the configuration based on the new network state. This may involve reconfiguring threadpools or defining ports as active that were previously closed due to conflict, unresolving previously-reachable ports, or closing previously open ports as conflicted. If a reconfiguration from either Parse or Build phase is initiated, CheckReconfig returns a code to CheckSources/CheckDestinations to abort their attempts to reopen connections because the system is about to reconfigure itself.
Return to top

HouseKeeping

Return to top
The term "HouseKeeping refers to the management and execution of periodic, miscellaneous support-chores that support but do not directly interact with Syslogd2's primary processing threads and threadpools.

Just as physical housekeeping may involve chores that do not qualify as "full-time" tasks such as taking out the trash, restocking the pantry, sweeping and dusting, Syslogd2 requires some mechanism to periodically execute chores that do not justify dedicated threads but that would distract from operational threads and overwhelm the single parent thread. These chores include freeing unused (garbage) structures as the configuration is altered over time, rotating files, refreshing the name-cache, updating filters, reopening files that have been marked "down" and bypassed, writing "-- Mark --" messages to files, flushing spooled data after connections are re-established, etc.

The parent thread (in an operational network-management environment is easily overwhelmed by the aggregate of these chores-- especially since some of the tasks can easily run for extended time-periods during which the parent thread would be unable to tend to other housekeeping chores.

Syslogd2 introduces the concept of a pool of dedicated threads to handle these "housekeeping" chores, offloading them from the parent thread and allowing the parent thread to concentrate on scheduling the tasks, insuring that resource-conflicts are avoided.

Because of the existence of HouseKeeping threads, it becomes possible to dedicate one of these threads to acting as a "host" server to an external "command-tool client" (tsuc). This client is then able to directly obtain run-time values from the running Syslogd2 daemon process as well as initiate (simple) configuration changes without needing to stop and restart the entire Syslogd2 process. For security reasons, only a streaming Linux socket is supported for command connections (no IP so no off-platform access) and any streaming Linux socket used for command-access must also have a specific flag set (the command option). Furthermore, the binary itself must define the compile-time symbols CAP_STREAMIN, CAP_HOUSEKEEPING, and CAP_COMMAND.

As a result of the additional CPU bandwidth available from the pool of HouseKeeping threads, Syslogd2 can implement additional background activities such as Delayed Resolution and Dynamic Reconfiguration without unduly impacting the primary processing threads or overall system operation.
Return to top

CAP_*-abilities

Return to top
The term CAP_*-abilities refers to the feature sets that can be included in or excluded from the Syslogd2 binary at compile-time through selection and inclusion of various compiler-symbols that start with the prefix "CAP_". These capabilities collectively act to customize the configuration of Syslogd2 binaries to balance the overall operational footprint (memory and disk usage, overall complexity, and thread-count) against the specific syslog traffic levels and processing requirements of a given host.

Each CAP_*-ability (when enabled) activates various additional configuration settings and/or options to configure and utilize that ability. Few of the CAP_*-abilities become active simply by declaring them. Those that do utilize reasonable default settings or default to minimal performance increases unless further configuration steps are taken. Many of the feature-sets need to be enabled before they can be configured. All feature-sets either need to be configured before use or represent/activate additional options that can be used to modify other (more basic) capabilities.

The full set of CAP_*-abilities is listed one the home-page with links to more information about each.
Return to top

Sources, Selectors and Destinations

Return to top
For purposes of Syslogd2, the term Source refers to any source of input data. This could be a file being "tailed" with the --tailfile option , a Linux socket, an IP socket, the kernel's syslog ring-buffer read by a Kernel Thread using a system-call or the special pseudo-file "/proc/kmsg" being read using the "--tailfile" option. Sources may be contextualized into socket sources, file-source, etc. Depending on context, a "source may either refer to the access-method being used to read the data (the IP or Linux socket or the file-handle) or it may refer to the device providing the data (the remote IP host, the "remote" Linux application writing to the input socket, the kernel file-or-ring-buffer, etc.

A Destination is the "target" component of an output connection-spec that has been stripped of all options except "stream" vs "datagram" protocol and IP port specifiers. A Destination may be a list of users, a Linux socket path (with stream or datagram specification), a file or named-pipe filesystem path, or an IP-address-or-IP-hostname specification with IP port and TCP/UDP protocol specifier, etc.

A Selector (aka Selector-String) is a semi-colon-separated list of facility/priority specifications that (for any given facility and priority found in a message) is used to "select" the associated destination for output. Together with a destination, a selector and a list of selector and destination options make up a complete output connection-spec and comprise the contents of one complete line in the configuration file.
Return to top

Size-String

Return to top
A Size-String is an integer followed by an optional single (case-sensitive) character to indicate the "size units" to be used. Size-strings are used by Syslogd2 primarily where there may be a lot of zeros in a value and a high probably of user-error in counting them. The following values are defined for the "block-size units" in Size-Strings. The default unit if omitted is "bytes":

k: 1000 bytes (kilobytes)
K: 1024 bytes(Kilobytes)
m: <1000 x 1000 bytes (megabytes)
M: 1024 x 1024 bytes (Megabytes)
g: 1000 x 1000 x 1000 bytes (gigabytes)
G: 1024 x 1024 x 1024 bytes (Gigabytes)

Example: MaxFileSize=2g
Return to top

Time-String

Return to top
A Time-String is an integer followed by an optional single (case-sensitive) character to indicate the "time units" to be used. Time-strings are used by Syslogd2 primarily where there may be a lot of digits in a value that is cast in seconds and a high probably of user-error in counting them. The following values are defined for the "time-units" in Time-Strings. The default unit if omitted is "seconds":

s: Seconds
m: Minutes
h: Hours
d: Days

Example: MarkInterval=2h
Return to top

Return to Home page

Discussion

Anonymous