Syslogd2 Wiki

High capacity syslog data collection, filtering, and management.

Brought to you by: efreesmeyer

CAP_(misc)

-- If declared, CAP_STREAMIN enables code and data-structures to read and process data from TCP/IP or streaming Linux sockets. CAP_STREAMIN's role in Syslogd2 is more of a optional resource upon which additional connection-types and features can be built. For example, CAP_STREAMIN is required by the CAP_COMMAND feature and to terminate relay-connections that use TCP to pass through firewalls or to spool data in case of network outages (to support CAP_SPOOLFILES).
-- CAP_STREAMIN does not need to be explicitly enabled once it has been declared.
-- There are no settings associated with CAP_STREAMIN other than the "stream" or "TCP" settings in individual --input command-line options.

CAP_STREAMOUT

Return to top
The CAP_STREAMOUT feature allows the use of TCP or streaming Linux output sockets in Syslogd2.

-- If declared, CAP_STREAMOUT enables code and data-structures to manage TCP/IP or streaming Linux socket connections. CAP_STREAMOUT's role in Syslogd2 is more of a optional resource upon which additional connection-types and features can be built. For example, CAP_STREAMOUT is required by the CAP_SPOOLFILES feature to detect broken connections and know when spooling of unwritten data is needed. -- CAP_STREAMOUT is not required for named-pipe output.
-- CAP_STREAMOUT does not need to be explicitly enabled once it has been declared.
-- There are no settings associated with CAP_STREAMOUT other than the "stream" or "TCP" settings in individual socket-type output-lines.

CAP_TAILFILES

Return to top
The CAP_TAILFILES feature allows the use of text-files as input sources in Syslogd2.

-- If declared, CAP_TAILFILES enables code and data-structures to treat text-files as input-sources in Syslogd2.
Two primary methods to 'tail' a file in Linux depend on the mechanism used to detect when the file has changed.

(1) Periodically poll the file to see if it has grown and (if so) read the new data. This is the "generic" approach and the one that the tailfile read-algorithm takes. Since this algorithm is fundamentally incompatible with the way that sockets are read, tailfile and socket input specifications cannot co-exist in the same thread-pool.
(2) Use the inotify() feature (unique to Linux) that sends socket notifications when a "watched" filesystem entry is modified. This appraoch promises to be more responsive and perhaps less resource-intensive.

-- The inotify socket-based notification of file-changes will allow text-files and generic socket input to share the same input algorithm and therefore the same reader-threadpool (when located on inotify-supporting Linux filesystems).
-- Because he inotify mechanism is non-portable not having a generic polling alternative poses portability problems for Syslogd2. It also prevents polling files that reside on mounted filesystems (NFS, CIFS, etc) or that reside on filesystems that do not support inotify FAT, gfs2, etc).
-- The inotify method has not yet been implemented in Syslogd2.
-- The duality of text-file input being both compatible (with inotify) and non-compatible (as a polling algorithm) with socket-input has generated the initial condition that text-file input and socket-input share the same set of threadpool-ids, yet no one threadpool is allowed to contain both types of input.
-- CAP_TAILFILES is required for use of the --tailfile command-line option.
-- CAP_TAILFILES does not need to be explicitly enabled once it has been declared.
-- There are several settings associated with CAP_TAILFILES:

-- The --threadmaps command-line option gains the thread-type "tailfile".

-- Global default values (resettable with the --defaults command-line option):

-- Individual sub-options to the --tailfile command-line option:

-- polltime, pollinterval: The new maximum time between pollnig "cycles" for all tailfile threadpools. A "cycle" ends when all files assigned to the theadpool have been examined, none of tham have more data to read, and there is no more data in any read-buffer.
-- poll: This input is to be polled and is not to use inotify even if the feature appears to be available.
-- append: At startup, start reading this input-file from current end-of-file instead of from start-of-file.
-- hostname, host: This assigns a "pseudo-hostname" for this file. Each line that is read (when converted to a syslog message) will appear to have come from this "host". This allows different log files to be identified with names such as "MySQL.hostname.domainname" or "AppLog.myhost.mydomain". This sub-option is also available for Linux socket input.
-- The idea of a pseudo-hostname has been extended to Linux input sockets as well to identify the entity on the local box that is being monitored to centralized tools.
Return to top

CAP_FILEROTATE

Return to top
The need for an internalized file-rotation capability first became apparent when prototype output file-sizes grew beyond 2GB in prototype testing within a matter of a couple of hours. The concept of this routine has changed little since then, though the implementation has evolved.

CAP_FILEROTATE enables a routine that (at specified intervals) walks the list of output files and checks their sizes.

-- If the size exceeds a certain threshold, the file is moved to a new name and Syslogd2 will open a new file with the original name.
-- By moving the file instead of copying it, the moved file retains the "last modified time" to indicate when it was last written.
-- By having Syslogd2 open the new files, file-ownership/group-ownership and permissions is maintained across files.
-- By having Syslogd2 rotate the files, they can be properly locked and closed before they are moved to insure data integrity.
-- By having Syslogd2 rotate the files, they can be moved when full rather than at fixed intervals that ignore the way traffic load differs throughout the day.

CAP_FILEROTATE enables and requires two global values: MaxFileSize and FileCheckInterval
.
When CAP_FILEROTATE is defined, The interrupt service routine "RotateFiles" becomes available for assignment to user-defined interrupts.
Return to top

CAP_USERTHREADS

Return to top
CAP_USERTHREADS provides the user threadpool-type that offloads the processing of user messages to a pre-allocated, persistent pool of dedicated threads. The user-threadpool is enabled by default when it is declared. There is never more than one user-threadpool. When the user-threadpool is declared and a message is to be written to user terminals, the message is placed into the user-threadpool queue for processing and processing of other destinations continues as normal.

When CAP_USERTHREADS is not declared or is disabled (--disable userthreads), any thread processing a user-terminal message will call this code as an in-line subroutine. Processing a single or list of users can be time-consuming since the entire /etc/utmp file must be traversed for each message to find and compare all logged-in user terminals. The time spent processing user-terminal output is time the threads cannot spend writing to sockets, files, named-pipes, etc.

The original (single-threaded) syslog daemon (sysklogd 1.x) would spawn an entirely new process to handle each user-terminal message. Syslogd2 relies on its multi-threaded design for infrequent requests or on the dedicated user-threadpool for more frequent requests.

There are no destination lines or options associated with this feature. In addition to the userthreads boolean keyword to disable this threadpool, Syslogd2 supports the userlogging boolean keyword to disable user-logging alltogether.

The user-threadpool is configured via the --defaults command-line option keyword using the kwyeords UserQueueWorkers and UserQueueLines to specify the thread-count and number of message-slots in the threadpool's FIFO queue respectively. Default values are 2 user-threads and 50 queue lines.
Return to top

Return to Home Page

Discussion

Anonymous