OutputProcessing

Return to Home page

Syslogd2 has integrated several new concepts in the way that it processes output connections. To some extent, these changes apply to Linux socket and Named-Pipe connections, but they are more obvious and extensive when applied to IP, so this page will concentrate on the IP changes with references to Linux connections (Linux socket and Named-Pipes) as / when they apply.

Historically, certain taboos have developed concerning the creation of Syslog configuration files. Among these are:

(1) No two output lines should ever be created for the same host. Attempts to do so will result in duplicate message delivery.
(2) All IP hosts should be specified using IP addresses only because DNS resolution cannot be relied to properly resolve hostnames.
(3) The number of output destinations should be kept to an absolute minimum because the more destinations that are specified, the more time the (usually single-threaded) application spends writing to destinations and the less time it spends waiting for or reading input traffic or processing that traffic.
(4) With the ability to pass syslog data over TCP in Linux, the number of TCP connections should be kept to an absolute minimum because network timeouts with TCP will typically cause all available syslog threads to go "out to lunch" while they wait on each other, then on the TCP socket that cannot re-establish its connection because the handshake packets continue to time-out with each new message.

"Selector" vs "Location" and "Usable" vs "Resolved"

Before going further, Some terminology is in order. Syslogd2 considers each output line of the configuration file to consist of 3 distinct components (not counting comments). These are the selector-string, the location and the option-list. The option-list is a combination of options that are associated with the selector and options that are associated with the location. Each location is considered to be just one of possibly several identifiers for a given host.

Selector options are those that do not distinguish one location from another, but that may vary based on the content of the data selected by the selector-string(s):
     filter This option specifies the file (in the ConfigDir directory) that contains a filter specification to apply to all messages that match this selector-string going to this location. Different selector-strings may specify different filters to apply to data (presumably from different sources) -- that is going to the same location.
     spoolfile This option component is only the decision whether or not data that matches this particular selector-string should be spooled if the connection to the associated location is not immediately available. The actual file, the filename, the connection and the connection state are all attributes or options associated with the location. The decision to spool data or not to spool data may reasonably change based on the content and volatility of the data that is being selected by the selector-string.
     network Each selector-string can select the network states in which that selector-string entry is valid. This is consistent with the general operation of the --network= command-line option.
     version The version option applies to IP connections only. In recognition of the fact that a location may specify a host that contains multiple interfaces (each with a separate name and address(es)) or multiple addresses on a single interface or multiple address-families on a single (or multiple) interface(s), Syslogd2 attempts to resolve all output line locations in the configuration file to unique hosts.

Syslogd2 makes the assumption that an application is uniquely defined on any given host by a combination of port and protocol regardless of the address or address-family used to reach that host. The reasoning is that varying either the port or protocol potentially changes the application (such as Port 514/udp and port 514/tcp being used by different applications). The use of the IP address '*' meaning "all addresses and interfaces on this host" supports this assumption. The configuration of different applications using the same port and protocol but different addresses or network-interfaces on the same host is a complexity that is vigorously avoided and never recommended. Indeed, most IP references and developers consider "port + protocol" to be a unique identifier for an application while configuring an application to use both IPv4 and IPv6 addresses on the same port and protocol (using the '*' address) is quite common.

At the same time, Syslogd2 recognizes that data coming from different sources may have different output-processing requirements (output filtering for network-management data vs locally-generated data for instance), yet need to be sent to the same location. This is accomplished in Syslogd2 by specifying multiple output lines with the same location using different selector-strings with different selector-options

With the recognition that the purpose of the selector-string and the location are different and that there may be multiple ways to reach the same location (different network-interfaces, different address-families or multiple addresses on the same interface) comes the recognition that the selector-string and location can and should be treated as distinct elements. Syslogd2's goal is to deliver the syslog traffic to the appropriate application on the remote host. It should not matter which address or address-family or network-interface is used to reach that host (and application). Syslogd2 attempts to consolidate all locations that resolve to the same host, ideally using any and all known addresses (and address-families) to get the message to the proper host and application, recognizing that intermediate network components may at times block (or fail to pass) one or another address family or address. This consolidation of locations allows Syslogd2 to know when a message has been delivered (to a given application) so it can avoid sending the same message multiple times (such as to different addresses of the same host).

Syslog will use a combination of the /etc/hosts file, the specified cachefile and DNS query results to attempt to RESOLVE each location (to obtain all host-names and IP addresses for each host) referenced by the configuration file and to combine (or "link") selector-string-lines in the configuration file that reference the same host.

When processing message traffic, Syslogd2 threads will walk the string of host identifiers, checking the list of selector-strings associated with each host. The first matching selector-string (lowest line number) for a given host causes the message to be sent to that host utilizing the selector-options of that selector-string. Remaining selector-strings associated with that host are then ignored (the message having already been delivered).

Syslogd2 associates the following output-line options with the location component of the configuration-file output-line because they affect elements of the location that cannot be changed without changing the location itself or causing significant operational problems with transmitting data to the ultimate application:
     IP Port Changing the IP port number selects a different application on the remote host. Doing so indicates a change of location.
     IP Protocol Changing from UDP to TCP may also indicate a change of application (location) and though not guaranteed to do so, cannot be relied upon.
     SpoolFile Name, max SpoolFile size, and SpoolFile Action Only one spool-file can effectively exist for any given host, therefore spool-file parameters are location-related.
     Connection Handle, Connection Status, Spooling availability These are all characteristics of the single connection that will be made to each unique host in the configuration file. Spooling availability refers to whether spooling is defined for this location.. Note that spooling for a location may be available even if a given selector-string chooses not to spool data in the event of connection failure.
     Time Format The relay (boolean) keyword changes the output time-field-format of the transmitted line to "{" + <system-clock-value> + "}" which is both easier for an application to parse and more accurate/complete (it contains the year of the message with no adjustments (and consequent loss of accuracy) for local daylight-saving-time or time-zone settings). Use of the relay option is recommended for Syslogd2-to-Syslogd2 relaying of messages or for transmission to other applications that recognize the relay time format, but should be avoided otherwise. This is a "location-specific" setting to avoid mixing time-field-formats in the same traffic-stream to the same application.
     The uid, gid, and mode for file locations In the event of conflicting settings between multiple selector-strings specifying the same file, the lowest configuration line-number "wins".

Syslogd2 considers each output-line (selector-line) in the configuration file to be in of 3 states:

     Resolved: A successful IP lookup has been made in the internal cache or in the DNS and all known addresses and hostname aliases have been identified and added to the internal cache. The port name (if non-numeric) was successfully resolved for the specified protocol via /etc/services. A file, named-pipe, terminal device or Linux Socket was given and all directories are mounted and readable.

     Usable: The address cannot be resolved, but there is sufficient information in the location to effect a connection. (A numeric IP address and numeric IP port were provided or the numeric IP-location option-list did not specify a protocol or port (defaulting to UDP/514). A usable selector is built with the expectation that some future attempt at resolving the entry will succeed, allowing it to be positively identified (and merged into) a given host or resolved to as host-identification of its own.

     UnUsable: These entries either use port-names that cannot be resolved or have hostnames or address-names that cannot be resolved. They might also be files, devices, named-pipes or Linux sockets with currently-invalid pathnames that may later become valid if or when remote filesystems are mounted on the local box. UnUsable entries are stored in an off-line list until such time as some future check is able to resolve them (such as when the network or DNS server becomes available). There is no known situation to account for a transition from UnUsable to Usable.

NOTE: There may be a period of time when locations are marked as "Usable". During this time selector-string entries are NOT consolidated into host groups. and duplicate syslog event delivery may occur. To prevent this, it is recommended that network-management deployments define the CAP_CACHE symbol at compile-time, enable it and provide a cache-file that provides all names and address information for all output locations to insure that all selector-lines immediately Resolve and do not linger in the Usable state.

Recovering From Connection Errors

Syslogd2 connection recovery is built around the recognition that it is both pointless and counterproductive to continue attempting to re-open a failed TCP connection with every message -- especially if the host is unreachable and Syslogd2 must wait for TCP-handshake timeouts. Attempting to reopen connections with every message usually results in all threads in the given threadpool backing up behind the one thread that is trying to re-open the failed connection at any given time. Syslogd2's approach monitors the state of every output connection and once a connection is determined to be down and at least one attempt to reopen it has failed, Syslogd2 marks that connection as "invalid" and make no further attempt to reopen it, but will continue to log messages to spool files (if available). Instead, (and until otherwise indicated) Syslogd2 will skip that location (or immediately log the data to the designated spool file) and move on to the next location.

On some user-defined schedule, Syslogd2 will execute the CheckDestinations() housekeeping routine. This routine starts by calling the CheckReconfig() routine if CheckReconfig() is due to be run in an attempt to resolve and deconflict currently unresolved output locations. CheckDestinations() (a non-time-critical "housekeeping" function) will then walk the list of output locations, attempting to reopen them even though timeouts may occur. Any time a connection is re-established, it is marked "available" (immediately allowing production threads to write to it). If there is an associated spool file, the spool file is flushed over the newly-re-established connection, then deleted.

Whether the non-responding connection is a slow (or crashed) application listening on a Named-Pipe or a Linux Socket or whether the connection is to an unreachable, remote IP host, the logic employed by Syslogd2 mitigates (most) timeout issues for TCP (and other) connections. Prior to writing data to a location, Syslogd2 checks the "state" of that connection to see what action to take. An output connection in Syslogd2 can be in one of several "states":

down: For connections in this state, no connection is currently active and no attempt has yet been made to establish the connection. This may be because Syslogd2 is just starting or because an IP location has recently been either resolved or made usable, but not yet opened for use. Syslogd2 will attempt to open the connection before writing to it. If the connection cannot be opened, it is marked "invalid" and Syslogd2 checks to see if spooling is defined for that destination and if spooling is desired by that selector's options. If yes, the data is written to the spool file, otherwise the data is discarded. If the connection is opened, but the write attempt fails, one attempt to re-open the connection is made. If this attempt fails, the same recovery-procedure is applied.

invalid: At least one attempt to (re-)open the connection has failed. Syslogd2 should not write to this connection until it is re-established (by the CheckDestinations() housekeeping routine). Check to see if spooling is defined for this location and if spooling is desired by this selector. Spool if yes, discard the message otherwise.

available: The connection is availabe - write to it. If a write error occurs, make one attempt to re-open the connection. If that attempt fails, mark the connection as "invalid" and check to see if spooling is enabled for this location and if the selector-options authorize spooling. If so, write to the spool file. If not, discard the message.

conflicted: This state is set and cleared by the "output-resolution" routine. It signifies a conflict between this location and another location specified earlier in the configuration file. When a connection-state is conflicted (or "in conflict") the connection is completely ignored. No spooling, no attempts to reopen it, Nothing. An example of conflicts for illustration purposes would be:

          <selector-string> /tmp/conflict ## a file in all network states
          <selector-string> @/tmp/conflict, network=other ## a Linux datagram Socket using the same path - valid only when the network state is detected as "other" (network is fully operational)
          <selector-string> @/tmp/conflict, stream, network=local ## a Linux stream Socket using the same path - valid only when the network state is detected as "local" (loopback interface only)
                    -- OR --
          #~ --disable IPv6 ## IPv6 is disabled by command-line.
          <selector-string> @fec0:3::80 ## This (datagram) location provides no alternative IPv4 connection option by virtue of the IPv6 address.

          File-system locations can also be conflict if there is a pre-existing filesystem entry of the wrong file-system type (file vs Linux socket vs named-pipe, etc.)

As a result of the Syslogd2 connection-recovery approach, deleting an output file while Syslogd2 is writing to it causes no harm other than the loss of the logged data. Syslogd2 will simply reopen the file. Applications that crash while Syslogd2 is writing to them may cause Syslogd2 to spool data until they are restored (depending on connection type), but will not otherwise significantly impact Syslogd2 performance or reliability. Unmounting directories that Syslogd2 is writing to (or reading from) should not have significant impact on Syslogd2 operation either (though this has not been tested yet). Furthermore Syslogd2 has a great deal of "self-healing" ability to recover from broken connections and crashed application interfaces without needing to be constantly re-started.

Return to Home page