Configuring Syslogd2

The configuration of Syslgod2 starts with an assessment of the type and volume of data to be collected and what is to be done with the data after collection. This assessment will determine the multi-thread model [Syslogd2 MultiThread Models] that most effectively balances performance against resource utilization and the selection of which Syslogd2's CAP_-abilities [Syslogd2 CAP_-abilities] are appropriate for your purpose.

No syslog collector can be a successful one-size-fits-all solution since the logging requirements of a stand-alone personal laptop (where the priority is on small size and minimal resource usage) are much different than those of a corporate firewall-monitor or syslog-based network collection / data-reduction deployment where the priority is on high throughput, and data-filtering at the network edge. Personal Linux users need only minimal feature-sets. Corporate desktops may need more features while corporate servers need yet more features and distributed network-collection systems or network-collection deployments usually need a variety of syslog-collection options and services.

Compile-time Considerations

As of version 1.0.0, Syslogd2 has 20 different CAP_-abilities that must be selected into a binary prior to compilation of the code. The CAP_-ability values each trigger inclusion or exclusion of specific code and data-structures that cause Syslogd2 to scale in both throughput and in feature-based capability. The CAP_-abilities for a given binary can always be found through the use of the --version command-ine option. The default distribution files compile 7 different Syslogd2 binaries [Configuring CAP_-abilities] - demonstrating each of the 5 multi-thread models supported by Syslogd2.

• tiny binary supports Model 1 -- a single threadpool with a single working thread.
• small, medium & large binaries support Model 2 -- a single threadpool with multiple working threads.
• small, medium & large binariessupport Model 2 -- multiple threadpools with multiple working threads.
• huge binary suports Model 3 -- multiple input threadpools + multiple dedicated output threadpools to offload routing and writing of data.
• demo binary supports Model 4 -- multiple threadmpols with dedicated reader thread-groups to improve throughput for high input traffic loads.
• mega binary supports Model 5 -- multiple instances of dedicated input and output threadpools for maximum overall throughput. This model has the highest resource usage which is the tradeoff for having the greatest performance level.

Run-time considerations

All Syslogd2 binaries (regardless of multithread model being used or features selected are interoperable with each other provided the required CAP_-abilities are present in both binaries wherever used for communications. (A TCP outbound connection cannot be made unless the receiving server can acceptTCP inbound connections).

Once the required features have been selected and the binaries compiled, the various run-time options need to be reviewed. In general, all non-traditional features are disabled or turned off by default. To use these features, they need to be enabled. Threadcounts may need to be adjusted for expected traffic levels, etc. Run unknown binaries with the --versionoption to get a list of the compiled-in CAP_-abilities. If CAP_WHATIF is included (recommended for most uses), run the binary with --testconfig. This will (by default) display the entire configuration that will be built as it looks just prior to allocating run-time resources. To route this information to a file for bug reporting use values of either stderr or stdout to reout to the error file or the output file respectively: --testconfig=stdout. (To get only parts of the output, add a hex or decimal value that is the sum of the desired display values found in the full output display (0 == full display): --testconfig=0x6.)

The --enable and --disable commands control the global flags that 'switch' features on and off.

The --defaults parameterallows over-riding compiled-in numeric and string values. The only paramter to the --defaults option is a comma-separated list of name=value pairs.

Note:

All command-line parameters can be moved to the configuration file except `--configfile` (to specify a
configuration file other than "/etc/syslog.conf"), `--help`, `--version`, and `--testconfig` because
these all exit immediately upon completion.

Use the tilde ('~') character at the start of a line to distinguish command-line input from output
specifications or comments. When placing command-line options into the configuration file, remove all
quoting and escape characters that would otherwise protect the strings from the shell's parser.  Syslogd2
will generally forgive the absence of equal signs and exraneous whitespace whereever the actual value of
parameters is not affected.

Refer to [Syslogd2 Configuration Reference] for a full list of configuration options. So far, your configuration file might look something like this:

---

~ --stderr = /var/log/parseerrors.log,uid=root,gid=root,level=7  --stdout=/var/log/output.log,level=7
~ --defaults = cachefile=/etc/syslog.d/cachefile.dat, markinterval=0, readers=3, workers=10,
inputlines=1000, outputworkers=5, outputlines=500
~ --enable inet, forwarding, cache, dns, allmessages, applicationmode
~ --defaults configdir=/etc/syslog.d, spooldir=/var/spool/syslog
kern.* /var/log/kernel.log
mail.* /var/log/mail.log
ftp.* /var/log/ftp.log
*.crit;*.alert /dev/console

---

Much of the power and capability of Syslogd2 stems from its ability to support multiple instances of multiple types of threadpools to optimize performance. To configure threadpool parameters, the administrator has several options. The actual values used at run-time are the maximum of the compiled-in / run-time-default values, the values given by the threadmaps option and the values provided to individual input and output declarations assigned to that threadpool. The final threadpool parameters are displayed in the --testconfig output display.

1. Accept the compile-time values or recompile the code each time they need to be changed.
2. Accept the compile-time values where appropriate but override them where necessary using the --defaults command-line option.
3. Use the --threadmaps option to centrally define and all threadpools definitions.
4. Allow indiviudal input and output declarations to increase the threadcounts & FIFO line-counts for the threadpools they are assigned to.
5. All of the above.

~ --threadmaps = input,id=0,readers=4,workers=12,lines=1000; \
                 output,id=3, workers=15,lines=800; \
                 tailfile,id=0, readers=4, lines=800, workers=12, polltime=10

~ --input *, port=1024,stream, id=0, version = 4
local0.*                 - /var/log/local0.log,id=3,uid=root,gid=staff,mode=640,filter=outputfilter1
local1.*                 |/var/log/pipeoutput

The above statements define 3 empty threadpools, then create an input TCP socket assigned to input threadpool-id 0 and an output file assigned to output threadpool-id 3. Note that the 2nd output line defaults to output threadpool 0 which will use global default threadpool values (If not defined in a --defaults option, the compiled-in values will be used). Note also the use of backslash ('\') to concatenate the successive --threadmaps lines for clarity.

---