Related Projects

Retrun to Home page

External Projects Related to Syslogd2

Return to top
* CheckNet:

* Status; Proposal (demo code in Syslogd2 *checknet* tool and shared/checknet.[ch] library files.)
* Vision: As Linux continues to move from servers to engineering workstations to desktops to laptops, the state-of-the-art needs to start dealing with how to handle changing network conditions to enable smooth transition from one network environment to another.

* Work users may change from docking-station wired connection to wireless connection on a 'guest network' for a work meeting and back.
* Mobile users may change from VPN connection to at-home internet service to coffee-shop to friend's network environment.
* When these network transitions are made, the users may be able to tolerate short-term performance degredation and delays, but Linux currently has no way to detect and 'close' invalid network connections, links & mounts. The CheckNet project would be to develop an application (or methodology) to monitor network state (not activity) and to report (or trigger) the state to other configuration elements.
* It should provide a library that can be embedded into various Linux services (like Syslogd2) with (perhaps) a service that periodically checks the network state and reconfigures network mounts and services accordingly based on the state of the IP network stack and available network services. Ideally, this CheckNet project could trigger mounting and dismounting of NFS & CIFS filesystems change of DNS & NIS configurations, startup & shutdown of applications that can now / no-longer connect to their host servers, etc.
* The preliminary (stub) form of CheckNet is currently at the core of Syslogd2's "network awareness" and the basic concept behind the *--netowrk=* option where the return of an identified network identifier other than "Down", "Local" and "Other" becomes possible and would drive different syslog configurations.

* pOpen:

* Status: Development (Approx 80% complete)
* pOPen is as much a separate spin-off of Syslogd2 as it is a 'tool' of Syslogd2.
* It is written and provided as a multi-threaded 'sample' Syslogd2 post-processor for calling external programs similar in concept to the omprogram plugin for rsyslog. I chose this functionality because I've never used the popen() system call before and because I found that the LibreNMS project relies on such a call to build its database. Ultimately, the DBD project may prove faster for database insertion, but pOpen sounds simple enough to make a good sample application and learning tool for aspiring multi-thread developers.

* pOpen demonstrates multi-threaded reading from a Linux socket, and multi-threaded output handling ( & deconflicting ) multiple connections, and flow-control using a library of simplified code from Syslogd2.

* DBD2 (DataBase Daemon 2):

* Status: Development (Approx 40% complete)
* Vision: It is my current intent to make DBD Syslogd2's "database insertion" post-processor-of-choice due to its (multi-threaded) high speed and flexibility.
* DBD2 was originally conceived as a means of recording job status and movement through HPC Clusters. The scheduling software (at that time) was able to move jobs between clusters but not track where or when a job ran or which cluster it ran on in any centralized sort of way.

* The DBD2 project consists of two primary components: a server that runs on a host with database drivers installed and a lightweight client that passes data provided on the command-line via TCP to the server via a series of *name=value* strings.
* The strings the DBD2 server gets from the client include which database and SQL template to use to insert the rest of the strings into the correct database fields.
* The DBD2 server is able to obtain envelope information (node-name if client specifies "hostname" in back-ticks as a field parameter), the cluster ID (from the connection source-address), and the time (from the system clock). If the client also supplies the job ID, the user, and the jub status (based on when it is called in the job-scheduling script), it is possible to track job information across a number of disparate clusters.
* As Syslogd2 advanced, I realized that DBD2 is a natural fit for a generic database-insertion engine for syslog data. DBD2 was still mostly unwritten and had already been designed to be multi-threaded for read and write, so all that was needed were extensions to read and parse TCP syslog data (UDP in the future).

* Syslog Parser (SLP):

* Status: Stalled (Approx 30% of design complete, some proof-of-concept code that is already obsolete and will be re-written)
* Vision: SLP is to be the central "coordinator" for a syslog-based network management system, receiving syslog input from local UDP and TCP sources as well as from remote TCP Syslogd2 concentrators.

* SLP is designed to provide additional features not available at the network edge where Syslogd2 lives.

* Timezone-aware -- SLP will be able to tanslate data across timezones for companies whose monitoring extends across timezones.
* Language-aware -- SLP is targeting the ability to translate input envelope information in different languages and to output envelope information in different languages and locale formats (primarily time-strings such as months-of-the-year and formats)
* User-definable output -- SLP is being designed to allow users full control of the output format for each output.
* User-definable message-field breakout -- This goal is to allow users to specify which components of syslog messages are to be isolated into database fields to be sent to DBD.

* As current Syslogd2 filters work by recognizing the static components of syslog messages, this SLP goal is to recognize and extract the variable components.
* Device-based routing -- SLP will be able to use device-type tags for internal routing of messages, providing the ability to distinguish and isolate traffic from different device "types" (VPN vs firewall vs router vs switch, etc). This ability is a pre-requisite to being able to break out variables from message content in any reliable fashion.

Return to top

---

Tools That Ship with Syslogd2

Return to top
* Stub programs useful for sending test data into or receiving data from Syslogd2. These programs can also act as sample code for those wishing to learn socket programming. Any of these programs will give parameters and usage if run with no parameters.

* Stubs with 3-letter names:

* XYZ where

* X is 't'ransmit or 'r'eceive
* Y is 's' (stream / TCP), 'd' (datagram / UDP)
* Z is '4' (IPv4), '6' (IPv6), 'p' (Pipe), 'u' (Unix)

* Stubs with 4-letter names:

* XYZf where

* XYZ as above
* f is for 'file' -- these programs take a file-name parameter and transmit the contents of that file to the applicable socket or pipe.

* Command tool (tsucS) where

* tsu are as above (Transmit, Stream, Unix)
* c is for 'command'
* S is the suffix assigned in the makefile for a given version of the binary (in sample code, the tsucg tool is designed for the syslogd2g binary which is the suffix associated with the 'MEGA' definition in shared/defines .h.

* checknet: This is a demonstration prototype for the CheckNet project proposed below. When you run this with an empty string as a filename (""), you get a text output of the state of your host's network stack. You can test this by trying variations on your linux stack (shut it down to force 'down', disconnect the network cable to force 'local' mode, unconfigure IPv6 or remove the IPv4 external addresses & IPv4 loopback address, etc. The checknet tool uses the same code library that Syslogd2 uses to find the current status of the IP network stack before attempting to initialize IP ports. The meanings of the 'status' value bits are as follows:

* bit value 0x01: IPv4 headers & libraries are installed
* bit value 0x02: IPv4 service is up and running on local interface (IP stack is up and running)
* bit value 0x04: IPv4 is configured and active on at least one external interface
* bit value 0x08: reserved
* bit value 0x10: IPv6 headers & libraries are installed
* bit value 0x20: IPv6 is up and runnign on local interface (IP stack is up and running)
* bit value 0x40: IPv6 is configured and active on at least one external interface
* bit value 0x80: reserved
* bit value 0x100 a configuration file was specified on the command-line but not found or not readable.

* popen: This is a wok-in-progress sample Syslogd2 post-processor. I filled it with comments and used simplified code from Syslogd2 to implement a multi-threaded application that reads from a Unix socket and uses the popen() system call to exec an external program (popentest), connecting the external program's stdin and stdout to a socket the parent can read and write to. There were to drivers for the creation of this program:

* The first driver was to demonstrate code that can function as a syslogd2 post-processor.
* The second drive was to provide sample code that other developers can study, use and emulate to learn multi-threaded programming to improve their skills and products.

Return to top

Return to Home page