DBD2 Wiki
A multi-threaded, multi-database tcp-based database insertion app.
Status: Beta
Brought to you by:
efreesmeyer
Menu
▾
▴
Operation
Contents
| Basic Concepts and Command-Line Options |
| Basic Operation |
| Syslog Support |
| Communication Protocols in DBD2 |
| Configuration Tips |
Basic Concepts and Command-Line Options
Understanding DBD2 need not be as complicated as it sounds. Specific terms and concepts are defined and used in this wiki to insure that similar (but distinct) concepts are not confused with each other (concepts such as "database-connection-threads" vs "DBD2 operational threads" for example).
At the highest level, DBD2 is simply a client-server project that allows any network host to create database entries without needing database drivers installed. It does this by sending named database fields to a central server (running DBD2 server) where those named-fields are processed and inserted into one or more databases. DBD2 has been designed with the goal of maximum flexibility and performance under extreme load conditions such as would be found in production environments.
Home-grown applications may (at their option) choose to bypass the dbcl client code and communicate directly with the DBD2 server. The protocols for doing so can be found here.
As is often the case with software, the complexities of DBD2 are in the details.
The DBD2 Client
The DBD2 client program (dbcl) is basically a C-language "shell" around a single TCP-protocol transmission of the parameters required to make one database-input line.
Since UDP provides no way to confirm receipt of network packets, the DBD2 client connection to the DBD2 server is only defined over TCP or Streaming Linux connections. The DBD2 client program requires no configuration file and no supporting software libraries. It is completely self-contained in the compiled binary image using command-line values for all input. It supports IPv4, IPv6 and Linux socket connections to the DBD2 server using TCP/Stream connections. Usage information from the DBD2 client program can be obtained by running it with no options or with "--help", "--usage", "-?" or "-h" options.
The DBD2 client program (dbcl) takes 2 or more parameters on the command-line. The last parameter is always a string containing one or more <name=value> pairs:
The first one (or two) parameter(s)::=These must be connection specifications -- either an absolute filesystem path to a Linux socket or an IP-host/IP-address and an IP-port-name/IP-port-number.
These parameters can be named using the --host= and --port= option-names as applicable. Linux sockets use only the --host= option.
These parameters can also be un-named giving the <location> followed by the IP-port-string if using an IP connection.
Optional (named) options are:
--sleeptime=<timespec> -- or -- -s <timespec>::= How long to sleep between attempts to contact the DBD2 server. (Default: 5s)
--runtime=<timespec> -- or -- -r <timespec>::= How long to attempt to reach the DBD2 server before abandoning the attempt and terminating the dbcl program. (Default: 10s)
A <timespec> is an integer value followed by an optional time-unit-identification character. If no time-unit is given, seconds are assumed. Time-units are: 's' for seconds, 'm' for minutes, 'h' for hours or 'd' for days. (for example: -s "10" --runtime="2d" Note the quotes around numeric values to protect from the Linux shell interpreter.
The rest of the command-line parameters consist of either a single quoted string consisting of a comma-separated list of unquoted <name>=<value> parameters, individual (quoted) <name>=<value> parameters or some combination of the two. At least one of the <name>=<value> strings must have a <name> that can be found in the server's "templatenames" list (Default: "template, t"). The <value> component of that pair must be a template-string.
dbcl -h /tmp/linux_dbd2_socket "template=a1b2, eventtime=`date +%s`, myhost = `gethostname`, userid=`whoami`" jobid="$jobid" file="file with spaces"
dbcl 192.168.2.4 1023 template=a3b4 eventtime="`date +%s`" myhost="`gethostname`" userid="`whoami`"
The examples above demonstrate the various options for specifying parameters to the dbcl client.
Bear in mind the way the Linux shell interpreter treats quotes:
Single quotes (" ' ") disable shell interpretation and provide literal values (For example: The shell-script variable $jobid gets interpreted as the string "$jobid" when single quotes are used.)
Double quotes (' " ') enable shell interpretation of quoted values. (For example: the shell-script variable $jobid gets replaced by its current value when double quotes are used.)
Back-ticks (" ` ") when surrounding Linux commands on the command-line replace the command-line value with the output of the command because of shell interpretation. When single-quotes are used, this replacement does not occur.
Don't forget to separate <name>=<value> parameter strings with commas if using the single-string approach shown at the start of the first example parameter-list.
The DBD2 client program provides for both connection- and write-error detection, back-off wait time, true backgrounding operation, and re-transmissions in the event that the server is down or flooded with connection attempts to the point that it cannot respond. It does this through the use of 2 optional timers.
The DBD2 Server
The DBD2 server is locked into
Configuration of the DBD2 server is covered elsewhere in this wiki. The command-line parameters to the server process are covered here.
The command-line options for DBD2 are:
--help (-h) : Print a help screen and exit. The help screen includes the number of "extra" syslog facilities supported (if CAP_SYSLOG is enabled) and a summary of the CAP_*-abilities compiled into the binary.
--version (-v): Print version information and exit. The version screen includes the number of "extra" syslog facilities supported (if CAP_SYSLOG is enabled) and a summary of the CAP_*-abilities compiled into the binary.
--configfile (-c): An alternate configuration to use. (Default: /etc/dbd2.conf).
--stderr (-E): A LogSpec that defines a global error file for DBD2. Defaults: uid=0, gid=0, mode=600, level=7.
--logfile (-L): A LogSpec that defines a global log file for DBD2. Defaults: uid=0, gid=0, mode=600, level=7.
--Testconfig (-T): Test (check) the configuration, print a summary description and exit without affecting the running daemon process in any way.
The --Testconfig option above requires more explanation as it takes several parameters:
What processing stage to display (default is AsBuilt):
AsParsed or p: Results are displayed after the parsing phase but before any deconfliction / validation steps that occur during the Build phase of startup.
AsBuilt or b: Results are displayed after all values have been validated, de-conflicted, converted into production data structures and just before the actual processing threads are started.
Where to display the results (Default is to the screen):
stderr or E: Send output to the global error file (defined on cmd-line or in configuration file).
logfile or L: Send output to the global log file (defined on cmd-line or in configuration file).
Not Specified: Send output to terminal screen.
What to display (Default is 0):
This is a bit-map value (best expressed in hexadecimal) specifying the bits that correspond to the section or sections to be contained in the display. A value of 0 displays all sections. Each display section contains a header-line that specifies which bit identifies that section.
--testconfig=b,0x13
--testconfig=L --> provides good remote-support or archiving output (especially if combined with the error-file containing parse & build errors).
Basic Configuration
DBD2 configuration starts at compile-time with the specification of which CAP_*-abilities to support in the server binary. To determine what is currently defined in pre-compiled binary image, use the --version command-line option. To determine what will be compiled prior to compilation, see instructions here.
The DBD2 command-set is divided roughly into two sets: global options and output options. Global options are those that are found "outside" of "output-section blocks". An "output section block" in the configuration file starts with an open-square-bracket character (' [ ') as the first character of a line and terminates at either the start of the next output-section-block or at the end of the file.
Global Options
Input sources are defined using the Input= and Pipe= options both of which take a ConnectionSpec as a parameter.
System default values are set (over-riding compiled-in values) with the Defaults option:
Parameter is a comma-separated list of (unquoted) name=value pairs.
Output Options
Syslog Support
The implementation of syslog support into DBD2 is fully compatible with all Syslogd2 options since the primary purpose of DBD2 syslog integration is to work as a "post-processor" for the Syslogd2 application.
When the CAP_SYSLOG declaration is set for DBD2, several additional (global) settings become valid for use.
Communication Protocols
Configuration Tips
