Recent changes to CommandReference

CommandReference modified by Ed Freesmeyer

Ed Freesmeyer — Sat, 02 Jan 2021 00:31:06 -0000

--- v4
+++ v5
@@ -5,9 +5,9 @@
 
-            
+            
-            
+            
     
         Global Commands Global Command Summary
         
Output Commands Output Command Summary
         
     
 
@@ -31,14 +31,14 @@
     OutputLines ::= Minimum (default) number of lines (msg-slots) to allocate for each Output-Threadpool

     Connections ::= Minimum (default) number of database connections to support for each output-type of mysql.

     
-    MaxInputFields, InputFieldCount ::= Maximum number of <name>=<value> pairs in any incoming data string (not counting the template-string definition). If CAP_Syslog is defined, this value has a floor of "4" (facpri, time, host, msg fields).  Incoming messages that exceed this number will have only the first <n> <name>=<value> pairs used.  Each additional value adds 12 bytes to each message-structure.

+    MaxInputFields, InputFieldCount ::= Maximum number of <name>=<value> pairs in any incoming data string (not counting the template-string definition). If CAP_Syslog is defined, this value has a floor of "4" (facpri, time, host, msg fields).  Incoming messages that exceed this number will have only the first <n> <name>=<value> pairs used.  Each additional value adds 12 bytes to each message-structure.

     MsgStructCount, GlobalMsgCount ::= The total number of message-structures allocated for system use.  The message structures are allocated at startup and then issued to active threadpools on request.  If this number is too low, data may be lost while waiting for msg-structures to be freed up after writing contents to output-destinations.  If too high, some of the allocated structures may not be used. (Better too many than not enough)

     HostName ::= The hostname of the DBD2 server.  DBD2 will initialize this value using the gethostname() system call. This option allows the administrator to over-ride the results of the gethostname() system-call.

     DomainName, Domain ::= The domain-name to use when forming Fully Qualified Domain Name (FQDN) values. Not all Linux hostnames contain correct domain-names. This allows the administrator to over-ride the domain-name contained in the results of the gethostname() system-call.

     TemplateKeyNames, Templatenames ::= Changes the default name-list used by the DBD2 server to identify which <name>=<value> pair contains the <template-string>.

     

     input, socket ::= A <ConnectionSpec> that defines an IP or Linux socket for data input. (Currently only TCP or Streaming Linux sockets are supported.)

-    pipe ::= A <ConnectionSpec> that defines a Named-Pipe for data input. (Not yet implemented)

+
     enable ::= Enable Global boolean values.

     disable, suppress, no ::= Disable Global boolean values.

     

@@ -104,11 +104,11 @@
     

     The following output options are available only when either CAP_MYSQL or CAP_DBI is compiled into the binary:
     

-        host ::=

-        database, dbname, name, N ::=

-        username, dbuser, user, U ::=

-        password, dbpass, pass, pwd, W ::=

-        connections ::=

+    host ::= This option specifies a <HostSpec> that specifies how to connect to the database. The <HostSpec> does not define a specific connection -- rather the <HostSpec> parameters are translated into database API syntax where they are then used to establish each of the multiple connections to that database.  The <HostSpec> syntax was chosen for consistency with other DBD2 parameters such as <FileSpec> and <InputSpec> and to allow for inclusion (again for consistency) of DBD2 parameters such as threadpool control and configuration settings as well as future " output "connection-related" values (such as filters and spooling).

+        database, dbname, name, N ::= The name of the database to connect to.

+        username, dbuser, user, U ::= The database-user-account to connect to the database as.

+        password, dbpass, pass, pwd, W ::= The password for the database-user-account.

+        connections ::= The number of concurrent connections this output definition will create to this database.

     

     The following output option(s) are available only when CAP_SYSLOG is compiled into the binary:
     

</n>

CommandReference modified by Ed Freesmeyer

Ed Freesmeyer — Sat, 02 Jan 2021 00:00:44 -0000

--- v3
+++ v4
@@ -14,80 +14,91 @@
 
 Return to top
 
-Global Commands

+Global Commands
 
     Return to top

         The following global options are always available:
-    

+    

     defaults ::= Set global default values. (Over-rides corresponding compiled-in values)

-    

-    SpoolDir ::= Not yet implemented or used.

+    

+    
     ConfigDir ::= Location of the cache-file if cache-file is given as a relative filename.

     CacheFile ::= User-supplied name-cache file for use in building envelope-variables and resolving incoming IP-Addresses to hostnames.

-    Readers ::= 

-    Workers ::= 

-    Writers, OutputWriters, OutputWorkers ::= 

-    lines ::= 

-    OutputLines ::= 

-    Connections ::= 

-    MaxFileSize, mfs ::= 

-    MaxInputFields, InputFieldCount ::= 

-    MsgStructCount, GlobalMsgCount ::= 

-    HostName ::= 

-    DomainName, Domain ::= 

-    TemplateKeyNames, Templatenames ::= 

-    

+    Readers ::= Minimum (default) number of threads to create for each Reader-Threadpool

+    Workers ::= Minimum (default) number of threads to create for each Worker-Threadpool

+    Writers, OutputWriters, OutputWorkers ::= Minimum (default) number of threads to create for each Output-Threadpool

+    lines ::= Minimum (default) number of lines (msg-slots) to allocate for each Worker-Threadpool

+    OutputLines ::= Minimum (default) number of lines (msg-slots) to allocate for each Output-Threadpool

+    Connections ::= Minimum (default) number of database connections to support for each output-type of mysql.

+    
+    MaxInputFields, InputFieldCount ::= Maximum number of <name>=<value> pairs in any incoming data string (not counting the template-string definition). If CAP_Syslog is defined, this value has a floor of "4" (facpri, time, host, msg fields).  Incoming messages that exceed this number will have only the first <n> <name>=<value> pairs used.  Each additional value adds 12 bytes to each message-structure.

+    MsgStructCount, GlobalMsgCount ::= The total number of message-structures allocated for system use.  The message structures are allocated at startup and then issued to active threadpools on request.  If this number is too low, data may be lost while waiting for msg-structures to be freed up after writing contents to output-destinations.  If too high, some of the allocated structures may not be used. (Better too many than not enough)

+    HostName ::= The hostname of the DBD2 server.  DBD2 will initialize this value using the gethostname() system call. This option allows the administrator to over-ride the results of the gethostname() system-call.

+    DomainName, Domain ::= The domain-name to use when forming Fully Qualified Domain Name (FQDN) values. Not all Linux hostnames contain correct domain-names. This allows the administrator to over-ride the domain-name contained in the results of the gethostname() system-call.

+    TemplateKeyNames, Templatenames ::= Changes the default name-list used by the DBD2 server to identify which <name>=<value> pair contains the <template-string>.

+    </n>

     input, socket ::= A <ConnectionSpec> that defines an IP or Linux socket for data input. (Currently only TCP or Streaming Linux sockets are supported.)

     pipe ::= A <ConnectionSpec> that defines a Named-Pipe for data input. (Not yet implemented)

     enable ::= Enable Global boolean values.

     disable, suppress, no ::= Disable Global boolean values.

-    

+    

     IPv4, 4 ::= Default: Enabled

     IPv6, 6 ::= Default: Enabled

-    log, l ::= DBD2 will log all incoming messages to either output-specific or global log file.  Default: Disabled.  Log output is of the form:
 Template <template-string> "<raw input>" ==> "<processed-output>"

+    log, l ::= DBD2 will log all incoming messages to either output-specific or global log file.  Default: Disabled.  Log output is of the form:

+        Template <template-string>: Input String: "<raw input>" ==> "<processed-output>"

     dns, d ::= DBD2 will use DNS queries if the name-cache is disabled or when name-cache queries fail. Default: Enabled

     inet, i ::= DBD2 will allow IP input and output connections. Default: Enabled

     cache, c ::= DBD2 will use it's internal name-cache information to resolve IP addresses and to create envelope-variables. If no cache-file is provided (or an incomplete one), and if DNS is enabled, DNS responses will be added to the name-cache. Default: Enabled

     syslog, s ::= DBD2 will accept and act on syslog input when received if otherwise configured. Default: Disabled

     queue, q ::= DBD2 will utilize alternate queueing algorithms and will wait for a queue slot to become available whenever a queue is full.  The default action is to discard messages when queueing to a full queue.  Default: Disabled

     interfacequery, query ::= DBD2 will query local network interfaces and use the results to validate input IP-Addresses. Default: Enabled

-    

-    timestring ::=

-    selfaddress, self ::=

-    threads ::=

-    var ::=

-    $ ::=

+    

+        timestring ::= Defines an indexed time-format using strftime format specifiers. (See Linux manual page for strftime).  timestring = 100 = <unquoted format string>

+    selfaddress, self ::= This option allows an administrator to manually specify that an IP address is present on an interface. This is not currently a known issue for Linux.  Apple's Mac OSX does not properly interpret IPv6 addresses such as "fec0:2::90", making this option necessary on my Mac.  Ports to other operating systems may experience similar failures.

+    threads ::= A semi-colon (' ; ') separated list of threadpool declarations. The threads option  configures threadpools. It can increase (but not decrease) the default  thread-counts or buffer-line-counts for any threadpool of any type or it can define new threadpools of any type.

+    

+    type, t =<threadpool-type> ::= <Threadpool-type> is one of " input ", " worker " or " output "

+    id, i=<numeric-threadpool-id> ::= The threadpool-id being configured. <Threadpool-Id> values are non-negative integer identifiers. The default threadpool-id for each threadpool-type is zero (0).

+    queue, q=<numeric-threadpool-id> ::= Valid for input-type declarations only. Defines which worker-threadpool a given reader-threadpool will queue it's data to.

+    readers, r ::= Valid for input-type threadpools only. Increases the default (minimum) number of input-threads to create for the threadpool being defined.

+    workers, writers, w ::= Valid for worker-type or output-type threadpools only. Increases the default (minimum) number of threads to create for the threadpool being defined.

+    lines, l ::= Valid for worker-type or output-type threadpools only.  Increases the default (minimum) number of buffer (queue) entries for the threadpool being defined.

+    threads= id=0, readers=4, q = 1, type=input; id=1, type=worker, lines=500, workers=5; type=output, id=0, workers=8, lines=200;

+    threads= id=10, readers=6, q = 2, type=input; id=2, type=worker, lines=300, workers=6; type=output, id=1, workers=12, lines=300;

+    

+    var ::= Defines an sql (user-string-provided) variable

+    $ ::= Defines an envelope (calculated) variable

     stderr, E ::= A <LogSpec> that defines the global error file.

     logfile, L ::= A <LogSpec> that defines the global log file.

     

     The following global options are available only when CAP_SYSLOG is compiled into the binary:
     

-    <Syslog-style selector-string> ::=

-    

+    <Syslog-style selector-string> ::= The "keyword" of this option is a syslog-style selector-string, the <value> component is a <template-string>.

+    

     Return to top
     

-Output Commands

-    
-    Return to top
+
Output Commands
+    
+    Return to top

         The following output options are always available:
     

         destid, i ::= A single (non-case-sensitive) letter to identify this output section in template-strings.

         description, desc ::=

         enable ::=

         disable, suppress, no ::=

+    

+        logfile, log, l ::= A boolean keyword to enable logging of dbcl (non-syslog) data to this database.

     

-        logfile, log, l ::= A boolean keyword to enable logging of dbcl (non-syslog) data to this database.

-    

         file ::= A <FileSpec> that defines the primary output file of this threadpool. (Only valid for output-destinations of type " file ").

         threadpoolid, id, p ::= The identifier of the output threadpool that services this output section.

         lines ::= The minimum number of buffer-lines to allocate in the input buffer of the output threadpool that services this output section.

         workers, writers, w ::= The minimum number of threads to define in the output threadpool that services this output section.

-    type, t ::= Type of output defined by this section.  One of " file ", " mysql " or " dbi ".

+        type, t ::= Type of output defined by this section.  One of " file ", " mysql " or " dbi ".

         template ::= Defines an SQL template. The parameter to this option is a " <template-id> [=] <template-string> " pair. where <template-id> is a positive integer and <template-string> is an unquoted format string that substitutes "$+sql_id" placeholders for  variable values (where sql_id is a positive integer):
-    

+    

     template = 1 = insert into jobs (jobid, user, myhost) values ('$1', '$56', '$88')

     template  12 insert into otherjobs (clusterid, clusternode, jobid) values ('$4', '$5', '$1')  --> equal signs are optional and have been omitted in this example
-    

+    

     stderr, E ::= A <LogSpec> that defines a dedicated error file for this output. If not defined the global error file is used.  (Parsing errors always go to the global error file.)

         logfile, L ::= A <LogSpec> that defines a dedicated logfile for this output. If not defined and boolean logging variable(s) are enabled, the global logfile is used.

     

@@ -102,7 +113,8 @@
     The following output option(s) are available only when CAP_SYSLOG is compiled into the binary:
     

         syslog, s ::= A boolean keyword that enables or disables logging of syslog traffic to this database. Default: Disabled.

-    

-    Return to top
-    
+    

+    Return to top


+Return to Home page
+

CommandReference modified by Ed Freesmeyer

Ed Freesmeyer — Fri, 01 Jan 2021 09:18:04 -0000

--- v2
+++ v3
@@ -21,34 +21,36 @@
     

     defaults ::= Set global default values. (Over-rides corresponding compiled-in values)

     

-     ::= 

-     ::= 

-     ::= 

-     ::= 

-     ::= 

-     ::= 

-     ::= 

-     ::= 

-     ::= 

-     ::= 

-     ::= 

-     ::= 

-     ::= 

+    SpoolDir ::= Not yet implemented or used.

+    ConfigDir ::= Location of the cache-file if cache-file is given as a relative filename.

+    CacheFile ::= User-supplied name-cache file for use in building envelope-variables and resolving incoming IP-Addresses to hostnames.

+    Readers ::= 

+    Workers ::= 

+    Writers, OutputWriters, OutputWorkers ::= 

+    lines ::= 

+    OutputLines ::= 

+    Connections ::= 

+    MaxFileSize, mfs ::= 

+    MaxInputFields, InputFieldCount ::= 

+    MsgStructCount, GlobalMsgCount ::= 

+    HostName ::= 

+    DomainName, Domain ::= 

+    TemplateKeyNames, Templatenames ::= 

     

     input, socket ::= A <ConnectionSpec> that defines an IP or Linux socket for data input. (Currently only TCP or Streaming Linux sockets are supported.)

-    pipe ::=
 A <ConnectionSpec> that defines a Named-Pipe for data input. (Not yet implemented)
+    pipe ::= A <ConnectionSpec> that defines a Named-Pipe for data input. (Not yet implemented)

     enable ::= Enable Global boolean values.

     disable, suppress, no ::= Disable Global boolean values.

     

     IPv4, 4 ::= Default: Enabled

     IPv6, 6 ::= Default: Enabled

-    log, l ::= Default: Disabled

-    dns, d ::= Default: Enabled

-    inet, i ::= Default: Enabled

-    cache, c ::= Default: Enabled

-    syslog, s ::= Default: Disabled

-    queue, q ::= Default: Disabled

-    interfacequery, query ::= 

+    log, l ::= DBD2 will log all incoming messages to either output-specific or global log file.  Default: Disabled.  Log output is of the form:
 Template <template-string> "<raw input>" ==> "<processed-output>"

+    dns, d ::= DBD2 will use DNS queries if the name-cache is disabled or when name-cache queries fail. Default: Enabled

+    inet, i ::= DBD2 will allow IP input and output connections. Default: Enabled

+    cache, c ::= DBD2 will use it's internal name-cache information to resolve IP addresses and to create envelope-variables. If no cache-file is provided (or an incomplete one), and if DNS is enabled, DNS responses will be added to the name-cache. Default: Enabled

+    syslog, s ::= DBD2 will accept and act on syslog input when received if otherwise configured. Default: Disabled

+    queue, q ::= DBD2 will utilize alternate queueing algorithms and will wait for a queue slot to become available whenever a queue is full.  The default action is to discard messages when queueing to a full queue.  Default: Disabled

+    interfacequery, query ::= DBD2 will query local network interfaces and use the results to validate input IP-Addresses. Default: Enabled

     

     timestring ::=

     selfaddress, self ::=

CommandReference modified by Ed Freesmeyer

Ed Freesmeyer — Fri, 01 Jan 2021 08:13:33 -0000

--- v1
+++ v2
@@ -19,18 +19,44 @@
     Return to top

         The following global options are always available:
     

-    defaults ::=

-    input, socket ::=

-    pipe ::=

-    enable ::=

-    disable, suppress, no ::=

+    defaults ::= Set global default values. (Over-rides corresponding compiled-in values)

+    

+     ::= 

+     ::= 

+     ::= 

+     ::= 

+     ::= 

+     ::= 

+     ::= 

+     ::= 

+     ::= 

+     ::= 

+     ::= 

+     ::= 

+     ::= 

+    

+    input, socket ::= A <ConnectionSpec> that defines an IP or Linux socket for data input. (Currently only TCP or Streaming Linux sockets are supported.)

+    pipe ::=
 A <ConnectionSpec> that defines a Named-Pipe for data input. (Not yet implemented)
+    enable ::= Enable Global boolean values.

+    disable, suppress, no ::= Disable Global boolean values.

+    

+    IPv4, 4 ::= Default: Enabled

+    IPv6, 6 ::= Default: Enabled

+    log, l ::= Default: Disabled

+    dns, d ::= Default: Enabled

+    inet, i ::= Default: Enabled

+    cache, c ::= Default: Enabled

+    syslog, s ::= Default: Disabled

+    queue, q ::= Default: Disabled

+    interfacequery, query ::= 

+    

     timestring ::=

     selfaddress, self ::=

     threads ::=

     var ::=

     $ ::=

-    stderr, E ::=

-    logfile, L ::=

+    stderr, E ::= A <LogSpec> that defines the global error file.

+    logfile, L ::= A <LogSpec> that defines the global log file.

     

     The following global options are available only when CAP_SYSLOG is compiled into the binary:
     

@@ -43,18 +69,25 @@
     Return to top
         The following output options are always available:
     

-        destid, i ::=

+        destid, i ::= A single (non-case-sensitive) letter to identify this output section in template-strings.

         description, desc ::=

         enable ::=

         disable, suppress, no ::=

-        file ::=

-        threadpoolid, id, p ::=

-        lines ::=

-        workers, writers, w ::=

-        type, t ::=

-        template ::=

-        stderr, E ::=

-        logfile, L ::=

+    

+        logfile, log, l ::= A boolean keyword to enable logging of dbcl (non-syslog) data to this database.

+    

+        file ::= A <FileSpec> that defines the primary output file of this threadpool. (Only valid for output-destinations of type " file ").

+        threadpoolid, id, p ::= The identifier of the output threadpool that services this output section.

+        lines ::= The minimum number of buffer-lines to allocate in the input buffer of the output threadpool that services this output section.

+        workers, writers, w ::= The minimum number of threads to define in the output threadpool that services this output section.

+    type, t ::= Type of output defined by this section.  One of " file ", " mysql " or " dbi ".

+        template ::= Defines an SQL template. The parameter to this option is a " <template-id> [=] <template-string> " pair. where <template-id> is a positive integer and <template-string> is an unquoted format string that substitutes "$+sql_id" placeholders for  variable values (where sql_id is a positive integer):
+    

+    template = 1 = insert into jobs (jobid, user, myhost) values ('$1', '$56', '$88')

+    template  12 insert into otherjobs (clusterid, clusternode, jobid) values ('$4', '$5', '$1')  --> equal signs are optional and have been omitted in this example
+    

+    stderr, E ::= A <LogSpec> that defines a dedicated error file for this output. If not defined the global error file is used.  (Parsing errors always go to the global error file.)

+        logfile, L ::= A <LogSpec> that defines a dedicated logfile for this output. If not defined and boolean logging variable(s) are enabled, the global logfile is used.

     

     The following output options are available only when either CAP_MYSQL or CAP_DBI is compiled into the binary:
     

@@ -64,6 +97,10 @@
         password, dbpass, pass, pwd, W ::=

         connections ::=

     

+    The following output option(s) are available only when CAP_SYSLOG is compiled into the binary:
+    

+        syslog, s ::= A boolean keyword that enables or disables logging of syslog traffic to this database. Default: Disabled.

+    

     Return to top

CommandReference modified by Ed Freesmeyer

Ed Freesmeyer — Fri, 01 Jan 2021 04:35:28 -0000

Global Commands

Return to top
The following global options are always available:

defaults ::=
input, socket ::=
pipe ::=
enable ::=
disable, suppress, no ::=
timestring ::=
selfaddress, self ::=
threads ::=
var ::=
$ ::=
stderr, E ::=
logfile, L ::=

The following global options are available only when CAP_SYSLOG is compiled into the binary:

<Syslog-style selector-string> ::=

Return to top

Output Commands

Return to top The following output options are always available:

destid, i ::=
description, desc ::=
enable ::=
disable, suppress, no ::=
file ::=
threadpoolid, id, p ::=
lines ::=
workers, writers, w ::=
type, t ::=
template ::=
stderr, E ::=
logfile, L ::=

The following output options are available only when either CAP_MYSQL or CAP_DBI is compiled into the binary:

host ::=
database, dbname, name, N ::=
username, dbuser, user, U ::=
password, dbpass, pass, pwd, W ::=
connections ::=

Return to top

Global Commands	Global Command Summary
Output Commands	Output Command Summary

Recent changes to CommandReference

CommandReference modified by Ed Freesmeyer

CommandReference modified by Ed Freesmeyer

Global Commands

Global Commands

Output Commands

Output Commands

CommandReference modified by Ed Freesmeyer

CommandReference modified by Ed Freesmeyer

CommandReference modified by Ed Freesmeyer

Contents

Global Commands

Output Commands