Recent changes to Home

Home modified by Ed Freesmeyer

Ed Freesmeyer — Sun, 01 Aug 2021 04:39:43 -0000

--- v10
+++ v11
@@ -23,7 +23,9 @@

 
 
-This project is currently in beta.  It needs stress-testing and validation under live conditions to be moved from beta to production status.

+This project is currently in beta for Apple OSX.  It has been reverted to a pre-beta state as it needs an overhaul as a result of its port to Linux and subsequent work to insure it works on both OS's.

+ 

+Once it has been re-verified for Linux, it will need stress-testing and validation under live conditions to be moved from beta to production status.

 

 This wiki needs a lot of work (and will receive it as time permits).  I'm also scrambling to clean up the documentation for Syslogd2. I've uploaded a (working copy) of a "dbd2 manual" to the files section of this project but anticipate using this wiki as the primary source of documentation for some time to come.

Home modified by Ed Freesmeyer

Ed Freesmeyer — Fri, 01 Jan 2021 03:43:18 -0000

--- v9
+++ v10
@@ -6,9 +6,13 @@
         
             Abstract and Introduction
         
+            Theory of Operation
+        
             Terms and Concepts
         
             Sample Configuration File
+        
+            Command Reference

Home modified by Ed Freesmeyer

Ed Freesmeyer — Thu, 31 Dec 2020 11:58:09 -0000

--- v8
+++ v9
@@ -6,7 +6,7 @@
         
             Abstract and Introduction
         
-            Terms and Concepts in DBD2
+            Terms and Concepts
         
             Sample Configuration File

Home modified by Ed Freesmeyer

Ed Freesmeyer — Thu, 31 Dec 2020 11:57:43 -0000

--- v7
+++ v8
@@ -5,6 +5,8 @@
     
         
             Abstract and Introduction
+        
+            Terms and Concepts in DBD2
         
             Sample Configuration File

Home modified by Ed Freesmeyer

Ed Freesmeyer — Thu, 31 Dec 2020 09:03:58 -0000

--- v6
+++ v7
@@ -1,54 +1,89 @@
-This project is currently in beta.  It needs stress-testing and validation under live conditions to be moved from beta to production status.
-This wiki needs a lot of work (and will receive it as time permits).  I'm also scrambling to clean up the documentation for Syslogd2. I've uploaded a (working copy) of a "manual" to the files section of this project but anticipate using this wiki as the primary source of documentation for some time to come.
+##Contents
+    
+
+
+    
+        
+            
+        
+            
+        
+    
+Abstract and Introduction
Sample Configuration File
+
+Return to Top
+##Abstract 
+Return to Top

-The DBD2 project consists of 2 binaries: a client component (dbcl) and a server component (dbd2).
-Both components are written completely in C.
-
-The prototype for this project was designed to track user jobs running on multiple Linux clusters.  The dbcl executable was called by the job scripts and sent internal cluster node-id, job-id, user-id, job-status and status-change-time to the server.  The dbcl (client executable) is used only for non-syslog input and can be used for any purpose in any situation where the client is running the Linux OS and desires to provide a logging record in some remote (MySql) database.  The IP protocol used by the client is TCP. If the connection cannot be immediately established, the client backgrounds itself and continues to try until its life-time timer expires.
-
-The server reads the command-line input, then does a reverse-name-lookup (via cache-file or DNS), and records the time-of-receipt.
-
-All the client data as well as the source-envelope information become available database field variables for use in user-defined SQL templates.
-
-
+
 
+This project is currently in beta.  It needs stress-testing and validation under live conditions to be moved from beta to production status.

+

+This wiki needs a lot of work (and will receive it as time permits).  I'm also scrambling to clean up the documentation for Syslogd2. I've uploaded a (working copy) of a "dbd2 manual" to the files section of this project but anticipate using this wiki as the primary source of documentation for some time to come.

+

+The DBD2 project consists of 2 binaries:
+
+
+    (1) A client component (dbcl)

+
+
+  The client has no pre-requisites or required supporting software.
+
+
+    (2) A server component (dbd2)

+
+
+    The sever component needs to have the mysqlclient library installed if native MySql support is desired.

+    If dbi support for multiple database types is desired, the "C"-language dbi-library API from the SourceForge project of the same name must be installed.

+    Other than the database library(ies), the dbd2 server has no other software requirements (besides the obvious such as a database of an appropriate type running somewhere on the network).
+
+
+
+
+Both components are written completely in C.

+

+The prototype for this project was designed to track user jobs running on multiple Linux clusters.  The dbcl executable was called by the job scripts and sent internal cluster node-id, job-id, user-id, job-status and status-change-time to the server.  The dbcl (client executable) is used only for non-syslog input and can be used for any purpose in any situation where the client is running the Linux OS and desires to provide a logging record in some remote (MySql) database.  The IP protocol used by the client is TCP. If the connection cannot be immediately established, the client backgrounds itself and continues to try until its life-time timer expires.

+

+The server reads the command-line input, then does a reverse-name-lookup (via cache-file or DNS), and records the time-of-receipt.

+

+All the client data as well as the source-envelope information become available database field variables for use in user-defined SQL templates.

+

 In the current design iteration, I have added:
 
-
+

     Interface code for Syslogd2:
     
-
+

         Allows syslog-priority-selector-strings to select the sql templates and databases (since syslog data is standard format).
         Treat syslog envelope fields (facility, priority, event-receipt-time, event-host, etc as usable database fields.
-    
+
 
 I plan for future releases to add additional features as requested or identified over time. These features include but are not limited to:
 
-
+

     Security features to prevent malicious insertions
 
+
+
+The system configuration file is divided into a global section and one or more output sections. (See Configuration File Format).

+

+The system supports multiple input threadpools each of which uses 1 or more threads to support one or more input sockets (Linux, IP or a mix of both -- TCP/Streaming Linux only for now).

+

+The system also supports one or more "worker" threadpools, each of which processes the incoming traffic from one or more "reader-threadpools".

+

+The system supports up to 26 databases (identified as 'a' - 'z') with 1 or more threads per threadpool.

+

+One or more databases per threadpool are supported with each database allowed to have one or more connections for parallel data insertion. Each database may also specify one or more "sql templates" defined by the administrator for insertion of one or more "sql variable sets" into an arbitrary number of database tables.

+

+Syslog data input is optionally supported. When not declared, the syslog components are not compiled into the binary, resulting in a smaller binary image.

+

+Only the dbd2 server needs to have the MySql client libraries --or-- the 'C' dbi drivers (found on sourceforge) -- or both installed locally.

+

+Output is specified as type `mysql`, type `dbi`, or type `file`. `File` is simply a text file containing the output SQL lines. (The output file may also  be useable as a database input-file for non-MySql database periodic updates).

+

+If there is developer interest or any interest in assisting with documentation, I will happily accept.
+
 
-    
-
-
-The system configuration file is divided into a global section and one or more output sections. (See Configuration File Format).
-
-The system supports multiple input threadpools each of which uses 1 or more threads to support one or more input sockets (Linux, IP or a mix of both -- TCP/Streaming Linux only for now).
-
-The system also supports one or more "worker" threadpools, each of which processes the incoming traffic from one or more "reader-threadpools".
-
-The system supports up to 26 databases (identified as 'a' - 'z') with 1 or more threads per threadpool.
-
-One or more databases per threadpool are supported with each database allowed to have one or more connections for parallel data insertion. Each database may also specify one or more "sql templates" defined by the administrator for insertion of one or more "sql variable sets" into an arbitrary number of database tables.
-
-Syslog data input is optionally supported. When not declared, the syslog components are not compiled into the binary, resulting in a smaller binary image.
-
-Only the dbd2 server needs to have the MySql client libraries --or-- the 'C' dbi drivers (found on sourceforge) -- or both installed locally.
-
-Output is specified as type `mysql`, type `dbi`, or type `file`. `File` is simply a text file containing the output SQL lines. (The output file may also  be useable as a database input-file for non-MySql database periodic updates).
-
-If there is developer interest or any interest in assisting with documentation, I will happily accept.
-
-
 [[members limit=20]]
-[[download_button]]
+
+Return to Top

Home modified by Ed Freesmeyer

Ed Freesmeyer — Wed, 30 Dec 2020 23:58:32 -0000

--- v5
+++ v6
@@ -1,4 +1,5 @@
 This project is currently in beta.  It needs stress-testing and validation under live conditions to be moved from beta to production status.
+This wiki needs a lot of work (and will receive it as time permits).  I'm also scrambling to clean up the documentation for Syslogd2. I've uploaded a (working copy) of a "manual" to the files section of this project but anticipate using this wiki as the primary source of documentation for some time to come.

 The DBD2 project consists of 2 binaries: a client component (dbcl) and a server component (dbd2).
 Both components are written completely in C.

Home modified by Ed Freesmeyer

Ed Freesmeyer — Wed, 30 Dec 2020 23:54:32 -0000

--- v4
+++ v5
@@ -1,30 +1,52 @@
-This project is currently in beta. I am actively working on it off-line, but wanted to put some descriptions up early.
+This project is currently in beta.  It needs stress-testing and validation under live conditions to be moved from beta to production status.

-The DBD2 project will consist primarily of 2 binaries: a client component (dbcl) and a server component.
+The DBD2 project consists of 2 binaries: a client component (dbcl) and a server component (dbd2).
 Both components are written completely in C.

-My preferred name for this application is DBD (DataBase Daemon), but that name was already taken on SourceForge, so in the spirit of Syslogd2 being a 2nd "envisioning" of a syslog daemon and considering that this tool is a different (2nd "envisioning") of a database daemon or database-insertion daemon, I think the (modified) name DBD2 is appropriate.
-
-The original prototype for this project was designed to track user jobs running on multiple Linux clusters.  The client was added into the job scripts and sent internal cluster node-id, job-id, user-id, job-status and status-change-time to the server.
+The prototype for this project was designed to track user jobs running on multiple Linux clusters.  The dbcl executable was called by the job scripts and sent internal cluster node-id, job-id, user-id, job-status and status-change-time to the server.  The dbcl (client executable) is used only for non-syslog input and can be used for any purpose in any situation where the client is running the Linux OS and desires to provide a logging record in some remote (MySql) database.  The IP protocol used by the client is TCP. If the connection cannot be immediately established, the client backgrounds itself and continues to try until its life-time timer expires.

 The server reads the command-line input, then does a reverse-name-lookup (via cache-file or DNS), and records the time-of-receipt.

 All the client data as well as the source-envelope information become available database field variables for use in user-defined SQL templates.

-In the re-design, I have added:
+
+
+In the current design iteration, I have added:
+
+
+    Interface code for Syslogd2:
+    
+
+        Allows syslog-priority-selector-strings to select the sql templates and databases (since syslog data is standard format).
+        Treat syslog envelope fields (facility, priority, event-receipt-time, event-host, etc as usable database fields.
+    
+
+I plan for future releases to add additional features as requested or identified over time. These features include but are not limited to:
+
+
     Security features to prevent malicious insertions
-    Interface code for Syslogd2:
-        Allow syslog-priority-selector-strings to select the sql templates and databases (since syslog data is standard format.
-        Treat syslog evelope fields (facility, prioity, event-receipt-time, event-host, etc as usable database fields.
+
+
+    


-Each defined database is allowed to have <n> user-defined simultaneous connections which are shared among a user-defined number of database-threads (<t>).  <n> and <t> are that are defined on a per-database basis.  Multiple databases can be defined for a single DBD2 server which allows for a single input request to be inserted into multiple databases or multiple database tables.
+The system configuration file is divided into a global section and one or more output sections. (See Configuration File Format).

-Only the server needs to have either the MySql client libraries --or-- the 'C' dbi drivers (found on sourceforge) -- or both.
+The system supports multiple input threadpools each of which uses 1 or more threads to support one or more input sockets (Linux, IP or a mix of both -- TCP/Streaming Linux only for now).

-Output is specified as type `mysql`, type `dbi`, or type `file`. `File` is simply a text file containing the output SQL lines. The output file may then  be useable as a database input-file.
+The system also supports one or more "worker" threadpools, each of which processes the incoming traffic from one or more "reader-threadpools".

-If there is developer interest or any interest in assisting with documenation (once the code is baselined and stable), I will move my subversion respository from my local host to sourceforge for shared access.
+The system supports up to 26 databases (identified as 'a' - 'z') with 1 or more threads per threadpool.
+
+One or more databases per threadpool are supported with each database allowed to have one or more connections for parallel data insertion. Each database may also specify one or more "sql templates" defined by the administrator for insertion of one or more "sql variable sets" into an arbitrary number of database tables.
+
+Syslog data input is optionally supported. When not declared, the syslog components are not compiled into the binary, resulting in a smaller binary image.
+
+Only the dbd2 server needs to have the MySql client libraries --or-- the 'C' dbi drivers (found on sourceforge) -- or both installed locally.
+
+Output is specified as type `mysql`, type `dbi`, or type `file`. `File` is simply a text file containing the output SQL lines. (The output file may also  be useable as a database input-file for non-MySql database periodic updates).
+
+If there is developer interest or any interest in assisting with documentation, I will happily accept.


 [[members limit=20]]
</t></n></t></n>

Home modified by Ed Freesmeyer

Ed Freesmeyer — Wed, 30 Dec 2020 23:28:40 -0000

--- v3
+++ v4
@@ -1,4 +1,4 @@
-This project is currently in pre-alpha. I am actively working on it off-line, but wanted to put some descriptions up early.
+This project is currently in beta. I am actively working on it off-line, but wanted to put some descriptions up early.

 The DBD2 project will consist primarily of 2 binaries: a client component (dbcl) and a server component.
 Both components are written completely in C.

Home modified by Ed Freesmeyer

Ed Freesmeyer — Fri, 21 Aug 2020 09:18:35 -0000

--- v2
+++ v3
@@ -2,6 +2,8 @@

 The DBD2 project will consist primarily of 2 binaries: a client component (dbcl) and a server component.
 Both components are written completely in C.
+
+My preferred name for this application is DBD (DataBase Daemon), but that name was already taken on SourceForge, so in the spirit of Syslogd2 being a 2nd "envisioning" of a syslog daemon and considering that this tool is a different (2nd "envisioning") of a database daemon or database-insertion daemon, I think the (modified) name DBD2 is appropriate.

 The original prototype for this project was designed to track user jobs running on multiple Linux clusters.  The client was added into the job scripts and sent internal cluster node-id, job-id, user-id, job-status and status-change-time to the server.

Home modified by Ed Freesmeyer

Ed Freesmeyer — Fri, 21 Aug 2020 09:14:31 -0000

--- v1
+++ v2
@@ -1,8 +1,29 @@
-Welcome to your wiki!
+This project is currently in pre-alpha. I am actively working on it off-line, but wanted to put some descriptions up early.

-This is the default page, edit it as you see fit. To add a new page simply reference it within brackets, e.g.: [SamplePage].
+The DBD2 project will consist primarily of 2 binaries: a client component (dbcl) and a server component.
+Both components are written completely in C.

-The wiki uses [Markdown](/p/dbd2/wiki/markdown_syntax/) syntax.
+The original prototype for this project was designed to track user jobs running on multiple Linux clusters.  The client was added into the job scripts and sent internal cluster node-id, job-id, user-id, job-status and status-change-time to the server.
+
+The server reads the command-line input, then does a reverse-name-lookup (via cache-file or DNS), and records the time-of-receipt.
+
+All the client data as well as the source-envelope information become available database field variables for use in user-defined SQL templates.
+
+In the re-design, I have added:
+    Security features to prevent malicious insertions
+    Interface code for Syslogd2:
+        Allow syslog-priority-selector-strings to select the sql templates and databases (since syslog data is standard format.
+        Treat syslog evelope fields (facility, prioity, event-receipt-time, event-host, etc as usable database fields.
+
+
+Each defined database is allowed to have <n> user-defined simultaneous connections which are shared among a user-defined number of database-threads (<t>).  <n> and <t> are that are defined on a per-database basis.  Multiple databases can be defined for a single DBD2 server which allows for a single input request to be inserted into multiple databases or multiple database tables.
+
+Only the server needs to have either the MySql client libraries --or-- the 'C' dbi drivers (found on sourceforge) -- or both.
+
+Output is specified as type `mysql`, type `dbi`, or type `file`. `File` is simply a text file containing the output SQL lines. The output file may then  be useable as a database input-file.
+
+If there is developer interest or any interest in assisting with documenation (once the code is baselined and stable), I will move my subversion respository from my local host to sourceforge for shared access.
+

 [[members limit=20]]
 [[download_button]]
</t></n></t></n>