File	Date	Author	Commit
src	2003-04-03	muthanna	[f8f7fc] - Fixed segfault in looperdb.c. LooperDB would ...
support	2003-03-03	muthanna	[67aa0a] - New memory allocation system using dlmalloc.
.hgtags	2010-03-15	convert-repo	[85a1b1] update tags
AUTHORS	2003-02-10	muthanna	[1eb53e] Initial revision
COPYING	2003-02-10	muthanna	[1eb53e] Initial revision
ChangeLog	2003-02-10	muthanna	[e4488b] - New certificate scripts.
INSTALL	2003-02-10	muthanna	[1eb53e] Initial revision
Makefile.am	2003-02-11	muthanna	[c6cf57] - Makefile portability fixes.
NEWS	2003-03-06	muthanna	[49fda5] - updated NEWS README
README	2003-03-06	muthanna	[49fda5] - updated NEWS README
backup.sh	2003-02-10	muthanna	[1eb53e] Initial revision
configure.in	2003-03-06	muthanna	[b02873] - Updated README.
cvs.sh	2003-02-10	muthanna	[1eb53e] Initial revision

Read Me

LooperDB 0.5
Secure Realtime Alert Database.
$Id$

Mohit Muthanna <mohit@muthanna.com>
--

LooperDB is a realtime, secure, multithreaded database server for the collection and consolidation of alerts and messages. When coupled with Looper, LooperDB can provide the framework for a sophisticated and flexible network/security management infrastructure. A looper module for LooperDB is also available.

Features:

- Highly secure. All data transfer over Secure Sockets Layer (SSL).
- Installs under a non-root user account. 
- All passwords 160-bit SHA1 encrypted.
- Lightweight multithreaded architecture. Minimum overhead.
- Small memory footprint.
- Extremely easy to use. Simple intuitive messaging protocol.
- A complete database with most features of SQL92 supported.
- All source code available. Can be used freely for any purpose subject to the conditions under the GPL.
- Support for alerts via SNMP, Syslog, Snort IDS, log files, Sockets, Pipes, Netcool, HPOV etc. via Looper.
- Service Monitors for ICMP, HTTP, POP3, IMAP, FTP, Telnet, LDAP, DNS, SMTP, RPC and lots more via Mon.

Supported Platforms
-------------------

LooperDB has been succesfully installed and tested on:

Linux/Intel 2.4 - Development Base
OpenBSD/Intel 3.0 - Port
Solaris/Sparc 2.8 - Port 

Porting LooperDB to other architectures should be trivial.

Home Page
---------

http://looperdb.muthanna.com

Prerequisites
-------------

OpenSSL 0.9.7 [www.openssl.org]
sqlite 2.7.6	[www.sqlite.org]
stunnel 4.0 [www.stunnel.org] - This is only needed for testing.

- Both libraries must be built with thread support. For sqlite add the option '-DTHREADSAFE' to TCC in Makefile after running configure.

- Both bin/ directories must be in $PATH. The installer needs access to 'openssl' and 'sqlite'.

- Remember to set LD_LIBRARY_PATH appropriately.

- On OpenBSD remember to install the pthreads library. Also for LooperDB to build on OpenBSD use gmake instead of make. 

Installing from source
----------------------

- Create a user and homedirs for LooperDB. LooperDB can run as root, but it is not recommended. 

- Untar the distribution package in a temp directory.

	$ cd /tmp; gzcat looperdb-0.1.tar.gz | tar -xvf -

- Setup the target directory. If OpenSSL or SQLite are installed in nonstandard directories, they can be specified in the --with-openssl and --with-sqlite arguments.

	$ ./configure --prefix=$HOME/looperdb --with-openssl=/opt/local/ssl --with-sqlite=/usr/local/sqlite

- Compile the software and create the certificates.

	$ make 
	(note on OpenBSD, use gmake)

  For the certificates to be successfully built, 'openssl' must be in the path. The 'makecerts.sh' script is executed to create the certificates. Note that the certs built here are only an example. You are free to build your own certificates or use real signed certificates from a trusted CA.

- Install the distribution.

	$ make install

Configure options
-----------------

--prefix=/path/to/looperdb
	Specifies looperdb installation target dir. Eg. /opt/looperdb.

--with-openssl=/path/to/openssl
	Specifies path to openssl installation dir.

--with-sqliter=/path/to/sqlite
	Specifies path to sqlite installation dir.

--enable-dlmalloc
	This can be used on older machines that have less efficient (or less reliable) memory allocation functions. New versions of linux don't need this.

Starting it up
--------------

1) Set LOOPERDB_HOME to the LooperDB base directory. This is not necessary if you specify a configuration file on the commandline as described later.

2) Edit the looperdb configuration file : $LOOPERDB_HOME/etc/looperdb.conf

3) Start looperdb.

	$ $LOOPERDB_HOME/looperdb 

If the private keys are encrypted, you will be requested for a password.

   To start looperdb using an alternate configuration file:

	$ $LOOPERDB_HOME/looperdb -c /path/to/conf

   In this case, remember to set 'home' in the configuration file to the LooperDB home directory. 'home' overrides $LOOPERDB_HOME.

4) All done. You can now visit the logs which are by default in $LOOPERDB_HOME/log.

Testing
-------

1) Get stunnel [www.stunnel.org].

2) Using the sample configuration file in $LOOPERDB_HOME/var/certs, start stunnel. If you made any changes to the LooperDB listen port, reflect it in the stunnel.conf file. You can also set the local listener port in the file.

	$ cd $LOOPERDB_HOME/var/certs
	$ stunnel stunnel.conf

3) Check stunnel.log. If it started successfully test the connection:

	$ telnet localhost 4000
	+HELO looperdb 0.1 [1023]

4) Now you can test the various looperdb features; but first, you must authenticate with LooperDB.

	auth system buggeroff
	
	... following returned by looperdb...

	+AUTH 
	+OK SQL 0 1 6
	+AUTH UID=0 GID=0 PRIVS=62 FILTER=''

5) Send an event to the database.

	event key='evt1uranusTrouble', node='uranus', severity=1, summary='Trouble', url='http://trouble/trouble.html'
	+OK SQL 0 0 0

   Only the 'key' field is mandatory, all else is optional. LooperDB uses the 'key' field to keep track duplicate events. To see the full database schema, check the create_ldb_database.sh script in $LOOPERDB_HOME/scripts.

6) List events in the database.

	list
	+OK SQL 0 1 23
	Key, Serial, Node, Agent, AlertKey, Severity, Summary, StateChange, FirstTime, LastTime, Type, Count, Class, SubKey, SubGroup, OtherInfo, Location, URL, UID, GID, Acknowledged, ServerName, ServerSerial 
	evt1uranusTrouble, 1, uranus, NULL, NULL, 1, , 1044906944, 1044906944, 1044906944, NULL, 1, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL 
	OR

	list severity=1 and node='uranus'

	OR

	list node='uranus' and (severity = 1 or (severity = 4 and acknowledged=1)

7) Run an sql statement (yes, you have the privs, you logged in as system)

	sql delete from events_state where node='uranus' and serial !=0;

   Note that you should never delete the event with serial number 0. This event is used to build views for looper connections. If you do delete it, you'll have to manually add it to the database with the 'sqlite' commandline tool. If you use the 'del' command, these checks are made for you.

Default passwords
-----------------

system/buggeroff	Permissions: SUPER 
client/buggeroff	Permissions: CLIENT, ACK
admin/buggeroff		Permissions: ADMIN, SQL, XSQL (XSQL ACL entry for 'housekeeping' only)
ackclient/buggeroff	Permissions: CLIENT, ACK, DEL
looper/buggeroff	Permissions: LOOPER

Also note that user 'client' can only modify events from the node 'uranus'. This is defined in the user's filter.

The permissions:

(Q) SQL: Run SQL statements on events database..
(X) XSQL: Run SQL statements on external databases (auth, admin, housekeeping).
(A) ADMIN: Event administration (unimplemented).
(K) ACK: Ack/Deack events.
(D) DEL: Delete events.
(L) LOOPER: Send events.
(C) CLIENT: List events.
(S) SUPER: Everything. User management.

Adding and removing users
-------------------------

To add/remove/list users in the auth database, use the 'xsql' command on the 'auth' database. The XSQL privs are required.

Listing users: xsql auth select * from looper_users

Adding a user: xsql auth insert into looper_users (uid, gid, name, passwd, priv, filter) values (NULL, 3, 'foobar', 'wlrVv5nqmSsqLl64DvGH1E4Dh3E=', 'CKD', '')

The password is a base64 encoded 160-bit SHA1 hash. To get the password use the 'passwd' cmd.

	passwd eatmyshorts
	+PASSWD SHA1 BASE64 oFgVIpVXe0VklV9OTmZyrnIb4yk=

Command List
------------

list [clause]
-------------

List events in database. [clause] could be the where part of an SQL statement. eg.:

list Node='uranus'

ack [1 | 0] [clause]
--------------------

Acknowledge / deacknowledge specified events. This sets the event's acknowledged field to 1 or 0 respectively. [clause] could be the where part of an SQL statement. eg.:

ack 1 Serial in (13, 14 19)

del [clause]
-------------

Delete specified events. [clause] could be the where part of an SQL statement. eg.:

del Severity < 2 and Acknowledged = 1

event [token=value, token=value ...]
------------------------------------

Send an event to the database. Any / all fields can be sent, the key field 'key' is manadatory. If an event with the same key is already in the database, it's count will be incremented. A good value for a key is "node+port+severity+any specific details". Adding a timestamp to the key will ensure that a unique record is generated for every event. Since this is a realtime alert database, this should be unnecessary.

auth [user] [password]
----------------------

Authenticate with LooperDB. 

sql [statement]
---------------

Send an SQL statement to the database. Make sure you do not delete events with Serial 0 as new events will not be created otherwise.

xsql [db] [statement]
---------------------

Execute an SQL command on an external database. XSQL permissions required to execute the command. In addition to this, and ACL entry for the database to be queried should be present. See sample entries in looper_acls and looper_objects. XSQL database objects have the type field of 'XSQL'. The supported databases (currently) are 'auth' and 'housekeeping'.

passwd [password]
-----------------

Generate a base 64 encoded SHA-1 encoded password string for creating new users.

stats
-----

Display server statistics and properties.

set [var] [value]
-----------------

Set session properties. With no arguments returns the list of properties and their values. Currently recognized properties are: 'delimiter'.

Integrating with Looper
-----------------------

1) Check out the looperdb_out module from the looper CVS tree. 

2) Setup stunnel on your system using the sample configuration file in $LOOPERDB_HOME/var/certs/stunnel.conf. 

3) Copy the client certificate "client.pem" (and if required the root CA "root.pem") to the machine running stunnel. Make sure that paths to these files are correctly referenced in stunnel.conf. Also make sure that stunnel is in client mode and forwards requests to the looperdb machines at the right address/port.

4) Start stunnel.

5) In the looper configuration file, specify the hostname and port of the stunnel machine, in the keys "host" and "port". Specify the username and password in "user" and "password". Other standard tokens are still available (eg. debuglevel etc.). Set debuglevel to 4 initially. Note that the user must have a minimum privelage setting of "L" (looper" to send events.

6) Bring it up.


Tuning
------

The current tunables are:

In common.h:
STRLEN: Max string len for sql statements, results, commands etc. 
NANOSLEEP: Macro which sleeps for a while between retry's. Can be reduced on fast machines.

sqlite:
The default_cache_size can be increased to substantially improve performace.


Is there a client event viewer?
-------------------------------

LooperWeb is distributed with this package. LooperWeb is a (mostly) fully functional Perl/CGI web client for LooperDB. Copy looperweb.pl to your CGI bin directory. Create a directory for looperweb to store session data and setup the properties in the %CONF hash of the looperweb.pl file. Setup stunnel as per the instructions in the previous section. Let it run.

Can I change the database backend?
----------------------------------

Modify the db.c and include/db.h files in the src/ tree. The ldb_db structure needs to be modified, the four db functions need to be implemented and the #defines need to be defined (LDB_DB_OK, LDB_DB_BUSY). Then all you need to do is set the database identifiers (filename, host:port etc.) in the looperdb configuration file instead of the SQLite filenames.

Is there an automation system?
------------------------------

Not yet. Use cron for now. The housekeeping database will be used to implement automated event escelation, correlation etc.


- fin - 
/foobar <foobar@muthanna.com>