ipfc-developer — Developer list for the IPFC framework (from wrapper to db-backend)

[Ipfc-developer] Documentation TODO : RFC, Install doc

[Alexandre D. <al...@co...>]

Hello,

The documentation part of IPFC seems to be very important (as we see in 
some email ;-). Here is a small TODO for the documentation : 

INSTALL

    * INSTALL-DBBACKEND
      Description: How to install a dbbackend
    * INSTALL-DRSERVER
      Description: How to install a dr-server.
    * INSTALL-WRAPPER
      Description: Guide for installing IPFC wrapper on various platform

RFC

    * DRAFT-IPFC-SECURE-LOGGING
      Description: IPFC Secure logging
    * DRAFT-IPFC-DTD
      Description: IPFC Document Type Definition
    * DRAFT-IPFC-EXCHANGE-PROTOCOL-HTTP
      Description: IPFC Exchange protocol using HTTP

You can directly edit/create/update these documents on the sourceforge 
site. https://sourceforge.net/docman/index.php?group_id=19759

If there are some people who want to care of the documentation part, don't 
hesitate. 

I hope we can have a complete documentation by the end of may. (ok this 
looks like a deadline ;-))

thanks.

alx



-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

Re: [Ipfc-developer] AIX errpt wrapper

[Alexandre D. <adu...@ma...>]

On Mon, 18 Feb 2002, Vincent JAMART wrote:

> After cooking my brain with some ideas, I guess I have a ~design to implement the IBM AIX error reporting on IPFC. You have two ways: you can redirect errpt to standard syslog, but you lost lots of usefull information or send all syslog events to errpt. I prefear the second option, but It's not an issue in the wrapper developement itself and more a sysadmin decision.
> This is how to do it:
> #/etc/syslog.conf
> *.debug		errlog
> 
> stupid isn't it ;-)
> 
> So, for design itself, I'm writing a perl script -based on the errreporter one- which extracts errpt -a info (full description) to a flat file each time an event occurs (some kind of daemon started with inittab, with a config file, selecting events to grab and destination file) and format it, removing unusefull fields. This file will be read by the wrapcol script and then sent to DR-server. 

OK. I have just one question : How can get from the errpt the number of 
returned logs ? It's quite important for the wrapper that runs into a loop 
waiting for data to have number of logging returned. (to not push same 
logging...)

So your daemon structure will be like that (for the collector part) : 

loop (until SIG-STOP/SIG-HUP)

     read new errpt 
     	after a certain time/amount 
	generate the IPFC message (that the net part will get from the out 
	directory)
    
Another easy solution it's to pipe the *.debug.

*.debug		|/opt/ipfc/dev/debugsyslogpipe

What is your opinion about that ? Have you other idea of implementation ? 
We can make an easy prototype from one of your AIX machine. (with the 
testing dr-server) I can give you account for your AIX maachine.  

Thanks a lot. 

Alx

 
 > 
> 

-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] AIX errpt wrapper

[Vincent J. <vin...@be...>]

After cooking my brain with some ideas, I guess I have a ~design to implement the IBM AIX error reporting on IPFC. You have two ways: you can redirect errpt to standard syslog, but you lost lots of usefull information or send all syslog events to errpt. I prefear the second option, but It's not an issue in the wrapper developement itself and more a sysadmin decision.
This is how to do it:
#/etc/syslog.conf
*.debug		errlog

stupid isn't it ;-)

So, for design itself, I'm writing a perl script -based on the errreporter one- which extracts errpt -a info (full description) to a flat file each time an event occurs (some kind of daemon started with inittab, with a config file, selecting events to grab and destination file) and format it, removing unusefull fields. This file will be read by the wrapcol script and then sent to DR-server. 


-- 
Vincent Jamart 
UNIX systems engineer
phone: +32-495/21.22.58
mail: vin...@be...
------------------------------
"My sister opened a computer store in Hawai.
She sells C shells by the seashore."

Re: [Ipfc-developer] Proposal : new in-memory representation of attributes

[Tycho F. <tyc...@co...>]

On Sun, 17 Feb 2002, Tycho Fruru wrote:

> > $attr; 				
> > 		is a hashref
> > $attr->{'key'}->[index] = value;  	
> > 		set the <index> occurance of attribute <key>
> > 		(text,int,int8,boolean,timestamp,null)
> > $attr->{'key'}->[index1]->{'key2'}->[index2] = value;
> > 		set a hash attribute (value is always text for now)
> 
> oops i forgot : 
> 
> if you're sure that you're only going to have one value for an attribute 
> (no multi-value) then you could omit the ->[index] part
> 
> When getting the data from the database, there will be an option on 
> whether you want to include the index for single valued attributes or not.
> 

the feedback I got (you know who you are) indicates that this is OK.
if not, speak up and thou shalt be heard ;-)

> > One open (and difficult !) point is the query facility.  Any ideas on how
> > to formulate the queries you want to perform on the database so that they
> > can be transformed to the database structure without having to worry about
> > the DB implementation details... ?
> > 
> > perhaps a simple SQL-like syntax would be cool (advantage is that's it's 
> > easy to use and already intuitive for db programmers)

but not for automatic query builders :-(

so my new suggestion is :

$query =
{
	op => 'select',
	select => [attr_1,attr_2,attr_3, ... ]
	where  => { 
			op => 'AND'
			t1 => {
				op => '=',
				t1 => { op => 'attribute',
					name => 'attr_4'
				      }
				t2 => { op => 'constant',
					value => '123'
				      }
			      }
			t2 => {
				op => '!=',
				t1 => { op => 'attribute',
					name => 'attr_5'
				      }
				t2 => { op => 'function',
					name => 'lower',
					parameters => [ { op => 'attribute',
						  	  t1 => 'attr_6'
							}
					      	      ]
				      }
			      }
		},
	order => [{ expression => {  op => 'attribute',name => 'attr_4' }, 
		    direction => 'asc'
		  } ],
	having => ... (expression) ...
	group => ... (expression) ...
	limit => 50,
	offset => 123
}

what do you think about the general idea ?
(of course, someone is always free to write a SQL to intermediary 
representation compiler ;-)

T.

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

Re: [Ipfc-developer] Proposal : new in-memory representation of attributes

[Tycho F. <tyc...@co...>]

On Sun, 17 Feb 2002, Tycho Fruru wrote:

> Hi,
> 
> a long-term goal of IPFC is (for me ;-) to have the in-memory 
> representation of attributes to be independent of the database 
> representation.  So this is why I suggest a possible (memory) 
> representation here.  I tried to have the following points covered :
> 
> - intuitive in the "simple" case (one attribute which is a normal value)
> - flexible to multiple occurences of the same attribute
> - supports attributes which are in themselves hashes
> 
> $attr; 				
> 		is a hashref
> $attr->{'key'}->[index] = value;  	
> 		set the <index> occurance of attribute <key>
> 		(text,int,int8,boolean,timestamp,null)
> $attr->{'key'}->[index1]->{'key2'}->[index2] = value;
> 		set a hash attribute (value is always text for now)

oops i forgot : 

if you're sure that you're only going to have one value for an attribute 
(no multi-value) then you could omit the ->[index] part

When getting the data from the database, there will be an option on 
whether you want to include the index for single valued attributes or not.


> 
> I'm writing routines which allow getting this data structure from the 
> database (handy to make the frondends independent of the db ...) and 
> getting it stored efficiently in the database with pipelining and all.
> 
> One open (and difficult !) point is the query facility.  Any ideas on how
> to formulate the queries you want to perform on the database so that they
> can be transformed to the database structure without having to worry about
> the DB implementation details... ?
> 
> perhaps a simple SQL-like syntax would be cool (advantage is that's it's 
> easy to use and already intuitive for db programmers)
> 
> what kinds of queries must be included on the events ? now is the time to
> speak up or you'll have to code it yourself :-)
> 
> Will the following query family fit the bill ?
> 
> * select [distinct] [count] e1.columns, e2.columns, ... 
>   from event as e1, event as e2, ...
>   where (simple conditions on attributes of e1,e2,...)
>   [order by (columns of e1,e2,...)]
>   [group by (columns of e1,e2,...)]
>   [having (simple conditions on columns of e1,e2,...)]
>   [limit number]
>   [offset number]
> 
> simple conditions mean : AND, OR, NOT, =, >, <, >=, <=, !=, LIKE, 
> 			 IN with a constant list
> simple conditions do NOT include : IN with a select subquery
> 				   EXISTS with a select subquery
> 
> I could (fairly) simply write a statement translator for this syntax ...
> 
> Feedback welcome, as usual !
> Tycho
> 
> 

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] Proposal : new in-memory representation of attributes

[Tycho F. <tyc...@co...>]

Hi,

a long-term goal of IPFC is (for me ;-) to have the in-memory 
representation of attributes to be independent of the database 
representation.  So this is why I suggest a possible (memory) 
representation here.  I tried to have the following points covered :

- intuitive in the "simple" case (one attribute which is a normal value)
- flexible to multiple occurences of the same attribute
- supports attributes which are in themselves hashes

$attr; 				
		is a hashref
$attr->{'key'}->[index] = value;  	
		set the <index> occurance of attribute <key>
		(text,int,int8,boolean,timestamp,null)
$attr->{'key'}->[index1]->{'key2'}->[index2] = value;
		set a hash attribute (value is always text for now)

I'm writing routines which allow getting this data structure from the 
database (handy to make the frondends independent of the db ...) and 
getting it stored efficiently in the database with pipelining and all.

One open (and difficult !) point is the query facility.  Any ideas on how
to formulate the queries you want to perform on the database so that they
can be transformed to the database structure without having to worry about
the DB implementation details... ?

perhaps a simple SQL-like syntax would be cool (advantage is that's it's 
easy to use and already intuitive for db programmers)

what kinds of queries must be included on the events ? now is the time to
speak up or you'll have to code it yourself :-)

Will the following query family fit the bill ?

* select [distinct] [count] e1.columns, e2.columns, ... 
  from event as e1, event as e2, ...
  where (simple conditions on attributes of e1,e2,...)
  [order by (columns of e1,e2,...)]
  [group by (columns of e1,e2,...)]
  [having (simple conditions on columns of e1,e2,...)]
  [limit number]
  [offset number]

simple conditions mean : AND, OR, NOT, =, >, <, >=, <=, !=, LIKE, 
			 IN with a constant list
simple conditions do NOT include : IN with a select subquery
				   EXISTS with a select subquery

I could (fairly) simply write a statement translator for this syntax ...

Feedback welcome, as usual !
Tycho

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] Graphical overview of the new db schema

[Alexandre D. <al...@co...>]

Dear All,

After the wonderful work of Tycho for the new db. Here is a  
graphical overview in EPS : 

http://www.conostix.com/ipfc/dist/db-example/overview/ipfc-db.eps
http://www.conostix.com/ipfc/dist/db-example/overview/ipfc-schema-version-1.0.4.html 
(in HTML)

The db will be available in the next release 1.0.4 (in 1 or 2 weeks). 
You can make a check out of the CVS if you want check this now (but it's 
cvs code ;-))

Don't hesitate to give feedback. 

Alex



-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] WIN32 event proposed ipfc data format

[Alexandre D. <al...@co...>]

Dear All,

Here is a example of a xml proposed format for the transport for WIN32 
Event : 

<?xml version='1' standalone='yes'?>
<ipfc version="1" type="events">
 <data version="1" type="events" subtype="win32eventlogline" transport="null">
  <win32eventlogline Computer="GERARD" Source="" EventType="" Description="" Category="" EventID="" User="">
   <message format="base64"></message>
   <strings format="base64"></strings>
  </win32eventlogline>
  </data>
 <agent date="2002-2-14 12:11:00" id="666" wrapperid="0" generationid="2002-2-14 12:11:0" sequenceid="1" transacid="2002021412110012480" />
</ipfc>
<ipfc-signature type="none" keyid="none" format="ascii"></ipfc-signature>

I have tried to include some exotic issue with Event (like the auditing 
part with some additional strings and so on...). 

If you have any comment, issue or a real WIN32 event log who doesn't fit 
this XML format, don't hesitate to give feedback. 

Thanks. 

Adulau

-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

Re: [Ipfc-developer] important: Big changes coming up

[Tycho F. <tyc...@co...>]

On Wed, 6 Feb 2002, Tycho Fruru wrote:

> Hello,
> 
> just a message to inform you that big changes are coming up !

I've tagged the latest current version (using old db) 
LAST_VERSION_WITH_OLD_DB in the CVS

Don't expect anything (db-backend related) to work until you see a 
positive confirmation from me that it works ;-)

For testing, please use the LAST_VERSION_WITH_OLD_DB or RELEASE_1_0_3
(except for the very brave)

I'll keep you posted
Tycho

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] important: Big changes coming up

[Tycho F. <tyc...@co...>]

Hello,

just a message to inform you that big changes are coming up !

First, what will remain the same :

- XML message passing format
- DR-server
- Wrappers
- db-wrapnet

so, what will change :

- db-backend schema
- db-backend-daemon
- alerter
- correlator
- frontend

You see ... lots of changes ... but what are they exactly ?  Basically we 
throw the database upside down and sort out the pieces ;-)

Situation now :

	Every event has its own record, containing all its data (messages, 
timestamps etc.). Every new type of event uses a new type of table in the 
database.  These tables form a hierarchical structure, enabling easy 
correlation on certain attributes (such as src IP address).

Problems now :

	Logs often contain the same messages over and over again (eg. "GET 
/ HTTP/1.0").  Right now we store each and every copy of these messages, 
resulting in an appreciable waste of space on the database.

	Correlation on "unexpected" types of data (such as url's for IDS, 
firewalls and webserver) is not so easy.

	Some logging formats could use a more expandable approach, in 
which some elements are not logged in some cases (eg. tcp syn flag status 
for udp packets).  Creating multiple types of logs is going to be 
difficult, so the use of "optional" attributes would be ... well, cool.

	Every new log/event/alert type needs a change to the database 
schema.  This can become cumbersome (and dangerous).

SO :

We have been thinking a lot (Alex & myself) about alternatives.  Here is 
the (for now) best result we came up with :

- each event has some basic information, present in every event
	- event_id
	- logfile_id
	- time
	- agent_id
	- event_type_id (numeric index to the different event types)

- all other attribute names are stored in a table attributes together with 
an attribute id :
	
	create table attributes
	(
	  attr_id	serial,
	  attr_name	text
			  unique
	)
	
	insert into attributes (attr_name) values ("src_address");
	insert into attributes (attr_name) values ("url");
	...

- a further table which defines per log/event type (identified with its
numeric id) which attributes are mandatory, optional or not allowed

- some tables (one per attribute storage type (int,int8,inet,...)

	create table attr_int
	(
	  event_id	int8
			  references events(event_id),
	  attr_id	int
			  references attributes(attr_id),
	  attr_value	int -- the actual value of the attribute
	)

- for text attributes it is still different :

	create table texts   -- a table to cache text entries
	(
	  text_id	serial8,
	  text_value	text
	)

	create table attr_text
	(
	  event_id	int8
			  references events(event_id),
	  attr_id	int
			  references attributes(attr_id),
	  attr_value	int8
			  references texts(text_id)
	)

	This setup allows us to re-use text messages from previous entries 
in order to save space.  Since there are no updates on these tables, only 
inserts and selects, this is not a problem. 

As always, please do not hesitate to send your remarks, comments or
suggestions !  Currently we are working like monkeys to get this stuff in,
and there's still a lot more to do so if anyone feels like it, just raise
your hand ok ;-)

Cheers,
Tycho

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] postgresql 7.2

[Tycho F. <tyc...@co...>]

Whoa,

finally postgresql 7.2 is ready, quickly go get it since you'll need it 
for ipfc ...

Cheerio
Tycho

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] WIN32 event

[Alexandre D. <al...@co...>]

Dear All,

We are making some test around the WIN32 events to create xml from events 
in ipfc format. 

Events are available :

Windows NT has 3 types of logs:

Application Log - Contains events reported by various applications 
installed on the Windows NT server. These can be Microsoft or 3-rd party 
applications. 

File: %SystemRoot%\System32\Config\AppEvent.evt

Security Log - Contains all the auditing and security events. 

File: %SystemRoot%\System32\Config\SecEvent.evt

System Log - Contains events reported by Windows NT system components 
(processes, kernel, drivers.

File: %SystemRoot%\System32\Config\SysEvent.evt

 

Windows 2000 Servers configured with Active Directory or just DNS has 3 
additional logs:

Directory Service - Contains events reported by Active Directory

File: %SystemRoot%\System32\Config\Director.evt

DNS Server - Contains events reported by Microsoft Windows 2000 DNS 
Server.

File: %SystemRoot%\System32\Config\DNSEvent.evt

File Replication Service - Contains events reported by Microsoft FRS 
Service.

File: %SystemRoot%\System32\Config\NTFrs.evt

Note: Win2K Professional cannot read any of the DNS/FRS/DS logs, unless 
the Admin pack is installed.

 

NT/2000 Event logs contain 5 types of events:

Information - An event that describes the successful operation of an 
application, driver, or service. For example, when a network driver loads 
successfully, an Information event will be
logged.

Error - A significant problem, such as loss of data or loss of 
functionality. For example, if a service fails to load during startup, an 
error will be logged.

Warning - An event that is not necessarily significant, but may indicate a 
possible future problem. For example, when disk space is low, a warning 
will be logged.

Success Audit - An audited security access attempt that succeeds. For 
example, a user's successful attempt to log on to the system will be 
logged as a Success Audit event.

Failure Audit - An audited security access attempt that fails. For 
example, if a user tries to access a network drive and fails, the attempt 
will be logged as a Failure Audit event

Event format : (if I understand everything)

Log type  : Application, Security or System
Source    : Program or component (for example mapi.dll)
Date/Time : Explicit
User      : Username when the events occured (system or administrator for 
            example)
Computer  : Hostname where the event occured
EventID   : An ID (check out eventid.net ;-)
Type      : The type could be Error, Warning, Information, Success Audit, 
	    or Failure Audit. Other things possible ?
Category  : A category seems to be used only for security auditing.
Message   : The messsage itself

The data format itself is like that : (the .evt file)




%EVENTLOGRECORD (
                Length                  =>      NULL
                RecordNumber            =>      NULL
                TimeGenerated           =>      NULL
                TimeWritten             =>      NULL
                EventID                 =>      NULL
                EventType               =>      NULL
                Category                =>      NULL
                ClosingRecordNumber     =>      NULL
                Source                  =>      NULL
                Computer                =>      NULL
                Strings                 =>      NULL
                Data                    =>      NULL
);


 Length
                 Specifies the length, in bytes, of this event record.
 RecordNumber
                 Contains a record number that can be used with the 
EVENTLOG_SEEK_READ flag
                 passed in a call to the Win32::Read function to begin 
reading at a specified record. 
 TimeGenerated
                 The time at which this entry was submitted. This time is 
measured in the number of
                 seconds elapsed since 00:00:00 January 1, 1970. 
 TimeWritten
                 Specifies the time at which this entry was received by 
the service to be written to the
                 logfile. This time is measured in the number of seconds 
elapsed since 00:00:00 January 1,
                 1970. 
 EventID
                 Identifies the event. This is specific to the source that 
generated the event log entry, and
                 is used, together with SourceName, to identify a message 
in a message file that is
                 presented to the user while viewing the log.
 EventType
                 Specifies the type of event. 
 Category
                 Specifies a subcategory for this event. This subcategory 
is source specific; the category
                 can have any value.
 Source
                 Contains the variable-length null-terminated string 
specifying the name of the source
                 (application, service, driver, subsystem) that generated 
the entry. This is the name used
                 to retrieve from the registry the name of the file 
containing the message strings for this
                 source. It is used, together with the event identifier, 
to get the message string that
                 describes this event. 
 Computer
                 Contains the variable-length null-terminated string 
specifying the name of the computer
                 that generated this event.
 Strings 
                 The event identifier together with SourceName and a 
language identifier identify a
                 message string that describes the event in more detail. 
The message strings are
                 contained in a message file specified in the source entry 
in the registry. 
 Data
                 The binary information is information that is specific to 
the event. It could be the
                 contents of the processor registers when a device driver 
got an error, a dump of an
                 invalid packet that was received from the network, a dump 
of all the structures in a
                 program (when the data area was detected to be corrupt), 
and so on.



                                         EventType
                          Value
                                                          Id
                                                                          
Meaning
  EVENTLOG_ERROR_TYPE
                                                          0
                                                               Error event
  EVENTLOG_WARNING_TYPE
                                                          2
                                                               Warning 
event
  EVENTLOG_INFORMATION_TYPE 
                                                          4
                                                               Information 
event
  EVENTLOG_AUDIT_SUCCESS
                                                          8
                                                               Success 
Audit event 
  EVENTLOG_

Now the question is what information do we need to get from event ? 
What is the best format for the ipfc xml ? line-by-line ? or full xml 
with all tag ? 

If you any comment idea...

alx
 





-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] IPFC release 1.0.3

[Alexandre D. <al...@co...>]

* Added initial support for security advisory 
* Improved frontend 
* New logging format 
* Added alerter (basic functionality for now) 
* Better polling in db-backend, db-wrapnet and alerter 
* Misc. fixes

tagged RELEASE_1_0_3


-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] Re: alerting

[Tycho F. <tyc...@co...>]

On Mon, 21 Jan 2002, Alexandre Dulaunoy wrote:

>
> On Mon, 21 Jan 2002, Tycho Fruru wrote:
>
> > Hello,
> >
> > I've been busy with the alerting part of ipfc.  To correctly implement
> > this we need to know exactly which events have not (yet) been processed.
> >
> > My first try was to have a table with <begin> <end> pairs, the existence
> > of a tuple means that all events between <begin> and <end> have been
> > processed (for alerts).
> >
> > Unfortunately this becomes *really slow* because we always need to scan
> > the whole " events " table and compare it with the begin and end tuples.
> >
> > So my next try was with triggers.  This seems to work better but there is
> > 1 problem (only one ;-) : now all alerts must be processed centrally, so
> > i'm looking for a simple alert dispatcher.  does someone already have such
> > a beast ?
>
> The Log::Dispatch module permits to send log/alert to multiple object.
>
> http://www.enstimac.fr/Perl/perl5.6.1/site_perl/5.6.1/Log/Dispatch.html
>
> Is it useful for you ?

Let me describe the control flow in more detail :

event is put in db-backend
-> trigger creates entry (reference to event_id) in " to-be-processed"
table

a daemon regularly looks into "to-be-processed" table, whenit finds a new
event_id :
	- do any processing that might be necessary ( -> may depend on eg.
syslog message, snort alert type etc )
	- queue the right alerts (in an alert-to-be-processed table ;-)
	- remove the event from the to-be-processed table

a second daemon looks into the "alert-to-be-processed" table
	- dispatches alerts
	- removes entry from table when alert successfully sent


I can merge the two daemons, but then ther will be less resilience in case
the alert was NOT successfully sent

What do you think ?

Tycho
-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] Re: alerting

[Alexandre D. <al...@co...>]

On Mon, 21 Jan 2002, Tycho Fruru wrote:

> Hello,
>
> I've been busy with the alerting part of ipfc.  To correctly implement
> this we need to know exactly which events have not (yet) been processed.
>
> My first try was to have a table with <begin> <end> pairs, the existence
> of a tuple means that all events between <begin> and <end> have been
> processed (for alerts).
>
> Unfortunately this becomes *really slow* because we always need to scan
> the whole " events " table and compare it with the begin and end tuples.
>
> So my next try was with triggers.  This seems to work better but there is
> 1 problem (only one ;-) : now all alerts must be processed centrally, so
> i'm looking for a simple alert dispatcher.  does someone already have such
> a beast ?

The Log::Dispatch module permits to send log/alert to multiple object.

http://www.enstimac.fr/Perl/perl5.6.1/site_perl/5.6.1/Log/Dispatch.html

Is it useful for you ?

alx


-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] alerting

[Tycho F. <tyc...@co...>]

Hello,

I've been busy with the alerting part of ipfc.  To correctly implement
this we need to know exactly which events have not (yet) been processed.

My first try was to have a table with <begin> <end> pairs, the existence
of a tuple means that all events between <begin> and <end> have been
processed (for alerts).

Unfortunately this becomes *really slow* because we always need to scan
the whole " events " table and compare it with the begin and end tuples.

So my next try was with triggers.  This seems to work better but there is
1 problem (only one ;-) : now all alerts must be processed centrally, so
i'm looking for a simple alert dispatcher.  does someone already have such
a beast ?

Thx
tycho

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] IPFC release 1.0.2

[Alexandre D. <al...@co...>]

A new release of IPFC (1.0.2 cvs tag : RELEASE_1_0_2) is available.

     This new release includes :

        * Added support for HMAC-SHA1 XML signatures
        * Added mon wrapper
        * general clean-ups and bugfixes

Now, you can sign with HMAC-SHA1  every message in the framework.
(from events (syslog, apachelog,...) to policy)

http://www.conostix.com/ipfc/
http://sourceforge.net/projects/ipfc/

Don't hesitate to give feedback.

See you soon.

alx




-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

Re: [Ipfc-developer] Oracle & AIX IPFC wrappers

[Tycho F. <tyc...@co...>]

On Fri, 11 Jan 2002, Vincent JAMART wrote:

> AIX
> ---
>
> Just take everything from the error report (errpt) (and syslog if
> enabled) alerts (H/S, T/P) with awk/perl/ksh and ODM-related commands
> to DB-backend as XML-formated & signed data through IPFC wrapper.
> Seems simple to do, using a-syslog-alike wrapper base.
I guess this can be done using either syslog-lines or log-lines transport.
Do any of your logs span multiple lines ?

> ORACLE
> ------
>
> | ORACLE DB&INSTANCE |      |   IPFC WRAPPER step1:filter    |     | IPFC WRAPPER step2 |
> ----------------------      |--------------------------------|=====|--------------------|
> |                    |      |                                |     |                    |
> | <DB: audit tables>-|---<--|-[perl/SQL querry on sys.aud$ ]-|-->--|--XML + signature---|
> | INSTANCE ALERT FILE|---<--|-awk/perl/ksh code--------------|-->--|--XML + signature---|->---DB-backend
> | INSTANCETrace files|---<--|-awk/perl/ksh code--------------|-->--|--XML + signature---|
> |____________________|      |--------------------------------|     |--------------------|
>
> Hum, short explanation and headache:
>
> Oracle logs errors to Alert file (ORA-xxxx + description), which is an
> ascii file. After analysis of this file, we can decide which errors
> need to be reported to ipfc. It also make trace of actions to trace
> file(s). Same as above to decide what to log & method.
Normally (but this depends on the amount of traffic that would be
generated) the idea behind ipfc is that the wrapper implements the minimum
amount of policy (eg. what to send, what priority to give it etc). Perhaps
this is not possible in this case, but then we need to foresee a (possibly
remote) configuration facility for the wrapper.

> Third, and more interresting one, the Auditing. Oracle, after running
> an internal SQL contained in rdbms options of $ORACLE_HOME can log
> lots of audits to the data dictionnary table of the DB, named
> sys.aud$. You can decide to audit all/some changes of a user, changes
> on a table and on a certain value inserted or failed to insert,
> ...(...) so lots of things..EVERYTHING. This configuration has to be
> done by the DBA with the analysts of the application in order to
> determine sensitive tables/data/users to audit. as SYSTEM user or
> SYSDBA, we can run an SQL pl/sql script of our own on that particular
> table to retrieve information and spool it to an ascii file, which
> will be XML-formated & signed by IPFC wrapper before sending to
> DB-backend.
OK.  It's even very easy to do if the file contains one log-entry per line
(see log-lines transport)

> There are few points I wanna focus & having your optinon on before going any further:
>
> -Security about pl/sql script: First line is "connect / as sysdba" or
> "connect system/manager" in 90% of cases. Even with chmod to 600 for
> oracle user, is it enough to include it 'as it' in IPFC??
tough one. It's better not to connect as sysdba or sys/manager if
possible.  Can't you just grant the necessary pl/sql (to dump the logs) to
a normal user ?

> -Do you think about another way to transport our view of Audit tables
> from Oracle to PostgreSQL (+XML/signature) than DB
> OBJECT->ASCII+changes->DB OBJECT again???
XML is not ASCII :-) If you really want you can serialize your DB OBJECT,
put it in the XML file then reconstitute at the other side.

> -Am I crazy/dumb/void with these propositions ? ;-)
Not at all.

Cheers,
Tycho

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

Re: [Ipfc-developer] Oracle & AIX IPFC wrappers

[Alexandre D. <al...@co...>]

On Fri, 11 Jan 2002, Vincent JAMART wrote:

> Hello ${all}
>
> I made the schema for Oracle wrapper in IPFC. It's stil draft and lots of code to generate but it's more clear in my brain.
>
> AIX
> ---
>
> Just take everything from the error report (errpt) (and syslog if enabled) alerts (H/S, T/P) with awk/perl/ksh and ODM-related commands to DB-backend as XML-formated & signed data through IPFC wrapper. Seems simple to do, using a-syslog-alike wrapper base.

Yes, the only thing to do is to describe that correctly in the XML format.
(transport null maybe ? or an aix-errpt transport ?)


>
>
> ORACLE
> ------
>
> | ORACLE DB&INSTANCE |      |   IPFC WRAPPER step1:filter    |     | IPFC WRAPPER step2 |
> ----------------------      |--------------------------------|=====|--------------------|
> |                    |      |                                |     |                    |
> | <DB: audit tables>-|---<--|-[perl/SQL querry on sys.aud$ ]-|-->--|--XML + signature---|
> | INSTANCE ALERT FILE|---<--|-awk/perl/ksh code--------------|-->--|--XML + signature---|->---DB-backend
> | INSTANCETrace files|---<--|-awk/perl/ksh code--------------|-->--|--XML + signature---|
> |____________________|      |--------------------------------|     |--------------------|
>
> Hum, short explanation and headache:
>
> Oracle logs errors to Alert file (ORA-xxxx + description), which is an ascii file. After analysis of this file, we can decide which errors need to be reported to ipfc.
> It also make trace of actions to trace file(s). Same as above to decide what to log & method.

Is there a lot of alerts files in a Oracle production system ?
We can integrate all alert into the IPFC framework. Depending of the
amount of alerts traffic.


>
> Third, and more interresting one, the Auditing. Oracle, after running an internal SQL contained in rdbms options of $ORACLE_HOME can log lots of audits to the data dictionnary table of the DB, named sys.aud$. You can decide to audit all/some changes of a user, changes on a table and on a certain value inserted or failed to insert, ...(...) so lots of things..EVERYTHING. This configuration has to be done by the DBA with the analysts of the application in order to determine sensitive tables/data/users to audit. as SYSTEM user or SYSDBA, we can run an SQL pl/sql script of our own on that particular table to retrieve information and spool it to an ascii file, which will be XML-formated & signed by IPFC wrapper before sending to DB-backend.
>
> There are few points I wanna focus & having your optinon on before going any further:
>
> -Security about pl/sql script: First line is "connect / as sysdba" or "connect system/manager" in 90% of cases. Even with chmod to 600 for oracle user, is it enough to include it 'as it' in IPFC??

Yes, this could be an issue for security. But is there a way to create a
special use for the wrapper that can get access to the necessary table.

>
> -Do you think about another way to transport our view of Audit tables from Oracle to PostgreSQL (+XML/signature) than DB OBJECT->ASCII+changes->DB OBJECT again???

You mean exporting SQL and encapsulate into the XML messaging for
integrating into the db-backend directly ? the IPFC db-schema is another
schema... I don't think this is useful but ...

 >
> -Am I crazy/dumb/void with these propositions ? ;-)

No, not at all. It's really interresting to have Oracle and AIX error
report into the IPFC framework.

Do you have any log example ?

Thanks a lot.

alx

-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] Oracle & AIX IPFC wrappers

[Vincent J. <vin...@li...>]

Hello ${all}

I made the schema for Oracle wrapper in IPFC. It's stil draft and lots of code to generate but it's more clear in my brain.

AIX
---

Just take everything from the error report (errpt) (and syslog if enabled) alerts (H/S, T/P) with awk/perl/ksh and ODM-related commands to DB-backend as XML-formated & signed data through IPFC wrapper. Seems simple to do, using a-syslog-alike wrapper base.


ORACLE
------

| ORACLE DB&INSTANCE |      |   IPFC WRAPPER step1:filter    |     | IPFC WRAPPER step2 |
----------------------      |--------------------------------|=====|--------------------|
|                    |      |                                |     |                    |
| <DB: audit tables>-|---<--|-[perl/SQL querry on sys.aud$ ]-|-->--|--XML + signature---|
| INSTANCE ALERT FILE|---<--|-awk/perl/ksh code--------------|-->--|--XML + signature---|->---DB-backend
| INSTANCETrace files|---<--|-awk/perl/ksh code--------------|-->--|--XML + signature---|
|____________________|      |--------------------------------|     |--------------------|

Hum, short explanation and headache:

Oracle logs errors to Alert file (ORA-xxxx + description), which is an ascii file. After analysis of this file, we can decide which errors need to be reported to ipfc.
It also make trace of actions to trace file(s). Same as above to decide what to log & method.
 
Third, and more interresting one, the Auditing. Oracle, after running an internal SQL contained in rdbms options of $ORACLE_HOME can log lots of audits to the data dictionnary table of the DB, named sys.aud$. You can decide to audit all/some changes of a user, changes on a table and on a certain value inserted or failed to insert, ...(...) so lots of things..EVERYTHING. This configuration has to be done by the DBA with the analysts of the application in order to determine sensitive tables/data/users to audit. as SYSTEM user or SYSDBA, we can run an SQL pl/sql script of our own on that particular table to retrieve information and spool it to an ascii file, which will be XML-formated & signed by IPFC wrapper before sending to DB-backend.

There are few points I wanna focus & having your optinon on before going any further:

-Security about pl/sql script: First line is "connect / as sysdba" or "connect system/manager" in 90% of cases. Even with chmod to 600 for oracle user, is it enough to include it 'as it' in IPFC??

-Do you think about another way to transport our view of Audit tables from Oracle to PostgreSQL (+XML/signature) than DB OBJECT->ASCII+changes->DB OBJECT again???

-Am I crazy/dumb/void with these propositions ? ;-)

-- 
Vincent Jamart 
UNIX systems engineer
phone: +32-495/21.22.58
mail: vin...@be...
------------------------------
"My sister opened a computer store in Hawai.
She sells C shells by the seashore."

Re: [Ipfc-developer] XML signatures

[Alexandre D. <al...@co...>]

For me, this seems good for the beginning. We can easy extend
the <ipfc-signature> to other things like encryption, extended middleware,
...

The only issue that is not an issue 8-) It's the multiple root node in the
XML document but the xml file should be taken as a stream.
As described in the XML 1.0 standard, the document starts with a root
entity or a document entity.

THis could be solved (for some validator parser) with adding a document
entity in the stream. But it's not really an issue for the moment because
we have a tight control of the framework.

For me, we can with that.

Is there any other comments ?


alx

 On Fri, 4 Jan 2002, Tycho Fruru
wrote:

> Dear all,
>
>
> for better authentication between db-backend and agents (both for logging
> as well as for pushing policies etc) it would be good to be able to sign
> the XML files.
>
> There is an XML signature standard which is unfortunately not easy to
> implement.  (http://www.w3.org/TR/xmldsig-core/)
>
> Good and free libraries to implement these specifications are not yet
> widely available.
>
> I propose to change the XML format (again) so that we have in each signed
> XML file the following :
>
> <ipfc version="1" type="status">
>   <data version="1" type="status" subtype="availability" transport="null">
>     <retrytime>60</retrytime>
>   </data>
>   <agent date="2001-12-17 17:56:25" id="3" generationid="2001-12-7 16:15:30" sequenceid="15807" transacid="2001121717562502246"/>
> </ipfc>
> <ipfc-signature type="HMAC-SHA1" keyid="testkey" format="base64">adFrfDwdFGd==</ipfc-signature>
>
>
> This means that the generated XML-files are not in themselves parseable
> (there are multiple root nodes) but they can easily be cut in the right
> pieces)
>
> the keyid is a freeform text identifier.
>
> Keyid, agent_id and wrapper_id give us a reference to the right key to
> use.
>
> As usual, let me hear what you think or it will be implemented like this
> ;-)
>
> Best regards,
> Tycho
>
>

-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] XML signatures

[Tycho F. <tyc...@co...>]

Dear all,


for better authentication between db-backend and agents (both for logging
as well as for pushing policies etc) it would be good to be able to sign
the XML files.

There is an XML signature standard which is unfortunately not easy to
implement.  (http://www.w3.org/TR/xmldsig-core/)

Good and free libraries to implement these specifications are not yet
widely available.

I propose to change the XML format (again) so that we have in each signed
XML file the following :

<ipfc version="1" type="status">
  <data version="1" type="status" subtype="availability" transport="null">
    <retrytime>60</retrytime>
  </data>
  <agent date="2001-12-17 17:56:25" id="3" generationid="2001-12-7 16:15:30" sequenceid="15807" transacid="2001121717562502246"/>
</ipfc>
<ipfc-signature type="HMAC-SHA1" keyid="testkey" format="base64">adFrfDwdFGd==</ipfc-signature>


This means that the generated XML-files are not in themselves parseable
(there are multiple root nodes) but they can easily be cut in the right
pieces)

the keyid is a freeform text identifier.

Keyid, agent_id and wrapper_id give us a reference to the right key to
use.

As usual, let me hear what you think or it will be implemented like this
;-)

Best regards,
Tycho

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] mon wrapper (ipfc.alert)

[Alexandre D. <al...@co...>]

I have added a simple mon wrapper (an alert script) that sends mon
alerting into IPFC.

There is a simple a extension of the xml format (in status).

You can check it out in the CVS :

./ipfc/src/wrapper/mon (the alert script)
./ipfc/docs/protocol/xml-examples/mon-alert.xml

If you have any idea to extend it or for other systems, don't hesitate.

With this example, you can see how it's easy to create new wrapper with
new data to gather.

Good wishes to everybody.

alx


-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] IPFC release 1.0.1

[Alexandre D. <al...@co...>]

A new release of IPFC (1.0.1 cvs tag : RELEASE_1_0_1) is available.

This new release includes :

	* changed XML format (which is more space efficient now)
	* added the concept of "transports"
	* changed db-backend and libipfc to support new format
	* general clean-ups and robustness improvements

url :   http://www.conostix.com/ipfc/
	http://www.sf.net/projects/ipfc/

Thanks.

The official announce :

Luxembourg, December 27, 2001
Conostix S.A. today announced the availability of IPFC (Inter Protocol
Flexible Control)
v1.0.1 IPFC v1.0.1 is a flexible software framework to manage, monitor
and control multiple devices in complex networked environments.

IPFC addresses the interoperability of heterogeneous corporate security
infrastructures for
	- Comprehensive entreprise infrastructure reporting
        - Through centralized secure logging and data correlation
	- Easier security operations
	- Securely outsourcing part or the global security infrastructure

IPFC is platform and product neutral. IPFC can monitor and manage
multiple devices such as routers, firewalls, embedded devices, Unix
servers,
Windows servers, IDS, etc)

The framework is scalable and based on a multi-tier architecture.
This makes it possible to use IPFC for the whole range of security
infrastructures,
from the smallest to massively distributed and redundant management
infrastructures.
The data exchange protocol between
the different zones in the IPFC architecture is secure, robust, stable and
standard.

IPFC can easily be integrated in any existing (or new) infrastructure.
IPFC is easy customizable: new devices and data formats can
be incorporated in a very short time period.

IPFC software was developed by Conostix to fit their needs for MSS
(Managed Security Services). The software was built to create the first
product-neutral off-the-shelf security management tool.

The complete software and protocol are released under the GNU General
Public License.
This means that the software can easily be customized to specific needs
and
can be audited by any authority.

Conostix provides consulting and services around the IPFC framework.
(from custom implementation to MSS services)
For more information about IPFC : http://www.conostix.com/ipfc/


About Conostix

Conostix is a young computer security company based in Luxembourg.
Conostix is a security services and software company providing solutions
for complex and distributed networked infrastructures. Conostix provides
Managed Security Services and Managed Security Monitoring (MSS-MSM) in
Europe. Conostix has taken a fundamentally new approach in their research
and
developement by using a Free Software (Open Source) methodology.
Research and development focuses are : distributed framework for managing
networked security devices, trusted operating systems and dedicated
security devices.

Conostix is located at Technoport Schlassgoart, rue de Luxembourg 66,
L4221 Esch-Sur-Alzette (Grand Duchy of Luxembourg)
Phone : +35226103061 Fax : +35226103062 Website: http://www.conostix.com/
Press contact : pr...@co...

All names and trademarks are the property of their respective owners.



-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] new XML format and libipfc.pl

[Alexandre D. <al...@co...>]

Hello all,

I have added three new functions for supporting the new XML format. The
three new functions are IpfcGenMessage(), IpfcPutDrserver() and IpfcFlushQueue().
In a really near future, the old function related to the old format will
be removed.
The argument of the three the new function are fixed (I hope so 8-)).
There will be some change inside in the future (for supporting more
transport and so on...).

We hope to tag the cvs at the end of the week with RELEASE_1_0_1 and make
a release using version 1 of the XML format.

If you have any comment, ideas, ... don't hesitate.

alx




-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/