ipfc-developer — Developer list for the IPFC framework (from wrapper to db-backend)

Re: [Ipfc-developer] IPFC Transport types

[Tycho F. <tyc...@co...>]

After some more thinking and a nice coffee :

<?xml version='1' standalone='yes'?>
<ipfc version="1" type="events">
  <data version="1" type="log" subtype="common-log" transport="log-line">
    <log-line>192.168.30.103 - - [20/Nov/2001:19:58:07 +0100] &quot;GET / HTTP/1.0&quot; 200 2890</log-line>
    <log-line>192.168.30.103 - - [20/Nov/2001:19:58:07 +0100] &quot;GET /poweredby.png HTTP/1.0&quot; 200 1154</log-line>
    <log-line>192.168.30.103 - - [20/Nov/2001:19:58:07 +0100] &quot;GET /icons/apache_pb.gif HTTP/1.0&quot; 200 2326</log-line>
    <log-line>192.168.30.103 - - [20/Nov/2001:19:58:16 +0100] &quot;GET /frontend/sql-test.html HTTP/1.0&quot; 200 458"</log-line>
    <log-line>192.168.30.103 - - [20/Nov/2001:19:58:16 +0100] &quot;GET /frontend/style.css HTTP/1.0&quot; 200 1215"</log-line>
  </data>
  <agent date="2001-11-20 19:58:17" id="17" sequenceid="530" generationid="2001-11-20 12:12:24" transacid="2001112019581706113"/>
</ipfc>

Any remarks are welcome
Tycho

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

Re: [Ipfc-developer] IPFC Transport types

[Tycho F. <tyc...@co...>]

On Thu, 20 Dec 2001, Tycho Fruru wrote:

> Due to the limited number of people using ipfc right now ;-) I dare change
> the XML format without bumping the version number ...
I changed the XML format (again ;-) to be more consistent.  The agent tag
now contains sequenceid, generationid, version and transacid.

I'm also working on a more concise XML representation, here is already a
primer ;-)

<?xml version='1' standalone='yes'?>
<ipfc version="1" type="events">
  <data version="1" type="log" subtype="common-log" transport="log-line">
    <log-line content="192.168.30.103 - - [20/Nov/2001:19:58:07 +0100] &quot;GET / HTTP/1.0&quot; 200 2890"/>
    <log-line content="192.168.30.103 - - [20/Nov/2001:19:58:07 +0100] &quot;GET /poweredby.png HTTP/1.0&quot; 200 1154"/>
    <log-line content="192.168.30.103 - - [20/Nov/2001:19:58:07 +0100] &quot;GET /icons/apache_pb.gif HTTP/1.0&quot; 200 2326"/>
    <log-line content="192.168.30.103 - - [20/Nov/2001:19:58:16 +0100] &quot;GET /frontend/sql-test.html HTTP/1.0&quot; 200 458"/>
    <log-line content="192.168.30.103 - - [20/Nov/2001:19:58:16 +0100] &quot;GET /frontend/style.css HTTP/1.0&quot; 200 1215"/>
  </data>
  <agent date="2001-11-20 19:58:17" id="17" sequenceid="530" generationid="2001-11-20 12:12:24" transacid="2001112019581706113"/>
</ipfc>

Cheerio
Tycho

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

Re: [Ipfc-developer] IPFC Transport types

[Tycho F. <tyc...@co...>]

On Thu, 20 Dec 2001, Tycho Fruru wrote:

From now on, only this format will be accepatble to the db-backend.

Due to the limited number of people using ipfc right now ;-) I dare change
the XML format without bumping the version number ...

See CVS for updated stuff
Tycho

> The proposed XML format is as follows :
>
> <ipfc>
>   <data>
> <!-- (note the lack of <entry> here !) -->
>     <type>log</type>
>     <category>common-log</category>
>     <version>1</version>
>     <transport>syslog-line</transport>
> <!-- (note the lack of <content> here !) -->
>     <syslog-line>Dec 17 13:03:41 tournesol su(pam_unix)[1590]: brol</syslog-line>
>     <syslog-line>Dec 31 23:59:59 tournesol kernel: one second to go</syslog-line>
>   </data>
>   <data>
>     <type>status</type>
>     <category>availability</category>
>     <version>1</version>
>     <transport>null</transport>
>   </data>
>   <agent>
>     <date>2001-12-17 13:03:42</date>
>     <id>1</id>
>   </agent>
>   <generationid>2001-12-17 12:44:39</generationid>
>   <sequenceid>27</sequenceid>
>   <version>1</version>
>   <transacid>2001121713034206833</transacid>
>   <type>events</type>
> </ipfc>

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] IPFC Transport types

[Tycho F. <tyc...@co...>]

Hello,

regarding the problem of transport types, here are some more thoughts...

- transport types are orthogonal to the event type (you can have apache in
syslog for example)

- some parsing can be done on the transport-type level (eg. parsing of
syslog-line date information), some needs to be done on
agent_type/event_type/event_category level.  This lower-level parsing can
override higher-level entries (eg. for apache, we have better timing
information in the apache log than in the syslog headers ...)

The proposed XML format is as follows :

<ipfc>
  <data>
<!-- (note the lack of <entry> here !) -->
    <type>log</type>
    <category>common-log</category>
    <version>1</version>
    <transport>syslog-line</transport>
<!-- (note the lack of <content> here !) -->
    <syslog-line>Dec 17 13:03:41 tournesol su(pam_unix)[1590]: brol</syslog-line>
    <syslog-line>Dec 31 23:59:59 tournesol kernel: one second to go</syslog-line>
  </data>
  <data>
    <type>status</type>
    <category>availability</category>
    <version>1</version>
    <transport>null</transport>
  </data>
  <agent>
    <date>2001-12-17 13:03:42</date>
    <id>1</id>
  </agent>
  <generationid>2001-12-17 12:44:39</generationid>
  <sequenceid>27</sequenceid>
  <version>1</version>
  <transacid>2001121713034206833</transacid>
  <type>events</type>
</ipfc>


comments are of course more than welcome !

Tycho

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

Re: [Ipfc-developer] basic logging format for application

[Alexandre D. <al...@co...>]

On Thu, 20 Dec 2001, Tycho Fruru wrote:

> > - One line per log entry (with a new line at the end)
> > - Each <tag>=<value> couple ends by a signle comma (srcport=22,)
> > - String with white space are surrounded by double quotes.
>
> What about strings which are supposed to begin with " ? Are these also
> surrounded by double quotes ?
> What about strings which end in "," ?
> > - Binary are encoded in an ascii format (like Base64 and others)
> > - Binary is described like that (data={base64}VklTSU8gMjAwMCBURU====,)
> > - Embedded double quotes inside an event have '\' inserted before it.
> - Embedded { are escaped with \
> - Embedded \ are escaped with \
> - Embedded , are escaped with \
>
> > An example log entry :
> >
> > srcip=192.168.1.1,srcport=2345,dstip=192.168.2.2,dstport=22,debutpayload={base64}VklTSU8gMjAwMCBURU====,proto=6
>
> foo="This is a happy string with \", \\ and \,",bar=ends_in_\,,baz=\{base64}This_is_ascii!
>
> Im looking forward to your updated rules ;-)
>
> Tycho
>


Ok, the best way is to quote only the double quote inside.

test="\"\{",payload={base64}"JEF45ES===="

Specifying the type with {} only between = " if not specified ascii.

Now we have to find consistent tag name 8-)


alx


-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

Re: [Ipfc-developer] basic logging format for application

[Tycho F. <tyc...@co...>]

> - One line per log entry (with a new line at the end)
> - Each <tag>=<value> couple ends by a signle comma (srcport=22,)
> - String with white space are surrounded by double quotes.

What about strings which are supposed to begin with " ? Are these also
surrounded by double quotes ?
What about strings which end in "," ?

> - Binary are encoded in an ascii format (like Base64 and others)
> - Binary is described like that (data={base64}VklTSU8gMjAwMCBURU====,)
> - Embedded double quotes inside an event have '\' inserted before it.
- Embedded { are escaped with \
- Embedded \ are escaped with \
- Embedded , are escaped with \

> An example log entry :
>
> srcip=192.168.1.1,srcport=2345,dstip=192.168.2.2,dstport=22,debutpayload={base64}VklTSU8gMjAwMCBURU====,proto=6

foo="This is a happy string with \", \\ and \,",bar=ends_in_\,,baz=\{base64}This_is_ascii!

Im looking forward to your updated rules ;-)

Tycho
-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] basic logging format for application

[Alexandre D. <al...@co...>]

Hello all,

As we dig more and more with interfacing software into IPFC framework.
Sometimes, we have some issue with some "application" and its log format.

For example with pf (that comes with OpenBSD 3.0), the logging format is
done in libpcap format and we can read it with a tcpdump -e.

The problem is that the tcpdump output is human friendly but not really
parser friendly.

So to find a solution for that, we are thinking of a basic log format to
extent application logging into something that could be understood be the
parser.

An idea :


- One line per log entry (with a new line at the end)
- Each <tag>=<value> couple ends by a signle comma (srcport=22,)
- String with white space are surrounded by double quotes.
- Binary are encoded in an ascii format (like Base64 and others)
- Binary is described like that (data={base64}VklTSU8gMjAwMCBURU====,)
- Embedded double quotes inside an event have '\' inserted before it.

An example log entry :

srcip=192.168.1.1,srcport=2345,dstip=192.168.2.2,dstport=22,debutpayload={base64}VklTSU8gMjAwMCBURU====,proto=6

Of course, we have to define a basic panel for the db-backend processing
and for matching the current issue (like pf, netfilter logs)

Have you any idea or comment about ?

thanks




alx





 --
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

Re: [Ipfc-developer] IPFC release 1.0 (cvs tag RELEASE_1_0)

[Tycho F. <tyc...@co...>]

On Mon, 17 Dec 2001, Alexandre Dulaunoy wrote:

> Hello All,
>
> As you have seen in the ipfc-announce the version 1.0 is released.
> The cvs tag is RELEASE_1_0 .
>
> So the fun is coming with new features, bug fixes, ...
it started already ;-) with small changes in the db-backend-daemon and
db-wrapnet to improve resilience in case someone (like your cleanup script
;-) decides to run on the exact same time we are parsing a directory.
Before, we might have missed some .processed files leading to duplicate
processing (with all problems following this event).

I'm thinking right now of rewriting the processing modules to be more
orthogonal.

So you''d have different TRANSPORTS and then different
EVENT_TYPE/EVENT_CATEGORY

eg. you could have a webserver which is sending apache common-logs through
a syslog (logged using syslog-lines)

or an application X sending logs line-by-line to syslog-lines.  In both
cases, we can already pluck the XML data apart until the syslog-line
level, where the event-type and event-category starts to play.

Feedback of course more than welcome ;-)

T.


-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] IPFC release 1.0 (cvs tag RELEASE_1_0)

[Alexandre D. <al...@co...>]

Hello All,

As you have seen in the ipfc-announce the version 1.0 is released.
The cvs tag is RELEASE_1_0 .

So the fun is coming with new features, bug fixes, ...

See ya

alx


-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] new CVS structure

[Alexandre D. <al...@co...>]

Dear all,

We have moved the CVS directory as follow :

/ipfc/docs			General Documentation directories
/ipfc/docs/db-backend		Documentation for db-backend (all that
				can process files from the dr-server)
/ipfc/docs/protocol		Documentation for protocol and data format
			        (from communication between zone to
				 xml data format)
/ipfc/docs/faq			FAQ and HOWTO about IPFC framework
/ipfc/docs/diagram		Diagram for the IPFC framework
/ipfc/src/db-backend		Source code for the DB-Backend
				(including SQL Schema and processing
scripts)
/ipfc/src/dr-server		Additional source code for the DR-Server
				(DR-Server is based upon Apache, this
				 include a building script)
/ipfc/src/frontend		Frontend to interact with the db-backend
/ipfc/src/lib			Source code of library that can be used
				with wrapper and other application using
				the IPFC framework.
/ipfc/src/wrapper		Wrapper source code.
				(one directory per type)

So the old current directory will be removed in a near future. (following
the 1.0 release 17 December)

We have moved the majority of source code to the new strucuture. But if
you have OLD FILES laying around the current directory. you SHOULD move it
to the new structure.

Don't hesitate to contact me if you got any issue with that.

See ya

adulau


-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] initial and alpha doc for protocol

[Alexandre D. <al...@co...>]

I have put an initial documentation for the protocol and the data
description of IPFC.

ipfc/current/docs/protocol/protocol.lyx

I have to extend the document and complete it. But if you want to
contribute to some part (from lexical analysis 8-) to chapter oriented to
your part...) don't hesitate...

see ya

alx

PS : ON sunday 2/12 , i will make a RC version. after two other RC and on
17/12 a 1.0 version.


-- 
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] Db-backend db overview and db future

[Alexandre D. <al...@co...>]

Here it is an overview of the db of the db-backend. The overview was done
with autodoc and need dia 0.88.1.

This is for your information only. The best way it's to get the full
schema (in the db-init sql files from the db-backend).

There will be a db for the security advisory too and for the agent
(hardware/software) inventory.




-- 
---
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

Re: [Ipfc-developer] xml data from dr-server (second try)

[Tycho F. <tyc...@co...>]

On Thu, 8 Nov 2001, Alexandre Dulaunoy wrote:

> Here it is the second try 8-)
And here is the database generated by the second try (done on the first
try ;-)

interesting for those who want to look at frond-end and processing stuff.

Cheers,
Tycho

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] xml data from dr-server (second try)

[Alexandre D. <al...@co...>]

Here it is the second try 8-)

The test documents act as reference for the xml data format. (we need to
make a complete documentation about that, will be done in the next weeks).

If you have any comment or addition about the format, don't hesitate.

Thanks

alx



-- 
---
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

Re: [Ipfc-developer] TODOs

[Tycho F. <tyc...@co...>]

On Wed, 7 Nov 2001, Tycho Fruru wrote:

> Hello All,
>
> here are my todos (for db-backend)
>
> - add new log/event formats
> 	- syslog-line works, needs some cleanup (esp. date processing)
> 	- apache-line works, needs further stress-testing with ugly data
> 	- snort
> 	- checkpoint
> 	- ISS RS
> 	- netfilter
> 	- pf / ipf
> 	- any other suggestions more than welcome
> - more consistency checking in the XML data (is the module ID the same as
> we expect etc)
> - better data validation (perhaps a separate library used to quote/unquote
> data for db storage ?)
> - test the stuff out with another database to see how dependent we are  on
> postgresql behaviour (FUTURE)
Oracle download takes *ages*

> - change the representation of the sql statement's fields and values (this
> is now an array, will become a hash making it easier to update already-set
> values)
DONE :-)

> positive feedback is more than welcome
> negative feedback also, but less so ;-)
>
> T.
>
>

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] TODOs

[Tycho F. <tyc...@co...>]

Hello All,

here are my todos (for db-backend)

- add new log/event formats
	- syslog-line works, needs some cleanup (esp. date processing)
	- apache-line works, needs further stress-testing with ugly data
	- snort
	- checkpoint
	- ISS RS
	- netfilter
	- pf / ipf
	- any other suggestions more than welcome
- more consistency checking in the XML data (is the module ID the same as
we expect etc)
- better data validation (perhaps a separate library used to quote/unquote
data for db storage ?)
- test the stuff out with another database to see how dependent we are  on
postgresql behaviour (FUTURE)
- change the representation of the sql statement's fields and values (this
is now an array, will become a hash making it easier to update already-set
values)

positive feedback is more than welcome
negative feedback also, but less so ;-)

T.

-- 
Tycho Fruru			tyc...@co...
Users' impressions of different operating systems, expressed as emoticons:
Linux:   :)
Windows: XP

[Ipfc-developer] Some status

[Alexandre D. <al...@co...>]

Hello All,

There is some experimental generic wrapper (wrapcol/wrapnet) available and
events based. (for the moment only FIFO file)
If you want to test it out (cvs co is your friend) and make comment on it
don't hesitate it's only a 2 hours hack....

TODO (for me):

* better error handling
* better configuration files
* better coding style (that's not in TODO it's my newyear's resolution)8-)
* adding SSL/TLS in libipfc
* adding more config style and parameter
* adding support for classical files and WIN32::Events::log

I'm testing it with 2 monitored hosts and 1 dr-server it's working quite
good.

PS : If perl module Event is rocksolid and portable, I will try to make
all wrapper events based. If you want other approach don't hesitate to code it by
yourself. 8-))))

If you have comments, new TODO and so on... don't hesitate.

Thanks.

Alex


-- 
---
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] Updated website

[Alexandre D. <al...@co...>]

Dear All,

The IPFC website was updated with current information.
http://www.foo.be/ipfc/

More has to be done like :

- FAQ (check out cvs for latest version)
- Schema
- Data description and xml description
- HOWTO

If you are volonteer don't hesitate...


alx


-- 
---
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/

[Ipfc-developer] New mailing list (who's who) and IPFC status

[Alexandre D. <al...@co...>]

Dear all,

We have created a new mailing list for the developer of IPFC. All the guys
who got CVS access, got a subscribe to this list.

The main purpose of the list is to discuss of :

* The current issue with the IPFC framework
* The current stage of each component of the IPFC framework
* Technical discussion (from big trouble to brilliant idea)
* .... and all related to IPFC.

Current active project & developer :

General coordination (adulau)
db-backend (fpmip)
dr-server (adulau)
wrapper (adulau, fpmip and others)
monitoring wrapper (fdulau)
frontend (walrave)
xml external schema (pcaesm)

Some random and possible milestone 8-) :

Beta release deadline  : 1 December 2001 !
RC release deadline    : 10 December 2001
release 1.0 	       : 15 December 2001
release 1.1 	       : 14 January 2001

If there is any other project or developer, don't hesitate to talk
about...

See you soon.

alx aka adulau aka alex

PS : I'm currently trying to make a new web site for IPFC.
PS2: IPFC need a new acronym (it's more than IP Filter now...):
     some possible acronyms are
	* Internet Protocol Functional Control
	* Inter Protocol Filter Control
	* .... any idea ? with the the generic framework in back ?

-- 
---
Alexandre Dulaunoy			ad...@co...
					http://www.conostix.com/