echelog-devel — Echelog devel list

Re: [Echelog-devel] Rewriting rules

[Kamil T. <to...@ar...>]

On Thu, Feb 20, 2003 at 09:40:54AM +0100, Benoit DOLEZ wrote:
> >>The Rewrite module can transform a messagetype in an other messagetype
> >>after doing some conversions on VARCHAR.
> >>
> >>sample:
> >>--------------------------------------------------------------------
> >>Filter is:
> >>LINE~"^(... [0-9]+ [0-9:]{8}) .*: ([^:]+): from=<?([^,])>?, 
> >>size=([0-9]+), relay=(.*)$",DATE=\1,SPOOL=\2,MAILFROM=\3,SIZE=\4,RELAY=\5
> >>--------------------------------------------------------------------
> >>For log as:
> >>Feb 17 12:22:18 local@server1 sendmail[15624]: h1HBMIVd015624: 
> >>from=<bd...@so...>, size=945, class=0, nrcpts=1, 
> >>msgid=<3E5...@so...>, proto=ESMTP, daemon=MTA, 
> >>relay=relay1.societe.local [172.16.0.2]
> >>--------------------------------------------------------------------
> >>The result is ....
> >>
> >>Do you have some ideas about the best method to use?
> >>
> >>Benoit
> >
> 
> >The rewriter module may be RULE based as filter is. Thus it could match
> >a regular expression, and assign the results of individual matches (\1,
> >\2, ...) to attributes in resulting messages
> >
> >The matched substrings can be parsed into echelog types using
> >get_type_instance like function and sscan features. However some
> >portions of filter module would be probably duplicated (or may be put
> >into a lib if carefully thought out ;)
> 
> ok
> >
> >What do you think?
> 
> The config file is like that :
> [Rewrite]
> Rule {
>     MSGFROM SimpleMessage,mes0
>     MSGTO   MailFromMessage,mai0
>     LINE~".....",MSG.msg_time=\1,ID=\2,FROM=\3,SIZE=\4
> }

OK, just make it readable enough ;) and document it please.

> 
> That seems to be good but what about perfs? I have to run lot of 
> filter/rewrite.

Yes, this could be a problem. But the rewriter module will hardly be
on a monitored production server. If the rules and regexps were
simple enough (no backtracking) the dedicated logging machine
should manage it. 

This seems to be the simpliest way how to implement such rewriting
module anyway. I think it is worth trying. If there were performance
problems, we would see what we can do about it.

Kamil

> 
> Benoit
> 

[Echelog-devel] Echelog 0.6.2pre2

[Kamil T. <to...@ar...>]

Echelog version advanced to 0.6.2pre2. CVS repository tag for this
developer's milestone is joint_0_6_2pre2.

Highlights:
-----------

- online filter capabilities enhanced by new header conditions
- regular expressions in chunk and header conditions

The new header filtering features conform to our previous proposal:
http://sourceforge.net/mailarchive/forum.php?thread_id=1674648&forum_id=2036

Note that new filter capabilites allow to workaround the message faking
problem described here:
http://sourceforge.net/mailarchive/message.php?msg_id=3852617
You can setup more receivers and attach filters to them. But it is not
very comfortable and the receiver input rules may need some changes
anyway.

Also note that new filter capabilites allow to drop empty messages,
sort them by size, ip adresses, agent instances and so forth.

Kamil

[Echelog-devel] About certificate

[Benoit D. <bd...@an...>]

Hi,

Why do you need ip addresses of the echelog sender in the receiver conf?
Do certificates not be sure to identify sender?

I can't use it for DHCP client.

Benoit

-- 
Benoit DOLEZ
GSM: +33 6 21 05 91 69                mailto:bd...@an...

Re: [Echelog-devel] Rewriting rules

[Benoit D. <bd...@an...>]

Kamil Toman wrote:
> On St, 2003-02-19 at 00:54, Benoit DOLEZ wrote:
> 
>>Hello again,
>>
>>I'm looking for doing stat on logged data like iptables log and maillog.
>>I want to execute these actions:
>>RemoteHost : getlog(/var/log/messages) -> Sender(Hub)
>>Hub : receiver(RemoteHost) -> Filter -l001->
>>         -> Rewriter(MailMessages) -> binstor/rrdtool
>>
>>The Rewrite module can transform a messagetype in an other messagetype
>>after doing some conversions on VARCHAR.
>>
>>sample:
>>--------------------------------------------------------------------
>>Filter is:
>>LINE~"^(... [0-9]+ [0-9:]{8}) .*: ([^:]+): from=<?([^,])>?, 
>>size=([0-9]+), relay=(.*)$",DATE=\1,SPOOL=\2,MAILFROM=\3,SIZE=\4,RELAY=\5
>>--------------------------------------------------------------------
>>For log as:
>>Feb 17 12:22:18 local@server1 sendmail[15624]: h1HBMIVd015624: 
>>from=<bd...@so...>, size=945, class=0, nrcpts=1, 
>>msgid=<3E5...@so...>, proto=ESMTP, daemon=MTA, 
>>relay=relay1.societe.local [172.16.0.2]
>>--------------------------------------------------------------------
>>The result is ....
>>
>>Do you have some ideas about the best method to use?
>>
>>Benoit
> 
> 
> Hi!
> 
> I think this module could work like a filter, which would pass all
> messages (or individual chunks) it doesn't know and translate known
> data to new type of messages.

I think so

> The rewriter module may be RULE based as filter is. Thus it could match
> a regular expression, and assign the results of individual matches (\1,
> \2, ...) to attributes in resulting messages
> 
> The matched substrings can be parsed into echelog types using
> get_type_instance like function and sscan features. However some
> portions of filter module would be probably duplicated (or may be put
> into a lib if carefully thought out ;)

ok
> 
> What do you think?

The config file is like that :
[Rewrite]
Rule {
     MSGFROM SimpleMessage,mes0
     MSGTO   MailFromMessage,mai0
     LINE~".....",MSG.msg_time=\1,ID=\2,FROM=\3,SIZE=\4
}

That seems to be good but what about perfs? I have to run lot of 
filter/rewrite.

Benoit

> 
> Kamil
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: SlickEdit Inc. Develop an edge.
> The most comprehensive and flexible code editor you can use.
> Code faster. C/C++, C#, Java, HTML, XML, many more. FREE 30-Day Trial.
> www.slickedit.com/sourceforge
> _______________________________________________
> Echelog-devel mailing list
> Ech...@li...
> https://lists.sourceforge.net/lists/listinfo/echelog-devel
> 
> 


-- 
Benoit DOLEZ
GSM: +33 6 21 05 91 69                mailto:bd...@an...

Re: [Echelog-devel] Rewriting rules

[Kamil T. <kt...@em...>]

On St, 2003-02-19 at 00:54, Benoit DOLEZ wrote:
> Hello again,
> 
> I'm looking for doing stat on logged data like iptables log and maillog.
> I want to execute these actions:
> RemoteHost : getlog(/var/log/messages) -> Sender(Hub)
> Hub : receiver(RemoteHost) -> Filter -l001->
>          -> Rewriter(MailMessages) -> binstor/rrdtool
> 
> The Rewrite module can transform a messagetype in an other messagetype
> after doing some conversions on VARCHAR.
> 
> sample:
> --------------------------------------------------------------------
> Filter is:
> LINE~"^(... [0-9]+ [0-9:]{8}) .*: ([^:]+): from=<?([^,])>?, 
> size=([0-9]+), relay=(.*)$",DATE=\1,SPOOL=\2,MAILFROM=\3,SIZE=\4,RELAY=\5
> --------------------------------------------------------------------
> For log as:
> Feb 17 12:22:18 local@server1 sendmail[15624]: h1HBMIVd015624: 
> from=<bd...@so...>, size=945, class=0, nrcpts=1, 
> msgid=<3E5...@so...>, proto=ESMTP, daemon=MTA, 
> relay=relay1.societe.local [172.16.0.2]
> --------------------------------------------------------------------
> The result is ....
> 
> Do you have some ideas about the best method to use?
> 
> Benoit

Hi!

I think this module could work like a filter, which would pass all
messages (or individual chunks) it doesn't know and translate known
data to new type of messages.

The rewriter module may be RULE based as filter is. Thus it could match
a regular expression, and assign the results of individual matches (\1,
\2, ...) to attributes in resulting messages.

The matched substrings can be parsed into echelog types using
get_type_instance like function and sscan features. However some
portions of filter module would be probably duplicated (or may be put
into a lib if carefully thought out ;)

What do you think?

Kamil

Re: [Echelog-devel] New type: regex

[Kamil T. <kt...@em...>]

On St, 2003-02-19 at 00:30, Benoit DOLEZ wrote:
> > Echelog types are designed primarily for serialization into messages.
> > This doesn't seem to be the case.
> 
> Ok, you're right. My problem was to not compile regex for each line.
> I move the compilation in the constructor and execute it in the
> test_chunk function and it seems that it is working :-)
> 
> It is not a big patch but it may be useful.
> 
> About it, I'm not expert in C++ developement, how can I say
> that tFilterCondRE::is_invalid() work like tFilterCondSS::is_invalid()?

I've modified the patch, so tFilterCondRE is based on tFilterCondSS and
as such it inherits is_invalid routine.

The suggested feature (matching regular expressions) is now included in
CVS.

Kamil

> 
> > 
> > You can check the attribute type for varchar,... (by is_compatible(attr)
> > for example) and perform any special operation you'd like to (like
> > regmatch to a compiled regular expression).
> > 
> Thanks,
> Benoit

[Echelog-devel] Rewriting rules

[Benoit D. <bd...@an...>]

Hello again,

I'm looking for doing stat on logged data like iptables log and maillog.
I want to execute these actions:
RemoteHost : getlog(/var/log/messages) -> Sender(Hub)
Hub : receiver(RemoteHost) -> Filter -l001->
         -> Rewriter(MailMessages) -> binstor/rrdtool

The Rewrite module can transform a messagetype in an other messagetype
after doing some conversions on VARCHAR.

sample:
--------------------------------------------------------------------
Filter is:
LINE~"^(... [0-9]+ [0-9:]{8}) .*: ([^:]+): from=<?([^,])>?, 
size=([0-9]+), relay=(.*)$",DATE=\1,SPOOL=\2,MAILFROM=\3,SIZE=\4,RELAY=\5
--------------------------------------------------------------------
For log as:
Feb 17 12:22:18 local@server1 sendmail[15624]: h1HBMIVd015624: 
from=<bd...@so...>, size=945, class=0, nrcpts=1, 
msgid=<3E5...@so...>, proto=ESMTP, daemon=MTA, 
relay=relay1.societe.local [172.16.0.2]
--------------------------------------------------------------------
The result is ....

Do you have some ideas about the best method to use?

Benoit

-- 
Benoit DOLEZ
GSM: +33 6 21 05 91 69                mailto:bd...@an...

Re: [Echelog-devel] New type: regex

[Benoit D. <bd...@an...>]

Kamil Toman wrote:
> On =DAt, 2003-02-18 at 21:27, Benoit DOLEZ wrote:
>=20
>>Hi,
>>
>>What about create a new data type: 'REGEX', that can be used in
>>filter module?
>>
>>Benoit
>>
>=20
>=20
> Echelog types are designed primarily for serialization into messages.
> This doesn't seem to be the case.

Ok, you're right. My problem was to not compile regex for each line.
I move the compilation in the constructor and execute it in the
test_chunk function and it seems that it is working :-)

It is not a big patch but it may be useful.

About it, I'm not expert in C++ developement, how can I say
that tFilterCondRE::is_invalid() work like tFilterCondSS::is_invalid()?

>=20
> You can check the attribute type for varchar,... (by is_compatible(attr=
)
> for example) and perform any special operation you'd like to (like
> regmatch to a compiled regular expression).
>=20
Thanks,

Benoit

> More generally, maybe attributes could be enhanced to handle regexp
> comparisons but it'd need a precisely defined behaviour for each type -=
-
> how would that look like?
>=20
> Kamil
>=20
>=20
>=20
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Echelog-devel mailing list
> Ech...@li...
> https://lists.sourceforge.net/lists/listinfo/echelog-devel
>=20
>=20


--=20
Benoit DOLEZ
GSM: +33 6 21 05 91 69                mailto:bd...@an...

Re: [Echelog-devel] New type: regex

[Kamil T. <kt...@em...>]

On =DAt, 2003-02-18 at 21:27, Benoit DOLEZ wrote:
> Hi,
>=20
> What about create a new data type: 'REGEX', that can be used in
> filter module?
>=20
> Benoit
>=20

Echelog types are designed primarily for serialization into messages.
This doesn't seem to be the case.

You can check the attribute type for varchar,... (by is_compatible(attr)
for example) and perform any special operation you'd like to (like
regmatch to a compiled regular expression).

More generally, maybe attributes could be enhanced to handle regexp
comparisons but it'd need a precisely defined behaviour for each type --
how would that look like?

Kamil

[Echelog-devel] New type: regex

[Benoit D. <bd...@an...>]

Hi,

What about create a new data type: 'REGEX', that can be used in
filter module?

Benoit

-- 
Benoit DOLEZ
GSM: +33 6 21 05 91 69                mailto:bd...@an...

Re: [Echelog-devel] Filter Attributes (message faking)

[Kamil T. <kt...@em...>]

On =C8t, 2003-01-16 at 11:48, Martin Ma=E8ok wrote:
> On Thu, Jan 16, 2003 at 10:30:32AM +0100, Marek Zoth wrote:
>=20
> > >    MSG.agent_id
> > >    MSG.agent_instance_id
> > >    MSG.agent_host_id
>=20
> > Maybe it will be possible then to workaround a small problem with
> > message faking.
>=20
> I assume that knowing those 3 attributes from one (compromised)
> monitored host gives you real chance to guess correct 3 attributes of
> messages from another (non-compromised) monitored host. That gives you
> real chance to send fake message from compromised agent that will be
> accepted by server as the real ones from other non-compromised host
> because the Filter doesn't know the IP/SSL channel through which the
> message really came from.
>=20

True. When the Filter is enhanced, then one may set up more than one
receivers and attach a Filter to each.

But obviously this identification filter rule should be also added to
Receiver (thus it would be able to drop messages with fake or invalid
headers). If there is a proxy (somewhere on the path) it also should
drop such forged packets. If it didn't you can consider the whole
segment compromised.


> Either the message should contain some (not easily forged) hosts's
> signature or at least the tests should be done earlier on the Receiver
> which knows the IP/SSL channel the message came from.
>=20
> Maybe, we can gain something just by adding some not easily guessed
> content of some message attribute, say MSG.password, which is 1-1
> related to host. If there are agents on hosts A and B and the server
> is C then when I compromise A I can make false messages that looks
> like those from B because I can know/guess B's *_id attributes. The
> server C now contains fake messages. If there is another not easily
> guessed attribute then I can't easily forge messages from B because
> message from B which doesn't have correct B's password attribute
> should be invalid (which can be checked online/offline by any Filter
> on C).
>=20
> --=20
>          Martin Ma=E8ok                 http://underground.cz/
>    mar...@un...        http://Xtrmntr.org/ORBman/

If the solution proposed above takes place there is no need for
MSG.password field as it doesn't increase inherent security.

Signing each message would be a very demanding task. It has been already
said that it is undesirable for agent systems.

Kamil

Re: [Echelog-devel] Filter Attributes (message faking)

[Martin <mar...@un...>]

On Thu, Jan 16, 2003 at 10:30:32AM +0100, Marek Zoth wrote:

> >    MSG.agent_id
> >    MSG.agent_instance_id
> >    MSG.agent_host_id

> Maybe it will be possible then to workaround a small problem with
> message faking.

I assume that knowing those 3 attributes from one (compromised)
monitored host gives you real chance to guess correct 3 attributes of
messages from another (non-compromised) monitored host. That gives you
real chance to send fake message from compromised agent that will be
accepted by server as the real ones from other non-compromised host
because the Filter doesn't know the IP/SSL channel through which the
message really came from.

Either the message should contain some (not easily forged) hosts's
signature or at least the tests should be done earlier on the Receiver
which knows the IP/SSL channel the message came from.

Maybe, we can gain something just by adding some not easily guessed
content of some message attribute, say MSG.password, which is 1-1
related to host. If there are agents on hosts A and B and the server
is C then when I compromise A I can make false messages that looks
like those from B because I can know/guess B's *_id attributes. The
server C now contains fake messages. If there is another not easily
guessed attribute then I can't easily forge messages from B because
message from B which doesn't have correct B's password attribute
should be invalid (which can be checked online/offline by any Filter
on C).

--=20
         Martin Ma=E8ok                 http://underground.cz/
   mar...@un...        http://Xtrmntr.org/ORBman/

Re: [Echelog-devel] Filter Attributes

[Marek Z. <mar...@vo...>]

On Thu, 16 Jan 2003, Kamil Toman wrote:

> > In the filter module, it is possible to filter with defined fields of
> > the message type. But each echelog entry having predefined fields because of the
> > echelog protocol (in the header): agent_ip, instance_id, date. 
> > I would like to known how to build filter like.
> >
> > Rule {
> >     MSG SimpleLog
> >     COND AGENT_INSTANCE_ID == 'rrd0'
> >     PASS Splitter
> > }
> 
> I'm not sure if the Filter is able to filter anything but message chunks
> by attributes. I think it'd need an enhancement to support this. Marek 
> should know more.

Yes, Kamil is right, it'd need an enhacement. Now the Filter is able only 
to select messages by message type and select chunks with specific 
condition from the message.

> I think there are also some minor issues concerning the complex echelog
> types (thus attributes) filtering conditions -- Filter doesn't allow
> simply testing complex types (attribute instances). That is you can't
> write:
> 
> #Incorrect
> Rule { 
>     MSG SessionMessage 
>     COND FOO == null foo2 foo3 
>     PASS out1 
> }
> 
> But you have to workaround this as
> 
> Rule {
>     MSG SessionMessage 
>     COND FOO.alt2 == foo2 
>     COND FOO.alt3 == foo3 
>     PASS out1 
> }

I have to note that it is possible to test complex types the way first 
rule was written, but values for all complex type "synonyms" must be 
specified on COND line.

#it is possible
COND FOO == foo1 foo2 foo3 

The part which is wrong is the word "null" in the meaning of "not 
initialized" or "doesn't matter" in this case you must use workaround as 
Kamil has written.


> In this context I'd suggest to enhance Rules MSG confition to support
> better message-by-header filtering not just by message type. 
> Thus condition
>    MSG SessionMessage 
> would be an alias for 
>    MSG.type == SessionMessage
> and other header condition rules would be:
>    MSG.msg_time
>    MSG.subrecords
>    MSG.agent_id
>    MSG.agent_instance_id
>    MSG.agent_host_id
>    MSG.flags -- allowed condition values: BASIC, INCREMENTAL, AGGREGABLE

It seems to be a good enhancement. Maybe it will be possible then to 
workaround a small problem with message faking. To the future, there is a 
question about creating "mighty" filter. Purpose of this Filter is only 
simple "online" filtering. There are other modules for "offline" usage 
like bingrep that can do "mighty" filtering now.

Marek

Re: [Echelog-devel] Filter Attributes

[Kamil T. <kt...@em...>]

On St, 2003-01-15 at 15:05, Benoit DOLEZ wrote:

Hi!

> In the filter module, it is possible to filter with defined fields of
> the message type. But each echelog entry having predefined fields because of the
> echelog protocol (in the header): agent_ip, instance_id, date. 
> I would like to known how to build filter like.
>
> Rule {
>     MSG SimpleLog
>     COND AGENT_INSTANCE_ID == 'rrd0'
>     PASS Splitter
> }
>
> Benoit

I'm not sure if the Filter is able to filter anything but message chunks
by attributes. I think it'd need an enhancement to support this. Marek 
should know more.

The filter rules must conform to following regular expression:

( MSG  msg_name  
  (COND  attribute_name cond_type  value ( value-part-n )*)* 
  ( ((PASS | PASS_DISCARD)  output1  ( outputN )*) | DISCARD ) ) 
| ( MSGALL PASS  output1  ( outputN )* ) )

where msg_name is a message type name,
attribute_name is an attribute name (of the selected message)

I think there are also some minor issues concerning the complex echelog
types (thus attributes) filtering conditions -- Filter doesn't allow
simply testing complex types (attribute instances). That is you can't
write:

#Incorrect
Rule { 
    MSG SessionMessage 
    COND FOO == null foo2 foo3 
    PASS out1 
}

But you have to workaround this as

Rule {
    MSG SessionMessage 
    COND FOO.alt2 == foo2 
    COND FOO.alt3 == foo3 
    PASS out1 
}


In this context I'd suggest to enhance Rules MSG confition to support
better message-by-header filtering not just by message type. 
Thus condition
   MSG SessionMessage 
would be an alias for 
   MSG.type == SessionMessage
and other header condition rules would be:
   MSG.msg_time
   MSG.subrecords
   MSG.agent_id
   MSG.agent_instance_id
   MSG.agent_host_id
   MSG.flags -- allowed condition values: BASIC, INCREMENTAL, AGGREGABLE

Kamil

Re: [Echelog-devel] Filter Attributes

[Kamil T. <kt...@em...>]

On =DAt, 2003-01-14 at 15:04, Benoit DOLEZ wrote:
> Hi, I'm a new echelog user (and developper). I'm french and have a bad en=
glish.
> But I don't speak czech. I hope you will understand me ...
>=20
> I'm looking for all existing attribut. I had found:

All attributes are defined in lib/messages/generate/typelist.conf.
See REAME.txt for code generation targets.

All attributes are based, directly or as complex types - structures, on
built-in echelog types. Nore: Complex types can be thought like
structures where each item holds information with the same semantics but
in a different format (like IP address and hostname). It means that
definitions of used types (thus attributes) must be compiled-in and
later registered (see conf/types.conf).

On the other hand, messages built from those attributes can be simply
added to a running echelog system (only new components and components
which need to understand them need reconfiguration).

Messages don't ever change. If there is need for a change, a new message
should be created. If someone creates a new module he can use an old
type of message or just to create a new one. See conf/messages.conf
for all message definitions.

Each message consists of a header and the message body. The header holds
information like message length, number of chunks, timestamp, agent and
instance id, flags etc. The message body consists of zero or more chunks
of the _same_ type which hold the actual collected data. Each chunk
consists of a sequence (accordingly to messsage type) specific
attributes.

See include/messages/msgbuf.h for interface to messages
See include/messages/msgheader.h for interface to message header
See include/messages/msgchunk.h for interface to message chunks
See include/messages/types for interfaces to all built-in types

You can try to examine files=20
test/test-kamil/msgwrite.cc and test/test-kamil/msgread.cc
to better understand the structure of messages.

>=20
> This is my questions:
> - Is there other attribut like INSTANCE, IP, DATE, MESSAGETYPE,etc. ?
>=20

I don't understand what exactly do you mean. There exist attributes
like LOCAL_ADDRESS which are typed as HOST which is the complex echelog=20
type able to hold IP adress or hostname. This concerns only transfered
data.

The origin of data (particular machine, time of creation, agent instance
etc.) can be extracted from message header.

> - I'm looking for splitting data into different modules after having
>    receive its. And I have the idea to parse message line from input
>    socket and build 'filterable' attributed data to output socket. Is
>    someone have an idea?

If I understand this correctly, you want sort of Translator/Parser
module.=20

It would receive messages like SimpleLog (unsorted lines, let's say),
parse them and send them to Filter where they would be sorted to=20
other modules by message type.

There is no such module. If you want to develop it it may be usefull to
start off with MsgSniff. Also it may be interesting how general this
parser should be -- more stupid simple parsing automatons or one larger
(highly configurable) parser...

>=20
> Thanks
>=20
> Benoit
>=20

Kamil
=20
P.S. Current (but still incomplete) developer's documentation can be
found here:
http://prdownloads.sourceforge.net/echelog/echelog-0.6-doc-dev.tar.gz?downl=
oad

[Echelog-devel] Filter Attributes

[Benoit D. <bd...@an...>]

Hi, I'm a new echelog user (and developper). I'm french and have a bad english.
But I don't speak czech. I hope you will understand me ...

I'm looking for all existing attribut. I had found:
- getlog:
    - LINE
- netstat:
    - PROTOCOL
    - LOCAL_ADDRESS
    - REMOTE_ADDRESS
    - LOCAL_PORT
    - REMOTE_PORT
    - STATE
- ps:
    - EOWNER
    - OWNER
    - PID
    - PPID
    - CPU
    - MEMSIZE
    - RSS
    - TTY
    - STATE
    - STARTTIME
    - CMDLINE
- utmp:
    - USER
    - HOST
    - TTY
    - SESSTYPE
    - IDLESEC
    - STARTTIME
- w:
    - UPTIMESEC
    - USERS
    - LOAD1
    - LOAD5
    - LOAD15
    - USER
    - HOST
    - TTY
    - SESSTYPE
    - IDLESEC
    - STARTTIME

In case of error LINE is set.

This is my questions:
- Is there other attribut like INSTANCE, IP, DATE, MESSAGETYPE,etc. ?

- I'm looking for splitting data into different modules after having
   receive its. And I have the idea to parse message line from input
   socket and build 'filterable' attributed data to output socket. Is
   someone have an idea?

Thanks

Benoit





-- 
Benoit DOLEZ
GSM: +33 6 21 05 91 69                mailto:bd...@an...

[Echelog-devel] [Fwd: SV: Admin tester]

[Kamil T. <kt...@em...>]

From: Stian Holm <st...@ai...>
To: 'Kamil Toman' <ka...@us...>
Subject: SV: Admin tester
Date: 30 Dec 2002 06:00:26 +0100

Hi Kamil!

I'm actually sitting in my room installing my 3 unix
servers right now, and I'll hope to be starting the
testing in early January 2003.

Just to give some information about the servers I run
and what os are on them:

Server 1: Pentium III 733mhz, 512mb SDRAM, 2x 4gb SCSI U160
Slackware 8.0

Server 2: Pentium I 262mhz, 256mb EDO RAM, 2x 10gb ide 5400
Slackware 8.0

Server 3: Dual Pentium III 1ghz, 1024 DDR RAM, 8x 18gb SCSI U160 (raid)
Slackware 8.1

I hope I can help you developing you product :)


---
Best regards,
Stian Holm

[Echelog-devel] ANNOUNCE: echelog-0.6.1 has been released

[Kamil T. <kt...@em...>]

Echelog 0.6.1 bugfix release has been made public. All users are
encouraged to upgrade. Here is the ChangeLog:

Version 0.6.1 (11-dec-2002)

  - fixes for gcc 3.2
  - fixes in setup-wizard (webtools installation, key handling)
  - fixes for alert (xmessage handling)
  - fixed ssl tests
  - message buffer cleanup
  - webtools update
    
  - expired testing keys replaced by new ones
  - new proxy instalation target in setup-wizard (useful for NAT)

  - updated documentation
  - Quick Start guide (en)

Enjoy,
Echelog Team.

[Echelog-devel] komentare k QuickStartu

[Pavel K. <pe...@mb...>]

Obecna poznamka 1: nepamatuju si uz to pravidlo, kdy pouzit "which" a kdy
"that", ale myslim, ze "which" spis uvozuje kriterium vyberu, zatimco
"that" popisuje vlastnosti, takze ve vetsine pripadu by bylo lepsi
"that" nez "which" (ktere je tam ted vsude).

Obecna poznamka 2: je treba uvadet cleny (ovsem jen na spravnych
mistech a ty spravne, ze ano <g>).

Obecna poznamka 3: frazi "modul XYZ" to chce prekladat nejak jednotne
(a ani mnou navrhovane formulace nejsou mozna v tomto smeru bez chyby).
IMHO by bylo nejlepsi vetsinou pouzivat jen nazev modulu (nejsem si jisty,
jestli s clenem nebo bez, mozna spis bez, podle kontextu) a maximalne pri
prvnim pouziti do toho cpat slovo "module" a to s urcitym clenem a zrejme
trochu lepe ve vetsine pripadu zni, kdyz je jmeno modulu uprostred,
tj. "the XYZ module".

Obecna poznamka 4: prislovecne urceni psane na zacatku vety se vetsinou
oddeluje carkou, ale obecne by se, pokud to neni na zavadu
srozumitelnosti, dodrzovat std. slovosled a psat ho na konec vety.

> Overview
> ========

> monitoring the network's and hosts' state.

Trochu divna formulace: lepsi by asi bylo "the state of hosts and the
network".

> Gained data are sent to a special server or servers 

"Gain" se chape vetsinou jako zvetsit hodnotu. Ja bych tady spis pouzil
"gathered", "collected" (coz by souznelo s jinde pouzivanym "collect")
nebo mozna "obtained".

> in order to store them.

Jednak to neni jen ukladani, jednak je ta formulace trochu kostrbata.
Napsal bych asi "where they are processed and stored" nebo mozna
"...archived".

> Modules and Messages
> --------------------

> Each of its modules carries out a precisely specified elemetar role.

"elementary role"

> network connections and sockets states (Miniagent Netstat)

"socket states"

> or they systematically send system logs (Miniagent GetLog).

To je trochu moc sroubovane. Neni mi jasne, co chces presne rict.
Mozna neco jako "or they watch system log files and send messages
with new log entries"

> Aggregator alters absolute messages (snapshots) from miniagents into
> relative messages and thus descreases the amount of data to be
> transfered.

Tady je ctenar po hlave hozen do toho, ze existuji snapshoty nebo
inkrementy. To by asi melo byt vysvetleno u zprav. Tady bych pak napsal
neco jako "Aggregator is a module transforming a sequence of snapshots
(generated by most miniagents) into a sequence of incremental messages,
eliminating redundant data."

> Filter sorts or duplicates messages and accordingly to given rules
> passes them to adjacent modules of Echelog.

To je take zrale na celkovou rekonstrukci. Rekneme treba "Filter forwards
incoming messages to one one or more channels or drops them according
to the given set of rules."

> For example to Alert module which notices an administrator about
> particular events by e-mail or SMS.

"to notice" je "vsimnout si". Melo by tam byt "notifies". Pak je to treba
take vhodne navazat na predchozi vetu.

> The data transmission is realized by Sender and Receiver modules. The 
> TCP data transmission over the net is secured using application protocol
> SSL.

"Sender and Receiver transmit data over the network. The communication is
secured using the SSL protocol." Mozna misto "secured" spis "protected".

> The module LogStore serves mainly for log reconstruction
> in their original text format (collected by agents on monitored hosts).

"...for the reconstruction of logs in their original..."

> The testing module MsgSniff takes (echelog) messages and on its output
> it writes them out in a human readable form.

Strasny slovosled a pritom by jen stacilo vynechat zbytecne "on its
output".

> MsgSniff can be also put between communication modules and used for
> message monitoring.

Spis bych rekl neco jako "Alternatively, it is possible to insert
MsgSniff between two other modules and monitor their communication."

> The binary log can be further processed by special set of auxiliary
> programs - BinTools.

Vyhodil bych "further" nebo zmenil slovosled "further processing of the
binary log can be done by...". Jo a pomlcky jsou dlouhe, v ASCII se bezne
pisou dvojite (--) a vetsinou se kolem nich nedelaji mezery.

> It is possible to search messages from a certain time interval,
> to search messages by a given tyoe or messages matching a regular
> expression (a grep analogy). 

"Search" mi tady v tom kontextu nejak nesedi. Ale ta veta vubec nejak
nenavazuje na tu predchozi, protoze by se nejdriv melo rict, co se vlastne
mysli tim further processing.

> Typical system configurations
> -----------------------------

> Agent
> -----

> Miniagents W, Ps, Utmp and Netstat collect interesting information 
> in regular time intervals

"at regular time intervals"

> and send them to aggregator which decreases the amount of data to be
> transfered over a network.

"reduces the amount...over the network"

> to Sender becouse there is no point to aggregate them.

"because their is not point in aggregating them"
    ^^^
> Sender transfers the data to a server

"the server" nebo "servers"

> and in case of problems it stores all messages into a spool and
> resends them upon a next successfull connection.

Melo by byt "in the case of...", ale tady to vubec nesedne.
Mozna ukoncit vetu a pokracovat "When network problems occur,
Sender starts saving incoming messages into a spool and resends
them..."

> The agent system is maintained by Overseer (implemented by echelogd,
> controled by echelogctl).

Tohle se skoro presne opakuje u serveru a pripada mi to trochu zbytecne.
V predchozi kapitole bych napsal neco jako ze "The modules running on an
agent or a server are monitored and controlled by the Overseer (echelogd)
that starts, stops, and reconfigures individual modules and maintains
communication channels connecting them."


> Server
> ------

> BinStor stores all incoming messages to economic binary log,

Mozna spis "to a compact binary log".

 LogStor restores
the transfered system logs and

> AlertMaster sends warnings in accordance with critical conditions set
> up by an administrator (by e-mail, by SMS, by a sound from a speaker
> etc.).

Alert master posila varovani v souladu s kritickymi podminkami nastavenymi
administratorem? :)

> Quick Start
> ===========

> After installation run Echelog by

Spis trochu kvetnateji: "Having completed the installation, you can start
Echelog by"

> the contents of file /var/log/echelog/echelog.log should

"file" je nadbytecne (a pokdu by bylo uvedeno, tak by asi muselo mit
urcity clen)

> (for simplicity the date is omitted):

"for the sake of simplicity, the date is omitted"
Ovsem tohle zjednoduseni povazuji za dosti diskutabilni.

> empty becouse none module has been configured so far.

"because" (uz zase)

> Now stop echelog by command

"Now, stop Echelog using"
A mozna by to chtelo mene opakovat "Now" na zacatku vety. :)

> This action should be visible in echelog.log as:

"...should appear in..."

> Alter modules/ps.conf values of time interval when the data are
> collected to:

"Alter the values in modules/ps.conf specifying time..."

> Now it neccessary to configure the Overseer module (daemon echelogd) to
> ensure the proper startup and communication between modules.

Snazis se to prekladat do anglictiny moc otrocky.

Look at the log contents of echelog.log.

> There should be logged the execution of both modules:

"The execution of both modules should be logged there:"

> and the file modules.log with a trace of executions of individual
> modules:

"with the trace of execution of individual"

> Every 30 seconds there should add a report about all currently running 
> processes.

Neco jako "Every 30 seconds a new report about running processes should
be appended to this files."

> into a incremental messages.

"an incremental message"

> so it'd aggregate messages of running processes

Stazene tvary jsou hovorove a nemely by se v podobnem textu pouzivat.
Navic je v tomto pripade lepsi "it will aggregate" nez kondicional.

> Then you need to register Aggregator by Overseer to correctly connect it
> to the system.

Coze? mozna "register ... at Overseer". A zbytek by mozna mel byt neco
jako "in order to connect it with the rest of the system properly"

> At last tell the system to reconfigurate

"to reconfigure" nebo mozna "to reconfigure itself". Pripadne "to refresh
its configuration"

> In /var/log/echelog/echelog.log you can see the course of events:

"the course of events" je pekna fraze

> Notice that the Overseer didn't touch MsgSniff during the

Tady by bylo asi lepsi "note" misto "notice"

> Small incremental messages now add up every 30 seconds into msgsniff.log
> and complete messages (snapshots) of all processes every 5 minutes.

"add up"? velmi divna formulace ("add up" znamena secist, davat vysledek)

Ja nevim, treba "Small incremental messages will be written to msgsniff.log
every 30 seconds, while large complete snaphots including all running
processes will appear every 5 minutes."

Ovsem mozna to neni uplne vyznamove spravne. (Ted si nejsem jisty, jak se
nakonec rozuzlilo nedorozumeni ohledne prazdnych zprav.)

> In this way you can create a configuration of all individual modules and

"This way, you can create..."

> Setup Wizard
> ------------

> The first complete initialization can be also simply done by the script
> setup-wizard.

"The easiest way to create an initial Echelog configuration is to run
the setup-wizard script."

>  User can select among executed miniagents, time intervals of data 

"select among"? Divne. Jestli chces rict, co vsechno muze uzivatel
nastavit, tak treba "The user can select running miniagents, set time
intervals..."

Uf...urcite jsem neco prehledl a urcite budou nektere pripominky zcestne,
ale vic uz po mne nechtejte :)

--Pavel Kankovsky aka Peak

[Echelog-devel] 0.6.1-RC1

[Kamil T. <kt...@em...>]

Cau!

Z toho, co se prave nachazi v cvs bych navrhoval udelat releasku 0.6.1.
Kdo ma nejaky komentar, necht se ozve ;)

K.

Re: [Echelog-devel] Quick Start Doc

[Pavel K. <pe...@mb...>]

On 5 Dec 2002, Kamil Toman wrote:

> Utrhnul jsem si pulhodku a prelozil jsem kapitolu o prvnim startu
> echelogu plus nejaky overview, takze to posilam k pripominkovani.

To jsem opravdovy borec, jestli jsi to napsal za pul hodiny.

V zasade to vypada dobre, i kdyz je tam pomerne dost zcestnych formulaci
(napr. "Filter sorts..."...striktne vzato je "sort" tridit ve smyslu
rozdelovat na skupiny, ale vetsina by to asi pochopila jako zmenu poradi).

Ted zrovna uz asi treti hodinu presvedcuju Nessus, aby se zkompiloval na
Red Hatu 8, ale az to dodelam (a az se zotavim), tak se na to podivam
poradne.

--Pavel Kankovsky aka Peak

[Echelog-devel] Quick Start Doc

[Kamil T. <kt...@em...>]

Utrhnul jsem si pulhodku a prelozil jsem kapitolu o prvnim startu
echelogu plus nejaky overview, takze to posilam k pripominkovani.

Asi bych to dal jako README.QuickStart do rootu instalace a do README
na to pridal odkaz.

Komentare?

Kamil

[Echelog-devel] Re: [ echelog-Bugs-648443 ] wizard problem with handling root certs

[Martin <mar...@un...>]

On Wed, Dec 04, 2002 at 07:36:34AM -0800, no...@so... wrote:

> Setup wizard creates an invalid configuration when user
> supplies all pre-generated certificates (sender or
> receiver then refuses to start up).

Zkus tenhle patch, zapomnel jsem zkopirovat ten podstrcenej root
certifikat do /etc/echelog/modules/keys

RFE: Sender/Receiver by do logu mohli zarvat, kterej soubor nemuzou
precist.

--=20
         Martin Ma=E8ok                 http://underground.cz/
   mar...@un...        http://Xtrmntr.org/ORBman/

Re: [Echelog-devel] Release 0.6.1

[Marek Z. <mar...@vo...>]

On Mon, 2 Dec 2002 kt...@em... wrote:

> Hola hola!
> 
> Vzhledem k tomu, ze aktualni verze echelogu nejde na modernejsich
> linuxovych distribucich ani prelozit bych navrhoval prostouchnout
> stojate vody vydanim nove mensi verze (treba 0.6.1) zalozene na
> aktualnim cvsku.
> 
> Kratce po odevzdani projektu se nekdo na schuzce kasal s nejakymi plany
> na alespon strucnou anglickou dokumentaci - jak to s tim vypada?
> Bude ji mozno prihodit?
> 
> Kamil

Prihodit ji asi mozny nebude ;-). Musim dat za pravdu Alesovi, kerej 
rikal, ze pracovni nadseni po obhajobe znacne opadne a na sobe to pozoruju 
perfektne. Na drunou stranu si taky myslim ze by nebylo od veci to 
dotahnout. Zatim sem k tomu nenasel vhodnou kombinaci chute do prace a 
casu. Ale zkusim se polepsit, no znate to .. bude ten Mikulas ..

M.

[Echelog-devel] [ echelog-Bugs-648443 ] wizard problem with handling root certs

[<no...@so...>]

Bugs item #648443, was opened at 2002-12-04 16:36
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=383858&aid=648443&group_id=25225

Category: user interface
Group: New
Status: Open
Resolution: None
Priority: 5
Submitted By: Kamil Toman (kato)
Assigned to: Martin Macok (martin_macok)
Summary: wizard problem with handling root certs

Initial Comment:
Setup wizard creates an invalid configuration when user
supplies all pre-generated certificates (sender or
receiver then refuses to start up).

Example situation, the pre-generated keys are in a
user's directory:
keys/
server_certificate.pem
server_dsa_key.pem
rootcert.pem

Q: Do you already have receiver's signed certificate
file and private key 
A: yes
Q: Signed certificate file 
A: keys/server_certificate.pem
Q: Private key file 
A: keys/server_dsa_key.pem
Q: Do you already have ROOT certificate file 
A: yes
Q: ROOT certificate file 
A: keys/rootcert.pem

Note: This issue should be fixed before release_0_6_1.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=383858&aid=648443&group_id=25225