Menu
▾
▴
echelog-devel
|
From: Benoit D. <bd...@an...> - 2003-02-18 23:54:19
|
Hello again,
I'm looking for doing stat on logged data like iptables log and maillog.
I want to execute these actions:
RemoteHost : getlog(/var/log/messages) -> Sender(Hub)
Hub : receiver(RemoteHost) -> Filter -l001->
-> Rewriter(MailMessages) -> binstor/rrdtool
The Rewrite module can transform a messagetype in an other messagetype
after doing some conversions on VARCHAR.
sample:
--------------------------------------------------------------------
Filter is:
LINE~"^(... [0-9]+ [0-9:]{8}) .*: ([^:]+): from=<?([^,])>?,
size=([0-9]+), relay=(.*)$",DATE=\1,SPOOL=\2,MAILFROM=\3,SIZE=\4,RELAY=\5
--------------------------------------------------------------------
For log as:
Feb 17 12:22:18 local@server1 sendmail[15624]: h1HBMIVd015624:
from=<bd...@so...>, size=945, class=0, nrcpts=1,
msgid=<3E5...@so...>, proto=ESMTP, daemon=MTA,
relay=relay1.societe.local [172.16.0.2]
--------------------------------------------------------------------
The result is ....
Do you have some ideas about the best method to use?
Benoit
--
Benoit DOLEZ
GSM: +33 6 21 05 91 69 mailto:bd...@an...
|
|
From: Kamil T. <kt...@em...> - 2003-02-19 16:39:53
|
On St, 2003-02-19 at 00:54, Benoit DOLEZ wrote:
> Hello again,
>
> I'm looking for doing stat on logged data like iptables log and maillog.
> I want to execute these actions:
> RemoteHost : getlog(/var/log/messages) -> Sender(Hub)
> Hub : receiver(RemoteHost) -> Filter -l001->
> -> Rewriter(MailMessages) -> binstor/rrdtool
>
> The Rewrite module can transform a messagetype in an other messagetype
> after doing some conversions on VARCHAR.
>
> sample:
> --------------------------------------------------------------------
> Filter is:
> LINE~"^(... [0-9]+ [0-9:]{8}) .*: ([^:]+): from=<?([^,])>?,
> size=([0-9]+), relay=(.*)$",DATE=\1,SPOOL=\2,MAILFROM=\3,SIZE=\4,RELAY=\5
> --------------------------------------------------------------------
> For log as:
> Feb 17 12:22:18 local@server1 sendmail[15624]: h1HBMIVd015624:
> from=<bd...@so...>, size=945, class=0, nrcpts=1,
> msgid=<3E5...@so...>, proto=ESMTP, daemon=MTA,
> relay=relay1.societe.local [172.16.0.2]
> --------------------------------------------------------------------
> The result is ....
>
> Do you have some ideas about the best method to use?
>
> Benoit
Hi!
I think this module could work like a filter, which would pass all
messages (or individual chunks) it doesn't know and translate known
data to new type of messages.
The rewriter module may be RULE based as filter is. Thus it could match
a regular expression, and assign the results of individual matches (\1,
\2, ...) to attributes in resulting messages.
The matched substrings can be parsed into echelog types using
get_type_instance like function and sscan features. However some
portions of filter module would be probably duplicated (or may be put
into a lib if carefully thought out ;)
What do you think?
Kamil
|
|
From: Benoit D. <bd...@an...> - 2003-02-20 08:41:09
|
Kamil Toman wrote:
> On St, 2003-02-19 at 00:54, Benoit DOLEZ wrote:
>
>>Hello again,
>>
>>I'm looking for doing stat on logged data like iptables log and maillog.
>>I want to execute these actions:
>>RemoteHost : getlog(/var/log/messages) -> Sender(Hub)
>>Hub : receiver(RemoteHost) -> Filter -l001->
>> -> Rewriter(MailMessages) -> binstor/rrdtool
>>
>>The Rewrite module can transform a messagetype in an other messagetype
>>after doing some conversions on VARCHAR.
>>
>>sample:
>>--------------------------------------------------------------------
>>Filter is:
>>LINE~"^(... [0-9]+ [0-9:]{8}) .*: ([^:]+): from=<?([^,])>?,
>>size=([0-9]+), relay=(.*)$",DATE=\1,SPOOL=\2,MAILFROM=\3,SIZE=\4,RELAY=\5
>>--------------------------------------------------------------------
>>For log as:
>>Feb 17 12:22:18 local@server1 sendmail[15624]: h1HBMIVd015624:
>>from=<bd...@so...>, size=945, class=0, nrcpts=1,
>>msgid=<3E5...@so...>, proto=ESMTP, daemon=MTA,
>>relay=relay1.societe.local [172.16.0.2]
>>--------------------------------------------------------------------
>>The result is ....
>>
>>Do you have some ideas about the best method to use?
>>
>>Benoit
>
>
> Hi!
>
> I think this module could work like a filter, which would pass all
> messages (or individual chunks) it doesn't know and translate known
> data to new type of messages.
I think so
> The rewriter module may be RULE based as filter is. Thus it could match
> a regular expression, and assign the results of individual matches (\1,
> \2, ...) to attributes in resulting messages
>
> The matched substrings can be parsed into echelog types using
> get_type_instance like function and sscan features. However some
> portions of filter module would be probably duplicated (or may be put
> into a lib if carefully thought out ;)
ok
>
> What do you think?
The config file is like that :
[Rewrite]
Rule {
MSGFROM SimpleMessage,mes0
MSGTO MailFromMessage,mai0
LINE~".....",MSG.msg_time=\1,ID=\2,FROM=\3,SIZE=\4
}
That seems to be good but what about perfs? I have to run lot of
filter/rewrite.
Benoit
>
> Kamil
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: SlickEdit Inc. Develop an edge.
> The most comprehensive and flexible code editor you can use.
> Code faster. C/C++, C#, Java, HTML, XML, many more. FREE 30-Day Trial.
> www.slickedit.com/sourceforge
> _______________________________________________
> Echelog-devel mailing list
> Ech...@li...
> https://lists.sourceforge.net/lists/listinfo/echelog-devel
>
>
--
Benoit DOLEZ
GSM: +33 6 21 05 91 69 mailto:bd...@an...
|
|
From: Kamil T. <to...@ar...> - 2003-02-20 14:02:29
|
On Thu, Feb 20, 2003 at 09:40:54AM +0100, Benoit DOLEZ wrote:
> >>The Rewrite module can transform a messagetype in an other messagetype
> >>after doing some conversions on VARCHAR.
> >>
> >>sample:
> >>--------------------------------------------------------------------
> >>Filter is:
> >>LINE~"^(... [0-9]+ [0-9:]{8}) .*: ([^:]+): from=<?([^,])>?,
> >>size=([0-9]+), relay=(.*)$",DATE=\1,SPOOL=\2,MAILFROM=\3,SIZE=\4,RELAY=\5
> >>--------------------------------------------------------------------
> >>For log as:
> >>Feb 17 12:22:18 local@server1 sendmail[15624]: h1HBMIVd015624:
> >>from=<bd...@so...>, size=945, class=0, nrcpts=1,
> >>msgid=<3E5...@so...>, proto=ESMTP, daemon=MTA,
> >>relay=relay1.societe.local [172.16.0.2]
> >>--------------------------------------------------------------------
> >>The result is ....
> >>
> >>Do you have some ideas about the best method to use?
> >>
> >>Benoit
> >
>
> >The rewriter module may be RULE based as filter is. Thus it could match
> >a regular expression, and assign the results of individual matches (\1,
> >\2, ...) to attributes in resulting messages
> >
> >The matched substrings can be parsed into echelog types using
> >get_type_instance like function and sscan features. However some
> >portions of filter module would be probably duplicated (or may be put
> >into a lib if carefully thought out ;)
>
> ok
> >
> >What do you think?
>
> The config file is like that :
> [Rewrite]
> Rule {
> MSGFROM SimpleMessage,mes0
> MSGTO MailFromMessage,mai0
> LINE~".....",MSG.msg_time=\1,ID=\2,FROM=\3,SIZE=\4
> }
OK, just make it readable enough ;) and document it please.
>
> That seems to be good but what about perfs? I have to run lot of
> filter/rewrite.
Yes, this could be a problem. But the rewriter module will hardly be
on a monitored production server. If the rules and regexps were
simple enough (no backtracking) the dedicated logging machine
should manage it.
This seems to be the simpliest way how to implement such rewriting
module anyway. I think it is worth trying. If there were performance
problems, we would see what we can do about it.
Kamil
>
> Benoit
>
|
Thanks for helping keep SourceForge clean.
X