simple-evcorr-users — Simple Event Correlator users

Re: [Simple-evcorr-users] SEC rules performance monitoring and tuning

[Risto V. <ris...@gm...>]

hi Richard,

as I understand from your post, you would like to create SEC dump files
periodically, in order to monitor performance of SEC based on these dump
files. Let me first provide some comments on performance related question.
Essentially, creation of dump file involves a pass over all major internal
data structures, so that summaries about internal data structures can be
written into dump file. In fact, SEC traverses all internal data structures
for housekeeping purposes anyway once a second (you can change this
interval with --cleantime command line option). Therefore, if you re-create
the dump file after reasonable time intervals (say, after couple of
minutes), it wouldn't noticeably increase CPU consumption (things would
become different if dumps would be generated many times a second).

For production use, I would provide couple of recommendations. Firstly, you
could consider generating dump files in JSON format which might make their
parsing easier. That feature was requested couple of years ago specifically
for monitoring purposes (
https://sourceforge.net/p/simple-evcorr/mailman/message/36334142/), and you
can activate JSON format for dump file with --dumpfjson command line
option. You could also consider using --dumpfts option which adds numeric
timestamp suffix to dump file names (e.g., /tmp/sec.dump.1580210466). SEC
does not overwrite the dump file if it already exists, so having timestamps
in file names avoids the need for dump file rotation (you would still need
to delete old dump files from time to time, though).

Hopefully this information is helpful.

kind regards,
risto

Kontakt Richard Ostrochovský (<ric...@gm...>) kirjutas
kuupäeval E, 27. jaanuar 2020 kell 22:04:

> Greetings, friends,
>
> there is an idea to monitor high-volume event flows (per rules or log
> files, e.g. in bytes per time unit) from dump files from *periodic
> dumping*.
>
> The question is, if this is recommended for *production use* - answer
> depends at least on if dump creation somehow slows down or stops SEC
> processing, or it has not significant overhead (also for thousands of
> rules, complex configurations).
>
> Are there some best practises for (or experiences with) performance
> self-monitoring and tuning of SEC?
>
> Thank you.
>
> Richard
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>

[Simple-evcorr-users] SEC rules performance monitoring and tuning

[Richard O. <ric...@gm...>]

Greetings, friends,

there is an idea to monitor high-volume event flows (per rules or log
files, e.g. in bytes per time unit) from dump files from *periodic dumping*.

The question is, if this is recommended for *production use* - answer
depends at least on if dump creation somehow slows down or stops SEC
processing, or it has not significant overhead (also for thousands of
rules, complex configurations).

Are there some best practises for (or experiences with) performance
self-monitoring and tuning of SEC?

Thank you.

Richard

Re: [Simple-evcorr-users] SEC + AI (machine learning)

[Richard O. <ric...@gm...>]

Thank you, Risto, such overview can serve as starting point, when designing
something yet more useful, than static configurations of correlations.

I am still not sure, what the optimal goal should be, I need to analyze and
study more about these topics.

I believe, that I will follow up on information you provided, in the
future, and further develop this thread.

št 23. 1. 2020 o 18:12 Risto Vaarandi <ris...@gm...> napísal(a):

> hi Richard,
>
> Next step would be integrating AI (machine learning) with SEC somehow, so
>> that user won't need to configure correlations statically, but they would
>> configure and self-optimize automatically. (There still could be some input
>> needed from the user, but system would be also able to react on changing
>> log traffic, and self-evolve.)
>>
>> Something like ELK+AI has usable in the log monitoring area.
>>
>> Maybe some integration with MXNet?
>>
>> http://blogs.perl.org/users/sergey_kolychev/2017/02/machine-learning-in-perl.html
>>
>> Does anybody have any experience in this area, to explain some more or
>> less theoretical or practical setup of AI-generated SEC rules? (I am pretty
>> sure, that this is out of scope of SEC itself, and SEC would'nt know, that
>> AI is dynamically generating its rules on the background and probably
>> nobody has working solution, but maybe we could invent something together.)
>>
>>
> Machine learning is a very wide area with a large number of different
> methods and algorithms around. These methods and algorithms are usually
> divided into two large classes:
> *) supervised algorithms which assume that you provide labeled data for
> learning (for example, a log file where some messages are labeled as
> "normal" and some messages as "system_fault"), so that the algorithm can
> learn from labeled examples how to distinguish normal messages from errors
> (note that in this simplified example, only two labels were used, but in
> more complex cases you could have more labels in play)
> *) unsupervised algorithms which are able to distinguish anomalous or
> abnormal messages without any previous training with labeled data
> So my first question is -- what is your actual setup and do you have the
> opportunity of using training data for supervised methods, or are
> unsupervised methods a better choice? After answering this question, you
> can start studying most promising methods more closely.
>
> Secondly, what is your actual goal? Do you want to:
> 1) detect an individual anomalous message or a time frame containing
> anomalous messages from event logs,
> 2) produce a warning if the number of messages from specific class (e.g.
> login failures) per N minutes increases suddenly to an unexpectedly large
> value,
> 3) use some tool for (semi)automated mining of new SEC rules,
> 4) something else?
>
> For achieving first goal, there is no silver bullet, but perhaps I can
> provide few pointers to some relevant research papers (note that there are
> many other papers in this area):
> https://ieeexplore.ieee.org/document/4781208
> https://ieeexplore.ieee.org/document/7367332
> https://dl.acm.org/doi/10.1145/3133956.3134015
>
> For achieving the second goal, you could consider using time series
> analysis methods. You could begin with a very simple moving average based
> method like the one described here:
>
> https://machinelearnings.co/data-science-tricks-simple-anomaly-detection-for-metrics-with-a-weekly-pattern-2e236970d77
> or you could employ more complex forecasting methods (before starting, it
> is probably a good idea to read this book on forecasting:
> https://otexts.com/fpp2/)
>
> If you want to mine new rules or knowledge for SEC (or for other tools)
> from event logs, I have actually done some previous research in this
> domain. Perhaps I can point you to a log mining utility called LogCluster (
> https://ristov.github.io/logcluster/) which allows for mining line
> patterns and outliers from textual events logs. Also, couple of years ago,
> an experimental system was created which was using LogCluster in a fully
> automated way for creating SEC Suppress rules, where these rules were
> essentially matching normal (expected) messages. Any message not matching
> these rules was considered an anomaly and was logged separately for manual
> review. Here is the paper that provides an overview of this system:
> https://ristov.github.io/publications/noms18-log-anomaly-web.pdf
>
> Hopefully these pointers will offer you some guidance what your precise
> research question could be, and what is the most promising avenue for
> continuing. My apologies if my answer was raising new questions, but
> machine learning is a very wide area with large number of methods for many
> different goals.
>
> kind regards,
> risto
>
>

Re: [Simple-evcorr-users] SEC + AI (machine learning)

[Risto V. <ris...@gm...>]

hi Richard,

Next step would be integrating AI (machine learning) with SEC somehow, so
> that user won't need to configure correlations statically, but they would
> configure and self-optimize automatically. (There still could be some input
> needed from the user, but system would be also able to react on changing
> log traffic, and self-evolve.)
>
> Something like ELK+AI has usable in the log monitoring area.
>
> Maybe some integration with MXNet?
>
> http://blogs.perl.org/users/sergey_kolychev/2017/02/machine-learning-in-perl.html
>
> Does anybody have any experience in this area, to explain some more or
> less theoretical or practical setup of AI-generated SEC rules? (I am pretty
> sure, that this is out of scope of SEC itself, and SEC would'nt know, that
> AI is dynamically generating its rules on the background and probably
> nobody has working solution, but maybe we could invent something together.)
>
>
Machine learning is a very wide area with a large number of different
methods and algorithms around. These methods and algorithms are usually
divided into two large classes:
*) supervised algorithms which assume that you provide labeled data for
learning (for example, a log file where some messages are labeled as
"normal" and some messages as "system_fault"), so that the algorithm can
learn from labeled examples how to distinguish normal messages from errors
(note that in this simplified example, only two labels were used, but in
more complex cases you could have more labels in play)
*) unsupervised algorithms which are able to distinguish anomalous or
abnormal messages without any previous training with labeled data
So my first question is -- what is your actual setup and do you have the
opportunity of using training data for supervised methods, or are
unsupervised methods a better choice? After answering this question, you
can start studying most promising methods more closely.

Secondly, what is your actual goal? Do you want to:
1) detect an individual anomalous message or a time frame containing
anomalous messages from event logs,
2) produce a warning if the number of messages from specific class (e.g.
login failures) per N minutes increases suddenly to an unexpectedly large
value,
3) use some tool for (semi)automated mining of new SEC rules,
4) something else?

For achieving first goal, there is no silver bullet, but perhaps I can
provide few pointers to some relevant research papers (note that there are
many other papers in this area):
https://ieeexplore.ieee.org/document/4781208
https://ieeexplore.ieee.org/document/7367332
https://dl.acm.org/doi/10.1145/3133956.3134015

For achieving the second goal, you could consider using time series
analysis methods. You could begin with a very simple moving average based
method like the one described here:
https://machinelearnings.co/data-science-tricks-simple-anomaly-detection-for-metrics-with-a-weekly-pattern-2e236970d77
or you could employ more complex forecasting methods (before starting, it
is probably a good idea to read this book on forecasting:
https://otexts.com/fpp2/)

If you want to mine new rules or knowledge for SEC (or for other tools)
from event logs, I have actually done some previous research in this
domain. Perhaps I can point you to a log mining utility called LogCluster (
https://ristov.github.io/logcluster/) which allows for mining line patterns
and outliers from textual events logs. Also, couple of years ago, an
experimental system was created which was using LogCluster in a fully
automated way for creating SEC Suppress rules, where these rules were
essentially matching normal (expected) messages. Any message not matching
these rules was considered an anomaly and was logged separately for manual
review. Here is the paper that provides an overview of this system:
https://ristov.github.io/publications/noms18-log-anomaly-web.pdf

Hopefully these pointers will offer you some guidance what your precise
research question could be, and what is the most promising avenue for
continuing. My apologies if my answer was raising new questions, but
machine learning is a very wide area with large number of methods for many
different goals.

kind regards,
risto

Re: [Simple-evcorr-users] RegExp modifiers

[Risto V. <ris...@gm...>]

hi Richard,

thanks for the suggestion! I have added some examples about commenting
regular expressions directly into FAQ which is probably more convenient for
the reader:
https://simple-evcorr.github.io/FAQ.html#13

kind regards,
risto

Kontakt Richard Ostrochovský (<ric...@gm...>) kirjutas
kuupäeval E, 20. jaanuar 2020 kell 16:57:

> Thank you for comprehensive answer, Risto. Maybe, hyperlink to it could be
> added to that FAQ item.
>
> (I found searching in this forum bit harder, due to traditional e-mail
> from, and maybe we could categorize threads in this forum into topics on ,
> I am inspired with this by
> https://www.perlmonks.org/?node=Categorized%20Questions%20and%20Answers.
> Maybe I would help with something like this, when I will have less busy
> time.)
>
> po 9. 12. 2019 o 16:43 Risto Vaarandi <ris...@gm...>
> napísal(a):
>
>> hi Richard,
>>
>> Kontakt Richard Ostrochovský (<ric...@gm...>) kirjutas
>> kuupäeval E, 9. detsember 2019 kell 01:57:
>>
>>> Hello colleagues,
>>>
>>> I was searching for the answer here:
>>> https://simple-evcorr.github.io/man.html
>>> https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/
>>> and haven't found the answer, so I'am putting new question here:
>>>
>>> Does SEC in pattern= parameters support RegExp modifiers (
>>> https://perldoc.perl.org/perlre.html#Modifiers) somehow?
>>>
>>
>> If you enclose a regular expression within /.../, SEC does not treat
>> slashes as separators but rather as parts of regular expression, therefore
>> you can't provide modifiers in the end of regular expression after /.
>> However, Perl regular expressions allow for modifiers to be provided with
>> (?<modifiers>) construct. For example, the following pattern matches the
>> string "test" in case insensitive way:
>> pattern=(?i)test
>> In addition, you can use such modifiers anywhere in regular expression
>> which makes them more flexible than modifiers after /. For example, the
>> following pattern matches strings "test" and "tesT":
>> pattern=tes(?i)t
>>
>> In SEC FAQ, there is also a short discussion on this topic:
>> https://simple-evcorr.github.io/FAQ.html#13)
>>
>>
>>> E.g. modifiers /x or /xx allow writing more readable expressions by
>>> ignoring unescaped whitespaces (implies possible multi-line regular
>>> expressions). It could be practical in case of more complex expressions, to
>>> let them being typed more legibly. Some simpler example:
>>>
>>> pattern=/\
>>> ^\s*([A-Z]\s+)?\
>>> (?<data_source_timestamp>\
>>>    (\
>>>       ([\[\d\-\.\:\s\]]*[\d\]]) |\
>>>       (\
>>>          (Mon|Tue|Wed|Thu|Fri|Sat|Sun) \s+\
>>>          (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) \s+ \d+ \s+
>>> \d\d:\d\d:\d\d \s+ ([A-Z]+\s+)?\d\d\d\d\
>>>       )\
>>>    )\
>>> ) (?<message>.*)/x
>>>
>>>
>> It is a tricky question, since SEC configuration file format allows to
>> natively provide regular expressions in multiple lines without the use of
>> (?x) modifier. If any line in rule definition ends with backslash, the
>> following line is appended to the current line and backslash is removed
>> during configuration file parsing. For example, the following two pattern
>> definitions are equivalent:
>>
>> pattern=test: \
>> (\S+) \
>> (\S+)$
>>
>> pattern=test: (\S+) (\S+)$
>>
>> However, it is important to remember that SEC converts multi-line rule
>> fields into single-line format before any other processing, and that
>> includes compiling regular expressions. In other words, if you consider the
>> first multi-line regular expression pattern definition above, SEC actually
>> sees it as "test: (\S+) (\S+)$" when it compiles this expression. This
>> introduces the following caveat -- when using (?x) modifier for introducing
>> a comment into multi-line regular expression, the expression is converted
>> into single line format before expression is compiled and (?x) has any
>> effect, and therefore the comment will unexpectedly run until the end of
>> regular expression. Consider the following example:
>>
>> pattern=(?x)test:\
>> # this is a comment \
>> (\S+)$
>>
>> Internally, this definition is first converted to single line format:
>> pattern=(?x)test:# this is a comment (\S+)$
>> However, this means that without the comment the expression looks like
>> (?x)test:
>> which is not what we want.
>>
>> To address this issue, you can limit the scope of comments with (?#...)
>> constructs that don't require (?x). For example:
>>
>> pattern=test:\
>> (?# this is a comment )\
>> (\S+)$
>>
>> During configuration file parsing this expression is converted into
>> "test:(?# this is a comment )(\S+)$", and after dropping the comment it
>> becomes "test:(\S+)$" as we expect.
>>
>> Hope this helps,
>> risto
>>
>>
>>
>

Re: [Simple-evcorr-users] using original message in TValue rule action

[Risto V. <ris...@gm...>]

hi Richard,

there are several pattern types like TValue and SubStr which have been
designed for fast matching and which do not support match variables
(including $0). Handling of match variables involves additional
computational cost, since after successful match, all variables in rule
definition have to be substituted with values from the match. So if you
want to get the entire line that reaches a given rule, the simplest
solution is to use $0 variable with RegExp pattern .? or .
Since you also mentioned the idea of defining special action list variable
(%e), it will involve the following subtle issue. Unlike match variables,
action list variables get substituted not immediately after the match, but
at the moment when action list gets executed. However, there are rules
where action list execution can happen much later than the match against
the rule -- just consider the 'action' field of PairWithWindow rule where
execution is triggered by reading from system clock, not a pattern match. A
similar issue will come up with Calendar rule that doesn't have a pattern
at all, or action lists of SEC contexts when contexts expire. If the action
list variable is defined as "input line currently under processing", such
variable will not make much sense when action list execution is triggered
by system clock. For this reason, using $0 is a better solution, since it
is substituted immediately after a successful pattern match.

kind regards,
risto

Kontakt Richard Ostrochovský (<ric...@gm...>) kirjutas
kuupäeval E, 20. jaanuar 2020 kell 17:40:

> Hello,
>
> I was find out the answer in manual and also archive of this forum, but
> without success, and the question seems very basic to me, so I assume 2 (3)
> possible alternative answers:
>
> - it is so easy, that I will bang my head
> - it is not possible at all (in current version of SEC)
> - (RegExp .* is equally efficient as TValue)
>
> Assuming, that using TValue instead of RegExp or any other rule type in
> cases, where I don't need filtering of or extraction from log messages, is
> most computing power efficient, I am trying to find out a straightforward
> way, how to use the original event text in event action of TValue rule.
>
> $0 seems not to be working for TValue (I understand, that it is
> RegExp-specific) in rule like this:
>
> type=Single
> ptype=TValue
> pattern=TRUE
> context=SVC_:tmp::home:user:somelog.log#MULTI-LINE &&
> SVC_:tmp::home:user:somelog.log #MULTI-LINE_MESSAGE
> desc= SVC_:tmp::home:user:somelog.log #MULTI-LINE_MESSAGE lines filter
> action=add ( SVC_:tmp::home:user:somelog.log #MULTI-LINE_MESSAGE) $0
>
> $0 literally is added to context in this case.
>
> ("#" is meant to be part of the context name, not any kind of comment.)
>
> Does somebody have any advice, how to use original event text in
> TValue-type rule, without "compromising" the performance? (Assuming, that
> the easiest solution, the replacement of TValue with RegExp nad TRUE with
> .* would do the job, but won't be as fast as TValue.) Maybe new predefined
> variable could be available (e.g. %e as event) independently on rule type.
>
> Thank you in advance.
>
> Richard
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>

[Simple-evcorr-users] SEC + AI (machine learning)

[Richard O. <ric...@gm...>]

Hello,

this is free continuation of
https://sourceforge.net/p/simple-evcorr/mailman/message/36867012/. That
post was about possibilities of user-friendly configurations of event
correlations outside of SEC (without knowing SEC syntax and low-level
principles), and generation of SEC rules from that externalized
configurations. But still manual process.

Next step would be integrating AI (machine learning) with SEC somehow, so
that user won't need to configure correlations statically, but they would
configure and self-optimize automatically. (There still could be some input
needed from the user, but system would be also able to react on changing
log traffic, and self-evolve.)

Something like ELK+AI has usable in the log monitoring area.

Maybe some integration with MXNet?
http://blogs.perl.org/users/sergey_kolychev/2017/02/machine-learning-in-perl.html

Does anybody have any experience in this area, to explain some more or less
theoretical or practical setup of AI-generated SEC rules? (I am pretty
sure, that this is out of scope of SEC itself, and SEC would'nt know, that
AI is dynamically generating its rules on the background and probably
nobody has working solution, but maybe we could invent something together.)

Thanks for any ideas.

Richard

[Simple-evcorr-users] using original message in TValue rule action

[Richard O. <ric...@gm...>]

Hello,

I was find out the answer in manual and also archive of this forum, but
without success, and the question seems very basic to me, so I assume 2 (3)
possible alternative answers:

- it is so easy, that I will bang my head
- it is not possible at all (in current version of SEC)
- (RegExp .* is equally efficient as TValue)

Assuming, that using TValue instead of RegExp or any other rule type in
cases, where I don't need filtering of or extraction from log messages, is
most computing power efficient, I am trying to find out a straightforward
way, how to use the original event text in event action of TValue rule.

$0 seems not to be working for TValue (I understand, that it is
RegExp-specific) in rule like this:

type=Single
ptype=TValue
pattern=TRUE
context=SVC_:tmp::home:user:somelog.log#MULTI-LINE &&
SVC_:tmp::home:user:somelog.log #MULTI-LINE_MESSAGE
desc= SVC_:tmp::home:user:somelog.log #MULTI-LINE_MESSAGE lines filter
action=add ( SVC_:tmp::home:user:somelog.log #MULTI-LINE_MESSAGE) $0

$0 literally is added to context in this case.

("#" is meant to be part of the context name, not any kind of comment.)

Does somebody have any advice, how to use original event text in
TValue-type rule, without "compromising" the performance? (Assuming, that
the easiest solution, the replacement of TValue with RegExp nad TRUE with
.* would do the job, but won't be as fast as TValue.) Maybe new predefined
variable could be available (e.g. %e as event) independently on rule type.

Thank you in advance.

Richard

Re: [Simple-evcorr-users] RegExp modifiers

[Richard O. <ric...@gm...>]

Thank you for comprehensive answer, Risto. Maybe, hyperlink to it could be
added to that FAQ item.

(I found searching in this forum bit harder, due to traditional e-mail
from, and maybe we could categorize threads in this forum into topics on ,
I am inspired with this by
https://www.perlmonks.org/?node=Categorized%20Questions%20and%20Answers.
Maybe I would help with something like this, when I will have less busy
time.)

po 9. 12. 2019 o 16:43 Risto Vaarandi <ris...@gm...> napísal(a):

> hi Richard,
>
> Kontakt Richard Ostrochovský (<ric...@gm...>) kirjutas
> kuupäeval E, 9. detsember 2019 kell 01:57:
>
>> Hello colleagues,
>>
>> I was searching for the answer here:
>> https://simple-evcorr.github.io/man.html
>> https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/
>> and haven't found the answer, so I'am putting new question here:
>>
>> Does SEC in pattern= parameters support RegExp modifiers (
>> https://perldoc.perl.org/perlre.html#Modifiers) somehow?
>>
>
> If you enclose a regular expression within /.../, SEC does not treat
> slashes as separators but rather as parts of regular expression, therefore
> you can't provide modifiers in the end of regular expression after /.
> However, Perl regular expressions allow for modifiers to be provided with
> (?<modifiers>) construct. For example, the following pattern matches the
> string "test" in case insensitive way:
> pattern=(?i)test
> In addition, you can use such modifiers anywhere in regular expression
> which makes them more flexible than modifiers after /. For example, the
> following pattern matches strings "test" and "tesT":
> pattern=tes(?i)t
>
> In SEC FAQ, there is also a short discussion on this topic:
> https://simple-evcorr.github.io/FAQ.html#13)
>
>
>> E.g. modifiers /x or /xx allow writing more readable expressions by
>> ignoring unescaped whitespaces (implies possible multi-line regular
>> expressions). It could be practical in case of more complex expressions, to
>> let them being typed more legibly. Some simpler example:
>>
>> pattern=/\
>> ^\s*([A-Z]\s+)?\
>> (?<data_source_timestamp>\
>>    (\
>>       ([\[\d\-\.\:\s\]]*[\d\]]) |\
>>       (\
>>          (Mon|Tue|Wed|Thu|Fri|Sat|Sun) \s+\
>>          (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) \s+ \d+ \s+
>> \d\d:\d\d:\d\d \s+ ([A-Z]+\s+)?\d\d\d\d\
>>       )\
>>    )\
>> ) (?<message>.*)/x
>>
>>
> It is a tricky question, since SEC configuration file format allows to
> natively provide regular expressions in multiple lines without the use of
> (?x) modifier. If any line in rule definition ends with backslash, the
> following line is appended to the current line and backslash is removed
> during configuration file parsing. For example, the following two pattern
> definitions are equivalent:
>
> pattern=test: \
> (\S+) \
> (\S+)$
>
> pattern=test: (\S+) (\S+)$
>
> However, it is important to remember that SEC converts multi-line rule
> fields into single-line format before any other processing, and that
> includes compiling regular expressions. In other words, if you consider the
> first multi-line regular expression pattern definition above, SEC actually
> sees it as "test: (\S+) (\S+)$" when it compiles this expression. This
> introduces the following caveat -- when using (?x) modifier for introducing
> a comment into multi-line regular expression, the expression is converted
> into single line format before expression is compiled and (?x) has any
> effect, and therefore the comment will unexpectedly run until the end of
> regular expression. Consider the following example:
>
> pattern=(?x)test:\
> # this is a comment \
> (\S+)$
>
> Internally, this definition is first converted to single line format:
> pattern=(?x)test:# this is a comment (\S+)$
> However, this means that without the comment the expression looks like
> (?x)test:
> which is not what we want.
>
> To address this issue, you can limit the scope of comments with (?#...)
> constructs that don't require (?x). For example:
>
> pattern=test:\
> (?# this is a comment )\
> (\S+)$
>
> During configuration file parsing this expression is converted into
> "test:(?# this is a comment )(\S+)$", and after dropping the comment it
> becomes "test:(\S+)$" as we expect.
>
> Hope this helps,
> risto
>
>
>

Re: [Simple-evcorr-users] Public Docker image for SEC

[Risto V. <ris...@gm...>]

hi Andres,

Added separate log feature. Write actions are written to container standard
> output. SEC logs are written to `/app/logs/sec.log`. To manage SEC logs
> then mount directory to host system `-v logs:/app/logs`. Probably log
> rotation can be done on the host machine side.
>

In order to properly rotate SEC logs, one needs to deliver USR2 signal to
SEC process during log rotation which forces the process to create a new
log file. But unfortunately the container doesn't seem to have logrotate
installed :(

For rotating the log on host machine, one needs to get SEC process ID from
container which normally requires --pid command line option (or some
scripting). However, there is also another approach -- since rsyslog starts
SEC with hup.signal="USR2" option, rsyslog delivers USR2 signal to SEC
process when rsyslog receives HUP during its own log rotation. On the other
hand, your container starts rsyslog with ENTRYPOINT directive which assigns
pid 1 to it. Therefore, you can execute the following command on host
machine when SEC log file is rotated:
docker exec <containerid> kill -1 1

I would personally handle log rotation inside the container, though, in
order to avoid creating additional dependencies on host machine. If you
don't want to install logrotate (or some other package) into container, you
can also configure SEC to rotate its own log with 'sigemul' action. For
example:

type=Calendar
time=0 0 * * *
desc=rotate our own log at midnight
action=lcall %o -> ( sub { rename("/app/logs/sec.log",
"/app/logs/sec.log.old") } ); \
           sigemul USR2

Hope this helps,
risto

Re: [Simple-evcorr-users] Public Docker image for SEC

[Andres P. <ap...@gm...>]

Added separate log feature. Write actions are written to container standard
output. SEC logs are written to `/app/logs/sec.log`. To manage SEC logs
then mount directory to host system `-v logs:/app/logs`. Probably log
rotation can be done on the host machine side.

In my usecase docker has been really useful in testing environment when I
need to test different rules. Deploying standard solution fast and easy is
the main benefit for using it. Nevertheless, rsyslog can be configured with
encryption also. I added simplest configuration I could.

Regards,

On Mon, Dec 16, 2019 at 5:22 PM Risto Vaarandi <ris...@gm...>
wrote:

> hi Andres,
> so far, official sec distribution has not had a docker image, since sec is
> packaged for common linux and bsd distributions, and it doesn't have many
> dependencies (just standard perl is needed without any exotic modules).
> That has made sec very easy to deploy.
> I had a quick look into the repository and it seems that the container
> runs sec as a process connected to rsyslog, with rsyslog acting as event
> collector for sec. Is my understanding correct? If so, is that container
> targeted for collecting logs in a small trusted network segment where
> encryption is not needed? I noticed that rsyslog has been configured to
> receive logs in plain text, hence the question.
> kind regards,
> risto
>
> Kontakt Andres Pihlak (<ap...@gm...>) kirjutas kuupäeval E, 16.
> detsember 2019 kell 12:40:
>
>> Hello,
>>
>> I had a need for SEC docker container because it makes life much easier.
>> Unfortunately, there isn't public image for that so I created it myself.
>> Furthermore, I added CI pipeline to build those docker images. Repository
>> is here: https://github.com/apihlak/SEC. Is there plan to add official
>> Dockerfile and image to simple-evcorr repository or is there any
>> suggestions to make my repository better to help out community?
>>
>> Regards,
>> Andres
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> Sim...@li...
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>

Re: [Simple-evcorr-users] Public Docker image for SEC

[Risto V. <ris...@gm...>]

hi Andres,
so far, official sec distribution has not had a docker image, since sec is
packaged for common linux and bsd distributions, and it doesn't have many
dependencies (just standard perl is needed without any exotic modules).
That has made sec very easy to deploy.
I had a quick look into the repository and it seems that the container runs
sec as a process connected to rsyslog, with rsyslog acting as event
collector for sec. Is my understanding correct? If so, is that container
targeted for collecting logs in a small trusted network segment where
encryption is not needed? I noticed that rsyslog has been configured to
receive logs in plain text, hence the question.
kind regards,
risto

Kontakt Andres Pihlak (<ap...@gm...>) kirjutas kuupäeval E, 16.
detsember 2019 kell 12:40:

> Hello,
>
> I had a need for SEC docker container because it makes life much easier.
> Unfortunately, there isn't public image for that so I created it myself.
> Furthermore, I added CI pipeline to build those docker images. Repository
> is here: https://github.com/apihlak/SEC. Is there plan to add official
> Dockerfile and image to simple-evcorr repository or is there any
> suggestions to make my repository better to help out community?
>
> Regards,
> Andres
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>

Re: [Simple-evcorr-users] Public Docker image for SEC

[Alberto C. P. <ac...@s2...>]

Hi,

Just curious. Are you making available the logs inside the container as a volume (bind mount) or are you reading log files already inside the container? Also, are you using log rotation?

Regards,

El 16/12/19 a las 11:39, Andres Pihlak escribió:
Hello,

I had a need for SEC docker container because it makes life much easier. Unfortunately, there isn't public image for that so I created it myself. Furthermore, I added CI pipeline to build those docker images. Repository is here: https://github.com/apihlak/SEC. Is there plan to add official Dockerfile and image to simple-evcorr repository or is there any suggestions to make my repository better to help out community?

Regards,
Andres




_______________________________________________
Simple-evcorr-users mailing list
Sim...@li...<mailto:Sim...@li...>
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users


--

[Simple-evcorr-users] Public Docker image for SEC

[Andres P. <ap...@gm...>]

Hello,

I had a need for SEC docker container because it makes life much easier.
Unfortunately, there isn't public image for that so I created it myself.
Furthermore, I added CI pipeline to build those docker images. Repository
is here: https://github.com/apihlak/SEC. Is there plan to add official
Dockerfile and image to simple-evcorr repository or is there any
suggestions to make my repository better to help out community?

Regards,
Andres

Re: [Simple-evcorr-users] RegExp modifiers

[Risto V. <ris...@gm...>]

hi Richard,

Kontakt Richard Ostrochovský (<ric...@gm...>) kirjutas
kuupäeval E, 9. detsember 2019 kell 01:57:

> Hello colleagues,
>
> I was searching for the answer here:
> https://simple-evcorr.github.io/man.html
> https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/
> and haven't found the answer, so I'am putting new question here:
>
> Does SEC in pattern= parameters support RegExp modifiers (
> https://perldoc.perl.org/perlre.html#Modifiers) somehow?
>

If you enclose a regular expression within /.../, SEC does not treat
slashes as separators but rather as parts of regular expression, therefore
you can't provide modifiers in the end of regular expression after /.
However, Perl regular expressions allow for modifiers to be provided with
(?<modifiers>) construct. For example, the following pattern matches the
string "test" in case insensitive way:
pattern=(?i)test
In addition, you can use such modifiers anywhere in regular expression
which makes them more flexible than modifiers after /. For example, the
following pattern matches strings "test" and "tesT":
pattern=tes(?i)t

In SEC FAQ, there is also a short discussion on this topic:
https://simple-evcorr.github.io/FAQ.html#13)


> E.g. modifiers /x or /xx allow writing more readable expressions by
> ignoring unescaped whitespaces (implies possible multi-line regular
> expressions). It could be practical in case of more complex expressions, to
> let them being typed more legibly. Some simpler example:
>
> pattern=/\
> ^\s*([A-Z]\s+)?\
> (?<data_source_timestamp>\
>    (\
>       ([\[\d\-\.\:\s\]]*[\d\]]) |\
>       (\
>          (Mon|Tue|Wed|Thu|Fri|Sat|Sun) \s+\
>          (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) \s+ \d+ \s+
> \d\d:\d\d:\d\d \s+ ([A-Z]+\s+)?\d\d\d\d\
>       )\
>    )\
> ) (?<message>.*)/x
>
>
It is a tricky question, since SEC configuration file format allows to
natively provide regular expressions in multiple lines without the use of
(?x) modifier. If any line in rule definition ends with backslash, the
following line is appended to the current line and backslash is removed
during configuration file parsing. For example, the following two pattern
definitions are equivalent:

pattern=test: \
(\S+) \
(\S+)$

pattern=test: (\S+) (\S+)$

However, it is important to remember that SEC converts multi-line rule
fields into single-line format before any other processing, and that
includes compiling regular expressions. In other words, if you consider the
first multi-line regular expression pattern definition above, SEC actually
sees it as "test: (\S+) (\S+)$" when it compiles this expression. This
introduces the following caveat -- when using (?x) modifier for introducing
a comment into multi-line regular expression, the expression is converted
into single line format before expression is compiled and (?x) has any
effect, and therefore the comment will unexpectedly run until the end of
regular expression. Consider the following example:

pattern=(?x)test:\
# this is a comment \
(\S+)$

Internally, this definition is first converted to single line format:
pattern=(?x)test:# this is a comment (\S+)$
However, this means that without the comment the expression looks like
(?x)test:
which is not what we want.

To address this issue, you can limit the scope of comments with (?#...)
constructs that don't require (?x). For example:

pattern=test:\
(?# this is a comment )\
(\S+)$

During configuration file parsing this expression is converted into
"test:(?# this is a comment )(\S+)$", and after dropping the comment it
becomes "test:(\S+)$" as we expect.

Hope this helps,
risto

[Simple-evcorr-users] RegExp modifiers

[Richard O. <ric...@gm...>]

Hello colleagues,

I was searching for the answer here:
https://simple-evcorr.github.io/man.html
https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/
and haven't found the answer, so I'am putting new question here:

Does SEC in pattern= parameters support RegExp modifiers (
https://perldoc.perl.org/perlre.html#Modifiers) somehow?

E.g. modifiers /x or /xx allow writing more readable expressions by
ignoring unescaped whitespaces (implies possible multi-line regular
expressions). It could be practical in case of more complex expressions, to
let them being typed more legibly. Some simpler example:

pattern=/\
^\s*([A-Z]\s+)?\
(?<data_source_timestamp>\
   (\
      ([\[\d\-\.\:\s\]]*[\d\]]) |\
      (\
         (Mon|Tue|Wed|Thu|Fri|Sat|Sun) \s+\
         (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) \s+ \d+ \s+
\d\d:\d\d:\d\d \s+ ([A-Z]+\s+)?\d\d\d\d\
      )\
   )\
) (?<message>.*)/x

Thank you.

Richard

Re: [Simple-evcorr-users] generate SEC configurations automatically

[Richard O. <ric...@gm...>]

po 2. 12. 2019 o 14:19 Risto Vaarandi <ris...@gm...> napísal(a):

>
>> This SEC admin, as I see it, is still also from user perspective, tightly
>> bound with SEC technology and user needs to know, how SEC rules and their
>> compositions work internally. I am imagining something, that is
>> technology-agnostic, just describing logic of correlations, and from that
>> "formal description" generating SEC (or something other,
>> technology-agnostic means, that it may not to be dependent on SEC or other
>> specific technology, user needs just "what", but not "how"). Therefore
>> configuration DB should not operate with SEC-specific entities, but rather
>> something more general. Maybe something, which could be described with
>> something like EMML (Event Management Markup Language - never used that, I
>> don't know it, I just found that it exists).
>>
>> Levels of formal description of correlation types:
>>
>> - *user (admin) level* (higher-level, less detailed) - containing enough
>> information to configure and use described correlation type
>>
>> - *developer level* (lower-level, more detailed) - containing enough
>> information, to generate lower-level code (SEC configurations) from formal
>> description (from configuration DB, maybe with support of something like
>> EMML)
>>
>> Hope this helps to undertand what I wanted to say :).
>>
>
> Developing another event correlation language and tools for supporting it
> sounds like a separate major software development project to me.
> Before embarking on that, have you considered using configuration
> management frameworks (e.g., Salt or Ansible)? Among other things, they
> allow you to create configuration files in automated fashion after the user
> has set some generic parameters. It might not be exactly what you have
> envisioned, but it is definitely cheaper workaround than rolling out an
> entirely new solution.
>

Thank you for sharing another look, Risto. I know Ansible, Salt and other
tools of this category, and I did not thought about  this possibility yet,
maybe it would be worth of it.

About separate major software development project (related to SEC or maybe
other event correlation engines): I see it this way too. Such project would
need not only core development of "configuration framework", but also
living community, submitting their "correlation modules" in specified
format (e.g. some kind of SEC configuration "templates"). Ideologically
something, as you have here: https://github.com/simple-evcorr/rulesets/,
plus some additional functionalities around:

   - interface for submitting correlation modules (for review, because
   submitted modules should be verified before publication on framework site)

   - catalogue of submitted (and verified) correlation modules, e.g. as
   table or set of tables (structured by some categories), containing
   information:

   - description WHAT module does (user perspective)

      - (HOW is that functionality implemented is not needed here,
      reviewer(s) of the module could see it in bundled technical
documentation,
      source code and comments, but I see these implementation details as not
      needed for publishing in such catalogue... maybe there may be also more
      alternative implementations of the same correlation algorithms, having
      their specifics)

      - high-level picture of correlation algorithm (without low-level
      concepts, e.g. SEC-related)

      - list of configurable parameters with their types (constraints)

      - event source preconditions (format, etc.)

 Richard

Re: [Simple-evcorr-users] generate SEC configurations automatically

[Risto V. <ris...@gm...>]

>
>
> This SEC admin, as I see it, is still also from user perspective, tightly
> bound with SEC technology and user needs to know, how SEC rules and their
> compositions work internally. I am imagining something, that is
> technology-agnostic, just describing logic of correlations, and from that
> "formal description" generating SEC (or something other,
> technology-agnostic means, that it may not to be dependent on SEC or other
> specific technology, user needs just "what", but not "how"). Therefore
> configuration DB should not operate with SEC-specific entities, but rather
> something more general. Maybe something, which could be described with
> something like EMML (Event Management Markup Language - never used that, I
> don't know it, I just found that it exists).
>
> Levels of formal description of correlation types:
>
> - *user (admin) level* (higher-level, less detailed) - containing enough
> information to configure and use described correlation type
>
> - *developer level* (lower-level, more detailed) - containing enough
> information, to generate lower-level code (SEC configurations) from formal
> description (from configuration DB, maybe with support of something like
> EMML)
>
> Hope this helps to undertand what I wanted to say :).
>

Developing another event correlation language and tools for supporting it
sounds like a separate major software development project to me.
Before embarking on that, have you considered using configuration
management frameworks (e.g., Salt or Ansible)? Among other things, they
allow you to create configuration files in automated fashion after the user
has set some generic parameters. It might not be exactly what you have
envisioned, but it is definitely cheaper workaround than rolling out an
entirely new solution.

kind regards,
risto


> Richard
>
>>
>> hope this helps,
>> risto
>>
>

Re: [Simple-evcorr-users] generate SEC configurations automatically

[Richard O. <ric...@gm...>]

hi John and Risto,

Being a "regular" administrator, I claim admins
>> that can't program/use programming methods won't
>> be admins much longer. If the companies you work
>> for are depending on (rapidly turning over) junior
>> admins to administer something as important as
>> monitoring and correlation you have a difficult
>> job ahead of you.
>>
>
Sometimes yes. We see this trend "everywhere" - the more are computers (and
smartphones) doing instead of us, the less we need to know. We see this in
schools and also in IT business, it is getting harder and harder to hire
personnel qualified to do everything needed, and when you train somebody,
then he or she can easily go away (because there is much more
opportunities, than qualified people), and you can train somebody else, and
the loop continues. Or there is a possiblity to have smarter technologies,
compensating less skilled and experienced personnel, and this is the way I
see, (un?)fortunately...

>
>> Knowing regular expressions at the very least is
>> required to use SEC.
>
>
> I agree with John's opinion. Also, regular expressions are not something
> specific to SEC, but they are used by many other event log processing and
> network monitoring tools such as syslog-ng, rsyslog, suricata, etc. While
> learning regular expressions requires some time from a newcomer, it is
> difficult to see how a monitoring engineer/admin would manage without this
> knowledge.
>

I agree too. Getting familiar with regular expressions is not that hard, as
it resembles file path wildcards which are notoricaly familiar (but regular
expressions are of course more powerful), and it is not that hard to
explain basics of regular expression also to unexperienced newcomer, how to
match simple / basic patterns. There are also a lot of self-teaching
materials to this topic, online tutorials and converters / checkers, as
they are widely used and therefore harder to avoid. For me, knowing regular
expession basics are definitely basics.

For example we can have HPOM admins, using simplified "regular expressions"
by HP:
http://www.softpanorama.org/Admin/HP_operations_manager/Policies/Conditions/patterns.shtml.
Not that hard to comprehend. Nor basics of Perl regular expressions, and
this also enables to gradually becoming regular expressions expert, step by
step, and use this knowledge in various domains. As I see it, it is more
about pattern matching, than about true programming.

SEC, as I see it, is another story, as there are layers of more
sophistication, building (not only) on regular expressions, but joining
that with event correlations algorithmization and (Perl) programming. I see
this much more demanding, than just regular expressions. The area for SEC
use is much narrower, than area of use of regular regular expressions, and
that may mean more motivation for successful avoiding learning it, with
greater chance, than one won't need it at the end (before he or she changes
work position, and then the knowledge remains in vacuum).


> I would also argue that creating more advanced event correlation schemes
> (not only for SEC, but for any other event correlation tool) definitely
> requires at least some development skills, since one has to essentially
> write down event processing algorithms here.
>

Also agree. I don't suggest total avoiding programming, but "hiding"
(packaging) it from people, that don't need it, but still being able to use
it. When I know what it does, I don't always need also know how it does it.
To ride a car, I don't need to be a car designer or motor mechanic. When I
want a car, I just configure and order it, I don't need to design, develop,
and manufacture it. I use it without that knowledge, but when I want to
repair, upgrade, something, or buy a new car, I am contacting specialist
again.

>
> Having said that, there are certainly ways to improve future versions of
> SEC -- for example, re-evaluating input file patterns after short time
> intervals (suggestion from one of the previous posts) is worth considering
> and I've written it down as an idea for the next version.
>
> Getting performance info out
>> of SEC is better, but still
>> difficult. E.G. finding and fixing expensive/poor
>> regular expressions can result in a significant
>> improvement of performance/throughput along with
>> hierarchical structuring of the rulesets.
>>
>> >Imagined concept illustration:
>> >
>> >[configuration DB] -> [generator(s)] -> [SEC configurations]
>> >
>> >The punch line is, that user won't need to know
>> >anything about SEC, but will need to understand
>> >logic of correlations employed, and their
>> >parameters
>>
>> I assume you mean regular expressions, threholds,
>> actions and commands etc.
>>
>
I meant:

   - *correlation type* (selected from some catalogue, where it is
   described, what it does, without needing to know how it does it... for
   example SEC supports 12 atomic rule types, but there are infinite
   possiblilities, what can I compose from them, and that is about that
   ever-growing catalogue... not reinventing the same logic every time, but
   reuse)

   - *regular expressions* (input/condition pattern, output pattern, line
   delimiter matching pattern, or message beginning/end matching pattern)

   - thresholds (and other *parameters* of chosen correlation)

   - *log file path* (or wildcard pattern matching more files)

   - maybe something other...

SEC actions and commands are quite low-level, and could be hidden from the
user in this concept.

The requirement is, that user needs to have some "collection of correlation
types" (catalogue) to choose from, and when no one suits his needs, then he
or she will call developer, to create that correlation type for him/her (or
download it from somewhere).

I am talking about some "framework", allowing to import correlation types
as some kind of "modules", and then to use them. And development would be
about creating modules, but using them not.

>
>> >(configuration DB may have some kind of GUI). In
>> >the background, higher-level correlations will be
>> >translated to respective SEC rules.
>>
>
> It is certainly possible to create such a configuration DB for a number of
> well known scenarios. For example, one could set up a database for common
> Cisco fault messages (maybe using examples from here:
> https://github.com/simple-evcorr/rulesets/blob/master/cisco/cisco-syslog.sec),
> and use it for automated creation of some specific rules, where the end
> user can change few simpler parameters (e.g., time windows and thresholds
> of event correlation rules).
>

Yes, this I see 100% compatible with my suggestion.


> However, it is also important to understand the limitations of this
> approach -- whenever new event processing algorithms need to be written for
> new scenarios and event log types, someone with developer skills needs to
> step in.
>

Yes, certainly.


> If you want to move the entire development work into GUI,
>

No, I don't want to move entire development work into GUI. I see
impossibility of that, resp. then it would be nothing else, than also some
kind of (visual) programming.


> maybe secadmin package that John has discussed below will provide some
> ideas?
>
>
>> There was a web interface referenced back in 2007
>> at:
>>
>>
>> https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/thread/op.tnxvfselqmj61d%40genius/#msg6200380
>>
>> The url's are dead but I have a copy of secadmin
>> from 2009 I have put it up at:
>>
>>   https://www.cs.umb.edu/~rouilj/sec/secadmin.tar.gz
>>
>> It doesn't use a back end db IIRC but it was
>> supposed to provide some guidance on creating the
>> correlation rules. Note that it would have to be
>> updated as new rules have been added since it was
>> developed. Also I think it ily supported the basic
>> sec correlations.
>>
>> Risto do you remember this?
>>
>
> After checking the link, I do recollect this post, but didn't have time to
> closely study this package at a time. However, I downloaded this package
> today and looked into it on a test virtual machine. This package is
> essentially a CGI script written for Apache web server, and despite being
> rolled out in 2006, it found it working on centos7 platform after few minor
> tweaks. The package offers a web based GUI for editing SEC rules, where
> most rule fields can be defined in text boxes, while values for some fields
> (e.g., rule type) can be selected from pull-down menus. I think it's a nice
> package for editing rules via web based interface, but one definitely has
> to update for newer SEC versions (since the GCI script was created in
> November 2006, it works with 2.4.X versions, while most recent sec-2.8.X
> contains many improvements over 2.4.X).
>

This SEC admin, as I see it, is still also from user perspective, tightly
bound with SEC technology and user needs to know, how SEC rules and their
compositions work internally. I am imagining something, that is
technology-agnostic, just describing logic of correlations, and from that
"formal description" generating SEC (or something other,
technology-agnostic means, that it may not to be dependent on SEC or other
specific technology, user needs just "what", but not "how"). Therefore
configuration DB should not operate with SEC-specific entities, but rather
something more general. Maybe something, which could be described with
something like EMML (Event Management Markup Language - never used that, I
don't know it, I just found that it exists).

Levels of formal description of correlation types:

- *user (admin) level* (higher-level, less detailed) - containing enough
information to configure and use described correlation type

- *developer level* (lower-level, more detailed) - containing enough
information, to generate lower-level code (SEC configurations) from formal
description (from configuration DB, maybe with support of something like
EMML)

Hope this helps to undertand what I wanted to say :).

Richard

>
> hope this helps,
> risto
>

Re: [Simple-evcorr-users] generate SEC configurations automatically

[Risto V. <ris...@gm...>]

hi Richard and John,

Hi Richard:
>
> In message
> <CAJ...@ma...>,
> Richard_Ostrochovsk writes:
> >this post loosely follows this one:
> >https://sourceforge.net/p/simple-evcorr/mailman/message/36867007/.
> >
> >Being monitoring consultant and developer, I have
> >an idea to hide complexity of SEC configurations,
> >and still to allow to configure it also for
> >"regular" administrators without any developer or
> >SEC background.
>
> Being a "regular" administrator, I claim admins
> that can't program/use programming methods won't
> be admins much longer. If the companies you work
> for are depending on (rapidly turning over) junior
> admins to administer something as important as
> monitoring and correlation you have a difficult
> job ahead of you.
>
> Knowing regular expressions at the very least is
> required to use SEC.


I agree with John's opinion. Also, regular expressions are not something
specific to SEC, but they are used by many other event log processing and
network monitoring tools such as syslog-ng, rsyslog, suricata, etc. While
learning regular expressions requires some time from a newcomer, it is
difficult to see how a monitoring engineer/admin would manage without this
knowledge. I would also argue that creating more advanced event correlation
schemes (not only for SEC, but for any other event correlation tool)
definitely requires at least some development skills, since one has to
essentially write down event processing algorithms here.

Having said that, there are certainly ways to improve future versions of
SEC -- for example, re-evaluating input file patterns after short time
intervals (suggestion from one of the previous posts) is worth considering
and I've written it down as an idea for the next version.

Getting performance info out
> of SEC is better, but still
> difficult. E.G. finding and fixing expensive/poor
> regular expressions can result in a significant
> improvement of performance/throughput along with
> hierarchical structuring of the rulesets.
>
> >Imagined concept illustration:
> >
> >[configuration DB] -> [generator(s)] -> [SEC configurations]
> >
> >The punch line is, that user won't need to know
> >anything about SEC, but will need to understand
> >logic of correlations employed, and their
> >parameters
>
> I assume you mean regular expressions, threholds,
> actions and commands etc.
>
> >(configuration DB may have some kind of GUI). In
> >the background, higher-level correlations will be
> >translated to respective SEC rules.
>

It is certainly possible to create such a configuration DB for a number of
well known scenarios. For example, one could set up a database for common
Cisco fault messages (maybe using examples from here:
https://github.com/simple-evcorr/rulesets/blob/master/cisco/cisco-syslog.sec),
and use it for automated creation of some specific rules, where the end
user can change few simpler parameters (e.g., time windows and thresholds
of event correlation rules). However, it is also important to understand
the limitations of this approach -- whenever new event processing
algorithms need to be written for new scenarios and event log types,
someone with developer skills needs to step in. If you want to move the
entire development work into GUI, maybe secadmin package that John has
discussed below will provide some ideas?


> There was a web interface referenced back in 2007
> at:
>
>
> https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/thread/op.tnxvfselqmj61d%40genius/#msg6200380
>
> The url's are dead but I have a copy of secadmin
> from 2009 I have put it up at:
>
>   https://www.cs.umb.edu/~rouilj/sec/secadmin.tar.gz
>
> It doesn't use a back end db IIRC but it was
> supposed to provide some guidance on creating the
> correlation rules. Note that it would have to be
> updated as new rules have been added since it was
> developed. Also I think it ily supported the basic
> sec correlations.
>
> Risto do you remember this?
>

After checking the link, I do recollect this post, but didn't have time to
closely study this package at a time. However, I downloaded this package
today and looked into it on a test virtual machine. This package is
essentially a CGI script written for Apache web server, and despite being
rolled out in 2006, it found it working on centos7 platform after few minor
tweaks. The package offers a web based GUI for editing SEC rules, where
most rule fields can be defined in text boxes, while values for some fields
(e.g., rule type) can be selected from pull-down menus. I think it's a nice
package for editing rules via web based interface, but one definitely has
to update for newer SEC versions (since the GCI script was created in
November 2006, it works with 2.4.X versions, while most recent sec-2.8.X
contains many improvements over 2.4.X).

hope this helps,
risto

Re: [Simple-evcorr-users] generate SEC configurations automatically

[John P. R. <ro...@cs...>]

Hi Richard:

In message
<CAJ...@ma...>,
Richard_Ostrochovsk writes:
>this post loosely follows this one:
>https://sourceforge.net/p/simple-evcorr/mailman/message/36867007/.
>
>Being monitoring consultant and developer, I have
>an idea to hide complexity of SEC configurations,
>and still to allow to configure it also for
>"regular" administrators without any developer or
>SEC background.

Being a "regular" administrator, I claim admins
that can't program/use programming methods won't
be admins much longer. If the companies you work
for are depending on (rapidly turning over) junior
admins to administer something as important as
monitoring and correlation you have a difficult
job ahead of you.

Knowing regular expressions at the very least is
required to use SEC. Getting performance info out
of SEC is better, but still
difficult. E.G. finding and fixing expensive/poor
regular expressions can result in a significant
improvement of performance/throughput along with
hierarchical structuring of the rulesets.

>Imagined concept illustration:
>
>[configuration DB] -> [generator(s)] -> [SEC configurations]
>
>The punch line is, that user won't need to know
>anything about SEC, but will need to understand
>logic of correlations employed, and their
>parameters

I assume you mean regular expressions, threholds,
actions and commands etc.

>(configuration DB may have some kind of GUI). In
>the background, higher-level correlations will be
>translated to respective SEC rules.

There was a web interface referenced back in 2007
at:

https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/thread/op.tnxvfselqmj61d%40genius/#msg6200380

The url's are dead but I have a copy of secadmin
from 2009 I have put it up at:

  https://www.cs.umb.edu/~rouilj/sec/secadmin.tar.gz

It doesn't use a back end db IIRC but it was
supposed to provide some guidance on creating the
correlation rules. Note that it would have to be
updated as new rules have been added since it was
developed. Also I think it ily supported the basic
sec correlations.

Risto do you remember this?

>Maybe there exists something similar as described
>- if somebody knows about something, I'd like if
>he or she will navigate me to it. If it does not
>exist, maybe this is potential opportunity for
>implementation, and this way also SEC could be
>more propagated, as still alive alternative to
>other newer solutions usable for event
>correlations, e.g. based on ELK (I see big
>advantage of SEC, that it does not need separate
>application infrastructure for log collection and
>processing).  Any opinions about this topic?

One thing I had played with was using templates to
create new correlation types by deploying a set of
basic correlations (pair, single, and single
with threshold for example) into a single unit
tied together by specifying fill-in parameters
(regular expressions, threshold counts etc.). I
used filepp to expand the correlation files into
something that sec could consume.

This was done at a previous employer. I don't
think I have any of that work anymore. But it was
able to generate more complex correlations with a
simpler interface for others to fill in.

Maybe this provides some ideas?

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

[Simple-evcorr-users] generate SEC configurations automatically

[Richard O. <ric...@gm...>]

Hello friends,

this post loosely follows this one:
https://sourceforge.net/p/simple-evcorr/mailman/message/36867007/.

Being monitoring consultant and developer, I have an idea to hide
complexity of SEC configurations, and still to allow to configure it also
for "regular" administrators without any developer or SEC background.

Imagined concept illustration:

[configuration DB] -> [generator(s)] -> [SEC configurations]

The punch line is, that user won't need to know anything about SEC, but
will need to understand logic of correlations employed, and their
parameters (configuration DB may have some kind of GUI). In the background,
higher-level correlations will be translated to respective SEC rules.

Maybe there exists something similar as described - if somebody knows about
something, I'd like if he or she will navigate me to it. If it does not
exist, maybe this is potential opportunity for implementation, and this way
also SEC could be more propagated, as still alive alternative to other
newer solutions usable for event correlations, e.g. based on ELK (I see big
advantage of SEC, that it does not need separate application infrastructure
for log collection and processing).

Any opinions about this topic?

Richard

Re: [Simple-evcorr-users] "multi-line" and multi-file logs - out of box

[Richard O. <ric...@gm...>]

hi Risto,

thank you for sharing. You are undoubtedly right, it is common, that
"delimiter" pattern uses to be beginning part of the message (we can't say
this about "\n", which is not "part of the beginning"... delimiting the end
of message is then optional, in cases where detecting the beginning of the
next message is satisfactory), so my original "specification" wasn't so
useful, as it could be.

Considering myself a Perl developer too, I could surely implement something
more or less as you suggested (and I probably will). But I consider also
important to explain, why I am asking such questions.

At first:

   - I like, what SEC does
   - I like, how SEC does it (although, it is not for "everybody", as
   describing below)

I operate as monitoring consultant and developer. My job is to implement
monitoring solutions and hand over them in documented state to "operations
guys" for further administration. These guys often are not developers, but
SEC configurations are more about programming, than about plain
configurations as "regular" administrators use to do (as I see it). At
first sight, good for me, because when those guys need to change something
and configuring SEC is out of their area of expertise, they depend on me,
and I have the next job... but: this is not the way I think, and I am
really trying to built such solutions, which will be comprehensible and
maintainable also by "regular" administrators by keeping them as simple as
possible, and then I can focus more on higher-level solutions, with more
added value, when I save some of my time by not doing these low level
configurations repeatedly for them (a prefer teaching fishing than selling
fishes). There is a lot of human resources turnover in IT, and SEC training
for its administrator's replacement sounds as science-fiction for me.
Therefore I am suggesting to be able to do most common things out-of-box
from command line options or some basic configuration, without need to
configure rules for everything (yes, I know, it is not easy nor clear to
tell, which things are "most common", as you said, infinite possibilities
:) ). This is also related with my next post (which am I currently
writing), maybe you will like that other ideas more than these ones
(multi-line & multi-file).

In this light, your suggestion with RegExpN is also interesting, as it can
decrease number of rules (less rules => less complexity) and scripting
complexity - in cases, where it is applicable.

Thank you again - for SEC availability at the first place :).

Richard

št 28. 11. 2019 o 11:33 Risto Vaarandi <ris...@gm...>
napísal(a):

> hi Richard,
>
> just one followup thought -- have you considered sec native multi-line
> patterns such as RegexpN for handling multi-line logs? Of course, there are
> scenarios where the value of N (max number of lines in a multi-line event)
> can be very large and is difficult to predict, and for such cases an event
> converter (like the one from the previous post) is probably the best
> approach. However, if you know the value of N and events can be matched
> with regular expressions, RegexpN pattern type can be used for this task.
> For example, if events have the brace separated format described in the
> previous post, and events can contain up to 20 lines, one could utilize the
> following Regexp20 pattern for matching:
>
> type=Single
> ptype=RegExp20
> pattern=(?s)^(?:.+\n)?\{\n(.+\n)?\}$
> desc=match a multiline event between braces
> action=write - $1
>
> Also, if you want to convert such multi-line events into a single-line
> format with builtin features, sec 'rewrite' action allows for that. In the
> following example, the first rule takes the multi-line data between braces
> and replaces each newline with a space character, and resulting single-line
> string (with a prefix "Converted event: ") is used for overwriting sec
> event buffer. The second rule is now able to match such converted events:
>
> type=Single
> ptype=RegExp20
> pattern=(?s)^(?:.+\n)?\{\n(?:(.+)\n)?\}$
> continue=TakeNext
> desc=convert a multiline event between braces to single-line format
> action=lcall %ret $1 -> ( sub { my($t) = $_[0]; $t =~ s/\n/ /g; return $t;
> } ); \
>        rewrite 20 Converted event: %ret
>
> type=Single
> ptype=RegExp
> pattern=Converted event: (.*)
> desc=match any event
> action=write - $1
>
> Maybe above examples are helpful for getting additional insights into
> different ways of processing multi-line events.
>
> kind regards,
> risto
>
>
> hi Richard,
>>>>
>>> ...
>>>
>>>> In the current code base, identifying the end of each line is done with
>>>> a simple search for newline character. The newline is searched not with a
>>>> regular expression, but rather with index() function which is much faster.
>>>> It is of course possible to change the code, so that a regular expression
>>>> pattern is utilized instead, but that would introduce a noticeable
>>>> performance penalty. For example, I made couple of quick tests with
>>>> replacing the index() function with a regular expression that identifies
>>>> the newline separator, and when testing modified sec code against log files
>>>> of 4-5 million events, cpu time consumption increased by 25%.
>>>>
>>>
>>> Hmm, this is interesting. The philosophically principial question came
>>> to my mind, if this penalty could be decreased (optimized), when doing
>>> replacements of these regular newline characters ("\n") and matching
>>> endings of "lines" with regexp - through rules (or by other more external
>>> way) - before further processing by subsequent rules, instead of potential
>>> built-in feature (used optionally on particular logfiles).
>>>
>>>
>> Perhaps I can add few thoughts here. Since the number of multi-line
>> formats is essentially infinite, converting multi-line format into
>> single-line representation externally (i.e., outside sec) offers most
>> flexibility. For instance, in many cases there is no delimiter as such
>> between messages, but beginning and end of the message contain different
>> character sequences that are part of the message. In addition, any lines
>> that are not between valid beginning and end should be discarded. It is
>> clear that using one regular expression for matching delimiters is not
>> addressing this scenario properly. Also, one can imagine many other
>> multi-line formats, and coming up with a single builtin approach for all of
>> them is not possible. On the other hand, a custom external converter allows
>> for addressing a given event format exactly as we like. For example,
>> suppose we are dealing with the following format, where multi-line event
>> starts with a lone opening brace on a separate line, and ends with a lone
>> closing brace:
>>
>> {
>>   line1
>>   line2
>>   ...
>> }
>>
>> For converting such events into a single line format, the following
>> simple wrapper could be utilized (written in 10 minutes):
>>
>> #!/usr/bin/perl -w
>> # the name of this wrapper is test.pl
>>
>> if (scalar(@ARGV) != 1) { die "Usage: $0 <file>\n"; }
>> $file = $ARGV[0];
>> if (!open(FILE, "tail -F $file |")) { die "Can't start tail for $file\n";
>> }
>> $| = 1;
>>
>> while (<FILE>) {
>>   chomp;
>>   if (/^{$/) { $message = $_; }
>>   elsif (/^}$/ && defined($message)) {
>>     $message .= $_;
>>     print $message, "\n";
>>     $message = undef;
>>   }
>>   elsif (defined($message)) {
>>     $message .= $_;
>>   }
>> }
>>
>> If this wrapper is then started from sec with 'spawn' or 'cspawn' action,
>> multi-line events from monitored file will appear as single-line synthetic
>> events for sec. For example:
>>
>> type=Single
>> ptype=RegExp
>> pattern=^(?:SEC_STARTUP|SEC_RESTART)$
>> context=SEC_INTERNAL_EVENT
>> desc=fork the converter when sec is started or restarted
>> action=spawn ./test.pl my.log
>>
>> type=Single
>> ptype=RegExp
>> pattern=\{importantmessage\}
>> desc=test
>> action=write - important message was received
>>
>> The second rule fires if the following 4-line event is written into
>> my.log:
>>
>> {
>> important
>> message
>> }
>>
>> My apologies if the above example is a bit laconic, but hopefully it
>> conveys the overall idea how to set up an event converter. And writing a
>> suitable converter is often taking not that much time, plus you get
>> something which is tailored exactly to your needs :)
>>
>> kind regards,
>> risto
>>
>>
>>

Re: [Simple-evcorr-users] "multi-line" and multi-file logs - out of box

[Risto V. <ris...@gm...>]

hi Richard,

just one followup thought -- have you considered sec native multi-line
patterns such as RegexpN for handling multi-line logs? Of course, there are
scenarios where the value of N (max number of lines in a multi-line event)
can be very large and is difficult to predict, and for such cases an event
converter (like the one from the previous post) is probably the best
approach. However, if you know the value of N and events can be matched
with regular expressions, RegexpN pattern type can be used for this task.
For example, if events have the brace separated format described in the
previous post, and events can contain up to 20 lines, one could utilize the
following Regexp20 pattern for matching:

type=Single
ptype=RegExp20
pattern=(?s)^(?:.+\n)?\{\n(.+\n)?\}$
desc=match a multiline event between braces
action=write - $1

Also, if you want to convert such multi-line events into a single-line
format with builtin features, sec 'rewrite' action allows for that. In the
following example, the first rule takes the multi-line data between braces
and replaces each newline with a space character, and resulting single-line
string (with a prefix "Converted event: ") is used for overwriting sec
event buffer. The second rule is now able to match such converted events:

type=Single
ptype=RegExp20
pattern=(?s)^(?:.+\n)?\{\n(?:(.+)\n)?\}$
continue=TakeNext
desc=convert a multiline event between braces to single-line format
action=lcall %ret $1 -> ( sub { my($t) = $_[0]; $t =~ s/\n/ /g; return $t;
} ); \
       rewrite 20 Converted event: %ret

type=Single
ptype=RegExp
pattern=Converted event: (.*)
desc=match any event
action=write - $1

Maybe above examples are helpful for getting additional insights into
different ways of processing multi-line events.

kind regards,
risto


hi Richard,
>>>
>> ...
>>
>>> In the current code base, identifying the end of each line is done with
>>> a simple search for newline character. The newline is searched not with a
>>> regular expression, but rather with index() function which is much faster.
>>> It is of course possible to change the code, so that a regular expression
>>> pattern is utilized instead, but that would introduce a noticeable
>>> performance penalty. For example, I made couple of quick tests with
>>> replacing the index() function with a regular expression that identifies
>>> the newline separator, and when testing modified sec code against log files
>>> of 4-5 million events, cpu time consumption increased by 25%.
>>>
>>
>> Hmm, this is interesting. The philosophically principial question came to
>> my mind, if this penalty could be decreased (optimized), when doing
>> replacements of these regular newline characters ("\n") and matching
>> endings of "lines" with regexp - through rules (or by other more external
>> way) - before further processing by subsequent rules, instead of potential
>> built-in feature (used optionally on particular logfiles).
>>
>>
> Perhaps I can add few thoughts here. Since the number of multi-line
> formats is essentially infinite, converting multi-line format into
> single-line representation externally (i.e., outside sec) offers most
> flexibility. For instance, in many cases there is no delimiter as such
> between messages, but beginning and end of the message contain different
> character sequences that are part of the message. In addition, any lines
> that are not between valid beginning and end should be discarded. It is
> clear that using one regular expression for matching delimiters is not
> addressing this scenario properly. Also, one can imagine many other
> multi-line formats, and coming up with a single builtin approach for all of
> them is not possible. On the other hand, a custom external converter allows
> for addressing a given event format exactly as we like. For example,
> suppose we are dealing with the following format, where multi-line event
> starts with a lone opening brace on a separate line, and ends with a lone
> closing brace:
>
> {
>   line1
>   line2
>   ...
> }
>
> For converting such events into a single line format, the following simple
> wrapper could be utilized (written in 10 minutes):
>
> #!/usr/bin/perl -w
> # the name of this wrapper is test.pl
>
> if (scalar(@ARGV) != 1) { die "Usage: $0 <file>\n"; }
> $file = $ARGV[0];
> if (!open(FILE, "tail -F $file |")) { die "Can't start tail for $file\n"; }
> $| = 1;
>
> while (<FILE>) {
>   chomp;
>   if (/^{$/) { $message = $_; }
>   elsif (/^}$/ && defined($message)) {
>     $message .= $_;
>     print $message, "\n";
>     $message = undef;
>   }
>   elsif (defined($message)) {
>     $message .= $_;
>   }
> }
>
> If this wrapper is then started from sec with 'spawn' or 'cspawn' action,
> multi-line events from monitored file will appear as single-line synthetic
> events for sec. For example:
>
> type=Single
> ptype=RegExp
> pattern=^(?:SEC_STARTUP|SEC_RESTART)$
> context=SEC_INTERNAL_EVENT
> desc=fork the converter when sec is started or restarted
> action=spawn ./test.pl my.log
>
> type=Single
> ptype=RegExp
> pattern=\{importantmessage\}
> desc=test
> action=write - important message was received
>
> The second rule fires if the following 4-line event is written into my.log:
>
> {
> important
> message
> }
>
> My apologies if the above example is a bit laconic, but hopefully it
> conveys the overall idea how to set up an event converter. And writing a
> suitable converter is often taking not that much time, plus you get
> something which is tailored exactly to your needs :)
>
> kind regards,
> risto
>
>
>

Re: [Simple-evcorr-users] "multi-line" and multi-file logs - out of box

[Risto V. <ris...@gm...>]

hi Richard,

Risto, thank you for your pre-analysis about multi-lines with regexp, and
> also for suggestions about multi-files yet more sophisticated solution.
>
> My comments are also inline:
>
> st 27. 11. 2019 o 15:07 Risto Vaarandi <ris...@gm...>
> napísal(a):
>
>> hi Richard,
>>
> ...
>
>> In the current code base, identifying the end of each line is done with a
>> simple search for newline character. The newline is searched not with a
>> regular expression, but rather with index() function which is much faster.
>> It is of course possible to change the code, so that a regular expression
>> pattern is utilized instead, but that would introduce a noticeable
>> performance penalty. For example, I made couple of quick tests with
>> replacing the index() function with a regular expression that identifies
>> the newline separator, and when testing modified sec code against log files
>> of 4-5 million events, cpu time consumption increased by 25%.
>>
>
> Hmm, this is interesting. The philosophically principial question came to
> my mind, if this penalty could be decreased (optimized), when doing
> replacements of these regular newline characters ("\n") and matching
> endings of "lines" with regexp - through rules (or by other more external
> way) - before further processing by subsequent rules, instead of potential
> built-in feature (used optionally on particular logfiles).
>
>
Perhaps I can add few thoughts here. Since the number of multi-line formats
is essentially infinite, converting multi-line format into single-line
representation externally (i.e., outside sec) offers most flexibility. For
instance, in many cases there is no delimiter as such between messages, but
beginning and end of the message contain different character sequences that
are part of the message. In addition, any lines that are not between valid
beginning and end should be discarded. It is clear that using one regular
expression for matching delimiters is not addressing this scenario
properly. Also, one can imagine many other multi-line formats, and coming
up with a single builtin approach for all of them is not possible. On the
other hand, a custom external converter allows for addressing a given event
format exactly as we like. For example, suppose we are dealing with the
following format, where multi-line event starts with a lone opening brace
on a separate line, and ends with a lone closing brace:

{
  line1
  line2
  ...
}

For converting such events into a single line format, the following simple
wrapper could be utilized (written in 10 minutes):

#!/usr/bin/perl -w
# the name of this wrapper is test.pl

if (scalar(@ARGV) != 1) { die "Usage: $0 <file>\n"; }
$file = $ARGV[0];
if (!open(FILE, "tail -F $file |")) { die "Can't start tail for $file\n"; }
$| = 1;

while (<FILE>) {
  chomp;
  if (/^{$/) { $message = $_; }
  elsif (/^}$/ && defined($message)) {
    $message .= $_;
    print $message, "\n";
    $message = undef;
  }
  elsif (defined($message)) {
    $message .= $_;
  }
}

If this wrapper is then started from sec with 'spawn' or 'cspawn' action,
multi-line events from monitored file will appear as single-line synthetic
events for sec. For example:

type=Single
ptype=RegExp
pattern=^(?:SEC_STARTUP|SEC_RESTART)$
context=SEC_INTERNAL_EVENT
desc=fork the converter when sec is started or restarted
action=spawn ./test.pl my.log

type=Single
ptype=RegExp
pattern=\{importantmessage\}
desc=test
action=write - important message was received

The second rule fires if the following 4-line event is written into my.log:

{
important
message
}

My apologies if the above example is a bit laconic, but hopefully it
conveys the overall idea how to set up an event converter. And writing a
suitable converter is often taking not that much time, plus you get
something which is tailored exactly to your needs :)

kind regards,
risto