simple-evcorr-users — Simple Event Correlator users

Re: [Simple-evcorr-users] RV: IP correlation with EventGroup

[John P. R. <ro...@cs...>]

Hi Risto:

In message
<CAG...@ma...> ,
Risto Vaarandi writes:
>hi Agustin,
>> Hi Risto,
>>
>> Thank you very much for your help.
>> I have another question related to this problem.
>>
>> Suppose we have the next entry in less than 60 seconds:
>> EVENT_TYPE_A 1.1.1.1     <--- the beginning of input for SEC
>> EVENT_TYPE_A 2.2.2.2
>> EVENT_TYPE_B 1.1.1.1
>> EVENT_TYPE_B 2.2.2.2
>> EVENT_TYPE_C 1.1.1.1
>> FINISH                                 <--- (FINISH is also an event) the
>> end of input for SEC
>>
>> We have the following rule:
>> Rule 1:
>> type=EventGroup2
>> ptype=RegExp
>> pattern=EVENT_TYPE_A ([\d.]+)
>> continue=TakeNext
>> ptype2=RegExp
>> pattern2=EVENT_TYPE_B ([\d.]+)
>> continue2=TakeNext
>> desc=Events A and B observed for IP $1 within 60 seconds
>> action=logonly Events A and B observed for IP $1
>> window=60
>>
>> Rule 2:
>> type=EventGroup3
>> ptype=RegExp
>> pattern=EVENT_TYPE_A ([\d.]+)
>> continue=TakeNext
>> ptype2=RegExp
>> pattern2=EVENT_TYPE_B ([\d.]+)
>> continue2=TakeNext
>> ptype3=RegExp
>> pattern3=EVENT_TYPE_C ([\d.]+)
>> continue3=TakeNext
>> desc=Events A, B and C observed for IP $1 within 60 seconds
>> action=logonly Events A , B and C observed for IP $1
>> window=60
>>
>> We get the following output:
>>  Events A and B observed for IP 1.1.1.1
>>  Events A and B observed for IP 2.2.2.2
>>  Events A , B and C observed for IP 1.1.1.1
>>
>> I'm waiting for the next output:
>>  Events A and B observed for IP 2.2.2.2
>>  Events A , B and C observed for IP 1.1.1.1
>>
>> The idea is to reduce the output.
>>
>
>One approach that is relatively easy to implement is the following -- when
>the first message for an IP address is generated, a context is created for
>this IP address which will prevent further matches by rules for this IP.
>Also, in all rule definitions, pattern* fields would be complemented with
>context* fields which verify that the context for current IP address does
>not exist. For example, in the following ruleset repeated messages for the
>same IP address are suppressed for 300 seconds (that's the lifetime of
>SUPPRESS_<ip> contexts):
>
>type=EventGroup2
>ptype=RegExp
>pattern=EVENT_TYPE_A ([\d.]+)
>context=!SUPPRESS_$1
>continue=TakeNext
>ptype2=RegExp
>pattern2=EVENT_TYPE_B ([\d.]+)
>context2=!SUPPRESS_$1
>continue2=TakeNext
>desc=Events A and B observed for IP $1 within 60 seconds
>action=write - %s; create SUPPRESS_$1 300
>window=60
>
>type=EventGroup3
>ptype=RegExp
>pattern=EVENT_TYPE_A ([\d.]+)
>context=!SUPPRESS_$1
>continue=TakeNext
>ptype2=RegExp
>pattern2=EVENT_TYPE_B ([\d.]+)
>context2=!SUPPRESS_$1
>continue2=TakeNext
>ptype3=RegExp
>pattern3=EVENT_TYPE_C ([\d.]+)
>context3=!SUPPRESS_$1
>continue3=TakeNext
>desc=Events A, B and C observed for IP $1 within 60 seconds
>action=write - %s; create SUPPRESS_$1 300
>window=60
>
>With these rules, the following output events would be produced for your
>example input:
>Events A and B observed for IP 1.1.1.1 within 60 seconds
>Events A and B observed for IP 2.2.2.2 within 60 seconds
>
>However, if you would like to suppress the output message that is generated
>on 3rd input event and rather generate an output message "Events A , B and
>C observed for IP 1.1.1.1" on 5th input event, it is not possible to
>achieve that goal with EventGroup (or any other) rules, since after seeing
>the 3rd event, it is not possible to know in advance what events will
>appear in the future. In other words, SEC rules execute actions immediately
>when a first matching set of events has been seen, and it is neither
>possible to reprocess past events nor postpone actions in the hope of
>better future match (which might never occur).

I agree SEC has no built in delay mechanism for actions, you can
create a context that expires in the future and runs an action on
expiration.

How about creating a delayed reporting context rather than
writing:

   Events A and B observed for IP 1.1.1.1

in EventGroup2, use an action like:

  create report_$1_event_A_and_B 60 (write - %s)

and add the action

  destroy report_$1_event_A_and_B

to EventGroup3.

This should prevent the reporting of A and B when A, B and C are
seen. It will delay the reporting of A and B until the window to find
A, B and C is done. This may not be desirable, but the laws of physics
won't permit another option.

Is there a variable that records how much time is left in the window?
If so you can have the report_$1_event_A_and_B expire in that many
seconds rather than in 60 seconds. (I assume the window expiration
times of both EventGroups will be the same since they are triggered by
the same event.)

This is a bit of a hack but I think ti will work to delay the write
action.

Thoughts?

Have a great week.
--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

Re: [Simple-evcorr-users] RV: IP correlation with EventGroup

[Risto V. <ris...@gm...>]

>
>
> However, if you would like to suppress the output message that is
> generated on 3rd input event and rather generate an output message "Events
> A , B and C observed for IP 1.1.1.1" on 5th input event, it is not possible
> to achieve that goal with EventGroup (or any other) rules, since after
> seeing the 3rd event, it is not possible to know in advance what events
> will appear in the future. In other words, SEC rules execute actions
> immediately when a first matching set of events has been seen, and it is
> neither possible to reprocess past events nor postpone actions in the hope
> of better future match (which might never occur).
>
>
To add one remark here -- it is possible to configure rules to store their
output into context, and if a context contains more than one event, select
one of these events for reporting after some event aggregation period,
discarding other events. However, in this case reporting will happen with a
delay which might not be acceptable for more critical events.

kind regards,
risto

Re: [Simple-evcorr-users] RV: IP correlation with EventGroup

[Risto V. <ris...@gm...>]

hi Agustin,


> Hi Risto,
>
> Thank you very much for your help.
> I have another question related to this problem.
>
> Suppose we have the next entry in less than 60 seconds:
> EVENT_TYPE_A 1.1.1.1     <--- the beginning of input for SEC
> EVENT_TYPE_A 2.2.2.2
> EVENT_TYPE_B 1.1.1.1
> EVENT_TYPE_B 2.2.2.2
> EVENT_TYPE_C 1.1.1.1
> FINISH                                 <--- (FINISH is also an event) the
> end of input for SEC
>
> We have the following rule:
> Rule 1:
> type=EventGroup2
> ptype=RegExp
> pattern=EVENT_TYPE_A ([\d.]+)
> continue=TakeNext
> ptype2=RegExp
> pattern2=EVENT_TYPE_B ([\d.]+)
> continue2=TakeNext
> desc=Events A and B observed for IP $1 within 60 seconds
> action=logonly Events A and B observed for IP $1
> window=60
>
> Rule 2:
> type=EventGroup3
> ptype=RegExp
> pattern=EVENT_TYPE_A ([\d.]+)
> continue=TakeNext
> ptype2=RegExp
> pattern2=EVENT_TYPE_B ([\d.]+)
> continue2=TakeNext
> ptype3=RegExp
> pattern3=EVENT_TYPE_C ([\d.]+)
> continue3=TakeNext
> desc=Events A, B and C observed for IP $1 within 60 seconds
> action=logonly Events A , B and C observed for IP $1
> window=60
>
> We get the following output:
>  Events A and B observed for IP 1.1.1.1
>  Events A and B observed for IP 2.2.2.2
>  Events A , B and C observed for IP 1.1.1.1
>
> I'm waiting for the next output:
>  Events A and B observed for IP 2.2.2.2
>  Events A , B and C observed for IP 1.1.1.1
>
> The idea is to reduce the output.
>

One approach that is relatively easy to implement is the following -- when
the first message for an IP address is generated, a context is created for
this IP address which will prevent further matches by rules for this IP.
Also, in all rule definitions, pattern* fields would be complemented with
context* fields which verify that the context for current IP address does
not exist. For example, in the following ruleset repeated messages for the
same IP address are suppressed for 300 seconds (that's the lifetime of
SUPPRESS_<ip> contexts):

type=EventGroup2
ptype=RegExp
pattern=EVENT_TYPE_A ([\d.]+)
context=!SUPPRESS_$1
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_B ([\d.]+)
context2=!SUPPRESS_$1
continue2=TakeNext
desc=Events A and B observed for IP $1 within 60 seconds
action=write - %s; create SUPPRESS_$1 300
window=60

type=EventGroup3
ptype=RegExp
pattern=EVENT_TYPE_A ([\d.]+)
context=!SUPPRESS_$1
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_B ([\d.]+)
context2=!SUPPRESS_$1
continue2=TakeNext
ptype3=RegExp
pattern3=EVENT_TYPE_C ([\d.]+)
context3=!SUPPRESS_$1
continue3=TakeNext
desc=Events A, B and C observed for IP $1 within 60 seconds
action=write - %s; create SUPPRESS_$1 300
window=60

With these rules, the following output events would be produced for your
example input:
Events A and B observed for IP 1.1.1.1 within 60 seconds
Events A and B observed for IP 2.2.2.2 within 60 seconds

However, if you would like to suppress the output message that is generated
on 3rd input event and rather generate an output message "Events A , B and
C observed for IP 1.1.1.1" on 5th input event, it is not possible to
achieve that goal with EventGroup (or any other) rules, since after seeing
the 3rd event, it is not possible to know in advance what events will
appear in the future. In other words, SEC rules execute actions immediately
when a first matching set of events has been seen, and it is neither
possible to reprocess past events nor postpone actions in the hope of
better future match (which might never occur).

hope this helps,
risto


> Kind regards,
> Agustín
>
>
>

[Simple-evcorr-users] RV: IP correlation with EventGroup

[Agustín L. R. <ag...@ho...>]

Hi Risto,

Thank you very much for your help.
I have another question related to this problem.

Suppose we have the next entry in less than 60 seconds:
EVENT_TYPE_A 1.1.1.1     <--- the beginning of input for SEC
EVENT_TYPE_A 2.2.2.2
EVENT_TYPE_B 1.1.1.1
EVENT_TYPE_B 2.2.2.2
EVENT_TYPE_C 1.1.1.1
FINISH                                 <--- (FINISH is also an event) the end of input for SEC

We have the following rule:
Rule 1:
type=EventGroup2
ptype=RegExp
pattern=EVENT_TYPE_A ([\d.]+)
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_B ([\d.]+)
continue2=TakeNext
desc=Events A and B observed for IP $1 within 60 seconds
action=logonly Events A and B observed for IP $1
window=60

Rule 2:
type=EventGroup3
ptype=RegExp
pattern=EVENT_TYPE_A ([\d.]+)
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_B ([\d.]+)
continue2=TakeNext
ptype3=RegExp
pattern3=EVENT_TYPE_C ([\d.]+)
continue3=TakeNext
desc=Events A, B and C observed for IP $1 within 60 seconds
action=logonly Events A , B and C observed for IP $1
window=60

We get the following output:
 Events A and B observed for IP 1.1.1.1
 Events A and B observed for IP 2.2.2.2
 Events A , B and C observed for IP 1.1.1.1

I'm waiting for the next output:
 Events A and B observed for IP 2.2.2.2
 Events A , B and C observed for IP 1.1.1.1

The idea is to reduce the output.

Kind regards,
Agustín

 ----------------------------------------------------------------------------------------------------------------------
hi Agustin,

I have tried the rule from your e-mail, and I am able to get the output you are expecting:

/usr/bin/sec --conf=test4.sec --input=-
SEC (Simple Event Correlator) 2.8.2
Reading configuration from test4.sec
1 rules loaded from test4.sec
No --bufsize command line option or --bufsize=0, setting --bufsize to 1
Opening input file -
Interactive process, SIGINT can't be used for changing the logging level
EVENT_TYPE_A 1.1.1.1     <--- the beginning of input for SEC
EVENT_TYPE_A 2.2.2.2
EVENT_TYPE_B 1.1.1.1
EVENT_TYPE_B 2.2.2.2     <--- the end of input for SEC
Writing event 'Events A and B observed for IP 1.1.1.1 within 60 seconds' to file '-'
Events A and B observed for IP 1.1.1.1 within 60 seconds
Writing event 'Events A and B observed for IP 2.2.2.2 within 60 seconds' to file '-'
Events A and B observed for IP 2.2.2.2 within 60 seconds


Are you sure that events for IP address 2.2.2.2 are separated by at most 60 seconds? If there is a larger time gap between those two events, the event correlation operation for 2.2.2.2 will not produce expected output.

kind regards,
risto

[Simple-evcorr-users] log files existence and accessibility

[Richard O. <ric...@gm...>]

Hello friends,

I am thinking about how to monitor not only events from log files, but also
those files existence and accessibility (for user running SEC) - in cases,
where this is considered to be a problem.

As I saw in the past, these were logged into SEC log file, but higher debug
level was required, so it is not suitable for production.

There are handful of other options how to monitor it "externally", e.g. via
some script, or other monitoring agent, but this is not ultimate solution,
as e.g. SELinux may be configured, allowing external script or agent
(running under the same user as SEC) to see nad open file for reading, but
not SEC (theoretical caveat of such solution).

So, has somebody some "best practise" for reliable and production-ready
way, how to "self-monitor" log files being accessed by SEC, if:

   - they exist
   - they are accessible by SEC

?

Thanks in advance.

Richard

Re: [Simple-evcorr-users] IP correlation with EventGroup

[Risto V. <ris...@gm...>]

hi Agustin,

I have tried the rule from your e-mail, and I am able to get the output you
are expecting:

/usr/bin/sec --conf=test4.sec --input=-
SEC (Simple Event Correlator) 2.8.2
Reading configuration from test4.sec
1 rules loaded from test4.sec
No --bufsize command line option or --bufsize=0, setting --bufsize to 1
Opening input file -
Interactive process, SIGINT can't be used for changing the logging level
EVENT_TYPE_A 1.1.1.1     <--- the beginning of input for SEC
EVENT_TYPE_A 2.2.2.2
EVENT_TYPE_B 1.1.1.1
EVENT_TYPE_B 2.2.2.2     <--- the end of input for SEC
Writing event 'Events A and B observed for IP 1.1.1.1 within 60 seconds' to
file '-'
Events A and B observed for IP 1.1.1.1 within 60 seconds
Writing event 'Events A and B observed for IP 2.2.2.2 within 60 seconds' to
file '-'
Events A and B observed for IP 2.2.2.2 within 60 seconds


Are you sure that events for IP address 2.2.2.2 are separated by at most 60
seconds? If there is a larger time gap between those two events, the event
correlation operation for 2.2.2.2 will not produce expected output.

kind regards,
risto


Hi Risto,
> I'm sorry, I don't think I made myself clear.
> Thanks for your help, but it still doesn't work. Here's the problem:
>
> We have the following rule:
> type=EventGroup2
> ptype=RegExp
> pattern=EVENT_TYPE_A ([\d.]+)
> continue=TakeNext
> ptype2=RegExp
> pattern2=EVENT_TYPE_B ([\d.]+)
> continue2=TakeNext
> desc=Events A and B observed for IP $1 within 60 seconds
> action=write - %s
> window=60
>
>
> And if we have the following events (input):
>
> EVENT_TYPE_A 1.1.1.1
> EVENT_TYPE_A 2.2.2.2
> EVENT_TYPE_B 1.1.1.1
> EVENT_TYPE_B 2.2.2.2
> FINISH
>
>
> I'm waiting for the next output:
>
> Events A and B observed for IP 1.1.1.1 within 60 seconds
> Events A and B observed for IP 2.2.2.2 within 60 seconds
>
>
> But I get the next exit:
>
> Events A and B observed for IP 1.1.1.1 within 60 seconds
>
>
> In addition (in another example) if the following events were to occurred:
>
> EVENT_TYPE_A 1.1.1.1
> EVENT_TYPE_B 2.2.2.2
> FINISH
>
>
> The rule should not be activated because although events A and B have
> occurred, they have not been for the same IP.
>
>
> The main problem is that I want to correlate N differents events for the
> same IP, but there can be different IPs.
> I have tried to combine this rule with the creation of contexts containing
> the IP, but I cannot solve it.
>
> Kind regards,
> Agustín
>
> ------------------------------
>
>
>
> hi Agustin,
>
> and thanks for feedback! Instead of developing one rule which addresses
> all scenarios, it is better to write a separate rule for each case. For
> example, for the first case EVENT_TYPE_A && EVENT_TYPE_B the rule would
> look like this:
>
> type=EventGroup2
> ptype=RegExp
> pattern=EVENT_TYPE_A ([\d.]+)
> continue=TakeNext
> ptype2=RegExp
> pattern2=EVENT_TYPE_B ([\d.]+)
> continue2=TakeNext
> desc=Events A and B observed for IP $1 within 60 seconds
> action=write - %s
> window=60
>
> This rule is able to match events of type A and B and extract an IP
> address from these events. Whichever event occurs first, the rule will
> start an event correlation operation for extracted IP address, and the
> operation will wait for the event of second type to arrive within 60
> seconds. If expected event arrives on time, the message "Events A and B
> observed for IP <ip> within 60 seconds" will be written to standard output.
> Please note that after this message has been generated, the operation will
> continue to run until the end of 60 second window, and further events A and
> B will be silently consumed by the operation until the end of the window.
> If you want to avoid this message suppression, you can change the action
> list of the above rule as follows:
>
> action=write - %s; reset 0
>
> In the above action list, the 'reset' action will terminate the event
> correlation operation that invoked this action list, and message
> suppression will therefore not happen.
>
> In order to address the second scenario EVENT_TYPE_A && EVENT_TYPE_B &&
> EVENT_TYPE_C, you would use the following rule which is similar to the
> previous example:
>
> type=EventGroup3
> ptype=RegExp
> pattern=EVENT_TYPE_A ([\d.]+)
> continue=TakeNext
> ptype2=RegExp
> pattern2=EVENT_TYPE_B ([\d.]+)
> continue2=TakeNext
> ptype3=RegExp
> pattern3=EVENT_TYPE_C ([\d.]+)
> continue3=TakeNext
> desc=Events A, B and C observed for IP $1 within 60 seconds
> action=write - %s
> window=60
>
> And as for the final scenario (EVENT_TYPE_A || EVENT_TYPE_B &&
> EVENT_TYPE_D), everything depends on how to interpret it. Given the
> precedence of logical operators, I would interpret it as EVENT_TYPE_A ||
> (EVENT_TYPE_B && EVENT_TYPE_D), and in that case the following two rules
> would be sufficient:
>
> type=Single
> ptype=RegExp
> pattern=EVENT_TYPE_A ([\d.]+)
> continue=TakeNext
> desc=Event A observed for IP
> action=write - %s
>
> type=EventGroup2
> ptype=RegExp
> pattern=EVENT_TYPE_B ([\d.]+)
> continue=TakeNext
> ptype2=RegExp
> pattern2=EVENT_TYPE_D ([\d.]+)
> continue2=TakeNext
> desc=Events B and D observed for IP $1 within 60 seconds
> action=write - %s
> window=60
>
> However, if you are actually dealing with the scenario (EVENT_TYPE_A ||
> EVENT_TYPE_B) && EVENT_TYPE_D, you could use the following two rules:
>
> type=EventGroup2
> ptype=RegExp
> pattern=EVENT_TYPE_A ([\d.]+)
> continue=TakeNext
> ptype2=RegExp
> pattern2=EVENT_TYPE_D ([\d.]+)
> continue2=TakeNext
> desc=Events A and D observed for IP $1 within 60 seconds
> action=write - %s
> window=60
>
> type=EventGroup2
> ptype=RegExp
> pattern=EVENT_TYPE_B ([\d.]+)
> continue=TakeNext
> ptype2=RegExp
> pattern2=EVENT_TYPE_D ([\d.]+)
> continue2=TakeNext
> desc=Events B and D observed for IP $1 within 60 seconds
> action=write - %s
> window=60
>
> Finally, please note that I have used 'continue*=TakeNext' statements in
> all rule definitions, since they are matching the same set of events and
> the statements will allow events to be matched and processed by following
> rules in the rule base (I have assumed that all 4 rules are in the same
> rule file). The 'continue' statements are also necessary if in addition to
> above 4 rules, you have also other rules in the remaining rule file which
> need to match events A, B, C and D.
>
> hope this helps,
> risto
>
>
> Kontakt Agustín Lara Romero (<ag...@ho...>) kirjutas kuupäeval P,
> 5. aprill 2020 kell 02:58:
>
> Hi Risto,
>
> Yes, your suppositions in the points 1, 2 and 3 are correct.
> The events can appear in any order.
> The expected time window for this events is 60 seconds
>
> Kind regards,
> Agustín
>
>
> *Cc:* sim...@li... <
> sim...@li...>
> *Asunto:* Re: [Simple-evcorr-users] IP correlation with EventGroup
>
> hi Agustin,
>
> Hi Risto,
> My name is Agustín, I'm working with the SEC and I have a problem that I
> can't solve.
> I have different events such as:
> EVENT_TYPE_A FROM 1.1.1.1
> EVENT_TYPE_A FROM 2.2.2.2
> EVENT_TYPE_B FROM 1.1.1.1
> EVENT_TYPE_B FROM 2.2.2.2
> EVENT_TYPE_C FROM 1.1.1.1
> EVENT_TYPE_C FROM 2.2.2.2
> EVENT_TYPE_D FROM 2.2.2.2
> FINISH
>
> And I want to get SEC to correlate the events for each IP when the FINISH
> event comes in with the following logic:
>
>
>    - For each IP:
>       - (INPUT FOR SAME IP)
>          - EVENT_TYPE_A && EVENT_TYPE_B
>       - (OUPUT)
>          - MATCH_1 FOR IP
>       - (INPUT FOR SAME IP)
>          - EVENT_TYPE_A && EVENT_TYPE_B && EVENT_TYPE_C
>       - (OUPUT)
>          - MATCH_2 FOR IP
>       - (INPUT FOR SAME IP)
>          - EVENT_TYPE_A || EVENT_TYPE_B && EVENT_TYPE_D
>       - (OUPUT)
>          - MATCH_3 FOR IP
>
>
>
> Before suggesting anything, I'd like to clarify some details of the
> problem you have. Have I understood correctly that you are dealing with the
> following three scenarios?
>
> 1) if events of type A and type B are observed for the same IP address,
> you would like to trigger an action for this IP address,
> 2) if events of type A, B and C are observer for the same IP address, you
> would like to trigger an action for this IP address,
> 3) if you see either event of type A, or events of type B and D for the
> same IP address, you would like to trigger an action for this IP address.
>
> Also, is the order of events important or can they appear in any order?
> And what is the expected time window for these events? (Is it 60 seconds as
> your rule example suggests?)
>
> kind regards,
> risto
>
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>

Re: [Simple-evcorr-users] IP correlation with EventGroup

[Agustín L. R. <ag...@ho...>]

Hi Risto,
I'm sorry, I don't think I made myself clear.
Thanks for your help, but it still doesn't work. Here's the problem:

We have the following rule:
type=EventGroup2
ptype=RegExp
pattern=EVENT_TYPE_A ([\d.]+)
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_B ([\d.]+)
continue2=TakeNext
desc=Events A and B observed for IP $1 within 60 seconds
action=write - %s
window=60

And if we have the following events (input):
EVENT_TYPE_A 1.1.1.1
EVENT_TYPE_A 2.2.2.2
EVENT_TYPE_B 1.1.1.1
EVENT_TYPE_B 2.2.2.2
FINISH

I'm waiting for the next output:
Events A and B observed for IP 1.1.1.1 within 60 seconds
Events A and B observed for IP 2.2.2.2 within 60 seconds

But I get the next exit:
Events A and B observed for IP 1.1.1.1 within 60 seconds

In addition (in another example) if the following events were to occurred:
EVENT_TYPE_A 1.1.1.1
EVENT_TYPE_B 2.2.2.2
FINISH

The rule should not be activated because although events A and B have occurred, they have not been for the same IP.


The main problem is that I want to correlate N differents events for the same IP, but there can be different IPs.
I have tried to combine this rule with the creation of contexts containing the IP, but I cannot solve it.

Kind regards,
Agustín

________________________________


hi Agustin,

and thanks for feedback! Instead of developing one rule which addresses all scenarios, it is better to write a separate rule for each case. For example, for the first case EVENT_TYPE_A && EVENT_TYPE_B the rule would look like this:

type=EventGroup2
ptype=RegExp
pattern=EVENT_TYPE_A ([\d.]+)
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_B ([\d.]+)
continue2=TakeNext
desc=Events A and B observed for IP $1 within 60 seconds
action=write - %s
window=60

This rule is able to match events of type A and B and extract an IP address from these events. Whichever event occurs first, the rule will start an event correlation operation for extracted IP address, and the operation will wait for the event of second type to arrive within 60 seconds. If expected event arrives on time, the message "Events A and B observed for IP <ip> within 60 seconds" will be written to standard output. Please note that after this message has been generated, the operation will continue to run until the end of 60 second window, and further events A and B will be silently consumed by the operation until the end of the window. If you want to avoid this message suppression, you can change the action list of the above rule as follows:

action=write - %s; reset 0

In the above action list, the 'reset' action will terminate the event correlation operation that invoked this action list, and message suppression will therefore not happen.

In order to address the second scenario EVENT_TYPE_A && EVENT_TYPE_B && EVENT_TYPE_C, you would use the following rule which is similar to the previous example:

type=EventGroup3
ptype=RegExp
pattern=EVENT_TYPE_A ([\d.]+)
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_B ([\d.]+)
continue2=TakeNext
ptype3=RegExp
pattern3=EVENT_TYPE_C ([\d.]+)
continue3=TakeNext
desc=Events A, B and C observed for IP $1 within 60 seconds
action=write - %s
window=60

And as for the final scenario (EVENT_TYPE_A || EVENT_TYPE_B && EVENT_TYPE_D), everything depends on how to interpret it. Given the precedence of logical operators, I would interpret it as EVENT_TYPE_A || (EVENT_TYPE_B && EVENT_TYPE_D), and in that case the following two rules would be sufficient:

type=Single
ptype=RegExp
pattern=EVENT_TYPE_A ([\d.]+)
continue=TakeNext
desc=Event A observed for IP
action=write - %s

type=EventGroup2
ptype=RegExp
pattern=EVENT_TYPE_B ([\d.]+)
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_D ([\d.]+)
continue2=TakeNext
desc=Events B and D observed for IP $1 within 60 seconds
action=write - %s
window=60

However, if you are actually dealing with the scenario (EVENT_TYPE_A || EVENT_TYPE_B) && EVENT_TYPE_D, you could use the following two rules:

type=EventGroup2
ptype=RegExp
pattern=EVENT_TYPE_A ([\d.]+)
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_D ([\d.]+)
continue2=TakeNext
desc=Events A and D observed for IP $1 within 60 seconds
action=write - %s
window=60

type=EventGroup2
ptype=RegExp
pattern=EVENT_TYPE_B ([\d.]+)
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_D ([\d.]+)
continue2=TakeNext
desc=Events B and D observed for IP $1 within 60 seconds
action=write - %s
window=60

Finally, please note that I have used 'continue*=TakeNext' statements in all rule definitions, since they are matching the same set of events and the statements will allow events to be matched and processed by following rules in the rule base (I have assumed that all 4 rules are in the same rule file). The 'continue' statements are also necessary if in addition to above 4 rules, you have also other rules in the remaining rule file which need to match events A, B, C and D.

hope this helps,
risto

Kontakt Agustín Lara Romero (<ag...@ho...<mailto:ag...@ho...>>) kirjutas kuupäeval P, 5. aprill 2020 kell 02:58:
Hi Risto,

Yes, your suppositions in the points 1, 2 and 3 are correct.
The events can appear in any order.
The expected time window for this events is 60 seconds

Kind regards,
Agustín


Cc: sim...@li...<mailto:sim...@li...> <sim...@li...<mailto:sim...@li...>>
Asunto: Re: [Simple-evcorr-users] IP correlation with EventGroup

hi Agustin,

Hi Risto,
My name is Agustín, I'm working with the SEC and I have a problem that I can't solve.
I have different events such as:
EVENT_TYPE_A FROM 1.1.1.1
EVENT_TYPE_A FROM 2.2.2.2
EVENT_TYPE_B FROM 1.1.1.1
EVENT_TYPE_B FROM 2.2.2.2
EVENT_TYPE_C FROM 1.1.1.1
EVENT_TYPE_C FROM 2.2.2.2
EVENT_TYPE_D FROM 2.2.2.2
FINISH

And I want to get SEC to correlate the events for each IP when the FINISH event comes in with the following logic:


  *   For each IP:
     *   (INPUT FOR SAME IP)
        *   EVENT_TYPE_A && EVENT_TYPE_B
     *   (OUPUT)
        *   MATCH_1 FOR IP
     *   (INPUT FOR SAME IP)
        *   EVENT_TYPE_A && EVENT_TYPE_B && EVENT_TYPE_C
     *   (OUPUT)
        *   MATCH_2 FOR IP
     *   (INPUT FOR SAME IP)
        *   EVENT_TYPE_A || EVENT_TYPE_B && EVENT_TYPE_D
     *   (OUPUT)
        *   MATCH_3 FOR IP


Before suggesting anything, I'd like to clarify some details of the problem you have. Have I understood correctly that you are dealing with the following three scenarios?

1) if events of type A and type B are observed for the same IP address, you would like to trigger an action for this IP address,
2) if events of type A, B and C are observer for the same IP address, you would like to trigger an action for this IP address,
3) if you see either event of type A, or events of type B and D for the same IP address, you would like to trigger an action for this IP address.

Also, is the order of events important or can they appear in any order? And what is the expected time window for these events? (Is it 60 seconds as your rule example suggests?)

kind regards,
risto

_______________________________________________
Simple-evcorr-users mailing list
Sim...@li...<mailto:Sim...@li...>
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] SEC CPU utilization

[Risto V. <ris...@gm...>]

hi Richard,


>
>> You mentioned that existing HPOM configuration was converted on 1:1 basis
>> -- does that mean that SEC is configured to use a number of rule files,
>> where each rule file corresponds to some HPOM policy, and all rules are
>> applied against all input messages from 50 input files?
>>
>
> 50 HPOM policies (of type LOGFILE) = 50 SEC rules files
>
> This I meant to be single level of hierarchy, applying conditions only on
> log files they should be applied.
>
> SEC is starting with -input=logfilepath=contextname for each of them
> (which should mean also implicit enablement of -intcontext, as it is not
> enabled explicitly).
>
> There is next level of hierarchy in each rules file, implemented by using
> context conditions (as each log file has not only its "default" context, as
> navigated from -input command option, but also other "derived" contexts,
> present only in that particular rules file... names of contexts beginning
> with the one used also on commandline from -input), and patterns routing
> events into another GoTo sections for further "parametrized deduplications".
>
> There are hundreds of rules in total in all rules files.
>
> Since such use of rules is not very efficient, perhaps you could introduce
>> hierarchical rule arrangement in the following way.
>>
>
> For now, I am not sure, if what I described above, is negative or positive
> according to your assumption.
>

As I can see from above description, each rule file has been optimized with
GoTo statements, so that irrelevant rules for any given event can be
skipped. Such optimizations are always very useful and can be used
*together* with hierarchical rule arrangement, yielding even greater
performance benefits (I have often combined both techniques in larger rule
sets, in order to make them more scalable).


> First, assign each rule file to a configuration file set with Options
>> rule. For example, if you have an HPOM policy called "sshd", you can use
>> the following Options rule in the beginning of corresponding rule file:
>>
>> type=Options
>> joincfset=sshd
>> procallin=no
>>
>> Secondly, run SEC with --intcontexts flag which will set an internal
>> context for each file (e.g., the file /var/log/secure will have the context
>> _FILE_EVENT_/var/log/secure by default, but you can also set a custom
>> context name in command line). Finally, create a special rule file (e.g.,
>> main.sec) which will route messages to relevant rule files.
>>
>
> In our setup, there is no "main.sec" and we rely on
> -input=logfilename=contextname for each input log file, and then context
> condtions in all rules, so patterns are being processed only on events from
> log file they belong to. It seems to me, that this is not as optimal as
> your "main.sec" jumping suggestion, as I understand, that context
> conditions are still being evaluated also for events they does not belong
> to, and maybe it could be further optimized (while preserving our concept)
> by adding a rule on the top of each rules file, continuing with DontCont
> with anything, what does not belong to contexts with names "beginning with"
> contextname (as we have practise each context name in rules file having the
> same prefix).
>

Did you mean adding a rule like this in top of each rule file:

type=Suppress
ptype=TValue
pattern=TRUE
context=[ ! _FILE_EVENT_/var/log/secure ]

<here come the rules which are relevant to /var/log/secure>
...

If so, this approach would also work for reducing the CPU load, and is
definitely much better that having all rules being matched against all
events. However, this approach means that all 50 rules in top of 50 rule
files would still be evaluated against all events. On the other hand, if
you have one main.sec file with 50 Jump rules, you can move most frequently
matching Jump rules to the top of main.sec, having a situation where for
many events only few Jump rules would have to be evaluated before match is
found.

Also, since you have 50 input files, perhaps you can also use another
optimization for polling input files. If most input files are infrequently
modified, you can use --check-timeout option for increasing time interval
between polling files with no new data. By default, SEC will sleep for 0.1
seconds if no new data have been read from input files, but if you use
option like --check-timeout=3, file that had no new data during last poll
will not be checked for the following 3 seconds. Using --check-timeout
option can also reduce the CPU load of your systems if most input files are
rarely updated (say, once per 10-20 minutes).


> But I will wait for your potential feedback yet, and then I will consider
> suggestions of further optimizations in our team.
>
> Thank you.
>
> Richard
>
>

Hopefully you will find this feedback is useful.

kind regards,
risto

Re: [Simple-evcorr-users] IP correlation with EventGroup

[Risto V. <ris...@gm...>]

hi Agustin,

and thanks for feedback! Instead of developing one rule which addresses all
scenarios, it is better to write a separate rule for each case. For
example, for the first case EVENT_TYPE_A && EVENT_TYPE_B the rule would
look like this:

type=EventGroup2
ptype=RegExp
pattern=EVENT_TYPE_A ([\d.]+)
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_B ([\d.]+)
continue2=TakeNext
desc=Events A and B observed for IP $1 within 60 seconds
action=write - %s
window=60

This rule is able to match events of type A and B and extract an IP address
from these events. Whichever event occurs first, the rule will start an
event correlation operation for extracted IP address, and the operation
will wait for the event of second type to arrive within 60 seconds. If
expected event arrives on time, the message "Events A and B observed for IP
<ip> within 60 seconds" will be written to standard output. Please note
that after this message has been generated, the operation will continue to
run until the end of 60 second window, and further events A and B will be
silently consumed by the operation until the end of the window. If you want
to avoid this message suppression, you can change the action list of the
above rule as follows:

action=write - %s; reset 0

In the above action list, the 'reset' action will terminate the event
correlation operation that invoked this action list, and message
suppression will therefore not happen.

In order to address the second scenario EVENT_TYPE_A && EVENT_TYPE_B &&
EVENT_TYPE_C, you would use the following rule which is similar to the
previous example:

type=EventGroup3
ptype=RegExp
pattern=EVENT_TYPE_A ([\d.]+)
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_B ([\d.]+)
continue2=TakeNext
ptype3=RegExp
pattern3=EVENT_TYPE_C ([\d.]+)
continue3=TakeNext
desc=Events A, B and C observed for IP $1 within 60 seconds
action=write - %s
window=60

And as for the final scenario (EVENT_TYPE_A || EVENT_TYPE_B &&
EVENT_TYPE_D), everything depends on how to interpret it. Given the
precedence of logical operators, I would interpret it as EVENT_TYPE_A ||
(EVENT_TYPE_B && EVENT_TYPE_D), and in that case the following two rules
would be sufficient:

type=Single
ptype=RegExp
pattern=EVENT_TYPE_A ([\d.]+)
continue=TakeNext
desc=Event A observed for IP
action=write - %s

type=EventGroup2
ptype=RegExp
pattern=EVENT_TYPE_B ([\d.]+)
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_D ([\d.]+)
continue2=TakeNext
desc=Events B and D observed for IP $1 within 60 seconds
action=write - %s
window=60

However, if you are actually dealing with the scenario (EVENT_TYPE_A ||
EVENT_TYPE_B) && EVENT_TYPE_D, you could use the following two rules:

type=EventGroup2
ptype=RegExp
pattern=EVENT_TYPE_A ([\d.]+)
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_D ([\d.]+)
continue2=TakeNext
desc=Events A and D observed for IP $1 within 60 seconds
action=write - %s
window=60

type=EventGroup2
ptype=RegExp
pattern=EVENT_TYPE_B ([\d.]+)
continue=TakeNext
ptype2=RegExp
pattern2=EVENT_TYPE_D ([\d.]+)
continue2=TakeNext
desc=Events B and D observed for IP $1 within 60 seconds
action=write - %s
window=60

Finally, please note that I have used 'continue*=TakeNext' statements in
all rule definitions, since they are matching the same set of events and
the statements will allow events to be matched and processed by following
rules in the rule base (I have assumed that all 4 rules are in the same
rule file). The 'continue' statements are also necessary if in addition to
above 4 rules, you have also other rules in the remaining rule file which
need to match events A, B, C and D.

hope this helps,
risto

Kontakt Agustín Lara Romero (<ag...@ho...>) kirjutas kuupäeval P,
5. aprill 2020 kell 02:58:

> Hi Risto,
>
> Yes, your suppositions in the points 1, 2 and 3 are correct.
> The events can appear in any order.
> The expected time window for this events is 60 seconds
>
> Kind regards,
> Agustín
>
>
> *Cc:* sim...@li... <
> sim...@li...>
> *Asunto:* Re: [Simple-evcorr-users] IP correlation with EventGroup
>
> hi Agustin,
>
> Hi Risto,
> My name is Agustín, I'm working with the SEC and I have a problem that I
> can't solve.
> I have different events such as:
> EVENT_TYPE_A FROM 1.1.1.1
> EVENT_TYPE_A FROM 2.2.2.2
> EVENT_TYPE_B FROM 1.1.1.1
> EVENT_TYPE_B FROM 2.2.2.2
> EVENT_TYPE_C FROM 1.1.1.1
> EVENT_TYPE_C FROM 2.2.2.2
> EVENT_TYPE_D FROM 2.2.2.2
> FINISH
>
> And I want to get SEC to correlate the events for each IP when the FINISH
> event comes in with the following logic:
>
>
>    - For each IP:
>       - (INPUT FOR SAME IP)
>          - EVENT_TYPE_A && EVENT_TYPE_B
>       - (OUPUT)
>          - MATCH_1 FOR IP
>       - (INPUT FOR SAME IP)
>          - EVENT_TYPE_A && EVENT_TYPE_B && EVENT_TYPE_C
>       - (OUPUT)
>          - MATCH_2 FOR IP
>       - (INPUT FOR SAME IP)
>          - EVENT_TYPE_A || EVENT_TYPE_B && EVENT_TYPE_D
>       - (OUPUT)
>          - MATCH_3 FOR IP
>
>
>
> Before suggesting anything, I'd like to clarify some details of the
> problem you have. Have I understood correctly that you are dealing with the
> following three scenarios?
>
> 1) if events of type A and type B are observed for the same IP address,
> you would like to trigger an action for this IP address,
> 2) if events of type A, B and C are observer for the same IP address, you
> would like to trigger an action for this IP address,
> 3) if you see either event of type A, or events of type B and D for the
> same IP address, you would like to trigger an action for this IP address.
>
> Also, is the order of events important or can they appear in any order?
> And what is the expected time window for these events? (Is it 60 seconds as
> your rule example suggests?)
>
> kind regards,
> risto
>
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>

Re: [Simple-evcorr-users] IP correlation with EventGroup

[Agustín L. R. <ag...@ho...>]

Hi Risto,

Yes, your suppositions in the points 1, 2 and 3 are correct.
The events can appear in any order.
The expected time window for this events is 60 seconds

Kind regards,
Agustín


Cc: sim...@li... <sim...@li...>
Asunto: Re: [Simple-evcorr-users] IP correlation with EventGroup

hi Agustin,

Hi Risto,
My name is Agustín, I'm working with the SEC and I have a problem that I can't solve.
I have different events such as:
EVENT_TYPE_A FROM 1.1.1.1
EVENT_TYPE_A FROM 2.2.2.2
EVENT_TYPE_B FROM 1.1.1.1
EVENT_TYPE_B FROM 2.2.2.2
EVENT_TYPE_C FROM 1.1.1.1
EVENT_TYPE_C FROM 2.2.2.2
EVENT_TYPE_D FROM 2.2.2.2
FINISH

And I want to get SEC to correlate the events for each IP when the FINISH event comes in with the following logic:


  *   For each IP:
     *   (INPUT FOR SAME IP)
        *   EVENT_TYPE_A && EVENT_TYPE_B
     *   (OUPUT)
        *   MATCH_1 FOR IP
     *   (INPUT FOR SAME IP)
        *   EVENT_TYPE_A && EVENT_TYPE_B && EVENT_TYPE_C
     *   (OUPUT)
        *   MATCH_2 FOR IP
     *   (INPUT FOR SAME IP)
        *   EVENT_TYPE_A || EVENT_TYPE_B && EVENT_TYPE_D
     *   (OUPUT)
        *   MATCH_3 FOR IP


Before suggesting anything, I'd like to clarify some details of the problem you have. Have I understood correctly that you are dealing with the following three scenarios?

1) if events of type A and type B are observed for the same IP address, you would like to trigger an action for this IP address,
2) if events of type A, B and C are observer for the same IP address, you would like to trigger an action for this IP address,
3) if you see either event of type A, or events of type B and D for the same IP address, you would like to trigger an action for this IP address.

Also, is the order of events important or can they appear in any order? And what is the expected time window for these events? (Is it 60 seconds as your rule example suggests?)

kind regards,
risto

Re: [Simple-evcorr-users] IP correlation with EventGroup

[Risto V. <ris...@gm...>]

hi Agustin,

Hi Risto,
> My name is Agustín, I'm working with the SEC and I have a problem that I
> can't solve.
> I have different events such as:
> EVENT_TYPE_A FROM 1.1.1.1
> EVENT_TYPE_A FROM 2.2.2.2
> EVENT_TYPE_B FROM 1.1.1.1
> EVENT_TYPE_B FROM 2.2.2.2
> EVENT_TYPE_C FROM 1.1.1.1
> EVENT_TYPE_C FROM 2.2.2.2
> EVENT_TYPE_D FROM 2.2.2.2
> FINISH
>
> And I want to get SEC to correlate the events for each IP when the FINISH
> event comes in with the following logic:
>
>
>    - For each IP:
>       - (INPUT FOR SAME IP)
>          - EVENT_TYPE_A && EVENT_TYPE_B
>       - (OUPUT)
>          - MATCH_1 FOR IP
>       - (INPUT FOR SAME IP)
>          - EVENT_TYPE_A && EVENT_TYPE_B && EVENT_TYPE_C
>       - (OUPUT)
>          - MATCH_2 FOR IP
>       - (INPUT FOR SAME IP)
>          - EVENT_TYPE_A || EVENT_TYPE_B && EVENT_TYPE_D
>       - (OUPUT)
>          - MATCH_3 FOR IP
>
>
>
Before suggesting anything, I'd like to clarify some details of the problem
you have. Have I understood correctly that you are dealing with the
following three scenarios?

1) if events of type A and type B are observed for the same IP address, you
would like to trigger an action for this IP address,
2) if events of type A, B and C are observer for the same IP address, you
would like to trigger an action for this IP address,
3) if you see either event of type A, or events of type B and D for the
same IP address, you would like to trigger an action for this IP address.

Also, is the order of events important or can they appear in any order? And
what is the expected time window for these events? (Is it 60 seconds as
your rule example suggests?)

kind regards,
risto

[Simple-evcorr-users] IP correlation with EventGroup

[Agustín L. R. <ag...@ho...>]

Hi Risto,
My name is Agustín, I'm working with the SEC and I have a problem that I can't solve.
I have different events such as:
EVENT_TYPE_A FROM 1.1.1.1
EVENT_TYPE_A FROM 2.2.2.2
EVENT_TYPE_B FROM 1.1.1.1
EVENT_TYPE_B FROM 2.2.2.2
EVENT_TYPE_C FROM 1.1.1.1
EVENT_TYPE_C FROM 2.2.2.2
EVENT_TYPE_D FROM 2.2.2.2
FINISH

And I want to get SEC to correlate the events for each IP when the FINISH event comes in with the following logic:


  *   For each IP:
     *   (INPUT FOR SAME IP)
        *   EVENT_TYPE_A && EVENT_TYPE_B
     *   (OUPUT)
        *   MATCH_1 FOR IP
     *   (INPUT FOR SAME IP)
        *   EVENT_TYPE_A && EVENT_TYPE_B && EVENT_TYPE_C
     *   (OUPUT)
        *   MATCH_2 FOR IP
     *   (INPUT FOR SAME IP)
        *   EVENT_TYPE_A || EVENT_TYPE_B && EVENT_TYPE_D
     *   (OUPUT)
        *   MATCH_3 FOR IP

To begin with, I've tried the following:

type=EventGroup3
ptype=RegExp
pattern=( EVENT_TYPE_A )(.*)
varmap= IP=2;
context= !EVENT_TYPE_A _$+{IP}
count=create EVENT_TYPE_A _$+{IP}
ptype2=RegExp
pattern2= ( EVENT_TYPE_B )(.*)
varmap2= IP=2;
context2=!EVENT_TYPE_B $+{IP}
count2=create EVENT_TYPE_B _$+{IP}
ptype3=RegExp
pattern3=FINISH
desc=TEST
action=logonly MATCH_1 FOR  $+{IP}
window=60

When I try to enter the following events:
EVENT_TYPE_A 1.1.1.1
EVENT_TYPE_A 2.2.2.2
EVENT_TYPE_B 1.1.1.1
EVENT_TYPE_B 2.2.2.2
FINISH

I have the next output:
MATCH_1 FOR 1.1.1.1

And I want to get the following output:
MATCH_1 FOR 1.1.1.1
MATCH_1 FOR 2.2.2.2

How can I solve this?

Thank you very much!
Agustín

Re: [Simple-evcorr-users] SEC CPU utilization

[Risto V. <ris...@gm...>]

>
> I mentiond as an offhand remark to Risto a profile mode that would
> count not only every rule that lead to an action, but every time the
> rule executed its regular expression. Having some sort of profile mode
> (not to be run in production) would help identify these sorts of
> issues.
>
>
I have a question about how to potentially implement this feature -- would
it be useful to calculate the total CPU time spent for processing the rule
in profile mode? That would not only include the CPU time for matching the
pattern against the event, but also the CPU time for evaluating the context
expression or executing the action list. Total CPU time would be more
useful if one is dealing with frequently matching rules (say, the ones
which match 80-90% of incoming events), and wants to identify most
expensive rule. Also, some pattern types like Cached and TValue are known
to have low computational cost, but they are sometimes used with more
complex context expressions (e.g., involving the execution of custom Perl
code). Total CPU time would allow to identify such bottlenecks.

I've added some alpha-level code into SEC and here is a small example case
for a rule with badly written regular expression:

type=single
ptype=regexp
pattern=^([0-9]+,?)*[[:punct:]]$
desc=test pattern
action=write - test pattern has matched

The regular expression expects an event which is consisting of numbers that
are optionally separated with commas, and the sequence of numbers must be
followed by a single punctuation mark. However, when one provides a very
long string consisting of digits only without terminating punctuation mark,
the regular expression engine will attempt to divide the sequence into all
possible sub-sequences during backtracking phase. Here is a fragment for
SEC dump file that highlights this issue:

Performance statistics:
============================================================
Run time: 142 seconds
User time: 63.92 seconds
System time: 0.05 seconds
Child user time: 0 seconds
Child system time: 0 seconds
Processed input lines: 1

Rule usage statistics:
============================================================

Statistics for the rules from test3.sec
(loaded at Fri Apr  3 18:23:03 2020)
------------------------------------------------------------
Rule 1 line 1 matched 0 events (test pattern) CPU time: 63.71

As one can see, SEC has processed just one event which has not matched its
only rule, but unsuccessful matching process took more than a minute. Of
course, it is possible to disable unnecessary backtracking by using
possessive quantifier:
^([0-9]+,?)*+[[:punct:]]$

Finding CPU time spent for processing a rule has one caveat -- processing
of Jump rules is a recursive process that might involve further jumps into
other rulesets, and CPU time of the Jump rule would indicate the cost of
entire recursive process. There is no easy way to subtract the CPU time of
child rules, since they might be invoked by other Jump rules that have
matched different events.

Does the idea outlined in this e-mail sound reasonable? I have also looked
into other implementation ideas, but the current one looks most useful and
involving least amount of overhead in the code base.

kind regards,
risto

Re: [Simple-evcorr-users] SEC CPU utilization

[Risto V. <ris...@gm...>]

hi Richard,


> We were doing log monitoring migration from HPOM to open-source monitoring
> tool, and using SEC for duplicate events flow reduction before passing to
> monitoring agent, in the manner as HPOM agent with built-in correlations
> was used, so the design of rules and correlations is tributary to how it
> was implemented in HPOM. There were hundreds to thousands of pattern
> conditions in HPOM per host, and the structure of their sections was as
> follows:
>
>    - HPOM: suppress unmatched conditions -> SEC: Suppress with NRegExp
>    - HPOM: suppress matched conditions -> SEC: Suppress with RegExp
>    - HPOM: message conditions (with configured time-based correlations)
>    -> SEC: Single with RegExp and GoTo -> duplicate suppress time-based
>    correlations, each consisting of 3-4 subsequent rules (Single,
>    PairWithWindow, SingleWithSuppress, depending on duplicate suppress
>    correlation type)
>
> We decided to automate conversion of HPOM configurations to SEC rules, so
> here was not too much space for conceptual improvements over HPOM concepts
> (e.g. by doing deeper analysis of configurations and actual log traffic),
> and we relied on the premise, that those HPOM configurations are OK, and
> tuned by years of development and operations, so the automated conversion
> was 1:1.
>
> Cca 50 log files per host are of several types (according to message
> structure), but each file was monitored in HPOM independently on each
> other, therefore after 1:1 conversion also in SEC is each file monitoring
> independently, however, there is some maybe uglier "configuration
> redundancy" for log files of the same type, as it was in HPOM. The static
> order of conditions in HPOM is preserved also in generated SEC rules.
>
>
Since I have used HPOM in the past, perhaps I can offer some comments and
advise here. As far as I remember, HPOM agent does not support hierarchical
arrangement of log file monitoring policies and rules. You mentioned that
existing HPOM configuration was converted on 1:1 basis -- does that mean
that SEC is configured to use a number of rule files, where each rule file
corresponds to some HPOM policy, and all rules are applied against all
input messages from 50 input files? Since such use of rules is not very
efficient, perhaps you could introduce hierarchical rule arrangement in the
following way. First, assign each rule file to a configuration file set
with Options rule. For example, if you have an HPOM policy called "sshd",
you can use the following Options rule in the beginning of corresponding
rule file:

type=Options
joincfset=sshd
procallin=no

Secondly, run SEC with --intcontexts flag which will set an internal
context for each file (e.g., the file /var/log/secure will have the context
_FILE_EVENT_/var/log/secure by default, but you can also set a custom
context name in command line). Finally, create a special rule file (e.g.,
main.sec) which will route messages to relevant rule files. For example,
suppose that HPOM policies "sshd" and "sudo" have been used for
/var/log/secure, and there are two SEC rule files that contain Options
rules as described above. For handling messages from /var/log/secure, you
can enter the following Jump rule into main.sec rule file:

type=Jump
context=[ _FILE_EVENT_/var/log/secure ]
ptype=TValue
pattern=True
desc=direct /var/log/secure events to relevant rules files
cfset=sshd sudo

Such configuration ensures that messages of /var/log/secure are being
matched against relevant rules only, and rule files not associated with
/var/log/secure will not be applied. Also, only the Jump rules from
main.sec are applied against the entire event stream, while rules from
other files receive their input from Jump rules in main.sec and are thus
applied only for relevant messages. Perhaps you have already applied this
technique, but if not, I would definitely recommend to use this
optimization, since it is very likely to reduce the CPU load.

hope this helps,
risto

Re: [Simple-evcorr-users] action-list checking if log file is already open by SEC

[Risto V. <ris...@gm...>]

hi Richard,

let me provide few comments here. Since you mentioned setting up an
associative array (Perl hash) that contains input file names, contexts are
actually implemented as a Perl hash as well in SEC code. Checking for the
presence of the context involves a search for some key in a hash, and it is
very fast in Perl. Therefore, if you are concerned about search speed,
using your own hash would not offer significant performance improvements.
However, using a custom hash has one important advantage -- internal hash
of SEC contexts has to be scanned periodically (by default once a second),
in order to detect expired contexts and execute their action lists. If you
have stored input file names in a custom hash, you don't have that
housekeeping overhead.

For estimating the impact of that overhead, you could run couple of tests
with a larger number of contexts in your environment. I have just done
couple of quick experiments on my 4 year old laptop (intel core i5) which
involve creating N contexts and letting SEC run without any other input for
about 15-20 minutes, measuring consumed CPU time at the end of the
experiment. With 1,000 contexts, CPU was utilized by 0.4%, with 10,000
contexts the utilization was 2.2%, and with 100,000 contexts the
utilization reached 16.3%. Given those figures, I would say that if you
have only few thousand contexts, housekeeping should not consume too much
of your CPU time. But if you want to employ contexts for different purposes
like implementing a large blacklist of IP addresses, custom Perl hash is
probably a better way to go (and it would be easier to search it from
PerlFunc patterns).

hope this helps,
risto

Kontakt Richard Ostrochovský (<ric...@gm...>) kirjutas
kuupäeval N, 2. aprill 2020 kell 01:18:

> Hi Risto,
>
> thank you for the solution. There is also concern about potential
> performance impact in case of e.g. thousands of files being added by
> addinput with creating extra context for each of them.
>
> Another way could be e.g. maintaining (associative) array in action list,
> with keys of paths to addinput files, reduced by dropinput, similarly as
> you suggest with context creation.
>
> And maybe the most efficient, having built-in way, how to access this list
> of open files, without custom implementation of it.
>
> How do you see these potential options in comparison by performance?
>
> Richard
>
> št 20. 2. 2020 o 21:23 Risto Vaarandi <ris...@gm...>
> napísal(a):
>
>> hi Richard,
>>
>> I think this scenario is best addressed by creating a relevant SEC
>> context when 'addinput' action is called. In fact, handling such scenarios
>> is one of the purposes of contexts, and here is an example rule which
>> illustrates this idea:
>>
>> type=single
>> ptype=regexp
>> pattern=start monitoring (\S+)
>> context=!FILE_$1_MONITORED
>> desc=add $1 to list of inputs
>> action=addinput $1; create FILE_$1_MONITORED
>>
>> Whenever "start monitoring <filename>" event appears, the rule will match
>> only if context FILE_<filename>_MONITORED does not exist. If rule matches,
>> it executes 'addinput' action for the given file and creates the context,
>> in order to manifest the fact that 'addinput' has already been executed for
>> the given file. Also, as you can see from the above rule, the presence of
>> the context for a file will prevent the execution of 'addinput' again for
>> this file. In order to keep contexts in sync with files that are monitored,
>> the context for a file should be deleted when 'dropinput' action is
>> executed for it.
>>
>> Note that when HUP signal is received, SEC will stop monitoring input
>> files set up with 'addinput' action. However, on receiving HUP signal SEC
>> will also drop all its contexts, so there is no need to take any extra
>> steps in that case.
>>
>> Hope this helps,
>> risto
>>
>> Kontakt Richard Ostrochovský (<ric...@gm...>) kirjutas
>> kuupäeval N, 20. veebruar 2020 kell 21:43:
>>
>>> Hello Risto and friends,
>>>
>>> having mechanism for dynamic opening (addinput) and closing (dropinput)
>>> files, I would like to be able to check, if the file is already opened,
>>> before trying to open it again, to avoid it. That way I would like to
>>> eliminate this error message from SEC log (present also in debug level 3):
>>>
>>> Dynamic input file '/path/to/file.log' already exists in the list of
>>> inputs, can't add
>>>
>>> This information is present in sec.dump, but maybe there exists more
>>> instant and straightforward way how to achieve it (without parsing
>>> intermediary files).
>>>
>>> Thank you.
>>>
>>> Richard
>>> _______________________________________________
>>> Simple-evcorr-users mailing list
>>> Sim...@li...
>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>>
>> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>

Re: [Simple-evcorr-users] SEC CPU utilization

[Clayton D. <cd...@lo...>]

Hi Richard,
Excellent feedback!
> Maybe we could make a teleconference call someday, as wanna-be participants of such community, and brainstorm
I would be happy to facilitate this – we use Zoom (enterprise).




[cid:image001.png@01D3FCD9.6DFA7670]First they'll say it's impossible, then they'll say it was inevitable
Clayton Dukes
CEO
LogZilla Corp
m:
919-600-3198
a:
4819 Emperor Boulevard
Raleigh, NC 27703
w:
logzilla.net<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.logzilla.net%2F&data=02%7C01%7Ccdukes%40logzilla.net%7C437136cdd3a14c8e3f3208d76a18c814%7C17fac5c255634489bef45cda2e65588f%7C0%7C0%7C637094527007060540&sdata=afr7xXoZZUxg1t%2Bw9QlnNeCThgwDIfJgK5SRypTrdS8%3D&reserved=0>  e: cd...@lo...<mailto:cd...@lo...>
[http://cdn2.hubspot.net/hubfs/184235/dev_images/signature_app/twitter_sig.png]<https://twitter.com/logzilla>  [http://cdn2.hubspot.net/hubfs/184235/dev_images/signature_app/linkedin_sig.png] <https://www.linkedin.com/in/lzcdukes/>


[signature_825941423]
The NEO platform delivers scalable, real-time, intelligent insight for enterprise networks worldwide





From: Richard Ostrochovský <ric...@gm...>
Date: Wednesday, April 1, 2020 at 6:15 PM
To: "sim...@li..." <sim...@li...>
Subject: Re: [Simple-evcorr-users] SEC CPU utilization

Hello guys,

thank you for many tips. I have carefully read through e-mails from Clayton, Risto, John, David, and Dusan, and I am summary reacting to all of it with this e-mail, as more reactions were in similar spirit.

To my primary question: I am satisfied, that log messages are not lost on "heavy load", but just (gradually) delayed (currently, delay is not the problem in our case, as 100% CPU consumption is not all the time, and SEC has the time "to catch the train in the next station"). These are application logs and our messages load is variable, but not the order of as high Dusan's is, and I am still not monitoring this flow (discussed here: https://sourceforge.net/p/simple-evcorr/mailman/message/36910235/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsourceforge.net%2Fp%2Fsimple-evcorr%2Fmailman%2Fmessage%2F36910235%2F&data=02%7C01%7Ccdukes%40logzilla.net%7C2b5460a6bac94718c7b608d7d68a3baa%7C17fac5c255634489bef45cda2e65588f%7C0%7C1%7C637213761535202266&sdata=MEG0tE0sDL3pRP8s6QCMxTurkj8SWx9Q%2F2%2F%2F1m9CGvY%3D&reserved=0>, medium priority task in my Jira).

To my secondary questions about finding bottlenecks and optimizations, I'll summarize the recommendations (with my comments), as some was mentioned by more of you:

  *   placing most "heavy-load" rules first (problem: unpredictability; potential solution: re-ordering rules dynamically, according to latest stats)
  *   hierarchical / tree-like setup (Jump rule in Risto's examples, or GoTo section ended with Suppress, as in our case)
  *   more instances of SEC (but there is also risk to consume not only 1 CPU, but as many CPUs, as many SEC instances running)
  *   preprocessing of log messages with some kind of classification / branching (out of SEC)

     *   LogZilla (Clayton,  http://demo.logzilla.net/help/event_correlation/intro_to_event_correlation<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdemo.logzilla.net%2Fhelp%2Fevent_correlation%2Fintro_to_event_correlation&data=02%7C01%7Ccdukes%40logzilla.net%7C2b5460a6bac94718c7b608d7d68a3baa%7C17fac5c255634489bef45cda2e65588f%7C0%7C1%7C637213761535202266&sdata=QEENl%2BQmig9eQmcd8KXnSV6ZDro1lKsOSont0rkCOUI%3D&reserved=0>)
     *   rsyslog mmnormalize (David)
     *   syslog-ng PatternDB (Dusan)

  *   RegExp optimization
  *   reducing count of contexts, active deletion where suitable
As some of you are interested in the design, I would need to ask for permission of my employer, to publish it, as I am not the author of rules design. I just created automation of their generation (topic discussed here: https://sourceforge.net/p/simple-evcorr/mailman/message/36867012/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsourceforge.net%2Fp%2Fsimple-evcorr%2Fmailman%2Fmessage%2F36867012%2F&data=02%7C01%7Ccdukes%40logzilla.net%7C2b5460a6bac94718c7b608d7d68a3baa%7C17fac5c255634489bef45cda2e65588f%7C0%7C1%7C637213761535212259&sdata=2FPuYoucQuOuBP8X1aiBPJsxbxGAqfPxwLxQDUe%2FCOc%3D&reserved=0>) and minor optimizations. Although I consulted a lot in this mailinglist, mainly with Risto, original pre-conslultations designs remained unchanged.  But for now, I will at least try to high-level describe it, but at first, it is important to mention, how it originated:

We were doing log monitoring migration from HPOM to open-source monitoring tool, and using SEC for duplicate events flow reduction before passing to monitoring agent, in the manner as HPOM agent with built-in correlations was used, so the design of rules and correlations is tributary to how it was implemented in HPOM. There were hundreds to thousands of pattern conditions in HPOM per host, and the structure of their sections was as follows:

  *   HPOM: suppress unmatched conditions -> SEC: Suppress with NRegExp
  *   HPOM: suppress matched conditions -> SEC: Suppress with RegExp
  *   HPOM: message conditions (with configured time-based correlations) -> SEC: Single with RegExp and GoTo -> duplicate suppress time-based correlations, each consisting of 3-4 subsequent rules (Single, PairWithWindow, SingleWithSuppress, depending on duplicate suppress correlation type)
We decided to automate conversion of HPOM configurations to SEC rules, so here was not too much space for conceptual improvements over HPOM concepts (e.g. by doing deeper analysis of configurations and actual log traffic), and we relied on the premise, that those HPOM configurations are OK, and tuned by years of development and operations, so the automated conversion was 1:1.

Cca 50 log files per host are of several types (according to message structure), but each file was monitored in HPOM independently on each other, therefore after 1:1 conversion also in SEC is each file monitoring independently, however, there is some maybe uglier "configuration redundancy" for log files of the same type, as it was in HPOM. The static order of conditions in HPOM is preserved also in generated SEC rules.

I think, branching multi-level cascade structure with GoTo is OK. What is harder is, that the majority of log files has multi-line messages, and some logs are multi-file (discussed here: https://sourceforge.net/p/simple-evcorr/mailman/message/36861921/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsourceforge.net%2Fp%2Fsimple-evcorr%2Fmailman%2Fmessage%2F36861921%2F&data=02%7C01%7Ccdukes%40logzilla.net%7C2b5460a6bac94718c7b608d7d68a3baa%7C17fac5c255634489bef45cda2e65588f%7C0%7C1%7C637213761535212259&sdata=VSlMdMD0IH9Z46bDgI2VAvtoPTc9xwUMJp43SWGN0tU%3D&reserved=0>).

Only one instance of SEC is running per host, so single CPU can be consumed maximum. No other preprocessing/classification is used, SEC is doing all the work.

Anyway, I am happy with Risto's answer, that 100% CPU utilization is not causing losses, just delays, and currently migration is over, and we'll see, if there will be demand for further optimizations in the future, so I don't want to burden you guys with some rules analysis and review, and concrete optimization advices.

But I am also interested in Risto's and Clayton's initiatives about rules sharing. I think, that further step to building user community with rules sharing could be some kind of formalization, "best practise rules templates", "rulesets / templates catalogue", possibly with possibility to choose correlation doing some specific task, fill in parameters, and generate SEC rules automatically... I see wide potential here, to enable SEC users not re-inventing wheels, but build on verified setups, as discussed here:  https://sourceforge.net/p/simple-evcorr/mailman/message/36867012/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsourceforge.net%2Fp%2Fsimple-evcorr%2Fmailman%2Fmessage%2F36867012%2F&data=02%7C01%7Ccdukes%40logzilla.net%7C2b5460a6bac94718c7b608d7d68a3baa%7C17fac5c255634489bef45cda2e65588f%7C0%7C1%7C637213761535222253&sdata=vY1Fmj7gpwraqeG2SKdoxqxnjMbfsmUWv4K3pfLu8b0%3D&reserved=0>. I may suggest my employer to join such initiative with our designs. Maybe we could make a teleconference call someday, as wanna-be participants of such community, and brainstorm about possibilities, as publishing of rulesets on GitHub or SourceForge is great, but maybe we could do yet more, and build ineractive catalogue beyond file sharing.

Richard

st 25. 3. 2020 o 17:52 Risto Vaarandi <ris...@gm...<mailto:ris...@gm...>> napísal(a):
hi Richard,

if CPU utilization has reached 100%, no rules or log file events would be skipped, but SEC would simply not be able to process events at their arrival rate and fall behind of events. If your events include timestamps, you would probably see events with past timestamps in dump file (among other data, SEC dump file reports the last processed event for each input file). As for debugging the reasons of high CPU utilization, I would recommend to have a look into rule match statistics, and make sure that rules with most matches appear in the top of their rule files (if possible). However, current versions of SEC do not report the CPU time spent on matching each pattern against events.

Just out of curiosity -- how many rules are you currently having in your rule base, and are all these rules connected to each other? How many events are you currently receiving per second? Also, are all 50 input files containing the same event types (e.g., httpd events) that need to be processed by all rules? If this is not the case, and each input file contains different events which are processed by different rules, I would strongly recommend to consider a hierarchical setup for your rule files. The principles of hierarchical setup have been described in SEC official documentation, for example: http://simple-evcorr.github.io/man.html#lbBE<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsimple-evcorr.github.io%2Fman.html%23lbBE&data=02%7C01%7Ccdukes%40logzilla.net%7C2b5460a6bac94718c7b608d7d68a3baa%7C17fac5c255634489bef45cda2e65588f%7C0%7C1%7C637213761535222253&sdata=Tjst%2BIdOnDt7EhzWSomDGHopaWHAw3iRhqCRmBNh4lg%3D&reserved=0>. Also, there is a recent paper which provides a relevant example: https://ristov.github.io/publications/cogsima15-sec-web.pdf<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fristov.github.io%2Fpublications%2Fcogsima15-sec-web.pdf&data=02%7C01%7Ccdukes%40logzilla.net%7C2b5460a6bac94718c7b608d7d68a3baa%7C17fac5c255634489bef45cda2e65588f%7C0%7C1%7C637213761535232247&sdata=slkTBZWNKvzlWER%2BABF7o7zRyuV%2FGUFuXWWALYeWteU%3D&reserved=0>. In addition, you could also consider running several instances of SEC for your input files. For example, if some input files contain messages from a specific application which are processed by few specific rule files, a separate SEC process could be started for handling these messages with given rule files. In that way, it might be possible to divide the rule files and input files into several independent groups, and having a separate SEC process for each group allows to balance the load across several CPU's.

hope this helps,
risto

Kontakt Richard Ostrochovský (<ric...@gm...<mailto:ric...@gm...>>) kirjutas kuupäeval K, 25. märts 2020 kell 17:07:
Hello friends,

I have SEC monitoring over 50 log files with various correlations, and it is consuming 100% of single CPU (luckily on 10-CPU machine, so not whole system affected, as SEC is single-CPU application).

This could mean, that SEC does not prosecute processing of all rules, and I am curious, what are possible effects, if this means increasing delays (first in, processing, first out), or skipping some lines from input files, or anything other (?).

And how to troubleshoot, finding bottlenecks. I can see quantities of log messages per contexts or log files in sec.dump, this is some indicator. Are there also other indicators? Is it possible, somehow, see also processing times of patterns (per rules)?

Thank you in advance.

Richard
_______________________________________________
Simple-evcorr-users mailing list
Sim...@li...<mailto:Sim...@li...>
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fsimple-evcorr-users&data=02%7C01%7Ccdukes%40logzilla.net%7C2b5460a6bac94718c7b608d7d68a3baa%7C17fac5c255634489bef45cda2e65588f%7C0%7C1%7C637213761535232247&sdata=E3QOGop0f4cIgauhYdYic72PfgwuSiQPM4mTsMaG8zc%3D&reserved=0>
IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Logzilla Corporation may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.

Re: [Simple-evcorr-users] action-list checking if log file is already open by SEC

[David L. <da...@la...>]

perl operations on a hash are surprisingly efficient. If you store your context 
in a hash, it can be very efficient to add/remove/check specific items. What is 
not efficient is aging things out based on time.

David Lang

On Thu, 2 Apr 2020, Richard Ostrochovský wrote:

> Date: Thu, 2 Apr 2020 00:17:40 +0200
> From: Richard Ostrochovský <ric...@gm...>
> To: sim...@li...
> Subject: Re: [Simple-evcorr-users] action-list checking if log file is already
>      open by SEC
> 
> Hi Risto,
>
> thank you for the solution. There is also concern about potential
> performance impact in case of e.g. thousands of files being added by
> addinput with creating extra context for each of them.
>
> Another way could be e.g. maintaining (associative) array in action list,
> with keys of paths to addinput files, reduced by dropinput, similarly as
> you suggest with context creation.
>
> And maybe the most efficient, having built-in way, how to access this list
> of open files, without custom implementation of it.
>
> How do you see these potential options in comparison by performance?
>
> Richard
>
> št 20. 2. 2020 o 21:23 Risto Vaarandi <ris...@gm...> napísal(a):
>
>> hi Richard,
>>
>> I think this scenario is best addressed by creating a relevant SEC context
>> when 'addinput' action is called. In fact, handling such scenarios is one
>> of the purposes of contexts, and here is an example rule which illustrates
>> this idea:
>>
>> type=single
>> ptype=regexp
>> pattern=start monitoring (\S+)
>> context=!FILE_$1_MONITORED
>> desc=add $1 to list of inputs
>> action=addinput $1; create FILE_$1_MONITORED
>>
>> Whenever "start monitoring <filename>" event appears, the rule will match
>> only if context FILE_<filename>_MONITORED does not exist. If rule matches,
>> it executes 'addinput' action for the given file and creates the context,
>> in order to manifest the fact that 'addinput' has already been executed for
>> the given file. Also, as you can see from the above rule, the presence of
>> the context for a file will prevent the execution of 'addinput' again for
>> this file. In order to keep contexts in sync with files that are monitored,
>> the context for a file should be deleted when 'dropinput' action is
>> executed for it.
>>
>> Note that when HUP signal is received, SEC will stop monitoring input
>> files set up with 'addinput' action. However, on receiving HUP signal SEC
>> will also drop all its contexts, so there is no need to take any extra
>> steps in that case.
>>
>> Hope this helps,
>> risto
>>
>> Kontakt Richard Ostrochovský (<ric...@gm...>) kirjutas
>> kuupäeval N, 20. veebruar 2020 kell 21:43:
>>
>>> Hello Risto and friends,
>>>
>>> having mechanism for dynamic opening (addinput) and closing (dropinput)
>>> files, I would like to be able to check, if the file is already opened,
>>> before trying to open it again, to avoid it. That way I would like to
>>> eliminate this error message from SEC log (present also in debug level 3):
>>>
>>> Dynamic input file '/path/to/file.log' already exists in the list of
>>> inputs, can't add
>>>
>>> This information is present in sec.dump, but maybe there exists more
>>> instant and straightforward way how to achieve it (without parsing
>>> intermediary files).
>>>
>>> Thank you.
>>>
>>> Richard
>>> _______________________________________________
>>> Simple-evcorr-users mailing list
>>> Sim...@li...
>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>>
>>
>

Re: [Simple-evcorr-users] SEC CPU utilization

[David L. <da...@la...>]

re: multi-line rules, my thought is to convert the newlines to some escape 
sequence and then treat the results as a single line.

I tend to not want to run SEC on each system as it can impact that system (both 
CPU utilization and disk I/O). Instead I make SEC part of the log processing 
infrastructure, ideally on a dedicated system (or set of systems) so having 
multiple instances, each eating a core, is a feature not a bug ;-)

David Lang

On Thu, 2 Apr 2020, Richard Ostrochovský wrote:

> Date: Thu, 2 Apr 2020 00:15:08 +0200
> From: Richard Ostrochovský <ric...@gm...>
> To: sim...@li...
> Subject: Re: [Simple-evcorr-users] SEC CPU utilization
> 
> Hello guys,
>
> thank you for many tips. I have carefully read through e-mails from
> Clayton, Risto, John, David, and Dusan, and I am summary reacting to all of
> it with this e-mail, as more reactions were in similar spirit.
>
> To my primary question: I am satisfied, that log messages are not lost on
> "heavy load", but just (gradually) delayed (currently, delay is not the
> problem in our case, as 100% CPU consumption is not all the time, and SEC
> has the time "to catch the train in the next station"). These are
> application logs and our messages load is variable, but not the order of as
> high Dusan's is, and I am still not monitoring this flow (discussed here:
> https://sourceforge.net/p/simple-evcorr/mailman/message/36910235/, medium
> priority task in my Jira).
>
> To my secondary questions about finding bottlenecks and optimizations, I'll
> summarize the recommendations (with my comments), as some was mentioned by
> more of you:
>
>   - placing most "heavy-load" rules first (problem: unpredictability;
>   potential solution: re-ordering rules dynamically, according to latest
>   stats)
>   - hierarchical / tree-like setup (Jump rule in Risto's examples, or GoTo
>   section ended with Suppress, as in our case)
>   - more instances of SEC (but there is also risk to consume not only 1
>   CPU, but as many CPUs, as many SEC instances running)
>   - preprocessing of log messages with some kind of classification /
>   branching (out of SEC)
>      - LogZilla (Clayton,
>      http://demo.logzilla.net/help/event_correlation/intro_to_event_correlation
>      )
>      - rsyslog mmnormalize (David)
>      - syslog-ng PatternDB (Dusan)
>   - RegExp optimization
>   - reducing count of contexts, active deletion where suitable
>
> As some of you are interested in the design, I would need to ask for
> permission of my employer, to publish it, as I am not the author of rules
> design. I just created automation of their generation (topic discussed
> here: https://sourceforge.net/p/simple-evcorr/mailman/message/36867012/)
> and minor optimizations. Although I consulted a lot in this mailinglist,
> mainly with Risto, original pre-conslultations designs remained unchanged.
> But for now, I will at least try to high-level describe it, but at first,
> it is important to mention, how it originated:
>
> We were doing log monitoring migration from HPOM to open-source monitoring
> tool, and using SEC for duplicate events flow reduction before passing to
> monitoring agent, in the manner as HPOM agent with built-in correlations
> was used, so the design of rules and correlations is tributary to how it
> was implemented in HPOM. There were hundreds to thousands of pattern
> conditions in HPOM per host, and the structure of their sections was as
> follows:
>
>   - HPOM: suppress unmatched conditions -> SEC: Suppress with NRegExp
>   - HPOM: suppress matched conditions -> SEC: Suppress with RegExp
>   - HPOM: message conditions (with configured time-based correlations) ->
>   SEC: Single with RegExp and GoTo -> duplicate suppress time-based
>   correlations, each consisting of 3-4 subsequent rules (Single,
>   PairWithWindow, SingleWithSuppress, depending on duplicate suppress
>   correlation type)
>
> We decided to automate conversion of HPOM configurations to SEC rules, so
> here was not too much space for conceptual improvements over HPOM concepts
> (e.g. by doing deeper analysis of configurations and actual log traffic),
> and we relied on the premise, that those HPOM configurations are OK, and
> tuned by years of development and operations, so the automated conversion
> was 1:1.
>
> Cca 50 log files per host are of several types (according to message
> structure), but each file was monitored in HPOM independently on each
> other, therefore after 1:1 conversion also in SEC is each file monitoring
> independently, however, there is some maybe uglier "configuration
> redundancy" for log files of the same type, as it was in HPOM. The static
> order of conditions in HPOM is preserved also in generated SEC rules.
>
> I think, branching multi-level cascade structure with GoTo is OK. What is
> harder is, that the majority of log files has multi-line messages, and some
> logs are multi-file (discussed here:
> https://sourceforge.net/p/simple-evcorr/mailman/message/36861921/).
>
> Only one instance of SEC is running per host, so single CPU can be consumed
> maximum. No other preprocessing/classification is used, SEC is doing all
> the work.
>
> Anyway, I am happy with Risto's answer, that 100% CPU utilization is not
> causing losses, just delays, and currently migration is over, and we'll
> see, if there will be demand for further optimizations in the future, so I
> don't want to burden you guys with some rules analysis and review, and
> concrete optimization advices.
>
> But I am also interested in Risto's and Clayton's initiatives about rules
> sharing. I think, that further step to building user community with rules
> sharing could be some kind of formalization, "best practise rules
> templates", "rulesets / templates catalogue", possibly with possibility to
> choose correlation doing some specific task, fill in parameters, and
> generate SEC rules automatically... I see wide potential here, to enable
> SEC users not re-inventing wheels, but build on verified setups, as
> discussed here:
> https://sourceforge.net/p/simple-evcorr/mailman/message/36867012/. I may
> suggest my employer to join such initiative with our designs. Maybe we
> could make a teleconference call someday, as wanna-be participants of such
> community, and brainstorm about possibilities, as publishing of rulesets on
> GitHub or SourceForge is great, but maybe we could do yet more, and build
> ineractive catalogue beyond file sharing.
>
> Richard
>
> st 25. 3. 2020 o 17:52 Risto Vaarandi <ris...@gm...> napísal(a):
>
>> hi Richard,
>>
>> if CPU utilization has reached 100%, no rules or log file events would be
>> skipped, but SEC would simply not be able to process events at their
>> arrival rate and fall behind of events. If your events include timestamps,
>> you would probably see events with past timestamps in dump file (among
>> other data, SEC dump file reports the last processed event for each input
>> file). As for debugging the reasons of high CPU utilization, I would
>> recommend to have a look into rule match statistics, and make sure that
>> rules with most matches appear in the top of their rule files (if
>> possible). However, current versions of SEC do not report the CPU time
>> spent on matching each pattern against events.
>>
>> Just out of curiosity -- how many rules are you currently having in your
>> rule base, and are all these rules connected to each other? How many events
>> are you currently receiving per second? Also, are all 50 input files
>> containing the same event types (e.g., httpd events) that need to be
>> processed by all rules? If this is not the case, and each input file
>> contains different events which are processed by different rules, I would
>> strongly recommend to consider a hierarchical setup for your rule files.
>> The principles of hierarchical setup have been described in SEC official
>> documentation, for example: http://simple-evcorr.github.io/man.html#lbBE.
>> Also, there is a recent paper which provides a relevant example:
>> https://ristov.github.io/publications/cogsima15-sec-web.pdf. In addition,
>> you could also consider running several instances of SEC for your input
>> files. For example, if some input files contain messages from a specific
>> application which are processed by few specific rule files, a separate SEC
>> process could be started for handling these messages with given rule files.
>> In that way, it might be possible to divide the rule files and input files
>> into several independent groups, and having a separate SEC process for each
>> group allows to balance the load across several CPU's.
>>
>> hope this helps,
>> risto
>>
>> Kontakt Richard Ostrochovský (<ric...@gm...>) kirjutas
>> kuupäeval K, 25. märts 2020 kell 17:07:
>>
>>> Hello friends,
>>>
>>> I have SEC monitoring over 50 log files with various correlations, and it
>>> is consuming 100% of single CPU (luckily on 10-CPU machine, so not whole
>>> system affected, as SEC is single-CPU application).
>>>
>>> This could mean, that SEC does not prosecute processing of all rules, and
>>> I am curious, what are possible effects, if this means increasing delays
>>> (first in, processing, first out), or skipping some lines from input files,
>>> or anything other (?).
>>>
>>> And how to troubleshoot, finding bottlenecks. I can see quantities of log
>>> messages per contexts or log files in sec.dump, this is some indicator. Are
>>> there also other indicators? Is it possible, somehow, see also processing
>>> times of patterns (per rules)?
>>>
>>> Thank you in advance.
>>>
>>> Richard
>>> _______________________________________________
>>> Simple-evcorr-users mailing list
>>> Sim...@li...
>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>>
>>
>

Re: [Simple-evcorr-users] action-list checking if log file is already open by SEC

[Richard O. <ric...@gm...>]

Hi Risto,

thank you for the solution. There is also concern about potential
performance impact in case of e.g. thousands of files being added by
addinput with creating extra context for each of them.

Another way could be e.g. maintaining (associative) array in action list,
with keys of paths to addinput files, reduced by dropinput, similarly as
you suggest with context creation.

And maybe the most efficient, having built-in way, how to access this list
of open files, without custom implementation of it.

How do you see these potential options in comparison by performance?

Richard

št 20. 2. 2020 o 21:23 Risto Vaarandi <ris...@gm...> napísal(a):

> hi Richard,
>
> I think this scenario is best addressed by creating a relevant SEC context
> when 'addinput' action is called. In fact, handling such scenarios is one
> of the purposes of contexts, and here is an example rule which illustrates
> this idea:
>
> type=single
> ptype=regexp
> pattern=start monitoring (\S+)
> context=!FILE_$1_MONITORED
> desc=add $1 to list of inputs
> action=addinput $1; create FILE_$1_MONITORED
>
> Whenever "start monitoring <filename>" event appears, the rule will match
> only if context FILE_<filename>_MONITORED does not exist. If rule matches,
> it executes 'addinput' action for the given file and creates the context,
> in order to manifest the fact that 'addinput' has already been executed for
> the given file. Also, as you can see from the above rule, the presence of
> the context for a file will prevent the execution of 'addinput' again for
> this file. In order to keep contexts in sync with files that are monitored,
> the context for a file should be deleted when 'dropinput' action is
> executed for it.
>
> Note that when HUP signal is received, SEC will stop monitoring input
> files set up with 'addinput' action. However, on receiving HUP signal SEC
> will also drop all its contexts, so there is no need to take any extra
> steps in that case.
>
> Hope this helps,
> risto
>
> Kontakt Richard Ostrochovský (<ric...@gm...>) kirjutas
> kuupäeval N, 20. veebruar 2020 kell 21:43:
>
>> Hello Risto and friends,
>>
>> having mechanism for dynamic opening (addinput) and closing (dropinput)
>> files, I would like to be able to check, if the file is already opened,
>> before trying to open it again, to avoid it. That way I would like to
>> eliminate this error message from SEC log (present also in debug level 3):
>>
>> Dynamic input file '/path/to/file.log' already exists in the list of
>> inputs, can't add
>>
>> This information is present in sec.dump, but maybe there exists more
>> instant and straightforward way how to achieve it (without parsing
>> intermediary files).
>>
>> Thank you.
>>
>> Richard
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> Sim...@li...
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>

Re: [Simple-evcorr-users] SEC CPU utilization

[Richard O. <ric...@gm...>]

Hello guys,

thank you for many tips. I have carefully read through e-mails from
Clayton, Risto, John, David, and Dusan, and I am summary reacting to all of
it with this e-mail, as more reactions were in similar spirit.

To my primary question: I am satisfied, that log messages are not lost on
"heavy load", but just (gradually) delayed (currently, delay is not the
problem in our case, as 100% CPU consumption is not all the time, and SEC
has the time "to catch the train in the next station"). These are
application logs and our messages load is variable, but not the order of as
high Dusan's is, and I am still not monitoring this flow (discussed here:
https://sourceforge.net/p/simple-evcorr/mailman/message/36910235/, medium
priority task in my Jira).

To my secondary questions about finding bottlenecks and optimizations, I'll
summarize the recommendations (with my comments), as some was mentioned by
more of you:

   - placing most "heavy-load" rules first (problem: unpredictability;
   potential solution: re-ordering rules dynamically, according to latest
   stats)
   - hierarchical / tree-like setup (Jump rule in Risto's examples, or GoTo
   section ended with Suppress, as in our case)
   - more instances of SEC (but there is also risk to consume not only 1
   CPU, but as many CPUs, as many SEC instances running)
   - preprocessing of log messages with some kind of classification /
   branching (out of SEC)
      - LogZilla (Clayton,
      http://demo.logzilla.net/help/event_correlation/intro_to_event_correlation
      )
      - rsyslog mmnormalize (David)
      - syslog-ng PatternDB (Dusan)
   - RegExp optimization
   - reducing count of contexts, active deletion where suitable

As some of you are interested in the design, I would need to ask for
permission of my employer, to publish it, as I am not the author of rules
design. I just created automation of their generation (topic discussed
here: https://sourceforge.net/p/simple-evcorr/mailman/message/36867012/)
and minor optimizations. Although I consulted a lot in this mailinglist,
mainly with Risto, original pre-conslultations designs remained unchanged.
But for now, I will at least try to high-level describe it, but at first,
it is important to mention, how it originated:

We were doing log monitoring migration from HPOM to open-source monitoring
tool, and using SEC for duplicate events flow reduction before passing to
monitoring agent, in the manner as HPOM agent with built-in correlations
was used, so the design of rules and correlations is tributary to how it
was implemented in HPOM. There were hundreds to thousands of pattern
conditions in HPOM per host, and the structure of their sections was as
follows:

   - HPOM: suppress unmatched conditions -> SEC: Suppress with NRegExp
   - HPOM: suppress matched conditions -> SEC: Suppress with RegExp
   - HPOM: message conditions (with configured time-based correlations) ->
   SEC: Single with RegExp and GoTo -> duplicate suppress time-based
   correlations, each consisting of 3-4 subsequent rules (Single,
   PairWithWindow, SingleWithSuppress, depending on duplicate suppress
   correlation type)

We decided to automate conversion of HPOM configurations to SEC rules, so
here was not too much space for conceptual improvements over HPOM concepts
(e.g. by doing deeper analysis of configurations and actual log traffic),
and we relied on the premise, that those HPOM configurations are OK, and
tuned by years of development and operations, so the automated conversion
was 1:1.

Cca 50 log files per host are of several types (according to message
structure), but each file was monitored in HPOM independently on each
other, therefore after 1:1 conversion also in SEC is each file monitoring
independently, however, there is some maybe uglier "configuration
redundancy" for log files of the same type, as it was in HPOM. The static
order of conditions in HPOM is preserved also in generated SEC rules.

I think, branching multi-level cascade structure with GoTo is OK. What is
harder is, that the majority of log files has multi-line messages, and some
logs are multi-file (discussed here:
https://sourceforge.net/p/simple-evcorr/mailman/message/36861921/).

Only one instance of SEC is running per host, so single CPU can be consumed
maximum. No other preprocessing/classification is used, SEC is doing all
the work.

Anyway, I am happy with Risto's answer, that 100% CPU utilization is not
causing losses, just delays, and currently migration is over, and we'll
see, if there will be demand for further optimizations in the future, so I
don't want to burden you guys with some rules analysis and review, and
concrete optimization advices.

But I am also interested in Risto's and Clayton's initiatives about rules
sharing. I think, that further step to building user community with rules
sharing could be some kind of formalization, "best practise rules
templates", "rulesets / templates catalogue", possibly with possibility to
choose correlation doing some specific task, fill in parameters, and
generate SEC rules automatically... I see wide potential here, to enable
SEC users not re-inventing wheels, but build on verified setups, as
discussed here:
https://sourceforge.net/p/simple-evcorr/mailman/message/36867012/. I may
suggest my employer to join such initiative with our designs. Maybe we
could make a teleconference call someday, as wanna-be participants of such
community, and brainstorm about possibilities, as publishing of rulesets on
GitHub or SourceForge is great, but maybe we could do yet more, and build
ineractive catalogue beyond file sharing.

Richard

st 25. 3. 2020 o 17:52 Risto Vaarandi <ris...@gm...> napísal(a):

> hi Richard,
>
> if CPU utilization has reached 100%, no rules or log file events would be
> skipped, but SEC would simply not be able to process events at their
> arrival rate and fall behind of events. If your events include timestamps,
> you would probably see events with past timestamps in dump file (among
> other data, SEC dump file reports the last processed event for each input
> file). As for debugging the reasons of high CPU utilization, I would
> recommend to have a look into rule match statistics, and make sure that
> rules with most matches appear in the top of their rule files (if
> possible). However, current versions of SEC do not report the CPU time
> spent on matching each pattern against events.
>
> Just out of curiosity -- how many rules are you currently having in your
> rule base, and are all these rules connected to each other? How many events
> are you currently receiving per second? Also, are all 50 input files
> containing the same event types (e.g., httpd events) that need to be
> processed by all rules? If this is not the case, and each input file
> contains different events which are processed by different rules, I would
> strongly recommend to consider a hierarchical setup for your rule files.
> The principles of hierarchical setup have been described in SEC official
> documentation, for example: http://simple-evcorr.github.io/man.html#lbBE.
> Also, there is a recent paper which provides a relevant example:
> https://ristov.github.io/publications/cogsima15-sec-web.pdf. In addition,
> you could also consider running several instances of SEC for your input
> files. For example, if some input files contain messages from a specific
> application which are processed by few specific rule files, a separate SEC
> process could be started for handling these messages with given rule files.
> In that way, it might be possible to divide the rule files and input files
> into several independent groups, and having a separate SEC process for each
> group allows to balance the load across several CPU's.
>
> hope this helps,
> risto
>
> Kontakt Richard Ostrochovský (<ric...@gm...>) kirjutas
> kuupäeval K, 25. märts 2020 kell 17:07:
>
>> Hello friends,
>>
>> I have SEC monitoring over 50 log files with various correlations, and it
>> is consuming 100% of single CPU (luckily on 10-CPU machine, so not whole
>> system affected, as SEC is single-CPU application).
>>
>> This could mean, that SEC does not prosecute processing of all rules, and
>> I am curious, what are possible effects, if this means increasing delays
>> (first in, processing, first out), or skipping some lines from input files,
>> or anything other (?).
>>
>> And how to troubleshoot, finding bottlenecks. I can see quantities of log
>> messages per contexts or log files in sec.dump, this is some indicator. Are
>> there also other indicators? Is it possible, somehow, see also processing
>> times of patterns (per rules)?
>>
>> Thank you in advance.
>>
>> Richard
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> Sim...@li...
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>

[Simple-evcorr-users] updates to SEC FAQ

[Risto V. <ris...@gm...>]

hi all,

SEC FAQ has received couple of updates:
*) Q24 (https://simple-evcorr.github.io/FAQ.html#24) that describes the use
of 'addinput' and 'dropinput' actions has been updated with a second
example about tracking log files with timestamps in file names,
*) new entry Q27 (https://simple-evcorr.github.io/FAQ.html#27) has been
added that provides an example about receiving input events with socat tool
from TCP and UDP ports.

kind regards,
risto

Re: [Simple-evcorr-users] SEC CPU utilization

[David L. <da...@la...>]

I set up SEC using the omprog option in rsyslog, so rsyslog started SEC and fed 
the logs in via stdin rather than writing to disk.

On Thu, 26 Mar 2020, Dusan Sovic wrote:

> I using similar approach as David mention. I processing syslog messages from network devices (various Vendors like Arista, Cisco, Juniper etc.).
>
> We have about 7k patterns defined for match. As SEC cannot handle such regexp volume, I using syslog-ng PatternDB for pattern matching, classification and tagging.
>
> Than all matched logs are written in normalized format into file what monitored by SEC and pre-defined correlation schema is applied. This help me to reduce flow into SEC to 30 msg/s while incoming syslog message flow is about ~20k msg/s (from about 20k unique network devices). My SEC instance is eating ~ 20 – 30 % of CPU (with about 500 active event correlation operations with ~ 4k active contexts in avg).
>
> ________________________________
> Od: David Lang <da...@la...>
> Odoslané: štvrtok 26. marca 2020 1:24
> Komu: Richard Ostrochovský <ric...@gm...>
> Kópia: sim...@li... <sim...@li...>
> Predmet: Re: [Simple-evcorr-users] SEC CPU utilization
>
> Much of the time you can trivially split your rules and then run multiple copies
> of SEC, each processing a subnet of the rules on the logs that will match them.
>
> Rsyslog has a very efficient parsing capability (mmnormalize), you can use it to
> just classify the logs, or you can go a step further and have it extract the
> relevant fields and pass them to SEC in ways that make it cheaper to parse and
> process.
>
> where you do have rules that correlate across different types of logs, you may
> be able to just combine those logs into one ruleset, or you can have parallel
> instances output messages to say that their part of the correlation has tested
> true and then have another instance that processes these partial correlation
> messages and decide if the combined correlation is matched.
>
> David Lang
>
>
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>

Re: [Simple-evcorr-users] SEC CPU utilization

[Dusan S. <dus...@ho...>]

I using similar approach as David mention. I processing syslog messages from network devices (various Vendors like Arista, Cisco, Juniper etc.).

We have about 7k patterns defined for match. As SEC cannot handle such regexp volume, I using syslog-ng PatternDB for pattern matching, classification and tagging.

Than all matched logs are written in normalized format into file what monitored by SEC and pre-defined correlation schema is applied. This help me to reduce flow into SEC to 30 msg/s while incoming syslog message flow is about ~20k msg/s (from about 20k unique network devices). My SEC instance is eating ~ 20 – 30 % of CPU (with about 500 active event correlation operations with ~ 4k active contexts in avg).

________________________________
Od: David Lang <da...@la...>
Odoslané: štvrtok 26. marca 2020 1:24
Komu: Richard Ostrochovský <ric...@gm...>
Kópia: sim...@li... <sim...@li...>
Predmet: Re: [Simple-evcorr-users] SEC CPU utilization

Much of the time you can trivially split your rules and then run multiple copies
of SEC, each processing a subnet of the rules on the logs that will match them.

Rsyslog has a very efficient parsing capability (mmnormalize), you can use it to
just classify the logs, or you can go a step further and have it extract the
relevant fields and pass them to SEC in ways that make it cheaper to parse and
process.

where you do have rules that correlate across different types of logs, you may
be able to just combine those logs into one ruleset, or you can have parallel
instances output messages to say that their part of the correlation has tested
true and then have another instance that processes these partial correlation
messages and decide if the combined correlation is matched.

David Lang


_______________________________________________
Simple-evcorr-users mailing list
Sim...@li...
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] SEC CPU utilization

[David L. <da...@la...>]

Much of the time you can trivially split your rules and then run multiple copies 
of SEC, each processing a subnet of the rules on the logs that will match them.

Rsyslog has a very efficient parsing capability (mmnormalize), you can use it to 
just classify the logs, or you can go a step further and have it extract the 
relevant fields and pass them to SEC in ways that make it cheaper to parse and 
process.

where you do have rules that correlate across different types of logs, you may 
be able to just combine those logs into one ruleset, or you can have parallel 
instances output messages to say that their part of the correlation has tested 
true and then have another instance that processes these partial correlation 
messages and decide if the combined correlation is matched.

David Lang

Re: [Simple-evcorr-users] SEC CPU utilization

[John P. R. <ro...@cs...>]

In message <CAJ69-dFBdhn7DyqSYYaO9QVHpdw0u1_Nsk3=x0EBc8=Ys...@ma...>
,
=?UTF-8?Q?Richard_Ostrochovsk=C3=BD?= writes:
>Hello friends,
>
>I have SEC monitoring over 50 log files with various correlations, and it
>is consuming 100% of single CPU (luckily on 10-CPU machine, so not whole
>system affected, as SEC is single-CPU application).
>
>This could mean, that SEC does not prosecute processing of all rules, and I
>am curious, what are possible effects, if this means increasing delays
>(first in, processing, first out), or skipping some lines from input files,
>or anything other (?).
>
>And how to troubleshoot, finding bottlenecks. I can see quantities of log
>messages per contexts or log files in sec.dump, this is some indicator. Are
>there also other indicators? Is it possible, somehow, see also processing
>times of patterns (per rules)?

I have found two operations that are large users of cpu in SEC.

1) If you have a lot of contexts, context lookups can eat cpu. I
   solved a problem where sec used 50% of a cpu by actively deleting
   contexts as soon as I could rather than lettng them time out. You
   can use the USR1 signal (IIRC see manpage) to get a dump of all
   contexts.  I don't remember how many I had but it was a lot. Once I
   started actively deleting contexts the cpu dropped to 1-2% with
   ~100-500 events per second though the system.

2) Inefficient regexps that require a lot of backtracking can kill
   your cpu.  I don't know how you would find these.

You don't say how your rulesets are constructed, but if you have say
100 rules and the most often hit rules are at position 99 and 100, you
are going to waste a lot of time applying regexps (even if they are
efficient). Again a dump file will tell you which rules are being
hit. Move the most used rules so they are applied earlier in you
rulesets. Also you can create rules so they are applied in a tree like
structure. Look at the manpage for rulesets. This structure allows you
to have a layer of filter rules (e.g. look for ssh in the log line)
and only apply rulesets for ssh to the log event. This can cut down
dramaticaly on the number of regexps that have to be applied and
discarded.

Also you can parse the event in one rule and then use more efficient
means to trigger rules (matching on a subset of the original event).

I mentiond as an offhand remark to Risto a profile mode that would
count not only every rule that lead to an action, but every time the
rule executed its regular expression. Having some sort of profile mode
(not to be run in production) would help identify these sorts of
issues.

Good luck and when you figure it out let us know how you found and
fixed it.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.