simple-evcorr-users — Simple Event Correlator users

[Simple-evcorr-users] R: Duplicate suppression and rearming

[Spelta E. <Edo...@be...>]

Hello,
back with some questions:


  1.  As long as we exclude the longer window expiration case (so we just want to suppress events for 10 secs), why you suggest to use two Single rules instead of a SingleWithSuppress ?

I’m referring to these rules:
type=Single
ptype=RegExp
pattern=event_(\w+)
context=!SUPP_$1
desc=react to event $1 and set up suppression
action=write - We have observed event $1; create SUPP_$1 10

type=Single
ptype=RegExp
pattern=event_(\w+)
context=SUPP_$1
desc=do not react to event $1 and only update its suppression context
action=set SUPP_$1 10



  1.  Does SingleWithSuppress use a sliding window ? According to my latest tests it does not, so it should be equivalent to your two Single rules..shouldn’t it ?





  1.  For the more complex scenario, when you  introduced also a MAXLIFE context of 60 secs, so i’m referring to:

type=Single
ptype=RegExp
pattern=event_(\w+)
context=!SUPP_$1
desc=react to event $1 and set up suppression
action=write - We have observed event $1; \
       create SUPP_$1_MAXLIFE 60 ( delete SUPP_$1 ); \
       create SUPP_$1 10 ( delete SUPP_$1_MAXLIFE )

type=Single
ptype=RegExp
pattern=event_(\w+)
context=SUPP_$1
desc=do not react to event $1 and only update its suppression context
action=set SUPP_$1 10



..as far as i can see, the shortest 10secs context will always expire first, thus deleting the MAXLIFE as well…so in the end this behaves just like the longer context does not exist at all. It seems equivalent to your rules in bullet 1), am i wrong ?

I’ve been generating identical events every two seconds, and with those two rules i get 1 line every 10 seconds…instead i would expect to get one every 60 seconds…because i expected the 10sec window to be constantly sliding thus suppressing everything until the 60 seconds windows expires and basically reset everything.

What am i missing ?

M.

Da: Spelta Edoardo
Inviato: martedì 14 marzo 2023 17:21
A: Eric V. Smith <er...@tr...>; Risto Vaarandi <ris...@gm...>
Cc: sim...@li...
Oggetto: R: [Simple-evcorr-users] Duplicate suppression and rearming

Absolutely amazing!

@Risto Vaarandi<mailto:ris...@gm...> I need some time to understand deeply what you propose and if it’s applicable to hundreds of conditions, but it sounds really promising !!

I’ll post some feedback as soon as i can

Thanks a lot !

M.

Da: Eric V. Smith <er...@tr...<mailto:er...@tr...>>
Inviato: martedì 14 marzo 2023 17:06
A: Risto Vaarandi <ris...@gm...<mailto:ris...@gm...>>; Spelta Edoardo <Edo...@be...<mailto:Edo...@be...>>
Cc: sim...@li...<mailto:sim...@li...>
Oggetto: Re: [Simple-evcorr-users] Duplicate suppression and rearming


Risto:

You (and sec) really are a treasure. Thanks for all you do. I don't currently have the issue that Mugugno mentions, but I might in the future. Your explanation below is excellent.

Eric
On 3/14/2023 11:52 AM, Risto Vaarandi wrote:
hi Mugugno,

thanks for clarifying your scenario! Perhaps I can first explain a simpler ruleset that is addressing a part of your scenario, and then provide a slightly more complex solution that addresses your scenario fully.

Let's assume that you want to react to the first instance of some event, and then keeping repeated instances of the same event suppressed, provided that the time frame between consecutive suppressed events is at most 10 seconds. In order to do that, you could utilize this ruleset of 2 Single rules:

type=Single
ptype=RegExp
pattern=event_(\w+)
context=!SUPP_$1
desc=react to event $1 and set up suppression
action=write - We have observed event $1; create SUPP_$1 10

type=Single
ptype=RegExp
pattern=event_(\w+)
context=SUPP_$1
desc=do not react to event $1 and only update its suppression context
action=set SUPP_$1 10

These two rules process events in the following format: event_<sometext>  (for example, event_X). The rules utilize the contexts SUPP_* for suppressing events of a given type. For example, for suppressing repeated instances of event_X, the context SUPP_X is used.

The first rule matches the event and checks if the suppression context is present. If the suppression context is not found, we are dealing with the first event we have seen, or the first event after previous suppression has ended. Therefore, the rule prints a notice "We have observed event *" to standard output. Also, after executing this action, the rule activates the suppression for the given event type. For example, if the event was event_X, the context SUPP_X is created. Note that the lifetime of the SUPP_X context is 10 seconds. If no subsequent events event_X will be observed during these 10 seconds, SUPP_X context will simply expire, and any further event_X will again trigger a notice written to standard output.

However, if some event_X will be seen within 10 seconds, it will match the second rule instead of the first one. The second rule does not produce any notice (in other words, it is suppressing event_X without action), but it is rather extending the lifetime of the suppression context SUPP_X for additional 10 seconds starting from the *current moment*. This means that as long as events event_X will keep arriving with the time internal not larger than 10 seconds between them, the lifetime of the SUPP_X context will get extended for another 10 seconds on the arrival of each event_X.

Obviously, the above ruleset has the following problem -- if events event_X keep on arriving at least once per 10 seconds indefinitely, the suppression of event_X will never end. For addressing this problem, you can use a further development of the previous ruleset that sets an upper limit of 60 seconds for the suppression:

type=Single
ptype=RegExp
pattern=event_(\w+)
context=!SUPP_$1
desc=react to event $1 and set up suppression
action=write - We have observed event $1; \
       create SUPP_$1_MAXLIFE 60 ( delete SUPP_$1 ); \
       create SUPP_$1 10 ( delete SUPP_$1_MAXLIFE )

type=Single
ptype=RegExp
pattern=event_(\w+)
context=SUPP_$1
desc=do not react to event $1 and only update its suppression context
action=set SUPP_$1 10

When event_X is observed and the suppression is currently not active, the first rule matches and writes a notice to standard output. In addition to context SUPP_X, the context SUPP_X_MAXLIFE gets created with the lifetime of 60 seconds (that is the maximum lifetime of the event suppression process). Note that the lifetime of this context is *not* extended by the ruleset, and also, when this context expires, it will delete the SUPP_X context that implements the suppression. This means that the suppression can never be active for more than 60 seconds after we saw the first event that triggered a notice message to standard output.

Also, note that when SUPP_X context expires, it will delete the SUPP_X_MAXLIFE context, since this context is no longer necessary. In other words, whatever context (SUPP_X or SUPP_X_MAXLIFE) expires first, it will always delete the other context, in order to continue with the clean state for the given event type.

By using the above technique with two contexts, you can implement the scenario you want. Also, I hope that the examples I have provided are clear enough and helpful for illustrating the main point.

kind regards,
risto


Kontakt Spelta Edoardo (<Edo...@be...<mailto:Edo...@be...>>) kirjutas kuupäeval T, 14. märts 2023 kell 16:58:
Hello,
you definitely described my scenario better than me.

This is the one i’m after:
0, 2, 23, 54 -- events at minutes 2 and 23 should be suppressed, but the event at minute 54 should trigger an action, since it happens 31 minutes after the event 23
0, 11, 32, 44, 61 -- events at minutes 11, 32 and 44 should be suppressed, since they are separated by less than 30 minutes, but the event at minute 61 should trigger an action, since the suppression can't last for longer than 1 hour
Any ideas ?
M.

Da: Risto Vaarandi <ris...@gm...<mailto:ris...@gm...>>
Inviato: martedì 14 marzo 2023 15:52
A: Spelta Edoardo <Edo...@be...<mailto:Edo...@be...>>
Cc: sim...@li...<mailto:sim...@li...>
Oggetto: Re: [Simple-evcorr-users] Duplicate suppression and rearming

hi Mugugno,

let me clarify your scenario a bit, considering the diagram from your post:

T1---------------------------T27-----------T30-------------------T57-------------T60----------T61
Event    suppr suppr    suppr          suppr                     suppr                                     Event

Do you want the suppression to start after the time moment T1 when the first event was observed and run for 1 hour, so that this window would not be sliding?

Or do you rather want the time window between two *consecutive* events to be less than N minutes in order to suppress them, so that suppression would work during max 1 hour? For example, suppose N=30minutes and consider the following events happening at these minutes:

0, 2, 23, 54 -- events at minutes 2 and 23 should be suppressed, but the event at minute 54 should trigger an action, since it happens 31 minutes after the event 23
0, 11, 32, 44, 61 -- events at minutes 11, 32 and 44 should be suppressed, since they are separated by less than 30 minutes, but the event at minute 61 should trigger an action, since the suppression can't last for longer than 1 hour

I would appreciate it if you could provide some additional comments on your scenario.

kind regards,
risto

Hello,
i’ve recently been struggling with SEC to implement something for this specific use case: suppressing specific entries read from a syslog for a certain amount of time (say 30min) but make sure that after a longer time (say 1h), if they are still being received, i’m getting a new one.

If these events are being received continously they will always be suppressed because the windows will be sliding accordingly but i’m trying to find a way to make it stop after 1h

The sequence should look like this:

T1---------------------------T27-----------T30-------------------T57-------------T60----------T61
Event    suppr suppr    suppr          suppr                     suppr                                     Event

I tried combining a SingleWithSuppress (which is ok for the suppression part) and context expiration but i cannot find a working solution.
Anybody already faced this use case ?

Any help appreciated!


Thanks and regards,
Mugugno


_______________________________________________
Simple-evcorr-users mailing list
Sim...@li...<mailto:Sim...@li...>
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users



_______________________________________________

Simple-evcorr-users mailing list

Sim...@li...<mailto:Sim...@li...>

https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] Duplicate suppression and rearming

[Risto V. <ris...@gm...>]

> Absolutely amazing!
>

it is great that I was able to help :)


>
>
> @Risto Vaarandi <ris...@gm...> I need some time to understand
> deeply what you propose and if it’s applicable to hundreds of conditions,
> but it sounds really promising !!
>

As a side note: this technique is applicable for any number of conditions
in parallel. Just make sure that both context names contain a unique ID for
each different condition. In the example from my previous post, the $1
variable plays that role and allows to process many event types
simultaneously (e.g., event_A, event_B, event_AB, event_ABC, etc.).

kind regards,
risto

Re: [Simple-evcorr-users] Duplicate suppression and rearming

[Eric V. S. <er...@tr...>]

Risto:

You (and sec) really are a treasure. Thanks for all you do. I don't 
currently have the issue that Mugugno mentions, but I might in the 
future. Your explanation below is excellent.

Eric

On 3/14/2023 11:52 AM, Risto Vaarandi wrote:
> hi Mugugno,
>
> thanks for clarifying your scenario! Perhaps I can first explain a 
> simpler ruleset that is addressing a part of your scenario, and then 
> provide a slightly more complex solution that addresses your scenario 
> fully.
>
> Let's assume that you want to react to the first instance of some 
> event, and then keeping repeated instances of the same event 
> suppressed, provided that the time frame between consecutive 
> suppressed events is at most 10 seconds. In order to do that, you 
> could utilize this ruleset of 2 Single rules:
>
> type=Single
> ptype=RegExp
> pattern=event_(\w+)
> context=!SUPP_$1
> desc=react to event $1 and set up suppression
> action=write - We have observed event $1; create SUPP_$1 10
>
> type=Single
> ptype=RegExp
> pattern=event_(\w+)
> context=SUPP_$1
> desc=do not react to event $1 and only update its suppression context
> action=set SUPP_$1 10
>
> These two rules process events in the following format: 
> event_<sometext>  (for example, event_X). The rules utilize the 
> contexts SUPP_* for suppressing events of a given type. For example, 
> for suppressing repeated instances of event_X, the context SUPP_X is used.
>
> The first rule matches the event and checks if the suppression context 
> is present. If the suppression context is not found, we are dealing 
> with the first event we have seen, or the first event after previous 
> suppression has ended. Therefore, the rule prints a notice "We have 
> observed event *" to standard output. Also, after executing this 
> action, the rule activates the suppression for the given event type. 
> For example, if the event was event_X, the context SUPP_X is created. 
> Note that the lifetime of the SUPP_X context is 10 seconds. If no 
> subsequent events event_X will be observed during these 10 seconds, 
> SUPP_X context will simply expire, and any further event_X will again 
> trigger a notice written to standard output.
>
> However, if some event_X will be seen within 10 seconds, it will match 
> the second rule instead of the first one. The second rule does not 
> produce any notice (in other words, it is suppressing event_X without 
> action), but it is rather extending the lifetime of the suppression 
> context SUPP_X for additional 10 seconds starting from the *current 
> moment*. This means that as long as events event_X will keep arriving 
> with the time internal not larger than 10 seconds between them, the 
> lifetime of the SUPP_X context will get extended for another 10 
> seconds on the arrival of each event_X.
>
> Obviously, the above ruleset has the following problem -- if events 
> event_X keep on arriving at least once per 10 seconds indefinitely, 
> the suppression of event_X will never end. For addressing this 
> problem, you can use a further development of the previous ruleset 
> that sets an upper limit of 60 seconds for the suppression:
>
> type=Single
> ptype=RegExp
> pattern=event_(\w+)
> context=!SUPP_$1
> desc=react to event $1 and set up suppression
> action=write - We have observed event $1; \
>        create SUPP_$1_MAXLIFE 60 ( delete SUPP_$1 ); \
>        create SUPP_$1 10 ( delete SUPP_$1_MAXLIFE )
>
> type=Single
> ptype=RegExp
> pattern=event_(\w+)
> context=SUPP_$1
> desc=do not react to event $1 and only update its suppression context
> action=set SUPP_$1 10
>
> When event_X is observed and the suppression is currently not active, 
> the first rule matches and writes a notice to standard output. In 
> addition to context SUPP_X, the context SUPP_X_MAXLIFE gets created 
> with the lifetime of 60 seconds (that is the maximum lifetime of the 
> event suppression process). Note that the lifetime of this context is 
> *not* extended by the ruleset, and also, when this context expires, it 
> will delete the SUPP_X context that implements the suppression. This 
> means that the suppression can never be active for more than 60 
> seconds after we saw the first event that triggered a notice message 
> to standard output.
>
> Also, note that when SUPP_X context expires, it will delete the 
> SUPP_X_MAXLIFE context, since this context is no longer necessary. In 
> other words, whatever context (SUPP_X or SUPP_X_MAXLIFE) expires 
> first, it will always delete the other context, in order to continue 
> with the clean state for the given event type.
>
> By using the above technique with two contexts, you can implement the 
> scenario you want. Also, I hope that the examples I have provided are 
> clear enough and helpful for illustrating the main point.
>
> kind regards,
> risto
>
>
> Kontakt Spelta Edoardo (<Edo...@be...>) kirjutas 
> kuupäeval T, 14. märts 2023 kell 16:58:
>
>     Hello,
>
>     you definitely described my scenario better than me.
>
>     This is the one i’m after:
>
>     0, 2, 23, 54 -- events at minutes 2 and 23 should be suppressed,
>     but the event at minute 54 should trigger an action, since it
>     happens 31 minutes after the event 23
>
>     0, 11, 32, 44, 61 -- events at minutes 11, 32 and 44 should be
>     suppressed, since they are separated by less than 30 minutes, but
>     the event at minute 61 should trigger an action, since the
>     suppression can't last for longer than 1 hour
>
>     Any ideas ?
>
>     M.
>
>     *Da:* Risto Vaarandi <ris...@gm...>
>     *Inviato:* martedì 14 marzo 2023 15:52
>     *A:* Spelta Edoardo <Edo...@be...>
>     *Cc:* sim...@li...
>     *Oggetto:* Re: [Simple-evcorr-users] Duplicate suppression and
>     rearming
>
>     hi Mugugno,
>
>     let me clarify your scenario a bit, considering the diagram from
>     your post:
>
>     T1---------------------------T27-----------T30-------------------T57-------------T60----------T61
>
>     Event    suppr suppr    suppr          suppr
>     suppr                                     Event
>
>     Do you want the suppression to start after the time moment T1 when
>     the first event was observed and run for 1 hour, so that this
>     window would not be sliding?
>
>     Or do you rather want the time window between two *consecutive*
>     events to be less than N minutes in order to suppress them, so
>     that suppression would work during max 1 hour? For example,
>     suppose N=30minutes and consider the following events happening at
>     these minutes:
>
>     0, 2, 23, 54 -- events at minutes 2 and 23 should be suppressed,
>     but the event at minute 54 should trigger an action, since it
>     happens 31 minutes after the event 23
>
>     0, 11, 32, 44, 61 -- events at minutes 11, 32 and 44 should be
>     suppressed, since they are separated by less than 30 minutes, but
>     the event at minute 61 should trigger an action, since the
>     suppression can't last for longer than 1 hour
>
>     I would appreciate it if you could provide some additional
>     comments on your scenario.
>
>     kind regards,
>
>     risto
>
>         Hello,
>
>         i’ve recently been struggling with SEC to implement something
>         for this specific use case: suppressing specific entries read
>         from a syslog for a certain amount of time (say 30min) but
>         make sure that after a longer time (say 1h), if they are still
>         being received, i’m getting a new one.
>
>         If these events are being received continously they will
>         always be suppressed because the windows will be sliding
>         accordingly but i’m trying to find a way to make it stop after 1h
>
>         The sequence should look like this:
>
>         T1---------------------------T27-----------T30-------------------T57-------------T60----------T61
>
>         Event    suppr suppr    suppr suppr suppr Event
>
>         I tried combining a SingleWithSuppress (which is ok for the
>         suppression part) and context expiration but i cannot find a
>         working solution.
>
>         Anybody already faced this use case ?
>
>         Any help appreciated!
>
>         Thanks and regards,
>
>         Mugugno
>
>         _______________________________________________
>         Simple-evcorr-users mailing list
>         Sim...@li...
>         https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
>
>
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] R: Duplicate suppression and rearming

[Spelta E. <Edo...@be...>]

Absolutely amazing!

@Risto Vaarandi<mailto:ris...@gm...> I need some time to understand deeply what you propose and if it’s applicable to hundreds of conditions, but it sounds really promising !!

I’ll post some feedback as soon as i can

Thanks a lot !

M.

Da: Eric V. Smith <er...@tr...>
Inviato: martedì 14 marzo 2023 17:06
A: Risto Vaarandi <ris...@gm...>; Spelta Edoardo <Edo...@be...>
Cc: sim...@li...
Oggetto: Re: [Simple-evcorr-users] Duplicate suppression and rearming


Risto:

You (and sec) really are a treasure. Thanks for all you do. I don't currently have the issue that Mugugno mentions, but I might in the future. Your explanation below is excellent.

Eric
On 3/14/2023 11:52 AM, Risto Vaarandi wrote:
hi Mugugno,

thanks for clarifying your scenario! Perhaps I can first explain a simpler ruleset that is addressing a part of your scenario, and then provide a slightly more complex solution that addresses your scenario fully.

Let's assume that you want to react to the first instance of some event, and then keeping repeated instances of the same event suppressed, provided that the time frame between consecutive suppressed events is at most 10 seconds. In order to do that, you could utilize this ruleset of 2 Single rules:

type=Single
ptype=RegExp
pattern=event_(\w+)
context=!SUPP_$1
desc=react to event $1 and set up suppression
action=write - We have observed event $1; create SUPP_$1 10

type=Single
ptype=RegExp
pattern=event_(\w+)
context=SUPP_$1
desc=do not react to event $1 and only update its suppression context
action=set SUPP_$1 10

These two rules process events in the following format: event_<sometext>  (for example, event_X). The rules utilize the contexts SUPP_* for suppressing events of a given type. For example, for suppressing repeated instances of event_X, the context SUPP_X is used.

The first rule matches the event and checks if the suppression context is present. If the suppression context is not found, we are dealing with the first event we have seen, or the first event after previous suppression has ended. Therefore, the rule prints a notice "We have observed event *" to standard output. Also, after executing this action, the rule activates the suppression for the given event type. For example, if the event was event_X, the context SUPP_X is created. Note that the lifetime of the SUPP_X context is 10 seconds. If no subsequent events event_X will be observed during these 10 seconds, SUPP_X context will simply expire, and any further event_X will again trigger a notice written to standard output.

However, if some event_X will be seen within 10 seconds, it will match the second rule instead of the first one. The second rule does not produce any notice (in other words, it is suppressing event_X without action), but it is rather extending the lifetime of the suppression context SUPP_X for additional 10 seconds starting from the *current moment*. This means that as long as events event_X will keep arriving with the time internal not larger than 10 seconds between them, the lifetime of the SUPP_X context will get extended for another 10 seconds on the arrival of each event_X.

Obviously, the above ruleset has the following problem -- if events event_X keep on arriving at least once per 10 seconds indefinitely, the suppression of event_X will never end. For addressing this problem, you can use a further development of the previous ruleset that sets an upper limit of 60 seconds for the suppression:

type=Single
ptype=RegExp
pattern=event_(\w+)
context=!SUPP_$1
desc=react to event $1 and set up suppression
action=write - We have observed event $1; \
       create SUPP_$1_MAXLIFE 60 ( delete SUPP_$1 ); \
       create SUPP_$1 10 ( delete SUPP_$1_MAXLIFE )

type=Single
ptype=RegExp
pattern=event_(\w+)
context=SUPP_$1
desc=do not react to event $1 and only update its suppression context
action=set SUPP_$1 10

When event_X is observed and the suppression is currently not active, the first rule matches and writes a notice to standard output. In addition to context SUPP_X, the context SUPP_X_MAXLIFE gets created with the lifetime of 60 seconds (that is the maximum lifetime of the event suppression process). Note that the lifetime of this context is *not* extended by the ruleset, and also, when this context expires, it will delete the SUPP_X context that implements the suppression. This means that the suppression can never be active for more than 60 seconds after we saw the first event that triggered a notice message to standard output.

Also, note that when SUPP_X context expires, it will delete the SUPP_X_MAXLIFE context, since this context is no longer necessary. In other words, whatever context (SUPP_X or SUPP_X_MAXLIFE) expires first, it will always delete the other context, in order to continue with the clean state for the given event type.

By using the above technique with two contexts, you can implement the scenario you want. Also, I hope that the examples I have provided are clear enough and helpful for illustrating the main point.

kind regards,
risto


Kontakt Spelta Edoardo (<Edo...@be...<mailto:Edo...@be...>>) kirjutas kuupäeval T, 14. märts 2023 kell 16:58:
Hello,
you definitely described my scenario better than me.

This is the one i’m after:
0, 2, 23, 54 -- events at minutes 2 and 23 should be suppressed, but the event at minute 54 should trigger an action, since it happens 31 minutes after the event 23
0, 11, 32, 44, 61 -- events at minutes 11, 32 and 44 should be suppressed, since they are separated by less than 30 minutes, but the event at minute 61 should trigger an action, since the suppression can't last for longer than 1 hour
Any ideas ?
M.

Da: Risto Vaarandi <ris...@gm...<mailto:ris...@gm...>>
Inviato: martedì 14 marzo 2023 15:52
A: Spelta Edoardo <Edo...@be...<mailto:Edo...@be...>>
Cc: sim...@li...<mailto:sim...@li...>
Oggetto: Re: [Simple-evcorr-users] Duplicate suppression and rearming

hi Mugugno,

let me clarify your scenario a bit, considering the diagram from your post:

T1---------------------------T27-----------T30-------------------T57-------------T60----------T61
Event    suppr suppr    suppr          suppr                     suppr                                     Event

Do you want the suppression to start after the time moment T1 when the first event was observed and run for 1 hour, so that this window would not be sliding?

Or do you rather want the time window between two *consecutive* events to be less than N minutes in order to suppress them, so that suppression would work during max 1 hour? For example, suppose N=30minutes and consider the following events happening at these minutes:

0, 2, 23, 54 -- events at minutes 2 and 23 should be suppressed, but the event at minute 54 should trigger an action, since it happens 31 minutes after the event 23
0, 11, 32, 44, 61 -- events at minutes 11, 32 and 44 should be suppressed, since they are separated by less than 30 minutes, but the event at minute 61 should trigger an action, since the suppression can't last for longer than 1 hour

I would appreciate it if you could provide some additional comments on your scenario.

kind regards,
risto

Hello,
i’ve recently been struggling with SEC to implement something for this specific use case: suppressing specific entries read from a syslog for a certain amount of time (say 30min) but make sure that after a longer time (say 1h), if they are still being received, i’m getting a new one.

If these events are being received continously they will always be suppressed because the windows will be sliding accordingly but i’m trying to find a way to make it stop after 1h

The sequence should look like this:

T1---------------------------T27-----------T30-------------------T57-------------T60----------T61
Event    suppr suppr    suppr          suppr                     suppr                                     Event

I tried combining a SingleWithSuppress (which is ok for the suppression part) and context expiration but i cannot find a working solution.
Anybody already faced this use case ?

Any help appreciated!


Thanks and regards,
Mugugno


_______________________________________________
Simple-evcorr-users mailing list
Sim...@li...<mailto:Sim...@li...>
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users




_______________________________________________

Simple-evcorr-users mailing list

Sim...@li...<mailto:Sim...@li...>

https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] Duplicate suppression and rearming

[Risto V. <ris...@gm...>]

hi Mugugno,

thanks for clarifying your scenario! Perhaps I can first explain a simpler
ruleset that is addressing a part of your scenario, and then provide a
slightly more complex solution that addresses your scenario fully.

Let's assume that you want to react to the first instance of some event,
and then keeping repeated instances of the same event suppressed, provided
that the time frame between consecutive suppressed events is at most 10
seconds. In order to do that, you could utilize this ruleset of 2 Single
rules:

type=Single
ptype=RegExp
pattern=event_(\w+)
context=!SUPP_$1
desc=react to event $1 and set up suppression
action=write - We have observed event $1; create SUPP_$1 10

type=Single
ptype=RegExp
pattern=event_(\w+)
context=SUPP_$1
desc=do not react to event $1 and only update its suppression context
action=set SUPP_$1 10

These two rules process events in the following format: event_<sometext>
(for example, event_X). The rules utilize the contexts SUPP_* for
suppressing events of a given type. For example, for suppressing repeated
instances of event_X, the context SUPP_X is used.

The first rule matches the event and checks if the suppression context is
present. If the suppression context is not found, we are dealing with the
first event we have seen, or the first event after previous suppression has
ended. Therefore, the rule prints a notice "We have observed event *" to
standard output. Also, after executing this action, the rule activates the
suppression for the given event type. For example, if the event was
event_X, the context SUPP_X is created. Note that the lifetime of the
SUPP_X context is 10 seconds. If no subsequent events event_X will be
observed during these 10 seconds, SUPP_X context will simply expire, and
any further event_X will again trigger a notice written to standard output.

However, if some event_X will be seen within 10 seconds, it will match the
second rule instead of the first one. The second rule does not produce any
notice (in other words, it is suppressing event_X without action), but it
is rather extending the lifetime of the suppression context SUPP_X for
additional 10 seconds starting from the *current moment*. This means that
as long as events event_X will keep arriving with the time internal not
larger than 10 seconds between them, the lifetime of the SUPP_X context
will get extended for another 10 seconds on the arrival of each event_X.

Obviously, the above ruleset has the following problem -- if events event_X
keep on arriving at least once per 10 seconds indefinitely, the suppression
of event_X will never end. For addressing this problem, you can use a
further development of the previous ruleset that sets an upper limit of 60
seconds for the suppression:

type=Single
ptype=RegExp
pattern=event_(\w+)
context=!SUPP_$1
desc=react to event $1 and set up suppression
action=write - We have observed event $1; \
       create SUPP_$1_MAXLIFE 60 ( delete SUPP_$1 ); \
       create SUPP_$1 10 ( delete SUPP_$1_MAXLIFE )

type=Single
ptype=RegExp
pattern=event_(\w+)
context=SUPP_$1
desc=do not react to event $1 and only update its suppression context
action=set SUPP_$1 10

When event_X is observed and the suppression is currently not active, the
first rule matches and writes a notice to standard output. In addition to
context SUPP_X, the context SUPP_X_MAXLIFE gets created with the lifetime
of 60 seconds (that is the maximum lifetime of the event suppression
process). Note that the lifetime of this context is *not* extended by the
ruleset, and also, when this context expires, it will delete the SUPP_X
context that implements the suppression. This means that the suppression
can never be active for more than 60 seconds after we saw the first event
that triggered a notice message to standard output.

Also, note that when SUPP_X context expires, it will delete the
SUPP_X_MAXLIFE context, since this context is no longer necessary. In other
words, whatever context (SUPP_X or SUPP_X_MAXLIFE) expires first, it will
always delete the other context, in order to continue with the clean state
for the given event type.

By using the above technique with two contexts, you can implement the
scenario you want. Also, I hope that the examples I have provided are clear
enough and helpful for illustrating the main point.

kind regards,
risto


Kontakt Spelta Edoardo (<Edo...@be...>) kirjutas kuupäeval
T, 14. märts 2023 kell 16:58:

> Hello,
>
> you definitely described my scenario better than me.
>
>
>
> This is the one i’m after:
>
> 0, 2, 23, 54 -- events at minutes 2 and 23 should be suppressed, but the
> event at minute 54 should trigger an action, since it happens 31 minutes
> after the event 23
>
> 0, 11, 32, 44, 61 -- events at minutes 11, 32 and 44 should be suppressed,
> since they are separated by less than 30 minutes, but the event at minute
> 61 should trigger an action, since the suppression can't last for longer
> than 1 hour
>
> Any ideas ?
>
> M.
>
>
>
> *Da:* Risto Vaarandi <ris...@gm...>
> *Inviato:* martedì 14 marzo 2023 15:52
> *A:* Spelta Edoardo <Edo...@be...>
> *Cc:* sim...@li...
> *Oggetto:* Re: [Simple-evcorr-users] Duplicate suppression and rearming
>
>
>
> hi Mugugno,
>
>
>
> let me clarify your scenario a bit, considering the diagram from your post:
>
>
>
>
> T1---------------------------T27-----------T30-------------------T57-------------T60----------T61
>
> Event    suppr suppr    suppr          suppr
> suppr                                     Event
>
>
>
> Do you want the suppression to start after the time moment T1 when the
> first event was observed and run for 1 hour, so that this window would not
> be sliding?
>
>
>
> Or do you rather want the time window between two *consecutive* events to
> be less than N minutes in order to suppress them, so that suppression would
> work during max 1 hour? For example, suppose N=30minutes and consider the
> following events happening at these minutes:
>
>
>
> 0, 2, 23, 54 -- events at minutes 2 and 23 should be suppressed, but the
> event at minute 54 should trigger an action, since it happens 31 minutes
> after the event 23
>
> 0, 11, 32, 44, 61 -- events at minutes 11, 32 and 44 should be suppressed,
> since they are separated by less than 30 minutes, but the event at minute
> 61 should trigger an action, since the suppression can't last for longer
> than 1 hour
>
>
>
> I would appreciate it if you could provide some additional comments on
> your scenario.
>
>
>
> kind regards,
>
> risto
>
>
>
> Hello,
>
> i’ve recently been struggling with SEC to implement something for this
> specific use case: suppressing specific entries read from a syslog for a
> certain amount of time (say 30min) but make sure that after a longer time
> (say 1h), if they are still being received, i’m getting a new one.
>
>
>
> If these events are being received continously they will always be
> suppressed because the windows will be sliding accordingly but i’m trying
> to find a way to make it stop after 1h
>
>
>
> The sequence should look like this:
>
>
>
>
> T1---------------------------T27-----------T30-------------------T57-------------T60----------T61
>
> Event    suppr suppr    suppr          suppr
> suppr                                     Event
>
>
>
> I tried combining a SingleWithSuppress (which is ok for the suppression
> part) and context expiration but i cannot find a working solution.
>
> Anybody already faced this use case ?
>
>
>
> Any help appreciated!
>
>
>
>
>
> Thanks and regards,
>
> Mugugno
>
>
>
>
>
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
>

[Simple-evcorr-users] R: Duplicate suppression and rearming

[Spelta E. <Edo...@be...>]

Hello,
you definitely described my scenario better than me.

This is the one i’m after:
0, 2, 23, 54 -- events at minutes 2 and 23 should be suppressed, but the event at minute 54 should trigger an action, since it happens 31 minutes after the event 23
0, 11, 32, 44, 61 -- events at minutes 11, 32 and 44 should be suppressed, since they are separated by less than 30 minutes, but the event at minute 61 should trigger an action, since the suppression can't last for longer than 1 hour
Any ideas ?
M.

Da: Risto Vaarandi <ris...@gm...>
Inviato: martedì 14 marzo 2023 15:52
A: Spelta Edoardo <Edo...@be...>
Cc: sim...@li...
Oggetto: Re: [Simple-evcorr-users] Duplicate suppression and rearming

hi Mugugno,

let me clarify your scenario a bit, considering the diagram from your post:

T1---------------------------T27-----------T30-------------------T57-------------T60----------T61
Event    suppr suppr    suppr          suppr                     suppr                                     Event

Do you want the suppression to start after the time moment T1 when the first event was observed and run for 1 hour, so that this window would not be sliding?

Or do you rather want the time window between two *consecutive* events to be less than N minutes in order to suppress them, so that suppression would work during max 1 hour? For example, suppose N=30minutes and consider the following events happening at these minutes:

0, 2, 23, 54 -- events at minutes 2 and 23 should be suppressed, but the event at minute 54 should trigger an action, since it happens 31 minutes after the event 23
0, 11, 32, 44, 61 -- events at minutes 11, 32 and 44 should be suppressed, since they are separated by less than 30 minutes, but the event at minute 61 should trigger an action, since the suppression can't last for longer than 1 hour

I would appreciate it if you could provide some additional comments on your scenario.

kind regards,
risto

Hello,
i’ve recently been struggling with SEC to implement something for this specific use case: suppressing specific entries read from a syslog for a certain amount of time (say 30min) but make sure that after a longer time (say 1h), if they are still being received, i’m getting a new one.

If these events are being received continously they will always be suppressed because the windows will be sliding accordingly but i’m trying to find a way to make it stop after 1h

The sequence should look like this:

T1---------------------------T27-----------T30-------------------T57-------------T60----------T61
Event    suppr suppr    suppr          suppr                     suppr                                     Event

I tried combining a SingleWithSuppress (which is ok for the suppression part) and context expiration but i cannot find a working solution.
Anybody already faced this use case ?

Any help appreciated!


Thanks and regards,
Mugugno


_______________________________________________
Simple-evcorr-users mailing list
Sim...@li...<mailto:Sim...@li...>
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] Duplicate suppression and rearming

[Risto V. <ris...@gm...>]

hi Mugugno,

let me clarify your scenario a bit, considering the diagram from your post:

T1---------------------------T27-----------T30-------------------T57-------------T60----------T61

Event    suppr suppr    suppr          suppr
suppr                                     Event


Do you want the suppression to start after the time moment T1 when the
first event was observed and run for 1 hour, so that this window would not
be sliding?


Or do you rather want the time window between two *consecutive* events to
be less than N minutes in order to suppress them, so that suppression would
work during max 1 hour? For example, suppose N=30minutes and consider the
following events happening at these minutes:


0, 2, 23, 54 -- events at minutes 2 and 23 should be suppressed, but the
event at minute 54 should trigger an action, since it happens 31 minutes
after the event 23

0, 11, 32, 44, 61 -- events at minutes 11, 32 and 44 should be suppressed,
since they are separated by less than 30 minutes, but the event at minute
61 should trigger an action, since the suppression can't last for longer
than 1 hour


I would appreciate it if you could provide some additional comments on your
scenario.


kind regards,

risto

Hello,
>
> i’ve recently been struggling with SEC to implement something for this
> specific use case: suppressing specific entries read from a syslog for a
> certain amount of time (say 30min) but make sure that after a longer time
> (say 1h), if they are still being received, i’m getting a new one.
>
>
>
> If these events are being received continously they will always be
> suppressed because the windows will be sliding accordingly but i’m trying
> to find a way to make it stop after 1h
>
>
>
> The sequence should look like this:
>
>
>
>
> T1---------------------------T27-----------T30-------------------T57-------------T60----------T61
>
> Event    suppr suppr    suppr          suppr
> suppr                                     Event
>
>
>
> I tried combining a SingleWithSuppress (which is ok for the suppression
> part) and context expiration but i cannot find a working solution.
>
> Anybody already faced this use case ?
>
>
>
> Any help appreciated!
>
>
>
>
>
> Thanks and regards,
>
> Mugugno
>
>
>
>
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>

[Simple-evcorr-users] Duplicate suppression and rearming

[Spelta E. <Edo...@be...>]

Hello,
i've recently been struggling with SEC to implement something for this specific use case: suppressing specific entries read from a syslog for a certain amount of time (say 30min) but make sure that after a longer time (say 1h), if they are still being received, i'm getting a new one.

If these events are being received continously they will always be suppressed because the windows will be sliding accordingly but i'm trying to find a way to make it stop after 1h

The sequence should look like this:

T1---------------------------T27-----------T30-------------------T57-------------T60----------T61
Event    suppr suppr    suppr          suppr                     suppr                                     Event

I tried combining a SingleWithSuppress (which is ok for the suppression part) and context expiration but i cannot find a working solution.
Anybody already faced this use case ?

Any help appreciated!


Thanks and regards,
Mugugno

Re: [Simple-evcorr-users] SEC tutorial

[Risto V. <ris...@gm...>]

hi Dusan,
many thanks for your kind words :)
risto

Kontakt Dusan Sovic (<dus...@ho...>) kirjutas kuupäeval R, 2.
detsember 2022 kell 23:07:

> Hi Risto,
>
>
>
> Thanks for this tutorial.
>
> It’s very handy (especially 4.5 Internal contexts).
>
> I’ve learned again something new.
>
> SEC helping me reliably more than 6 years without any issues.
>
> It’s simple, reliable and well-documented.
>
>
>
> Thank you,
>
> Dusan
>
>
>
> *Od: *Risto Vaarandi <ris...@gm...>
> *Odoslané: *štvrtok 1. decembra 2022 11:25
> *Komu: *sim...@li...
> *Predmet: *[Simple-evcorr-users] SEC tutorial
>
>
>
> hi all,
>
>
>
> during the last few weeks, I have written a new SEC tutorial paper which
> can be accessed through this web link:
> https://raw.githubusercontent.com/simple-evcorr/tutorial/main/SEC-tutorial.pdf
>
> Links to this tutorial and the relevant Github repository (
> https://github.com/simple-evcorr/tutorial) are also available in the SEC
> home page.
>
>
>
> This tutorial is primarily intended for people who haven't used SEC
> before, and it explains the essentials of SEC in a gentle way, starting
> from simple examples and gradually moving to more complex ones. After
> reading this tutorial, the reader should have a good understanding of event
> correlation rules, event correlation operations, contexts, synthetic
> events, pattern match caching, and many other things. And it might provide
> interesting insights even for experienced users :)
>
>
>
> As mentioned in the introduction, this tutorial does not intend to be a
> replacement for the official documentation. Therefore, it does not attempt
> to explain every possible detail, but rather provide an overview of the key
> concepts, so that it would be easier to look up more detailed information
> from the man page.
>
>
>
> Hopefully the tutorial will be useful and you will find it easy to follow
> :)
>
>
>
> kind regards,
>
> risto
>
>
>

Re: [Simple-evcorr-users] SEC tutorial

[Dusan S. <dus...@ho...>]

Hi Risto,

Thanks for this tutorial.
It’s very handy (especially 4.5 Internal contexts).
I’ve learned again something new.
SEC helping me reliably more than 6 years without any issues.
It’s simple, reliable and well-documented.

Thank you,
Dusan

Od: Risto Vaarandi<mailto:ris...@gm...>
Odoslané: štvrtok 1. decembra 2022 11:25
Komu: sim...@li...<mailto:sim...@li...>
Predmet: [Simple-evcorr-users] SEC tutorial

hi all,

during the last few weeks, I have written a new SEC tutorial paper which can be accessed through this web link: https://raw.githubusercontent.com/simple-evcorr/tutorial/main/SEC-tutorial.pdf
Links to this tutorial and the relevant Github repository (https://github.com/simple-evcorr/tutorial) are also available in the SEC home page.

This tutorial is primarily intended for people who haven't used SEC before, and it explains the essentials of SEC in a gentle way, starting from simple examples and gradually moving to more complex ones. After reading this tutorial, the reader should have a good understanding of event correlation rules, event correlation operations, contexts, synthetic events, pattern match caching, and many other things. And it might provide interesting insights even for experienced users :)

As mentioned in the introduction, this tutorial does not intend to be a replacement for the official documentation. Therefore, it does not attempt to explain every possible detail, but rather provide an overview of the key concepts, so that it would be easier to look up more detailed information from the man page.

Hopefully the tutorial will be useful and you will find it easy to follow :)

kind regards,
risto

Re: [Simple-evcorr-users] SEC tutorial

[Risto V. <ris...@gm...>]

hi Jim,
many thanks for your very kind offer! The tutorial is fairly large (around
50 pages), but if you have time, I would appreciate your feedback. Adding
the remarks and recommendations into PDF works fine for me, and you can
just send the modified PDF over to my private email address.
kind regards,
risto

Kontakt Jim Van Meggelen (<jim...@cl...>) kirjutas
kuupäeval N, 1. detsember 2022 kell 15:25:

> Risto,
>
> Thanks for doing that up. I'll work through it.
>
> Are you interested in feedback on it? If so, do you have a preferred
> mechanism for that? (I can just mark up the PDF as I read through and
> upload it somewhere when I'm done). Note that I might find myself mostly
> copyediting, rather than testing all the examples.
>
> Jim
>
>
> --
> Jim Van Meggelen
> ClearlyCore Inc.
>
>
>
> +1-416-639-6001 (DID)
> +1-877-253-2716 (Canada)
> +1-866-644-7729 (USA)
> +1-416-425-6111 x6001
> jim...@cl...
> http://www.clearlycore.com
>
>
> *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:*
> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf
>
> ------------------------------
>
> *From: *"Risto Vaarandi" <ris...@gm...>
> *To: *"simple-evcorr-users" <sim...@li...>
> *Sent: *Thursday, 1 December, 2022 05:23:48
> *Subject: *[Simple-evcorr-users] SEC tutorial
>
> hi all,
>
> during the last few weeks, I have written a new SEC tutorial paper which
> can be accessed through this web link:
> https://raw.githubusercontent.com/simple-evcorr/tutorial/main/SEC-tutorial.pdf
> Links to this tutorial and the relevant Github repository (
> https://github.com/simple-evcorr/tutorial) are also available in the SEC
> home page.
>
> This tutorial is primarily intended for people who haven't used SEC
> before, and it explains the essentials of SEC in a gentle way, starting
> from simple examples and gradually moving to more complex ones. After
> reading this tutorial, the reader should have a good understanding of event
> correlation rules, event correlation operations, contexts, synthetic
> events, pattern match caching, and many other things. And it might provide
> interesting insights even for experienced users :)
>
> As mentioned in the introduction, this tutorial does not intend to be a
> replacement for the official documentation. Therefore, it does not attempt
> to explain every possible detail, but rather provide an overview of the key
> concepts, so that it would be easier to look up more detailed information
> from the man page.
>
> Hopefully the tutorial will be useful and you will find it easy to follow
> :)
>
> kind regards,
> risto
>
>
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
>

Re: [Simple-evcorr-users] SEC tutorial

[Jim V. M. <jim...@cl...>]

Risto, 

Thanks for doing that up. I'll work through it. 

Are you interested in feedback on it? If so, do you have a preferred mechanism for that? (I can just mark up the PDF as I read through and upload it somewhere when I'm done). Note that I might find myself mostly copyediting, rather than testing all the examples. 

Jim 

-- 
Jim Van Meggelen 
ClearlyCore Inc. 

+1-416-639-6001 (DID) 
+1-877-253-2716 (Canada) 
+1-866-644-7729 (USA) 
+1-416-425-6111 x6001 
jim...@cl... 
[ http://www.clearlycore.com/ | http://www.clearlycore.com ] 

Asterisk: The Definitive Guide 
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: 
[ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] 

> From: "Risto Vaarandi" <ris...@gm...>
> To: "simple-evcorr-users" <sim...@li...>
> Sent: Thursday, 1 December, 2022 05:23:48
> Subject: [Simple-evcorr-users] SEC tutorial

> hi all,

> during the last few weeks, I have written a new SEC tutorial paper which can be
> accessed through this web link: [
> https://raw.githubusercontent.com/simple-evcorr/tutorial/main/SEC-tutorial.pdf
> |
> https://raw.githubusercontent.com/simple-evcorr/tutorial/main/SEC-tutorial.pdf
> ]
> Links to this tutorial and the relevant Github repository ( [
> https://github.com/simple-evcorr/tutorial |
> https://github.com/simple-evcorr/tutorial ] ) are also available in the SEC
> home page.

> This tutorial is primarily intended for people who haven't used SEC before, and
> it explains the essentials of SEC in a gentle way, starting from simple
> examples and gradually moving to more complex ones. After reading this
> tutorial, the reader should have a good understanding of event correlation
> rules, event correlation operations, contexts, synthetic events, pattern match
> caching, and many other things. And it might provide interesting insights even
> for experienced users :)

> As mentioned in the introduction, this tutorial does not intend to be a
> replacement for the official documentation. Therefore, it does not attempt to
> explain every possible detail, but rather provide an overview of the key
> concepts, so that it would be easier to look up more detailed information from
> the man page.

> Hopefully the tutorial will be useful and you will find it easy to follow :)

> kind regards,
> risto

> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

[Simple-evcorr-users] SEC tutorial

[Risto V. <ris...@gm...>]

hi all,

during the last few weeks, I have written a new SEC tutorial paper which
can be accessed through this web link:
https://raw.githubusercontent.com/simple-evcorr/tutorial/main/SEC-tutorial.pdf
Links to this tutorial and the relevant Github repository (
https://github.com/simple-evcorr/tutorial) are also available in the SEC
home page.

This tutorial is primarily intended for people who haven't used SEC before,
and it explains the essentials of SEC in a gentle way, starting from simple
examples and gradually moving to more complex ones. After reading this
tutorial, the reader should have a good understanding of event correlation
rules, event correlation operations, contexts, synthetic events, pattern
match caching, and many other things. And it might provide interesting
insights even for experienced users :)

As mentioned in the introduction, this tutorial does not intend to be a
replacement for the official documentation. Therefore, it does not attempt
to explain every possible detail, but rather provide an overview of the key
concepts, so that it would be easier to look up more detailed information
from the man page.

Hopefully the tutorial will be useful and you will find it easy to follow :)

kind regards,
risto

Re: [Simple-evcorr-users] add string to the event store of a context only if it doesn't exist

[Risto V. <ris...@gm...>]

hi Sean,

there are several ways to approach this problem and perhaps I will outline
two possible ways below.

Before adding a new ID to the context, one could search all existing IDs in
the context with the SEC 'while' action, looping over all IDs one by one,
and comparing each previously stored ID with the new ID value. However, it
is more efficient to implement that functionality in one 'lcall' action
that would take all stored IDs as an input parameter for a 3-line Perl
function, and do all processing inside that function. Here is an example
rule that illustrates that approach:

type=single
ptype=regexp
pattern=add ID (\d+)
desc=add new ID to perl hash
action=copy CONTEXT %id_list; \
       lcall %ret %id_list $1 -> ( sub { my(@keys) = split(/\n/, $_[0]); \
                                   my(%hash) = map { $_ => 1 } @keys; \
                                   return !exists($hash{$_[1]}); } ); \
       if %ret ( add CONTEXT $1 )

In this rule, all IDs that have been previously stored to CONTEXT are
written into the action list variable %id_list, so that IDs are separated
with newlines in that variable. In the Perl function that 'lcall' action
invokes, the list of stored IDs is the first parameter and the new ID value
the second one. Also, the function returns true if the new value is not
present in the list, and false if it is already there. Inside the function,
the list of IDs is turned into a hash table (%hash), and a new value is
searched from this hash table with a key lookup. Finally, the SEC 'if'
action checks the function return value and adds the new ID to CONTEXT only
if the function returned true.

This approach has the drawback that you have to build the Perl hash table
before each attempt to add a new ID to the context. It is much cheaper not
to use the context when you are storing the IDs, but keep them in a Perl
hash during this process, moving them over to a context when you start the
cleanup procedure. Here is an example how this could be done:

type=single
ptype=regexp
pattern=add ID (\d+)
desc=add new ID to perl hash
action=lcall %o $1 -> ( sub { $hash{$_[0]} = 1 } );

type=single
ptype=regexp
pattern=process IDs
desc=process all IDs from hash and clear the hash
action=lcall %id_list -> ( sub { keys(%hash) } ); \
       fill CONTEXT %id_list; \
       lcall %o -> ( sub { %hash = () } )

As you can see, the first rule creates a key with the value of 1 for each
observed ID in the hash table (if the key already exists in the table, this
operation simply overwrites its value, and is thus a no-op). The second
rule implements a cleanup procedure for all stored IDs. First, the Perl
function of the 'lcall' action returns a list of all keys from the hash
table. The keys will be joined into a string, using newlines as separators,
and this string is assigned to the action list variable %id_list. The
'fill' action will then write the value of the %id_list variable into
CONTEXT. Since %id_list holds a multiline string, each line from that
string will become a separate line in the event store of CONTEXT. Finally,
another 'lcall' action will clear the Perl hash table.

With the above approach, the ID values are tracked with the help of the
Perl hash table, and the context is created at a cleanup stage when all IDs
are processed together. Since I don't know the nature of the cleanup
process, the second rule is just setting up CONTEXT for that without any
further action. If the cleanup process involves some Perl code or invoking
an external script, having CONTEXT might not even be needed, but you could
directly pass the %id_list variable to the cleanup code or cleanup script.

Hopefully these examples were able to provide some ideas on how to tackle
the problem you have.

kind regards,
risto


It’s late and my Perl is years rusty. Does anyone already have a routine
> that will do this? Basically I’m using an event store to track of a list of
> id’s so I can clean something up. I need a way to add a string to the event
> store only if it’s not already in there.
>
>
>
> Thanks.
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>

[Simple-evcorr-users] add string to the event store of a context only if it doesn't exist

[Sean H. <sea...@me...>]

It's late and my Perl is years rusty. Does anyone already have a routine that will do this? Basically I'm using an event store to track of a list of id's so I can clean something up. I need a way to add a string to the event store only if it's not already in there.

Thanks.

Re: [Simple-evcorr-users] context w/ input file name embedded

[Sean H. <sea...@me...>]

Risto,

Man I feel like an idiot. That's what I get for copy / pasteing stuff around. I removed the ; in the test setup and it's working like I thought it would.

My real setup seems to be good now as well.

Thanks a million.

While I have you, one other quick question, can SEC do sub-second precision?

I've used SEC on and off for years so thanks for a great product. It always to be the tool I pull out when I have a really complicated problem to solve.

Sean

From: Risto Vaarandi <ris...@gm...>
Sent: Saturday, November 5, 2022 8:36 PM
To: Sean Hennessey <sea...@me...>
Cc: sim...@li...
Subject: Re: [Simple-evcorr-users] context w/ input file name embedded

hi Sean,
I was quite puzzled why the ruleset you have posted is not working, and after testing it several times and looking at the rules, I think I found the reason. When you look at the 'context' field of the second rule, there is an extra semicolon at the end of the context name (when the rule file is loaded and parsed, it is treated as a part of the context name). Therefore, the 'context' field is not checking the presence of the right context. After removing that semicolon, the second rule started to produce the matches :-)
kind regards,
risto

Kontakt Sean Hennessey (<sea...@me...<mailto:sea...@me...>>) kirjutas kuupäeval P, 6. november 2022 kell 02:06:
I'm trying to do something I think I possible, but it's not quite working. I have a use case where I need to watch one central log file to catch a system that is going to fire off a sep process that creates a log I'm really interested in. I the rules that do this just fine. It uses the addinput and dropinput to pull in and out files.

My issue comes in watching these other log files. Each one of these file is logging events I'm concerned about. Each file has threads. Those threads are unique in that file, but not across files. So thread 1 for example is going to be in all log files. I'm able to set a context that includes the filename and thread id w/ no problem on the rule that is picking up the first event I'm concerned about. My issue is then using the context= in the next rule to limit it to just that input.

Here's some examples I've tried:

type=single
ptype=regexp
pattern=Action one.*Thread="Thread([0-9]+)"
desc=Action one for $+{_inputsrc} thread $1
action=create LK_$+{_inputsrc}_$1 86400;fill LK_$+{_inputsrc}_$1 %t;

type=single
continue=takenext
ptype=regexp
pattern="Thread([0-9]+)"
context=LK_$+{_inputsrc}_$1;
desc=got a line from $+{_inputsrc} for thread $1 winner winner ===LK_$+{_intcontext}_$1===
action=delete LK_$+{_inputsrc}_$1;write - NL %t %var1 %s;

type=single
continue=takenext
ptype=regexp
pattern="Thread([0-9]+)"
context=LK_inp.text_21
desc=why only this one? got a line from $+{_inputsrc} for thread $1 +++$+{_intcontext}+++ ===$+{_inputsrc}=== _$1_
action=pop LK_$+{_inputsrc}_$1 %var1;delete LK_$+{_inputsrc}_$1;write - NL %t %var1 %s;


sec --conf=testing.sec --intevents --intcontexts --nochildterm --input=inp.text --debug=6
SEC (Simple Event Correlator) 2.8.2
Reading configuration from testing.sec
3 rules loaded from testing.sec
No --bufsize command line option or --bufsize=0, setting --bufsize to 1
Opening input file inp.text
Interactive process, SIGINT can't be used for changing the logging level
Creating SEC internal context 'SEC_INTERNAL_EVENT'
Creating SEC internal event 'SEC_STARTUP'
Deleting SEC internal context 'SEC_INTERNAL_EVENT'


Now I feed in one line:
echo 'INFO  [2022-11-05 16:24:58,506] Action one Thread="Thread21", "' >> inp.text

I get:
Creating context 'LK_inp.text_21'
Filling context 'LK_inp.text_21' with event(s) 'Sat Nov  5 19:24:55 2022'

If I create the sec.dump file I see this as far as what contexts are there:
List of contexts:
============================================================
Context Name: LK_inp.text_21
Creation Time: Sat Nov  5 19:24:55 2022
Lifetime: 86400 seconds
1 events associated with context:
Sat Nov  5 19:24:55 2022
------------------------------------------------------------
Total: 1 elements

I now feed in the other line:
echo 'INFO  [2022-11-05 16:24:58,506] Action two Thread="Thread21", "' >> inp.text

Rule 3 ends up firing and not rule 2:
Pop the last element of context 'LK_inp.text_21' event store into variable '%var1'
Variable '%var1' set to 'Sat Nov  5 19:24:55 2022'
Deleting context 'LK_inp.text_21'
Context 'LK_inp.text_21' deleted
Writing event 'NL Sat Nov  5 19:26:25 2022 Sat Nov  5 19:24:55 2022 why only this one? got a line from inp.text for thread 21 +++_FILE_EVENT_inp.text+++ ===inp.text=== _21_' to file '-'
NL Sat Nov  5 19:26:25 2022 Sat Nov  5 19:24:55 2022 why only this one? got a line from inp.text for thread 21 +++_FILE_EVENT_inp.text+++ ===inp.text=== _21_

Rule 3 is able to delete the context just fine, so I know the $+{_inputsrc} is being evaluated correctly. Can it not be evaluated in the context= line?

Can someone guide me w/ a way to get this work?

Thanks in advance
Sean
_______________________________________________
Simple-evcorr-users mailing list
Sim...@li...<mailto:Sim...@li...>
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Furl.avanan.click%2Fv2%2F___https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fsimple-evcorr-users___.YXAzOm1lcmN1cnlnYXRlOmE6bzowMTM5ZWJkMTc0OTQ0MTI1NjEyN2M5ZjFhYTVhNGQ2Yzo2OjdlMTM6ZmI5N2U3NzQ1MzhmYTVmZjUxYTBkM2E3NGMwZDUxYzY2OTJmY2IzN2YwZDdhYmM3OWVkNmVkZmQxNmRhZWEwODpoOlQ&data=05%7C01%7Csean.hennessey%40mercurygate.com%7C12647ac631c24d38966508dabf8efa1c%7Cd382f247fb2846cc88ecfe1341d2f058%7C0%7C0%7C638032918047103540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PQGlMCG2DYpwlMjyiY1AiNHMAhb8kq8ai3ce5ai7s1s%3D&reserved=0>

Re: [Simple-evcorr-users] context w/ input file name embedded

[Risto V. <ris...@gm...>]

hi Sean,

Risto,
>
>
>
> Man I feel like an idiot. That’s what I get for copy / pasteing stuff
> around. I removed the ; in the test setup and it’s working like I thought
> it would.
>
>
>
> My real setup seems to be good now as well.
>
>
>
> Thanks a million.
>

It's great I was able to help (such redundant characters are sometimes very
difficult to spot and I've run into the same issue several times in the
past :)


>
>
> While I have you, one other quick question, can SEC do sub-second
> precision?
>

Unfortunately not, the current code base utilizes the time() function for
all time measurements which returns seconds since Epoch (that limits the
precision to full seconds).


>
>
> I’ve used SEC on and off for years so thanks for a great product. It
> always to be the tool I pull out when I have a really complicated problem
> to solve.
>

Many thanks for using it :)

risto


>
>
> Sean
>
>
>
>

Re: [Simple-evcorr-users] context w/ input file name embedded

[Risto V. <ris...@gm...>]

hi Sean,
I was quite puzzled why the ruleset you have posted is not working, and
after testing it several times and looking at the rules, I think I found
the reason. When you look at the 'context' field of the second rule, there
is an extra semicolon at the end of the context name (when the rule file is
loaded and parsed, it is treated as a part of the context name). Therefore,
the 'context' field is not checking the presence of the right context.
After removing that semicolon, the second rule started to produce the
matches :-)
kind regards,
risto

Kontakt Sean Hennessey (<sea...@me...>) kirjutas
kuupäeval P, 6. november 2022 kell 02:06:

> I’m trying to do something I think I possible, but it’s not quite working.
> I have a use case where I need to watch one central log file to catch a
> system that is going to fire off a sep process that creates a log I’m
> really interested in. I the rules that do this just fine. It uses the
> addinput and dropinput to pull in and out files.
>
>
>
> My issue comes in watching these other log files. Each one of these file
> is logging events I’m concerned about. Each file has threads. Those threads
> are unique in that file, but not across files. So thread 1 for example is
> going to be in all log files. I’m able to set a context that includes the
> filename and thread id w/ no problem on the rule that is picking up the
> first event I’m concerned about. My issue is then using the context= in the
> next rule to limit it to just that input.
>
>
>
> Here’s some examples I’ve tried:
>
>
>
> type=single
>
> ptype=regexp
>
> pattern=Action one.*Thread="Thread([0-9]+)"
>
> desc=Action one for $+{_inputsrc} thread $1
>
> action=create LK_$+{_inputsrc}_$1 86400;fill LK_$+{_inputsrc}_$1 %t;
>
>
>
> type=single
>
> continue=takenext
>
> ptype=regexp
>
> pattern="Thread([0-9]+)"
>
> context=LK_$+{_inputsrc}_$1;
>
> desc=got a line from $+{_inputsrc} for thread $1 winner winner
> ===LK_$+{_intcontext}_$1===
>
> action=delete LK_$+{_inputsrc}_$1;write - NL %t %var1 %s;
>
>
>
> type=single
>
> continue=takenext
>
> ptype=regexp
>
> pattern="Thread([0-9]+)"
>
> context=LK_inp.text_21
>
> desc=why only this one? got a line from $+{_inputsrc} for thread $1
> +++$+{_intcontext}+++ ===$+{_inputsrc}=== _$1_
>
> action=pop LK_$+{_inputsrc}_$1 %var1;delete LK_$+{_inputsrc}_$1;write - NL
> %t %var1 %s;
>
>
>
>
>
> sec --conf=testing.sec --intevents --intcontexts --nochildterm
> --input=inp.text --debug=6
>
> SEC (Simple Event Correlator) 2.8.2
>
> Reading configuration from testing.sec
>
> 3 rules loaded from testing.sec
>
> No --bufsize command line option or --bufsize=0, setting --bufsize to 1
>
> Opening input file inp.text
>
> Interactive process, SIGINT can't be used for changing the logging level
>
> Creating SEC internal context 'SEC_INTERNAL_EVENT'
>
> Creating SEC internal event 'SEC_STARTUP'
>
> Deleting SEC internal context 'SEC_INTERNAL_EVENT'
>
>
>
>
>
> Now I feed in one line:
>
> echo 'INFO  [2022-11-05 16:24:58,506] Action one Thread="Thread21", "' >>
> inp.text
>
>
>
> I get:
> Creating context 'LK_inp.text_21'
>
> Filling context 'LK_inp.text_21' with event(s) 'Sat Nov  5 19:24:55 2022'
>
>
>
> If I create the sec.dump file I see this as far as what contexts are there:
> List of contexts:
>
> ============================================================
>
> Context Name: LK_inp.text_21
>
> Creation Time: Sat Nov  5 19:24:55 2022
>
> Lifetime: 86400 seconds
>
> 1 events associated with context:
>
> Sat Nov  5 19:24:55 2022
>
> ------------------------------------------------------------
>
> Total: 1 elements
>
>
>
> I now feed in the other line:
>
> echo 'INFO  [2022-11-05 16:24:58,506] Action two Thread="Thread21", "' >>
> inp.text
>
>
>
> Rule 3 ends up firing and not rule 2:
>
> Pop the last element of context 'LK_inp.text_21' event store into variable
> '%var1'
>
> Variable '%var1' set to 'Sat Nov  5 19:24:55 2022'
>
> Deleting context 'LK_inp.text_21'
>
> Context 'LK_inp.text_21' deleted
>
> Writing event 'NL Sat Nov  5 19:26:25 2022 Sat Nov  5 19:24:55 2022 why
> only this one? got a line from inp.text for thread 21
> +++_FILE_EVENT_inp.text+++ ===inp.text=== _21_' to file '-'
>
> NL Sat Nov  5 19:26:25 2022 Sat Nov  5 19:24:55 2022 why only this one?
> got a line from inp.text for thread 21 +++_FILE_EVENT_inp.text+++
> ===inp.text=== _21_
>
>
>
> Rule 3 is able to delete the context just fine, so I know the
> $+{_inputsrc} is being evaluated correctly. Can it not be evaluated in the
> context= line?
>
>
>
> Can someone guide me w/ a way to get this work?
>
>
>
> Thanks in advance
>
> Sean
>
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>

[Simple-evcorr-users] context w/ input file name embedded

[Sean H. <sea...@me...>]

I'm trying to do something I think I possible, but it's not quite working. I have a use case where I need to watch one central log file to catch a system that is going to fire off a sep process that creates a log I'm really interested in. I the rules that do this just fine. It uses the addinput and dropinput to pull in and out files.

My issue comes in watching these other log files. Each one of these file is logging events I'm concerned about. Each file has threads. Those threads are unique in that file, but not across files. So thread 1 for example is going to be in all log files. I'm able to set a context that includes the filename and thread id w/ no problem on the rule that is picking up the first event I'm concerned about. My issue is then using the context= in the next rule to limit it to just that input.

Here's some examples I've tried:

type=single
ptype=regexp
pattern=Action one.*Thread="Thread([0-9]+)"
desc=Action one for $+{_inputsrc} thread $1
action=create LK_$+{_inputsrc}_$1 86400;fill LK_$+{_inputsrc}_$1 %t;

type=single
continue=takenext
ptype=regexp
pattern="Thread([0-9]+)"
context=LK_$+{_inputsrc}_$1;
desc=got a line from $+{_inputsrc} for thread $1 winner winner ===LK_$+{_intcontext}_$1===
action=delete LK_$+{_inputsrc}_$1;write - NL %t %var1 %s;

type=single
continue=takenext
ptype=regexp
pattern="Thread([0-9]+)"
context=LK_inp.text_21
desc=why only this one? got a line from $+{_inputsrc} for thread $1 +++$+{_intcontext}+++ ===$+{_inputsrc}=== _$1_
action=pop LK_$+{_inputsrc}_$1 %var1;delete LK_$+{_inputsrc}_$1;write - NL %t %var1 %s;


sec --conf=testing.sec --intevents --intcontexts --nochildterm --input=inp.text --debug=6
SEC (Simple Event Correlator) 2.8.2
Reading configuration from testing.sec
3 rules loaded from testing.sec
No --bufsize command line option or --bufsize=0, setting --bufsize to 1
Opening input file inp.text
Interactive process, SIGINT can't be used for changing the logging level
Creating SEC internal context 'SEC_INTERNAL_EVENT'
Creating SEC internal event 'SEC_STARTUP'
Deleting SEC internal context 'SEC_INTERNAL_EVENT'


Now I feed in one line:
echo 'INFO  [2022-11-05 16:24:58,506] Action one Thread="Thread21", "' >> inp.text

I get:
Creating context 'LK_inp.text_21'
Filling context 'LK_inp.text_21' with event(s) 'Sat Nov  5 19:24:55 2022'

If I create the sec.dump file I see this as far as what contexts are there:
List of contexts:
============================================================
Context Name: LK_inp.text_21
Creation Time: Sat Nov  5 19:24:55 2022
Lifetime: 86400 seconds
1 events associated with context:
Sat Nov  5 19:24:55 2022
------------------------------------------------------------
Total: 1 elements

I now feed in the other line:
echo 'INFO  [2022-11-05 16:24:58,506] Action two Thread="Thread21", "' >> inp.text

Rule 3 ends up firing and not rule 2:
Pop the last element of context 'LK_inp.text_21' event store into variable '%var1'
Variable '%var1' set to 'Sat Nov  5 19:24:55 2022'
Deleting context 'LK_inp.text_21'
Context 'LK_inp.text_21' deleted
Writing event 'NL Sat Nov  5 19:26:25 2022 Sat Nov  5 19:24:55 2022 why only this one? got a line from inp.text for thread 21 +++_FILE_EVENT_inp.text+++ ===inp.text=== _21_' to file '-'
NL Sat Nov  5 19:26:25 2022 Sat Nov  5 19:24:55 2022 why only this one? got a line from inp.text for thread 21 +++_FILE_EVENT_inp.text+++ ===inp.text=== _21_

Rule 3 is able to delete the context just fine, so I know the $+{_inputsrc} is being evaluated correctly. Can it not be evaluated in the context= line?

Can someone guide me w/ a way to get this work?

Thanks in advance
Sean

[Simple-evcorr-users] new features introduced in SEC-2.9.1

[Risto V. <ris...@gm...>]

hi all,

this email provides an introduction to new features in the 2.9.1 version.

Starting from the 2.9.0 version (released last year), EventGroup rules are
supporting event group patterns which allow for matching specific event
sequences within predefined time windows. For example, suppose you want to
match a sequence of events A and B within the window of 60 seconds,
provided that this sequence ends with subsequence A B A. In order to
accomplish this, you could use the 'egpattern' field together with
EventGroup2 rule:

type=EventGroup2
ptype=SubStr
pattern=EVENT_A
ptype2=SubStr
pattern2=EVENT_B
desc=Sequence of A and B that ends with A B A
action=write - %s
egptype=RegExp
egpattern=1 2 1$
window=60

In the above rule, the 'egpattern' field ensures that the last three events
in the event sequence were matched by the 1st, 2nd, and 1st regular
expression pattern respectively.

Starting from SEC-2.9.1, one is no longer limited to numeric tokens that
indicate the matching pattern (for example, tokens 1 and 2 like in the
above rule), but it is possible to define custom tokens. This allows for
implementing useful event correlation schemes, even for EventGroup rules
involving a single event pattern. For example, consider the following rule:

type=EventGroup
ptype=RegExp
pattern=sshd\[\d+\]: Failed .+ for (\S+) from ([\d.]+) port \d+ ssh2
desc=SSH login failures from three different hosts within 1m for user $1
egtoken=$2
egptype=PerlFunc
egpattern=sub { my(%hosts) = map { $_ => 1 } @{$_[1]}; \
                return scalar(keys %hosts) >= 3; }
action=pipe '%t: %s' /bin/mail root@localhost
window=60
thresh=3

The above rule sends an email to root@localhost if at least three SSH login
failures were observed within 60 seconds for the same user account, so that
login failures originated from three unique client hosts. In this rule, the
'egtoken' field configures the use of client host IP addresses as tokens.
Also, the 'egpattern' field is a Perl function which takes the list of
tokens as its second input parameter ($_[1]), making sure that the list
contains at least three unique elements (IP addresses).

The above task can be accomplished with the help of contexts (e.g., this
paper https://ristov.github.io/publications/cogsima15-sec-web.pdf describes
one rule example), but the new features offer an opportunity for writing
more compact solutions.

Hopefully the rule examples from this email provided some insights how to
use the new features and what tasks they allow to address.

kind regards,
risto

[Simple-evcorr-users] SEC-2.9.1 released

[Risto V. <ris...@gm...>]

hi all,

SEC-2.9.1 has been released which is available from the SEC home page and
through the following download link:
https://github.com/simple-evcorr/sec/releases/download/2.9.1/sec-2.9.1.tar.gz

Here is the changelog for the new version:

--- version 2.9.1

* added support for 'egtoken*' fields in EventGroup rules.

* starting from this version, list of event group string tokens is passed
  to PerlFunc and NPerlFunc event group patterns as an additional parameter.


kind regards,
risto

Re: [Simple-evcorr-users] Parsing Asterisk log files for downstream reporting - so far so good!

[Jim V. M. <jim...@cl...>]

Risto,

I promise to do so!

Thanks again for your generous help, and for this amazing piece of software.



--
Jim Van Meggelen
ClearlyCore Inc
+14166396001

________________________________
From: Risto Vaarandi <ris...@gm...>
Sent: Friday, April 1, 2022 06:50
To: Jim Van Meggelen
Cc: simple-evcorr-users
Subject: Re: [Simple-evcorr-users] Parsing Asterisk log files for downstream reporting - so far so good!



hi Jim,

it is great to hear that the rule example was useful! As for supporting the
project, the best way of doing it is to post interesting questions to the
mailing list :) The discussions in the mailing list will be helpful for all
the subscribers, and will also provide new ideas for further development of
SEC.

kind regards,
risto


Risto,
>
> I tried that out and it looks to be doing what it should!
>
> Thank you so much for generously taking your time to help me with this.
> I'm still trying to wrap my head around how contexts and descriptions and
> such connect it all together, but I am learning!
>
> Is there some way I can support the project by way of thanks?
>
> Warm regards and thanks again. Aitäh!
>
> Jim
>
> --
> Jim Van Meggelen
> ClearlyCore Inc.
>
>
>
> +1-416-639-6001 (DID)
> +1-877-253-2716 (Canada)
> +1-866-644-7729 (USA)
> +1-416-425-6111 x6001
> jim...@cl...
> http://www.clearlycore.com
>
>
> *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:*
> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf
>
> ------------------------------
>
> *From: *"Risto Vaarandi" <ris...@gm...>
> *To: *"Jim Van Meggelen" <jim...@cl...>
> *Cc: *"simple-evcorr-users" <sim...@li...>
> *Sent: *Thursday, 31 March, 2022 12:50:46
> *Subject: *Re: [Simple-evcorr-users] Parsing Asterisk log files for
> downstream reporting - so far so good!
>
> hi Jim,
>
> if you want to match the "ActivationHelp" event and react to the earliest
> "User entered" event, provided that these events share the same caller ID,
> you could use the following rule:
>
> type=PAIR
> desc=IVR caller $4 offered activation or statement inquiry
> ptype=RegExp
> action= write - $4 activation or statement inquiry offered
> pattern=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*Executing
> \[(.+)\@(ivr-welcome:[0-9]{1,3})\] (.+)\(\"([A-Z]{3,10})\/(.+)\",
> \"ActivationHelp,prompts.*
> ptype2=regexp
> rem=pattern2=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*
> app_read.c: User entered '?(nothing|[0-9]*)'?.*
> pattern2=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[($4)\].*
> app_read.c: User entered '?(nothing|[0-9]*)'?.*
> desc2=Flexiti IVR - caller selection at activation or statement inquiry
> action2=assign %p flexity; \
>         assign %i -ivr ; \
>         write %p%i.csv $1,$2,$4,,,,,,,,,$5 ; closef %p%i.csv ; \
>         write - $4 ACTIVATION_STATEMENT %p %i $1 User entered: $5
> window=10800
>
> Let me also explain the changes I have made in the original version:
>
> 1) in the 'pattern' field, I have replaced "flexity-ivr-welcome" with
> "ivr-welcome", so that the example events would be properly matched. Also,
> I have set %p in the 'action2' field, since this variable was originally
> used without a value.
>
> 2) the 'desc' field of the rule has been set to the string 'IVR caller $4
> offered activation or statement inquiry'. Since the value of the 'desc'
> field determines the ID of the event correlation operation, the presence of
> the $4 variable (corresponds to the caller ID) in the 'desc' field ensures
> that the rule will start a separate Pair operation for each distinct caller
> ID. That will prevent situations where there is only one operation that
> mistakenly reacts to "ActivationHelp" and "User entered" events for
> different caller ID's.
>
> 3) in the 'pattern2' field, "\[(C-[0-9a-f]{8})\]" has been replaced with
> "\[($4)\]". The use of $4 variable in 'pattern2' field allows for narrowing
> the regular expression match to the particular caller ID you are expecting
> to see. For instance, after seeing the event
>
> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing
> [8005551234@ivr-welcome:34] Read("SIP/CHANNELNAME-000138ac",
> "ActivationHelp,prompts/902&prompts/063,1,,,15,") in new stack
>
> the Pair operation is started which expects another event matching the
> following regular expression (note that $4 has been replaced with the
> caller ID from the previous event):
>
> ^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C\-00004037)\].*
> app_read.c: User entered '?(nothing|[0-9]*)'?.*
>
> Without this modification to 'pattern2', the regular expression would
> match the "User entered" event for *any* caller ID (that is probably not
> what you want).
>
> 4) I have also set the "window" parameter to 3 hours (10800 seconds) -- if
> for some reason you would never be seeing "User entered" event for the
> given caller ID, the Pair operation will not exist forever but will rather
> time out after 3 hours.
>
> 5) If there is a chance of not seeing the "User entered event", you can
> also utilize the following Single rule after the Pair rule for terminating
> the hanging Pair operation when the call is ended:
>
> type=Single
> ptype=RegExp
> pattern=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*pbx\.c:
> Spawn extension \(.*\) exited non-zero on.*
> desc=End of call for caller $4
> action=reset -1 IVR caller $4 offered activation or statement inquiry
>
> Note that the first parameter of the 'reset' action (-1) indicates that an
> event correlation operation started by the previous rule (i.e., the Pair
> rule) has to be terminated. And the second parameter "IVR caller $4 offered
> activation or statement inquiry" provides the ID of this operation (in
> other words, whenever a call with some ID has ended, we will be terminating
> the Pair operation for this call ID). Also, if this Pair operation does not
> exist for the given call ID (maybe because the ActivationHelp event was
> never seen for this particular call), the 'reset' action is a no-op.
>
> However, if you are absolutely confident that the "User entered" event
> always occurs after "ActivationHelp", you don't need the above Single rule.
>
> Does this solution work for you? I hope I understood the problem
> description correctly, and if there are any other questions or something
> that needs improvement, feel free to ask.
>
> kind regards,
> risto
>
>
> Risto,
>>
>> Absolutely.
>>
>> The following pattern should catch it:
>>
>> ^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*
>> pbx.c: Spawn extension \(.*\) exited non-zero on.*
>>
>>
>> Here's a few lines leading up to it (interesting because here is a Read()
>> operation where the caller does not make a choice but instead hangs up
>> while the system is waiting for a response):
>>
>> [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] pbx.c: Executing
>> [8005551234@ivr-MainMenu:45] Read("SIP/CHANNELNAME-000138ac",
>> "BalanceRepeatPressed,prompts/004h&prompts/004i&prompts/043,1,,,15") in new
>> stack
>> [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] app_read.c: Accepting a
>> maximum of 1 digits.
>> [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] file.c:
>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004h.slin' (language 'en')
>> [2022-02-08-06:21:25:] VERBOSE[24858][C-00004037] file.c:
>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004i.slin' (language 'en')
>> [2022-02-08-06:21:28:] VERBOSE[24858][C-00004037] file.c:
>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/043.slin' (language 'en')
>> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] app_read.c: User
>> disconnected
>> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing
>> [h@ivr-MainMenu:1] NoOp("SIP/CHANNELNAME-000138ac", "Hangup handler
>> GlobalCallCount=0 CHANNELNAME=0 ") in new stack
>> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing
>> [h@ivr-MainMenu:2] NoOp("SIP/CHANNELNAME-000138ac", "SurveyType is ") in
>> new stack
>> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing
>> [h@ivr-MainMenu:3] ExecIf("SIP/CHANNELNAME-000138ac", "1?Hangup()") in
>> new stack
>> *[2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Spawn extension
>> (ivr-MainMenu, h, 3) exited non-zero on 'SIP/CHANNELNAME-000138ac'*
>>
>>
>>
>> --
>> Jim Van Meggelen
>> ClearlyCore Inc.
>>
>>
>>
>> +1-416-639-6001 (DID)
>> +1-877-253-2716 (Canada)
>> +1-866-644-7729 (USA)
>> +1-416-425-6111 x6001
>> jim...@cl...
>> http://www.clearlycore.com
>>
>>
>> *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:*
>> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf
>>
>
>

Re: [Simple-evcorr-users] Parsing Asterisk log files for downstream reporting - so far so good!

[Risto V. <ris...@gm...>]

hi Jim,

it is great to hear that the rule example was useful! As for supporting the
project, the best way of doing it is to post interesting questions to the
mailing list :) The discussions in the mailing list will be helpful for all
the subscribers, and will also provide new ideas for further development of
SEC.

kind regards,
risto


Risto,
>
> I tried that out and it looks to be doing what it should!
>
> Thank you so much for generously taking your time to help me with this.
> I'm still trying to wrap my head around how contexts and descriptions and
> such connect it all together, but I am learning!
>
> Is there some way I can support the project by way of thanks?
>
> Warm regards and thanks again. Aitäh!
>
> Jim
>
> --
> Jim Van Meggelen
> ClearlyCore Inc.
>
>
>
> +1-416-639-6001 (DID)
> +1-877-253-2716 (Canada)
> +1-866-644-7729 (USA)
> +1-416-425-6111 x6001
> jim...@cl...
> http://www.clearlycore.com
>
>
> *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:*
> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf
>
> ------------------------------
>
> *From: *"Risto Vaarandi" <ris...@gm...>
> *To: *"Jim Van Meggelen" <jim...@cl...>
> *Cc: *"simple-evcorr-users" <sim...@li...>
> *Sent: *Thursday, 31 March, 2022 12:50:46
> *Subject: *Re: [Simple-evcorr-users] Parsing Asterisk log files for
> downstream reporting - so far so good!
>
> hi Jim,
>
> if you want to match the "ActivationHelp" event and react to the earliest
> "User entered" event, provided that these events share the same caller ID,
> you could use the following rule:
>
> type=PAIR
> desc=IVR caller $4 offered activation or statement inquiry
> ptype=RegExp
> action= write - $4 activation or statement inquiry offered
> pattern=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*Executing
> \[(.+)\@(ivr-welcome:[0-9]{1,3})\] (.+)\(\"([A-Z]{3,10})\/(.+)\",
> \"ActivationHelp,prompts.*
> ptype2=regexp
> rem=pattern2=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*
> app_read.c: User entered '?(nothing|[0-9]*)'?.*
> pattern2=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[($4)\].*
> app_read.c: User entered '?(nothing|[0-9]*)'?.*
> desc2=Flexiti IVR - caller selection at activation or statement inquiry
> action2=assign %p flexity; \
>         assign %i -ivr ; \
>         write %p%i.csv $1,$2,$4,,,,,,,,,$5 ; closef %p%i.csv ; \
>         write - $4 ACTIVATION_STATEMENT %p %i $1 User entered: $5
> window=10800
>
> Let me also explain the changes I have made in the original version:
>
> 1) in the 'pattern' field, I have replaced "flexity-ivr-welcome" with
> "ivr-welcome", so that the example events would be properly matched. Also,
> I have set %p in the 'action2' field, since this variable was originally
> used without a value.
>
> 2) the 'desc' field of the rule has been set to the string 'IVR caller $4
> offered activation or statement inquiry'. Since the value of the 'desc'
> field determines the ID of the event correlation operation, the presence of
> the $4 variable (corresponds to the caller ID) in the 'desc' field ensures
> that the rule will start a separate Pair operation for each distinct caller
> ID. That will prevent situations where there is only one operation that
> mistakenly reacts to "ActivationHelp" and "User entered" events for
> different caller ID's.
>
> 3) in the 'pattern2' field, "\[(C-[0-9a-f]{8})\]" has been replaced with
> "\[($4)\]". The use of $4 variable in 'pattern2' field allows for narrowing
> the regular expression match to the particular caller ID you are expecting
> to see. For instance, after seeing the event
>
> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing
> [8005551234@ivr-welcome:34] Read("SIP/CHANNELNAME-000138ac",
> "ActivationHelp,prompts/902&prompts/063,1,,,15,") in new stack
>
> the Pair operation is started which expects another event matching the
> following regular expression (note that $4 has been replaced with the
> caller ID from the previous event):
>
> ^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C\-00004037)\].*
> app_read.c: User entered '?(nothing|[0-9]*)'?.*
>
> Without this modification to 'pattern2', the regular expression would
> match the "User entered" event for *any* caller ID (that is probably not
> what you want).
>
> 4) I have also set the "window" parameter to 3 hours (10800 seconds) -- if
> for some reason you would never be seeing "User entered" event for the
> given caller ID, the Pair operation will not exist forever but will rather
> time out after 3 hours.
>
> 5) If there is a chance of not seeing the "User entered event", you can
> also utilize the following Single rule after the Pair rule for terminating
> the hanging Pair operation when the call is ended:
>
> type=Single
> ptype=RegExp
> pattern=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*pbx\.c:
> Spawn extension \(.*\) exited non-zero on.*
> desc=End of call for caller $4
> action=reset -1 IVR caller $4 offered activation or statement inquiry
>
> Note that the first parameter of the 'reset' action (-1) indicates that an
> event correlation operation started by the previous rule (i.e., the Pair
> rule) has to be terminated. And the second parameter "IVR caller $4 offered
> activation or statement inquiry" provides the ID of this operation (in
> other words, whenever a call with some ID has ended, we will be terminating
> the Pair operation for this call ID). Also, if this Pair operation does not
> exist for the given call ID (maybe because the ActivationHelp event was
> never seen for this particular call), the 'reset' action is a no-op.
>
> However, if you are absolutely confident that the "User entered" event
> always occurs after "ActivationHelp", you don't need the above Single rule.
>
> Does this solution work for you? I hope I understood the problem
> description correctly, and if there are any other questions or something
> that needs improvement, feel free to ask.
>
> kind regards,
> risto
>
>
> Risto,
>>
>> Absolutely.
>>
>> The following pattern should catch it:
>>
>> ^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*
>> pbx.c: Spawn extension \(.*\) exited non-zero on.*
>>
>>
>> Here's a few lines leading up to it (interesting because here is a Read()
>> operation where the caller does not make a choice but instead hangs up
>> while the system is waiting for a response):
>>
>> [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] pbx.c: Executing
>> [8005551234@ivr-MainMenu:45] Read("SIP/CHANNELNAME-000138ac",
>> "BalanceRepeatPressed,prompts/004h&prompts/004i&prompts/043,1,,,15") in new
>> stack
>> [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] app_read.c: Accepting a
>> maximum of 1 digits.
>> [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] file.c:
>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004h.slin' (language 'en')
>> [2022-02-08-06:21:25:] VERBOSE[24858][C-00004037] file.c:
>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004i.slin' (language 'en')
>> [2022-02-08-06:21:28:] VERBOSE[24858][C-00004037] file.c:
>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/043.slin' (language 'en')
>> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] app_read.c: User
>> disconnected
>> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing
>> [h@ivr-MainMenu:1] NoOp("SIP/CHANNELNAME-000138ac", "Hangup handler
>> GlobalCallCount=0 CHANNELNAME=0 ") in new stack
>> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing
>> [h@ivr-MainMenu:2] NoOp("SIP/CHANNELNAME-000138ac", "SurveyType is ") in
>> new stack
>> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing
>> [h@ivr-MainMenu:3] ExecIf("SIP/CHANNELNAME-000138ac", "1?Hangup()") in
>> new stack
>> *[2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Spawn extension
>> (ivr-MainMenu, h, 3) exited non-zero on 'SIP/CHANNELNAME-000138ac'*
>>
>>
>>
>> --
>> Jim Van Meggelen
>> ClearlyCore Inc.
>>
>>
>>
>> +1-416-639-6001 (DID)
>> +1-877-253-2716 (Canada)
>> +1-866-644-7729 (USA)
>> +1-416-425-6111 x6001
>> jim...@cl...
>> http://www.clearlycore.com
>>
>>
>> *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:*
>> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf
>>
>
>

Re: [Simple-evcorr-users] Parsing Asterisk log files for downstream reporting - so far so good!

[Jim V. M. <jim...@cl...>]

Risto, 

I tried that out and it looks to be doing what it should! 

Thank you so much for generously taking your time to help me with this. I'm still trying to wrap my head around how contexts and descriptions and such connect it all together, but I am learning! 

Is there some way I can support the project by way of thanks? 

Warm regards and thanks again. A itäh! 

Jim 

-- 
Jim Van Meggelen 
ClearlyCore Inc. 

+1-416-639-6001 (DID) 
+1-877-253-2716 (Canada) 
+1-866-644-7729 (USA) 
+1-416-425-6111 x6001 
jim...@cl... 
[ http://www.clearlycore.com/ | http://www.clearlycore.com ] 

Asterisk: The Definitive Guide 
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: 
[ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] 

> From: "Risto Vaarandi" <ris...@gm...>
> To: "Jim Van Meggelen" <jim...@cl...>
> Cc: "simple-evcorr-users" <sim...@li...>
> Sent: Thursday, 31 March, 2022 12:50:46
> Subject: Re: [Simple-evcorr-users] Parsing Asterisk log files for downstream
> reporting - so far so good!

> hi Jim,

> if you want to match the "ActivationHelp" event and react to the earliest "User
> entered" event, provided that these events share the same caller ID, you could
> use the following rule:

> type=PAIR
> desc=IVR caller $4 offered activation or statement inquiry
> ptype=RegExp
> action= write - $4 activation or statement inquiry offered
> pattern=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*Executing
> \[(.+)\@(ivr-welcome:[0-9]{1,3})\] (.+)\(\"([A-Z]{3,10})\/(.+)\",
> \"ActivationHelp,prompts.*
> ptype2=regexp
> rem=pattern2=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*
> app_read.c: User entered '?(nothing|[0-9]*)'?.*
> pattern2=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[($4)\].*
> app_read.c: User entered '?(nothing|[0-9]*)'?.*
> desc2=Flexiti IVR - caller selection at activation or statement inquiry
> action2=assign %p flexity; \
> assign %i -ivr ; \
> write %p%i.csv $1,$2,$4,,,,,,,,,$5 ; closef %p%i.csv ; \
> write - $4 ACTIVATION_STATEMENT %p %i $1 User entered: $5
> window=10800

> Let me also explain the changes I have made in the original version:

> 1) in the 'pattern' field, I have replaced "flexity-ivr-welcome" with
> "ivr-welcome", so that the example events would be properly matched. Also, I
> have set %p in the 'action2' field, since this variable was originally used
> without a value.

> 2) the 'desc' field of the rule has been set to the string 'IVR caller $4
> offered activation or statement inquiry'. Since the value of the 'desc' field
> determines the ID of the event correlation operation, the presence of the $4
> variable (corresponds to the caller ID) in the 'desc' field ensures that the
> rule will start a separate Pair operation for each distinct caller ID. That
> will prevent situations where there is only one operation that mistakenly
> reacts to "ActivationHelp" and "User entered" events for different caller ID's.

> 3) in the 'pattern2' field, "\[(C-[0-9a-f]{8})\]" has been replaced with
> "\[($4)\]". The use of $4 variable in 'pattern2' field allows for narrowing the
> regular expression match to the particular caller ID you are expecting to see.
> For instance, after seeing the event

> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing
> [8005551234@ivr-welcome:34] Read("SIP/CHANNELNAME-000138ac",
> "ActivationHelp,prompts/902&prompts/063,1,,,15,") in new stack

> the Pair operation is started which expects another event matching the following
> regular expression (note that $4 has been replaced with the caller ID from the
> previous event):

> ^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C\-00004037)\].*
> app_read.c: User entered '?(nothing|[0-9]*)'?.*

> Without this modification to 'pattern2', the regular expression would match the
> "User entered" event for *any* caller ID (that is probably not what you want).

> 4) I have also set the "window" parameter to 3 hours (10800 seconds) -- if for
> some reason you would never be seeing "User entered" event for the given caller
> ID, the Pair operation will not exist forever but will rather time out after 3
> hours.

> 5) If there is a chance of not seeing the "User entered event", you can also
> utilize the following Single rule after the Pair rule for terminating the
> hanging Pair operation when the call is ended:

> type=Single
> ptype=RegExp
> pattern=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*pbx\.c:
> Spawn extension \(.*\) exited non-zero on.*
> desc=End of call for caller $4
> action=reset -1 IVR caller $4 offered activation or statement inquiry

> Note that the first parameter of the 'reset' action (-1) indicates that an event
> correlation operation started by the previous rule (i.e., the Pair rule) has to
> be terminated. And the second parameter "IVR caller $4 offered activation or
> statement inquiry" provides the ID of this operation (in other words, whenever
> a call with some ID has ended, we will be terminating the Pair operation for
> this call ID). Also, if this Pair operation does not exist for the given call
> ID (maybe because the ActivationHelp event was never seen for this particular
> call), the 'reset' action is a no-op.

> However, if you are absolutely confident that the "User entered" event always
> occurs after "ActivationHelp", you don't need the above Single rule.

> Does this solution work for you? I hope I understood the problem description
> correctly, and if there are any other questions or something that needs
> improvement, feel free to ask.

> kind regards,
> risto

>> Risto,

>> Absolutely.

>> The following pattern should catch it:

>> ^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*
>> pbx.c: Spawn extension \(.*\) exited non-zero on.*

>> Here's a few lines leading up to it (interesting because here is a Read()
>> operation where the caller does not make a choice but instead hangs up while
>> the system is waiting for a response):

>> [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] pbx.c: Executing
>> [8005551234@ivr-MainMenu:45] Read("SIP/CHANNELNAME-000138ac",
>> "BalanceRepeatPressed,prompts/004h&prompts/004i&prompts/043,1,,,15") in new
>> stack
>> [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] app_read.c: Accepting a
>> maximum of 1 digits.
>> [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] file.c:
>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004h.slin' (language 'en')
>> [2022-02-08-06:21:25:] VERBOSE[24858][C-00004037] file.c:
>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004i.slin' (language 'en')
>> [2022-02-08-06:21:28:] VERBOSE[24858][C-00004037] file.c:
>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/043.slin' (language 'en')
>> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] app_read.c: User disconnected
>> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing
>> [h@ivr-MainMenu:1] NoOp("SIP/CHANNELNAME-000138ac", "Hangup handler
>> GlobalCallCount=0 CHANNELNAME=0 ") in new stack
>> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing
>> [h@ivr-MainMenu:2] NoOp("SIP/CHANNELNAME-000138ac", "SurveyType is ") in new
>> stack
>> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing
>> [h@ivr-MainMenu:3] ExecIf("SIP/CHANNELNAME-000138ac", "1?Hangup()") in new
>> stack
>> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Spawn extension
>> (ivr-MainMenu, h, 3) exited non-zero on 'SIP/CHANNELNAME-000138ac'

>> --
>> Jim Van Meggelen
>> ClearlyCore Inc.

>> +1-416-639-6001 (DID)
>> +1-877-253-2716 (Canada)
>> +1-866-644-7729 (USA)
>> +1-416-425-6111 x6001
>> [ mailto:jim...@cl... | jim...@cl... ]
>> [ http://www.clearlycore.com/ | http://www.clearlycore.com ]

>> Asterisk: The Definitive Guide
>> FIFTH EDITION NOW AVAILABLE TO DOWNLOAD:
>> [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf |
>> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ]

Re: [Simple-evcorr-users] Parsing Asterisk log files for downstream reporting - so far so good!

[Risto V. <ris...@gm...>]

hi Jim,

if you want to match the "ActivationHelp" event and react to the earliest
"User entered" event, provided that these events share the same caller ID,
you could use the following rule:

type=PAIR
desc=IVR caller $4 offered activation or statement inquiry
ptype=RegExp
action= write - $4 activation or statement inquiry offered
pattern=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*Executing
\[(.+)\@(ivr-welcome:[0-9]{1,3})\] (.+)\(\"([A-Z]{3,10})\/(.+)\",
\"ActivationHelp,prompts.*
ptype2=regexp
rem=pattern2=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*
app_read.c: User entered '?(nothing|[0-9]*)'?.*
pattern2=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[($4)\].*
app_read.c: User entered '?(nothing|[0-9]*)'?.*
desc2=Flexiti IVR - caller selection at activation or statement inquiry
action2=assign %p flexity; \
        assign %i -ivr ; \
        write %p%i.csv $1,$2,$4,,,,,,,,,$5 ; closef %p%i.csv ; \
        write - $4 ACTIVATION_STATEMENT %p %i $1 User entered: $5
window=10800

Let me also explain the changes I have made in the original version:

1) in the 'pattern' field, I have replaced "flexity-ivr-welcome" with
"ivr-welcome", so that the example events would be properly matched. Also,
I have set %p in the 'action2' field, since this variable was originally
used without a value.

2) the 'desc' field of the rule has been set to the string 'IVR caller $4
offered activation or statement inquiry'. Since the value of the 'desc'
field determines the ID of the event correlation operation, the presence of
the $4 variable (corresponds to the caller ID) in the 'desc' field ensures
that the rule will start a separate Pair operation for each distinct caller
ID. That will prevent situations where there is only one operation that
mistakenly reacts to "ActivationHelp" and "User entered" events for
different caller ID's.

3) in the 'pattern2' field, "\[(C-[0-9a-f]{8})\]" has been replaced with
"\[($4)\]". The use of $4 variable in 'pattern2' field allows for narrowing
the regular expression match to the particular caller ID you are expecting
to see. For instance, after seeing the event

[2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing
[8005551234@ivr-welcome:34] Read("SIP/CHANNELNAME-000138ac",
"ActivationHelp,prompts/902&prompts/063,1,,,15,") in new stack

the Pair operation is started which expects another event matching the
following regular expression (note that $4 has been replaced with the
caller ID from the previous event):

^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C\-00004037)\].*
app_read.c: User entered '?(nothing|[0-9]*)'?.*

Without this modification to 'pattern2', the regular expression would match
the "User entered" event for *any* caller ID (that is probably not what you
want).

4) I have also set the "window" parameter to 3 hours (10800 seconds) -- if
for some reason you would never be seeing "User entered" event for the
given caller ID, the Pair operation will not exist forever but will rather
time out after 3 hours.

5) If there is a chance of not seeing the "User entered event", you can
also utilize the following Single rule after the Pair rule for terminating
the hanging Pair operation when the call is ended:

type=Single
ptype=RegExp
pattern=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*pbx\.c:
Spawn extension \(.*\) exited non-zero on.*
desc=End of call for caller $4
action=reset -1 IVR caller $4 offered activation or statement inquiry

Note that the first parameter of the 'reset' action (-1) indicates that an
event correlation operation started by the previous rule (i.e., the Pair
rule) has to be terminated. And the second parameter "IVR caller $4 offered
activation or statement inquiry" provides the ID of this operation (in
other words, whenever a call with some ID has ended, we will be terminating
the Pair operation for this call ID). Also, if this Pair operation does not
exist for the given call ID (maybe because the ActivationHelp event was
never seen for this particular call), the 'reset' action is a no-op.

However, if you are absolutely confident that the "User entered" event
always occurs after "ActivationHelp", you don't need the above Single rule.

Does this solution work for you? I hope I understood the problem
description correctly, and if there are any other questions or something
that needs improvement, feel free to ask.

kind regards,
risto


Risto,
>
> Absolutely.
>
> The following pattern should catch it:
>
> ^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*
> pbx.c: Spawn extension \(.*\) exited non-zero on.*
>
>
> Here's a few lines leading up to it (interesting because here is a Read()
> operation where the caller does not make a choice but instead hangs up
> while the system is waiting for a response):
>
> [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] pbx.c: Executing
> [8005551234@ivr-MainMenu:45] Read("SIP/CHANNELNAME-000138ac",
> "BalanceRepeatPressed,prompts/004h&prompts/004i&prompts/043,1,,,15") in new
> stack
> [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] app_read.c: Accepting a
> maximum of 1 digits.
> [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] file.c:
> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004h.slin' (language 'en')
> [2022-02-08-06:21:25:] VERBOSE[24858][C-00004037] file.c:
> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004i.slin' (language 'en')
> [2022-02-08-06:21:28:] VERBOSE[24858][C-00004037] file.c:
> <SIP/CHANNELNAME-000138ac> Playing 'prompts/043.slin' (language 'en')
> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] app_read.c: User
> disconnected
> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing
> [h@ivr-MainMenu:1] NoOp("SIP/CHANNELNAME-000138ac", "Hangup handler
> GlobalCallCount=0 CHANNELNAME=0 ") in new stack
> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing
> [h@ivr-MainMenu:2] NoOp("SIP/CHANNELNAME-000138ac", "SurveyType is ") in
> new stack
> [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing
> [h@ivr-MainMenu:3] ExecIf("SIP/CHANNELNAME-000138ac", "1?Hangup()") in
> new stack
> *[2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Spawn extension
> (ivr-MainMenu, h, 3) exited non-zero on 'SIP/CHANNELNAME-000138ac'*
>
>
>
> --
> Jim Van Meggelen
> ClearlyCore Inc.
>
>
>
> +1-416-639-6001 (DID)
> +1-877-253-2716 (Canada)
> +1-866-644-7729 (USA)
> +1-416-425-6111 x6001
> jim...@cl...
> http://www.clearlycore.com
>
>
> *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:*
> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf
>
>