simple-evcorr-users — Simple Event Correlator users

Re: [Simple-evcorr-users] sec question

[Risto V. <ris...@gm...>]

hi Rajeev,

the scheme you have proposed (taking timestamps from an input file and
using them to adjust an artificial clock) does not work with SEC as it
has been explained in the documentation and this tutorial:
https://simple-evcorr.github.io/SEC-tutorial.pdf. Implementing an
artificial clock is not easy and involves a number of advanced issues.
For example, the internal event correlation state is not only changing
on the appearance of new input events, but also on the expiration of
various timers (created and adjusted dynamically at runtime, and thus
difficult to track). To make things worse, timestamps are not always
increasing in log files (e.g., because of multi-threading of logging
applications, but also issues like unsynchronized system clocks,
etc.). As a result, artificial time will not always move forward, but
can also move backwards. Therefore, to implement such an artificial
clock would probably require a separate project, since it is less
connected to real-time event correlation.

Also, can you post further questions to the mailing list as instructed
in the SEC home page? (the list can be found here:
https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/)

kind regards,
risto


Kontakt Rajeev Raut (R) (<Ra...@op...>) kirjutas kuupäeval
R, 24. jaanuar 2025 kell 16:15:
>
> Hi Risto,
>
>
>
> I am typical support guy .. and have written few Domain Specific perl based regex’s to get my job done.. in near real time (automations).
>
>
>
> I find it interesting that your tool sec might come handy in my efforts to tackle further challenges w/o coding too much going forward.
>
>
>
> My problem is:
>
>
>
> I setup data in a file where say I keep some date/time part as data (that can act like actual-time of the events) and the rest of the line.. as dummy events to be handled by sec. I want sec to take this datee/time as if that is the wall clock ? is it possible ? reason to do is – this way we can test far more scenarios w/o really waiting for actual wall-clock. This will speedup doing test-cases.
>
>
>
> Let me know.
>
>
>
> PS: Please keep related-work-groups (CBS...@te...) in cc-list for better response times, as we may get side-tracked due to other tasks.
>
> Stay safe.
>
>
>
> Regards
> Rajeev Raut
> Desk: +27 12 311 4086
> Cell: 081 569 3343 / 061 902 0808
> Meet your future today
>
>
>
> ===== This e-mail and its contents are subject to the Openserve (Pty) Ltd E-mail legal notice https://connect.openserve.co.za/assets/documents/LegalEmail/OpenserveEMailLegalNotice.pdf =====

[Simple-evcorr-users] SEC-2.9.3 released

[Risto V. <ris...@gm...>]

hi all,

SEC-2.9.3 has been released which is available from SEC home page and
through the following download link:
https://github.com/simple-evcorr/sec/releases/download/2.9.3/sec-2.9.3.tar.gz

The new version adds support for built-in action list variables
%.chr32, ..., %.chr255 that are set to ASCII 32..255 characters. These
variables are useful for various encoding tasks in action lists -- for
example, the sequence %.chr195%.chr132 represents the A-umlaut
character in UTF-8 encoding.

kind regards,
risto

[Simple-evcorr-users] SEC historical home page will close on June 30

[Risto V. <ris...@gm...>]

hi all,

during the early days of SEC, its home page was located at
http://kodu.neti.ee/~risto/sec. That site is a part of a web portal owned
by Telia Estonia (www.telia.ee), and Telia has decided to close that portal
on June 30.

Note that http://kodu.neti.ee/~risto/sec has not been actively used since
2009, and when accessing that site, you will be redirected to the current
SEC home page at Github: https://simple-evcorr.github.io. Although the old
site is not likely to be used by end users, there is a slight chance the
old link is still present in some web pages and documentation which were
originally created before 2009. That email serves as a notification to
update the SEC home page address if you have been using the old link.

Finally, we thank Telia Estonia for keeping the kodu.neti.ee portal up and
running as a free service for 25+ years :)

kind regards,
risto

Re: [Simple-evcorr-users] Problem with action2

[Tom D. <to...@lo...>]

Thanks for the detailed explanation. Appreciate your work Risto.

Tom Damon
LogZilla

From: Risto Vaarandi <ris...@gm...>
Date: Thursday, April 11, 2024 at 5:38 PM
To: Tom Damon <to...@lo...>
Cc: sim...@li... <sim...@li...>
Subject: Re: [Simple-evcorr-users] Problem with action2
hi Tom,

the PairWithWindow rule works as follows (see also the documentation of the PairWithWindow rule in https://simple-evcorr.github.io/man.html#lbAP):

step1) if the incoming event matches the pattern defined with the 'pattern' field, the rule either (a) starts a new event correlation operation if it does not exist yet, or (b) if the operation exists, the rule sends the event to the operation which consumes the event silently.
step2) if the incoming event does not match the pattern defined with the 'pattern' field, the event is processed by all event correlation operations started by the rule, and the operations try to match this event against their 'pattern2' patterns. If any of the patterns matches, corresponding 'action2' of the relevant operation is executed.

Given the scheme described above, if the 'pattern' field matches all events that 'pattern2' matches, all events are handled during step1 and no event will reach step2. You are seeing this behavior, since both patterns are identical in your rule definition. To fix that issue, you need to make the 'pattern' and 'pattern2' fields different enough, so that the first pattern would only match the specific event which should start the event correlation operation, whereas the second pattern would only match the event which should end the operation.

kind regards,
risto

Kontakt Tom Damon via Simple-evcorr-users (<sim...@li...<mailto:sim...@li...>>) kirjutas kuupäeval N, 11. aprill 2024 kell 23:34:
Hello list,
  I’m trying to get this rule working.  The action works, but action2 does not. What am I missing?

type=PairWithWindow
ptype=regexp
pattern=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc=(WARNING) $1 is $3 from $2
action=pipe 'sending' /etc/logzilla/scripts/sec.sh '%s'
ptype2=regexp
pattern2=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc2=(NOTICE) You seeing this means, we have seen a recovery event.
action2=pipe 'sending' /etc/logzilla/scripts/sec.sh 'recovered'
window=5

Thanks,
Tom Damon
LogZilla
_______________________________________________
Simple-evcorr-users mailing list
Sim...@li...<mailto:Sim...@li...>
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Re: [Simple-evcorr-users] Problem with action2

[Tom D. <to...@lo...>]

Ah, didn’t catch that. Thank you!

Tom Damon
LogZilla

From: Frazier, Jon <Jon...@gm...>
Date: Thursday, April 11, 2024 at 4:41 PM
To: Tom Damon <to...@lo...>, sim...@li... <sim...@li...>
Subject: RE: [Simple-evcorr-users] Problem with action2
Could it be because the two patterns are identical thus you don’t have a “recovered” string for the second to match on.

I haven’t dug into this in years so I may be mistaken.


Regards,
Jon Frazier

From: Tom Damon via Simple-evcorr-users <sim...@li...>
Sent: Thursday, April 11, 2024 12:00 PM
To: sim...@li...
Subject: [External] [Simple-evcorr-users] Problem with action2

CAUTION: This email originated from outside of GM Financial and may contain unsafe content.
Hello list,
  I’m trying to get this rule working.  The action works, but action2 does not. What am I missing?

type=PairWithWindow
ptype=regexp
pattern=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc=(WARNING) $1 is $3 from $2
action=pipe 'sending' /etc/logzilla/scripts/sec.sh '%s'
ptype2=regexp
pattern2=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc2=(NOTICE) You seeing this means, we have seen a recovery event.
action2=pipe 'sending' /etc/logzilla/scripts/sec.sh 'recovered'
window=5

Thanks,
Tom Damon
LogZilla

Re: [Simple-evcorr-users] Problem with action2

[Risto V. <ris...@gm...>]

hi Tom,

the PairWithWindow rule works as follows (see also the documentation of the
PairWithWindow rule in https://simple-evcorr.github.io/man.html#lbAP):

step1) if the incoming event matches the pattern defined with the 'pattern'
field, the rule either (a) starts a new event correlation operation if it
does not exist yet, or (b) if the operation exists, the rule sends the
event to the operation which consumes the event silently.
step2) if the incoming event does not match the pattern defined with the
'pattern' field, the event is processed by all event correlation operations
started by the rule, and the operations try to match this event against
their 'pattern2' patterns. If any of the patterns matches, corresponding
'action2' of the relevant operation is executed.

Given the scheme described above, if the 'pattern' field matches all events
that 'pattern2' matches, all events are handled during step1 and no event
will reach step2. You are seeing this behavior, since both patterns are
identical in your rule definition. To fix that issue, you need to make the
'pattern' and 'pattern2' fields different enough, so that the first pattern
would only match the specific event which should start the event
correlation operation, whereas the second pattern would only match the
event which should end the operation.

kind regards,
risto

Kontakt Tom Damon via Simple-evcorr-users (<
sim...@li...>) kirjutas kuupäeval N, 11.
aprill 2024 kell 23:34:

> Hello list,
>
>   I’m trying to get this rule working.  The action works, but action2 does
> not. What am I missing?
>
>
>
> type=PairWithWindow
>
> ptype=regexp
>
> pattern=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
>
> desc=(WARNING) $1 is $3 from $2
>
> action=pipe 'sending' /etc/logzilla/scripts/sec.sh '%s'
>
> ptype2=regexp
>
> pattern2=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
>
> desc2=(NOTICE) You seeing this means, we have seen a recovery event.
>
> action2=pipe 'sending' /etc/logzilla/scripts/sec.sh 'recovered'
>
> window=5
>
>
>
> Thanks,
>
> Tom Damon
>
> LogZilla
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>

Re: [Simple-evcorr-users] Problem with action2

[Frazier, J. <Jon...@gm...>]

No problem.


Regards,
Jon Frazier


From: Tom Damon <to...@lo...>
Sent: Thursday, April 11, 2024 4:00 PM
To: Frazier, Jon <Jon...@gm...>; sim...@li...
Subject: [External] [Simple-evcorr-users] Problem with action2

CAUTION: This email originated from outside of GM Financial and may contain unsafe content.
Ah, didn’t catch that. Thank you!

Tom Damon
LogZilla

From: Frazier, Jon <Jon...@gm...<mailto:Jon...@gm...>>
Date: Thursday, April 11, 2024 at 4:41 PM
To: Tom Damon <to...@lo...<mailto:to...@lo...>>, sim...@li...<mailto:sim...@li...> <sim...@li...<mailto:sim...@li...>>
Subject: RE: [Simple-evcorr-users] Problem with action2
Could it be because the two patterns are identical thus you don’t have a “recovered” string for the second to match on.

I haven’t dug into this in years so I may be mistaken.


Regards,
Jon Frazier

From: Tom Damon via Simple-evcorr-users <sim...@li...<mailto:sim...@li...>>
Sent: Thursday, April 11, 2024 12:00 PM
To: sim...@li...<mailto:sim...@li...>
Subject: [External] [Simple-evcorr-users] Problem with action2

CAUTION: This email originated from outside of GM Financial and may contain unsafe content.
Hello list,
  I’m trying to get this rule working.  The action works, but action2 does not. What am I missing?

type=PairWithWindow
ptype=regexp
pattern=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc=(WARNING) $1 is $3 from $2
action=pipe 'sending' /etc/logzilla/scripts/sec.sh '%s'
ptype2=regexp
pattern2=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc2=(NOTICE) You seeing this means, we have seen a recovery event.
action2=pipe 'sending' /etc/logzilla/scripts/sec.sh 'recovered'
window=5

Thanks,
Tom Damon
LogZilla

Re: [Simple-evcorr-users] Problem with action2

[Frazier, J. <Jon...@gm...>]

Could it be because the two patterns are identical thus you don't have a "recovered" string for the second to match on.

I haven't dug into this in years so I may be mistaken.


Regards,
Jon Frazier


From: Tom Damon via Simple-evcorr-users <sim...@li...>
Sent: Thursday, April 11, 2024 12:00 PM
To: sim...@li...
Subject: [External] [Simple-evcorr-users] Problem with action2

CAUTION: This email originated from outside of GM Financial and may contain unsafe content.
Hello list,
  I'm trying to get this rule working.  The action works, but action2 does not. What am I missing?

type=PairWithWindow
ptype=regexp
pattern=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc=(WARNING) $1 is $3 from $2
action=pipe 'sending' /etc/logzilla/scripts/sec.sh '%s'
ptype2=regexp
pattern2=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc2=(NOTICE) You seeing this means, we have seen a recovery event.
action2=pipe 'sending' /etc/logzilla/scripts/sec.sh 'recovered'
window=5

Thanks,
Tom Damon
LogZilla

[Simple-evcorr-users] Problem with action2

[Tom D. <to...@lo...>]

Hello list,
  I’m trying to get this rule working.  The action works, but action2 does not. What am I missing?

type=PairWithWindow
ptype=regexp
pattern=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc=(WARNING) $1 is $3 from $2
action=pipe 'sending' /etc/logzilla/scripts/sec.sh '%s'
ptype2=regexp
pattern2=host.(\S+)\s+subtype=\S+\smessage=.*User-ID-Agent\s+(\S+)\s(\S+):
desc2=(NOTICE) You seeing this means, we have seen a recovery event.
action2=pipe 'sending' /etc/logzilla/scripts/sec.sh 'recovered'
window=5

Thanks,
Tom Damon
LogZilla

[Simple-evcorr-users] SEC tutorial has been updated

[Risto V. <ris...@gm...>]

hi all,

SEC tutorial has been updated and you can access the new version at
https://simple-evcorr.github.io/SEC-tutorial.pdf.

Many thanks to Jim Van Meggelen for suggestions on how to improve the
tutorial!

Happy Boxing Day to all SEC users :)

risto

Re: [Simple-evcorr-users] Storing a sequence counter in a context

[Jim V. M. <jim...@cl...>]

Risto, 

That's important advice from a performance perspective, so thank you for that. I would have missed it and suffered as a result. 

If we decide to run SEC like a daemon, as the data is generated (which we might do), then all that stale data would have definitely caused us problems (the callid DOES get re-used, as you surmised). I'm sure I can figure out how to put a handler in to drop everything relating to a call once it's done. I'll do some reading (I always learn new things as I re-read the docs, so that's time well spent). 

I like the option to use Perl to handle the counter; I'll see how the performance does and decide based on that. 

Thanks as always for the help. 

Jim 

-- 
Jim Van Meggelen 
ClearlyCore Inc. 

+1-416-639-6001 (DID) 
+1-877-253-2716 (Canada) 
+1-866-644-7729 (USA) 
+1-416-425-6111 x6001 
jim...@cl... 
[ http://www.clearlycore.com/ | http://www.clearlycore.com ] 

Asterisk: The Definitive Guide 
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: 
[ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] 

> From: "Risto Vaarandi" <ris...@gm...>
> To: "Jim Van Meggelen" <jim...@cl...>
> Cc: "simple-evcorr-users" <sim...@li...>
> Sent: Sunday, 24 September, 2023 04:35:50
> Subject: Re: [Simple-evcorr-users] Storing a sequence counter in a context

> hi Jim,
> many thanks for the clarification! I have a better understanding now about the
> nature of the problem.
> Because you are doing offline processing of an already existing log file,
> removing old contexts or old hash table keys (recommendation from the end of my
> post) might not be so important, provided that call IDs are not reused for
> different calls in the processed log file, and the number of call IDs is not
> very large. That would keep the solution a bit more simple.
> kind regards,
> risto

> Kontakt Jim Van Meggelen (< [ mailto:jim...@cl... |
> jim...@cl... ] >) kirjutas kuupäeval L, 23. september 2023
> kell 22:11:

>> Risto,

>> I cannot change the timestamps, as they are coded in the files I'm working with
>> (old data). Going forward, I also cannot guarantee that I will have control
>> over the source data, and it is likely that it will have a similar granularity.
>> Finally, there's no need for that level of granularity downstream; it is only
>> necessary that two events with the same timestamp are sequenced so that it is
>> clear which happened first.

>> It does seem a bit kludgy, but if I try to reduce the kludge in my SEC file, I
>> am almost certain to have to introduce kludge somewhere else, which is worse
>> (this way I only have SEC to maintain, rather than additional technology).

>> This way, I can feed relevant log files to SEC from any source, and trust that
>> I'll be able to handle them, so long as they follow standard Asterisk log
>> format.

>> Thanks again for your generous advice.

>> Jim

>> --
>> Jim Van Meggelen
>> ClearlyCore Inc.

>> +1-416-639-6001 (DID)
>> +1-877-253-2716 (Canada)
>> +1-866-644-7729 (USA)
>> +1-416-425-6111 x6001
>> [ mailto:jim...@cl... | jim...@cl... ]
>> [ http://www.clearlycore.com/ | http://www.clearlycore.com ]

>> Asterisk: The Definitive Guide
>> FIFTH EDITION NOW AVAILABLE TO DOWNLOAD:
>> [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf |
>> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ]

>>> From: "Risto Vaarandi" < [ mailto:ris...@gm... |
>>> ris...@gm... ] >
>>> To: "Jim Van Meggelen" < [ mailto:jim...@cl... |
>>> jim...@cl... ] >
>>> Cc: "simple-evcorr-users" < [ mailto:sim...@li... |
>>> sim...@li... ] >
>>> Sent: Saturday, 23 September, 2023 03:51:02
>>> Subject: Re: [Simple-evcorr-users] Storing a sequence counter in a context

>>> hi Jim,
>>> the solutions from my previous post were provided without knowing the root cause
>>> of the issue. If the main reason for the problem is the nature of the
>>> timestamps in the logs (that is, they are provided with an accuracy of a
>>> second), I would recommend using a different logging scheme with
>>> high-resolution timestamps. For example, as David has mentioned in his post,
>>> using syslog servers with RFC5424-based syslog protocol ( [
>>> https://datatracker.ietf.org/doc/html/rfc5424 |
>>> https://datatracker.ietf.org/doc/html/rfc5424 ] ) provides such timestamps
>>> out-of-the-box. I am not sure whether you can change the logging scheme for
>>> these particular application messages, but if you can, these timestamps are
>>> highly useful for many purposes like the investigation of past system faults
>>> and security incidents.
>>> kind regards,
>>> risto

>>>> hi Jim,

>>>> let me provide some suggestions how to accomplish this task. First, you could
>>>> utilize the context based approach that you have described in your post. In
>>>> that case, the numeral that has been retrieved from the context with the 'pop'
>>>> action needs to be incremented, and you can do it with the 'lcall' action that
>>>> invokes the relevant Perl code. Here is one example (I have added several of
>>>> extra actions into the rule for initializing the context):

>>>> type=single
>>>> ptype=regexp
>>>> pattern=call_id (\S+)
>>>> desc=call id $1
>>>> action=exists %e call_uniqueid_seq_$1; \
>>>> if %e ( none ) else ( add call_uniqueid_seq_$1 0); \
>>>> pop call_uniqueid_seq_$1 %sequence_num; \
>>>> lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \
>>>> add call_uniqueid_seq_$1 %sequence_num

>>>> The first two actions check if the context call_uniqueid_seq_* for the given
>>>> call exists, and if not, 0 is written into that context to initialize it:

>>>> exists %e call_uniqueid_seq_$1; \
>>>> if %e ( none ) else ( add call_uniqueid_seq_$1 0);

>>>> The 'pop' and 'add' actions (the 3rd and 5th action in the above rule) are
>>>> identical to the actions from your email.

>>>> The 'lcall' action (4th action in the above rule) increments the counter value
>>>> retrieved from the context by executing the following Perl one-line function:
>>>> sub { $_[0] + 1 }
>>>> That function takes the value of its parameter (that is, %sequence_num) and
>>>> returns an incremented value which is stored back into %sequence_num. Note that
>>>> %sequence_num appears twice in the 'lcall' action, since it acts both as an
>>>> input parameter for the Perl function and as the variable where the return
>>>> value from the function is stored.

>>>> There is also an alternative way for addressing the same assignment. This
>>>> approach involves keeping the counter for each call ID not in a SEC context,
>>>> but rather in a Perl hash table. That approach is more efficient, since the
>>>> counter value is not copied back and forth between the Perl function and the
>>>> SEC context each time it needs to be incremented. Instead, the counter value
>>>> stays in a Perl data structure during these increments. Here is a relevant rule
>>>> example:

>>>> type=single
>>>> ptype=regexp
>>>> pattern=call_id (\S+)
>>>> desc=call id $1
>>>> action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \
>>>> write - the new value for counter for call $1 is %sequence_num

>>>> The above example involves the use of a Perl hash (associative array) 'calls',
>>>> where the call ID serves as a key into the hash. For each key (call ID), the
>>>> hash stores the counter for that call ID. Each time the function
>>>> sub { ++$calls{$_[0]} }
>>>> gets called, the counter for the relevant call ID (the value of $1) is
>>>> incremented in the hash 'calls', and the result is stored to %sequence_num.

>>>> If you just want to retrieve the current value of the counter for some call ID
>>>> (the value of $1) without incrementing it, you can simply invoke the following
>>>> action:

>>>> lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } )

>>>> With both the context-based and Perl-hash-based approach, one needs to think
>>>> about dropping stale contexts (or stale hash keys) for calls that are no longer
>>>> relevant. With contexts, you can remove them with the 'delete' action. In the
>>>> case you use the Perl hash for storing the counters, you can use the following
>>>> 'lcall' action for deleting a key from the hash 'calls':

>>>> lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } )

>>>> I hope the above examples are useful.

>>>> kind regards,
>>>> risto

>>>> Kontakt Jim Van Meggelen (< [ mailto:jim...@cl... |
>>>> jim...@cl... ] >) kirjutas kuupäeval R, 22. september 2023
>>>> kell 19:50:

>>>>> I am using SEC for an atypical use case, where rather than using log events to
>>>>> correlate events for things like security events, I am using it to parse
>>>>> through Asterisk logs in order to capture a sequence of phone call events (call
>>>>> comes in, auto attendant answers, user selects a digit, call is transferred,
>>>>> etc). I have SEC write each to a CSV file, which can then be processed
>>>>> downstream for reporting (for example, I produced a Sankey chart using this
>>>>> data).

>>>>> SEC has proven fantastic for this, however one minor issue has been that the
>>>>> timestamps in the logs are not granular to smaller than a second, so it's
>>>>> possible for two or more events to occur within the same second. Generally this
>>>>> doesn't cause a problem, but when sorting the CSV file elsewhere, this can
>>>>> result in out-of-sequence lines, if they both contain the exact same timestamp.

>>>>> So, what I've been trying to figure out is how to store a counter that is tied
>>>>> to the uniqueid of the call, and increment that with each event. I figured I'd
>>>>> be able to do this by storing an integer in the event store of a context (tied
>>>>> to the uniqueid for that call). I can then increment it as the various log
>>>>> lines of the call are processed in turn.

Re: [Simple-evcorr-users] Storing a sequence counter in a context

[Risto V. <ris...@gm...>]

hi Jim,
many thanks for the clarification! I have a better understanding now about
the nature of the problem.
Because you are doing offline processing of an already existing log file,
removing old contexts or old hash table keys (recommendation from the end
of my post) might not be so important, provided that call IDs are not
reused for different calls in the processed log file, and the number of
call IDs is not very large. That would keep the solution a bit more simple.
kind regards,
risto

Kontakt Jim Van Meggelen (<jim...@cl...>) kirjutas
kuupäeval L, 23. september 2023 kell 22:11:

> Risto,
>
> I cannot change the timestamps, as they are coded in the files I'm working
> with (old data). Going forward, I also cannot guarantee that I will have
> control over the source data, and it is likely that it will have a similar
> granularity. Finally, there's no need for that level of granularity
> downstream; it is only necessary that two events with the same timestamp
> are sequenced so that it is clear which happened first.
>
> It does seem a bit kludgy, but if I try to reduce the kludge in my SEC
> file, I am almost certain to have to introduce kludge somewhere else, which
> is worse (this way I only have SEC to maintain, rather than additional
> technology).
>
> This way, I can feed relevant log files to SEC from any source, and trust
> that I'll be able to handle them, so long as they follow standard Asterisk
> log format.
>
> Thanks again for your generous advice.
>
> Jim
>
>
>
> --
> Jim Van Meggelen
> ClearlyCore Inc.
>
>
>
> +1-416-639-6001 (DID)
> +1-877-253-2716 (Canada)
> +1-866-644-7729 (USA)
> +1-416-425-6111 x6001
> jim...@cl...
> http://www.clearlycore.com
>
>
> *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:*
> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf
>
> ------------------------------
>
> *From: *"Risto Vaarandi" <ris...@gm...>
> *To: *"Jim Van Meggelen" <jim...@cl...>
> *Cc: *"simple-evcorr-users" <sim...@li...>
> *Sent: *Saturday, 23 September, 2023 03:51:02
> *Subject: *Re: [Simple-evcorr-users] Storing a sequence counter in a
> context
>
> hi Jim,
> the solutions from my previous post were provided without knowing the root
> cause of the issue. If the main reason for the problem is the nature of the
> timestamps in the logs (that is, they are provided with an accuracy of a
> second), I would recommend using a different logging scheme with
> high-resolution timestamps. For example, as David has mentioned in his
> post, using syslog servers with RFC5424-based syslog protocol (
> https://datatracker.ietf.org/doc/html/rfc5424) provides such timestamps
> out-of-the-box. I am not sure whether you can change the logging scheme for
> these particular application messages, but if you can, these timestamps are
> highly useful for many purposes like the investigation of past system
> faults and security incidents.
> kind regards,
> risto
>
>
> hi Jim,
>>
>> let me provide some suggestions how to accomplish this task. First, you
>> could utilize the context based approach that you have described in your
>> post. In that case, the numeral that has been retrieved from the context
>> with the 'pop' action needs to be incremented, and you can do it with the
>> 'lcall' action that invokes the relevant Perl code. Here is one example (I
>> have added several of extra actions into the rule for initializing the
>> context):
>>
>> type=single
>> ptype=regexp
>> pattern=call_id (\S+)
>> desc=call id $1
>> action=exists %e call_uniqueid_seq_$1; \
>>        if %e ( none ) else ( add call_uniqueid_seq_$1 0); \
>>        pop call_uniqueid_seq_$1 %sequence_num; \
>>        lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \
>>        add call_uniqueid_seq_$1 %sequence_num
>>
>> The first two actions check if the context call_uniqueid_seq_* for the
>> given call exists, and if not, 0 is written into that context to initialize
>> it:
>>
>> exists %e call_uniqueid_seq_$1; \
>> if %e ( none ) else ( add call_uniqueid_seq_$1 0);
>>
>> The 'pop' and 'add' actions (the 3rd and 5th action in the above rule)
>> are identical to the actions from your email.
>>
>> The 'lcall' action (4th action in the above rule) increments the counter
>> value retrieved from the context by executing the following Perl one-line
>> function:
>> sub { $_[0] + 1 }
>> That function takes the value of its parameter (that is, %sequence_num)
>> and returns an incremented value which is stored back into %sequence_num.
>> Note that %sequence_num appears twice in the 'lcall' action, since it acts
>> both as an input parameter for the Perl function and as the variable where
>> the return value from the function is stored.
>>
>> There is also an alternative way for addressing the same assignment. This
>> approach involves keeping the counter for each call ID not in a SEC
>> context, but rather in a Perl hash table. That approach is more efficient,
>> since the counter value is not copied back and forth between the Perl
>> function and the SEC context each time it needs to be incremented. Instead,
>> the counter value stays in a Perl data structure during these increments.
>> Here is a relevant rule example:
>>
>> type=single
>> ptype=regexp
>> pattern=call_id (\S+)
>> desc=call id $1
>> action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \
>>        write - the new value for counter for call $1 is %sequence_num
>>
>> The above example involves the use of a Perl hash (associative array)
>> 'calls', where the call ID serves as a key into the hash. For each key
>> (call ID), the hash stores the counter for that call ID. Each time the
>> function
>> sub { ++$calls{$_[0]} }
>> gets called, the counter for the relevant call ID (the value of $1) is
>> incremented in the hash 'calls', and the result is stored to %sequence_num.
>>
>> If you just want to retrieve the current value of the counter for some
>> call ID (the value of $1) without incrementing it, you can simply invoke
>> the following action:
>>
>> lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } )
>>
>> With both the context-based and Perl-hash-based approach, one needs to
>> think about dropping stale contexts (or stale hash keys) for calls that are
>> no longer relevant. With contexts, you can remove them with the 'delete'
>> action. In the case you use the Perl hash for storing the counters, you can
>> use the following 'lcall' action for deleting a key from the hash 'calls':
>>
>> lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } )
>>
>> I hope the above examples are useful.
>>
>> kind regards,
>> risto
>>
>>
>>
>>
>>
>>
>>
>>
>> Kontakt Jim Van Meggelen (<jim...@cl...>) kirjutas
>> kuupäeval R, 22. september 2023 kell 19:50:
>>
>>> I am using SEC for an atypical use case, where rather than using log
>>> events to correlate events for things like security events, I am using it
>>> to parse through Asterisk logs in order to capture a sequence of phone call
>>> events (call comes in, auto attendant answers, user selects a digit, call
>>> is transferred, etc). I have SEC write each to a CSV file, which can then
>>> be processed downstream for reporting (for example, I produced a Sankey
>>> chart using this data).
>>>
>>> SEC has proven fantastic for this, however one minor issue has been that
>>> the timestamps in the logs are not granular to smaller than a second, so
>>> it's possible for two or more events to occur within the same second.
>>> Generally this doesn't cause a problem, but when sorting the CSV file
>>> elsewhere, this can result in out-of-sequence lines, if they both contain
>>> the exact same timestamp.
>>>
>>> So, what I've been trying to figure out is how to store a counter that
>>> is tied to the uniqueid of the call, and increment that with each event. I
>>> figured I'd be able to do this by storing an integer in the event store of
>>> a context (tied to the uniqueid for that call). I can then increment it as
>>> the various log lines of the call are processed in turn.
>>>
>>>
>>>
>

Re: [Simple-evcorr-users] Storing a sequence counter in a context

[Jim V. M. <jim...@cl...>]

Risto, 

I cannot change the timestamps, as they are coded in the files I'm working with (old data). Going forward, I also cannot guarantee that I will have control over the source data, and it is likely that it will have a similar granularity. Finally, there's no need for that level of granularity downstream; it is only necessary that two events with the same timestamp are sequenced so that it is clear which happened first. 

It does seem a bit kludgy, but if I try to reduce the kludge in my SEC file, I am almost certain to have to introduce kludge somewhere else, which is worse (this way I only have SEC to maintain, rather than additional technology). 

This way, I can feed relevant log files to SEC from any source, and trust that I'll be able to handle them, so long as they follow standard Asterisk log format. 

Thanks again for your generous advice. 

Jim 

-- 
Jim Van Meggelen 
ClearlyCore Inc. 

+1-416-639-6001 (DID) 
+1-877-253-2716 (Canada) 
+1-866-644-7729 (USA) 
+1-416-425-6111 x6001 
jim...@cl... 
[ http://www.clearlycore.com/ | http://www.clearlycore.com ] 

Asterisk: The Definitive Guide 
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: 
[ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] 

> From: "Risto Vaarandi" <ris...@gm...>
> To: "Jim Van Meggelen" <jim...@cl...>
> Cc: "simple-evcorr-users" <sim...@li...>
> Sent: Saturday, 23 September, 2023 03:51:02
> Subject: Re: [Simple-evcorr-users] Storing a sequence counter in a context

> hi Jim,
> the solutions from my previous post were provided without knowing the root cause
> of the issue. If the main reason for the problem is the nature of the
> timestamps in the logs (that is, they are provided with an accuracy of a
> second), I would recommend using a different logging scheme with
> high-resolution timestamps. For example, as David has mentioned in his post,
> using syslog servers with RFC5424-based syslog protocol ( [
> https://datatracker.ietf.org/doc/html/rfc5424 |
> https://datatracker.ietf.org/doc/html/rfc5424 ] ) provides such timestamps
> out-of-the-box. I am not sure whether you can change the logging scheme for
> these particular application messages, but if you can, these timestamps are
> highly useful for many purposes like the investigation of past system faults
> and security incidents.
> kind regards,
> risto

>> hi Jim,

>> let me provide some suggestions how to accomplish this task. First, you could
>> utilize the context based approach that you have described in your post. In
>> that case, the numeral that has been retrieved from the context with the 'pop'
>> action needs to be incremented, and you can do it with the 'lcall' action that
>> invokes the relevant Perl code. Here is one example (I have added several of
>> extra actions into the rule for initializing the context):

>> type=single
>> ptype=regexp
>> pattern=call_id (\S+)
>> desc=call id $1
>> action=exists %e call_uniqueid_seq_$1; \
>> if %e ( none ) else ( add call_uniqueid_seq_$1 0); \
>> pop call_uniqueid_seq_$1 %sequence_num; \
>> lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \
>> add call_uniqueid_seq_$1 %sequence_num

>> The first two actions check if the context call_uniqueid_seq_* for the given
>> call exists, and if not, 0 is written into that context to initialize it:

>> exists %e call_uniqueid_seq_$1; \
>> if %e ( none ) else ( add call_uniqueid_seq_$1 0);

>> The 'pop' and 'add' actions (the 3rd and 5th action in the above rule) are
>> identical to the actions from your email.

>> The 'lcall' action (4th action in the above rule) increments the counter value
>> retrieved from the context by executing the following Perl one-line function:
>> sub { $_[0] + 1 }
>> That function takes the value of its parameter (that is, %sequence_num) and
>> returns an incremented value which is stored back into %sequence_num. Note that
>> %sequence_num appears twice in the 'lcall' action, since it acts both as an
>> input parameter for the Perl function and as the variable where the return
>> value from the function is stored.

>> There is also an alternative way for addressing the same assignment. This
>> approach involves keeping the counter for each call ID not in a SEC context,
>> but rather in a Perl hash table. That approach is more efficient, since the
>> counter value is not copied back and forth between the Perl function and the
>> SEC context each time it needs to be incremented. Instead, the counter value
>> stays in a Perl data structure during these increments. Here is a relevant rule
>> example:

>> type=single
>> ptype=regexp
>> pattern=call_id (\S+)
>> desc=call id $1
>> action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \
>> write - the new value for counter for call $1 is %sequence_num

>> The above example involves the use of a Perl hash (associative array) 'calls',
>> where the call ID serves as a key into the hash. For each key (call ID), the
>> hash stores the counter for that call ID. Each time the function
>> sub { ++$calls{$_[0]} }
>> gets called, the counter for the relevant call ID (the value of $1) is
>> incremented in the hash 'calls', and the result is stored to %sequence_num.

>> If you just want to retrieve the current value of the counter for some call ID
>> (the value of $1) without incrementing it, you can simply invoke the following
>> action:

>> lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } )

>> With both the context-based and Perl-hash-based approach, one needs to think
>> about dropping stale contexts (or stale hash keys) for calls that are no longer
>> relevant. With contexts, you can remove them with the 'delete' action. In the
>> case you use the Perl hash for storing the counters, you can use the following
>> 'lcall' action for deleting a key from the hash 'calls':

>> lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } )

>> I hope the above examples are useful.

>> kind regards,
>> risto

>> Kontakt Jim Van Meggelen (< [ mailto:jim...@cl... |
>> jim...@cl... ] >) kirjutas kuupäeval R, 22. september 2023
>> kell 19:50:

>>> I am using SEC for an atypical use case, where rather than using log events to
>>> correlate events for things like security events, I am using it to parse
>>> through Asterisk logs in order to capture a sequence of phone call events (call
>>> comes in, auto attendant answers, user selects a digit, call is transferred,
>>> etc). I have SEC write each to a CSV file, which can then be processed
>>> downstream for reporting (for example, I produced a Sankey chart using this
>>> data).

>>> SEC has proven fantastic for this, however one minor issue has been that the
>>> timestamps in the logs are not granular to smaller than a second, so it's
>>> possible for two or more events to occur within the same second. Generally this
>>> doesn't cause a problem, but when sorting the CSV file elsewhere, this can
>>> result in out-of-sequence lines, if they both contain the exact same timestamp.

>>> So, what I've been trying to figure out is how to store a counter that is tied
>>> to the uniqueid of the call, and increment that with each event. I figured I'd
>>> be able to do this by storing an integer in the event store of a context (tied
>>> to the uniqueid for that call). I can then increment it as the various log
>>> lines of the call are processed in turn.

Re: [Simple-evcorr-users] Storing a sequence counter in a context

[Jim V. M. <jim...@cl...>]

Risto, 

Thank you for this. It makes sense, and you've taken the trouble to explain all the nuances that I would have otherwise missed. Wow. 

I'm going to work through this on Monday. 

Jim 

-- 
Jim Van Meggelen 
ClearlyCore Inc. 

+1-416-639-6001 (DID) 
+1-877-253-2716 (Canada) 
+1-866-644-7729 (USA) 
+1-416-425-6111 x6001 
jim...@cl... 
[ http://www.clearlycore.com/ | http://www.clearlycore.com ] 

Asterisk: The Definitive Guide 
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: 
[ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] 

> From: "Risto Vaarandi" <ris...@gm...>
> To: "Jim Van Meggelen" <jim...@cl...>
> Cc: "simple-evcorr-users" <sim...@li...>
> Sent: Friday, 22 September, 2023 15:42:28
> Subject: Re: [Simple-evcorr-users] Storing a sequence counter in a context

> hi Jim,

> let me provide some suggestions how to accomplish this task. First, you could
> utilize the context based approach that you have described in your post. In
> that case, the numeral that has been retrieved from the context with the 'pop'
> action needs to be incremented, and you can do it with the 'lcall' action that
> invokes the relevant Perl code. Here is one example (I have added several of
> extra actions into the rule for initializing the context):

> type=single
> ptype=regexp
> pattern=call_id (\S+)
> desc=call id $1
> action=exists %e call_uniqueid_seq_$1; \
> if %e ( none ) else ( add call_uniqueid_seq_$1 0); \
> pop call_uniqueid_seq_$1 %sequence_num; \
> lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \
> add call_uniqueid_seq_$1 %sequence_num

> The first two actions check if the context call_uniqueid_seq_* for the given
> call exists, and if not, 0 is written into that context to initialize it:

> exists %e call_uniqueid_seq_$1; \
> if %e ( none ) else ( add call_uniqueid_seq_$1 0);

> The 'pop' and 'add' actions (the 3rd and 5th action in the above rule) are
> identical to the actions from your email.

> The 'lcall' action (4th action in the above rule) increments the counter value
> retrieved from the context by executing the following Perl one-line function:
> sub { $_[0] + 1 }
> That function takes the value of its parameter (that is, %sequence_num) and
> returns an incremented value which is stored back into %sequence_num. Note that
> %sequence_num appears twice in the 'lcall' action, since it acts both as an
> input parameter for the Perl function and as the variable where the return
> value from the function is stored.

> There is also an alternative way for addressing the same assignment. This
> approach involves keeping the counter for each call ID not in a SEC context,
> but rather in a Perl hash table. That approach is more efficient, since the
> counter value is not copied back and forth between the Perl function and the
> SEC context each time it needs to be incremented. Instead, the counter value
> stays in a Perl data structure during these increments. Here is a relevant rule
> example:

> type=single
> ptype=regexp
> pattern=call_id (\S+)
> desc=call id $1
> action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \
> write - the new value for counter for call $1 is %sequence_num

> The above example involves the use of a Perl hash (associative array) 'calls',
> where the call ID serves as a key into the hash. For each key (call ID), the
> hash stores the counter for that call ID. Each time the function
> sub { ++$calls{$_[0]} }
> gets called, the counter for the relevant call ID (the value of $1) is
> incremented in the hash 'calls', and the result is stored to %sequence_num.

> If you just want to retrieve the current value of the counter for some call ID
> (the value of $1) without incrementing it, you can simply invoke the following
> action:

> lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } )

> With both the context-based and Perl-hash-based approach, one needs to think
> about dropping stale contexts (or stale hash keys) for calls that are no longer
> relevant. With contexts, you can remove them with the 'delete' action. In the
> case you use the Perl hash for storing the counters, you can use the following
> 'lcall' action for deleting a key from the hash 'calls':

> lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } )

> I hope the above examples are useful.

> kind regards,
> risto

> Kontakt Jim Van Meggelen (< [ mailto:jim...@cl... |
> jim...@cl... ] >) kirjutas kuupäeval R, 22. september 2023
> kell 19:50:

>> I am using SEC for an atypical use case, where rather than using log events to
>> correlate events for things like security events, I am using it to parse
>> through Asterisk logs in order to capture a sequence of phone call events (call
>> comes in, auto attendant answers, user selects a digit, call is transferred,
>> etc). I have SEC write each to a CSV file, which can then be processed
>> downstream for reporting (for example, I produced a Sankey chart using this
>> data).

>> SEC has proven fantastic for this, however one minor issue has been that the
>> timestamps in the logs are not granular to smaller than a second, so it's
>> possible for two or more events to occur within the same second. Generally this
>> doesn't cause a problem, but when sorting the CSV file elsewhere, this can
>> result in out-of-sequence lines, if they both contain the exact same timestamp.

>> So, what I've been trying to figure out is how to store a counter that is tied
>> to the uniqueid of the call, and increment that with each event. I figured I'd
>> be able to do this by storing an integer in the event store of a context (tied
>> to the uniqueid for that call). I can then increment it as the various log
>> lines of the call are processed in turn.

>> The part I think I'm not getting is due to my lack of understanding of Perl (and
>> specifically perl syntax).

>> The first rule can create the context:

>> add call_uniqueid_seq_$4 1 (where $4 is the unique ID for that call)

>> But then in subsequent rules I want to do something like this:

>> call_uniqueid_seq_$4 ++
>> However I don't expect that to be valid syntax (especially since that store is
>> for strings), so I have to figure out something else.

>> I was working my head around just 'pop'ing the value off the event store, and
>> then adding 1 to it, but I realized I have no clue how to do this.

>> So, I am hoping that I am just missing something obvious, but, failing that, I
>> guess I need to understand how I would do something like:

>> pop call_uniqueid_seq_$4 %sequence_num
>> add call_uniqueid_seq_$4 %sequence_num+1
>> I feel like I'm on the wrong track here, but I'm not finding the answers in the
>> man page, or tutorial, or anywhere else, and my knowledge of Perl is
>> insufficient to help make sense of this.

>> Any thoughts or advice or clues would be greatly appreciated.

>> --
>> Jim Van Meggelen
>> ClearlyCore Inc.

>> +1-416-639-6001 (DID)
>> +1-877-253-2716 (Canada)
>> +1-866-644-7729 (USA)
>> +1-416-425-6111 x6001
>> [ mailto:jim...@cl... | jim...@cl... ]
>> [ http://www.clearlycore.com/ | http://www.clearlycore.com ]

>> Asterisk: The Definitive Guide
>> FIFTH EDITION NOW AVAILABLE TO DOWNLOAD:
>> [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf |
>> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ]
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> [ mailto:Sim...@li... |
>> Sim...@li... ]
>> [ https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users |
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users ]

Re: [Simple-evcorr-users] Storing a sequence counter in a context

[Risto V. <ris...@gm...>]

hi Jim,
the solutions from my previous post were provided without knowing the root
cause of the issue. If the main reason for the problem is the nature of the
timestamps in the logs (that is, they are provided with an accuracy of a
second), I would recommend using a different logging scheme with
high-resolution timestamps. For example, as David has mentioned in his
post, using syslog servers with RFC5424-based syslog protocol (
https://datatracker.ietf.org/doc/html/rfc5424) provides such timestamps
out-of-the-box. I am not sure whether you can change the logging scheme for
these particular application messages, but if you can, these timestamps are
highly useful for many purposes like the investigation of past system
faults and security incidents.
kind regards,
risto


hi Jim,
>
> let me provide some suggestions how to accomplish this task. First, you
> could utilize the context based approach that you have described in your
> post. In that case, the numeral that has been retrieved from the context
> with the 'pop' action needs to be incremented, and you can do it with the
> 'lcall' action that invokes the relevant Perl code. Here is one example (I
> have added several of extra actions into the rule for initializing the
> context):
>
> type=single
> ptype=regexp
> pattern=call_id (\S+)
> desc=call id $1
> action=exists %e call_uniqueid_seq_$1; \
>        if %e ( none ) else ( add call_uniqueid_seq_$1 0); \
>        pop call_uniqueid_seq_$1 %sequence_num; \
>        lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \
>        add call_uniqueid_seq_$1 %sequence_num
>
> The first two actions check if the context call_uniqueid_seq_* for the
> given call exists, and if not, 0 is written into that context to initialize
> it:
>
> exists %e call_uniqueid_seq_$1; \
> if %e ( none ) else ( add call_uniqueid_seq_$1 0);
>
> The 'pop' and 'add' actions (the 3rd and 5th action in the above rule) are
> identical to the actions from your email.
>
> The 'lcall' action (4th action in the above rule) increments the counter
> value retrieved from the context by executing the following Perl one-line
> function:
> sub { $_[0] + 1 }
> That function takes the value of its parameter (that is, %sequence_num)
> and returns an incremented value which is stored back into %sequence_num.
> Note that %sequence_num appears twice in the 'lcall' action, since it acts
> both as an input parameter for the Perl function and as the variable where
> the return value from the function is stored.
>
> There is also an alternative way for addressing the same assignment. This
> approach involves keeping the counter for each call ID not in a SEC
> context, but rather in a Perl hash table. That approach is more efficient,
> since the counter value is not copied back and forth between the Perl
> function and the SEC context each time it needs to be incremented. Instead,
> the counter value stays in a Perl data structure during these increments.
> Here is a relevant rule example:
>
> type=single
> ptype=regexp
> pattern=call_id (\S+)
> desc=call id $1
> action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \
>        write - the new value for counter for call $1 is %sequence_num
>
> The above example involves the use of a Perl hash (associative array)
> 'calls', where the call ID serves as a key into the hash. For each key
> (call ID), the hash stores the counter for that call ID. Each time the
> function
> sub { ++$calls{$_[0]} }
> gets called, the counter for the relevant call ID (the value of $1) is
> incremented in the hash 'calls', and the result is stored to %sequence_num.
>
> If you just want to retrieve the current value of the counter for some
> call ID (the value of $1) without incrementing it, you can simply invoke
> the following action:
>
> lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } )
>
> With both the context-based and Perl-hash-based approach, one needs to
> think about dropping stale contexts (or stale hash keys) for calls that are
> no longer relevant. With contexts, you can remove them with the 'delete'
> action. In the case you use the Perl hash for storing the counters, you can
> use the following 'lcall' action for deleting a key from the hash 'calls':
>
> lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } )
>
> I hope the above examples are useful.
>
> kind regards,
> risto
>
>
>
>
>
>
>
>
> Kontakt Jim Van Meggelen (<jim...@cl...>) kirjutas
> kuupäeval R, 22. september 2023 kell 19:50:
>
>> I am using SEC for an atypical use case, where rather than using log
>> events to correlate events for things like security events, I am using it
>> to parse through Asterisk logs in order to capture a sequence of phone call
>> events (call comes in, auto attendant answers, user selects a digit, call
>> is transferred, etc). I have SEC write each to a CSV file, which can then
>> be processed downstream for reporting (for example, I produced a Sankey
>> chart using this data).
>>
>> SEC has proven fantastic for this, however one minor issue has been that
>> the timestamps in the logs are not granular to smaller than a second, so
>> it's possible for two or more events to occur within the same second.
>> Generally this doesn't cause a problem, but when sorting the CSV file
>> elsewhere, this can result in out-of-sequence lines, if they both contain
>> the exact same timestamp.
>>
>> So, what I've been trying to figure out is how to store a counter that is
>> tied to the uniqueid of the call, and increment that with each event. I
>> figured I'd be able to do this by storing an integer in the event store of
>> a context (tied to the uniqueid for that call). I can then increment it as
>> the various log lines of the call are processed in turn.
>>
>>
>>

Re: [Simple-evcorr-users] Storing a sequence counter in a context

[David L. <da...@la...>]

most syslog servers have the ability to log with subsecond accuracy, and 
RFC-5424 requires it in the spec.

David Lang

Re: [Simple-evcorr-users] Storing a sequence counter in a context

[Clayton D. <cd...@lo...>]

Hi Jim,
Interesting use case!
If it helps, LogZilla (I am the founder) tracks events with millisecond accuracy and has SEC built in. Using them together, you could just ask LZ for the timestamp.
Might be something worth checking out.



From: Jim Van Meggelen <jim...@cl...>
Date: Friday, September 22, 2023 at 12:50 PM
To: simple-evcorr-users <sim...@li...>
Subject: [Simple-evcorr-users] Storing a sequence counter in a context
I am using SEC for an atypical use case, where rather than using log events to correlate events for things like security events, I am using it to parse through Asterisk logs in order to capture a sequence of phone call events (call comes in, auto attendant answers, user selects a digit, call is transferred, etc). I have SEC write each to a CSV file, which can then be processed downstream for reporting (for example, I produced a Sankey chart using this data).

SEC has proven fantastic for this, however one minor issue has been that the timestamps in the logs are not granular to smaller than a second, so it's possible for two or more events to occur within the same second. Generally this doesn't cause a problem, but when sorting the CSV file elsewhere, this can result in out-of-sequence lines, if they both contain the exact same timestamp.

So, what I've been trying to figure out is how to store a counter that is tied to the uniqueid of the call, and increment that with each event. I figured I'd be able to do this by storing an integer in the event store of a context (tied to the uniqueid for that call). I can then increment it as the various log lines of the call are processed in turn.

The part I think I'm not getting is due to my lack of understanding of Perl (and specifically perl syntax).

The first rule can create the context:

add call_uniqueid_seq_$4 1 (where $4 is the unique ID for that call)

But then in subsequent rules I want to do something like this:

call_uniqueid_seq_$4 ++

However I don't expect that to be valid syntax (especially since that store is for strings), so I have to figure out something else.


I was working my head around just 'pop'ing the value off the event store, and then adding 1 to it, but I realized I have no clue how to do this.

So, I am hoping that I am just missing something obvious, but, failing that, I guess I need to understand how I would do something like:

pop call_uniqueid_seq_$4 %sequence_num
add call_uniqueid_seq_$4 %sequence_num+1

I feel like I'm on the wrong track here, but I'm not finding the answers in the man page, or tutorial, or anywhere else, and my knowledge of Perl is insufficient to help make sense of this.

Any thoughts or advice or clues would be greatly appreciated.


--
Jim Van Meggelen
ClearlyCore Inc.

[cid:e655b21c1463e37f7bba0bc97c53e4cb81753f44@zimbra]

+1-416-639-6001 (DID)
+1-877-253-2716 (Canada)
+1-866-644-7729 (USA)
+1-416-425-6111 x6001
jim...@cl...
http://www.clearlycore.com

Asterisk: The Definitive Guide
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD:
https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf

Re: [Simple-evcorr-users] Storing a sequence counter in a context

[Risto V. <ris...@gm...>]

hi Jim,

let me provide some suggestions how to accomplish this task. First, you
could utilize the context based approach that you have described in your
post. In that case, the numeral that has been retrieved from the context
with the 'pop' action needs to be incremented, and you can do it with the
'lcall' action that invokes the relevant Perl code. Here is one example (I
have added several of extra actions into the rule for initializing the
context):

type=single
ptype=regexp
pattern=call_id (\S+)
desc=call id $1
action=exists %e call_uniqueid_seq_$1; \
       if %e ( none ) else ( add call_uniqueid_seq_$1 0); \
       pop call_uniqueid_seq_$1 %sequence_num; \
       lcall %sequence_num %sequence_num -> ( sub { $_[0] + 1 } ); \
       add call_uniqueid_seq_$1 %sequence_num

The first two actions check if the context call_uniqueid_seq_* for the
given call exists, and if not, 0 is written into that context to initialize
it:

exists %e call_uniqueid_seq_$1; \
if %e ( none ) else ( add call_uniqueid_seq_$1 0);

The 'pop' and 'add' actions (the 3rd and 5th action in the above rule) are
identical to the actions from your email.

The 'lcall' action (4th action in the above rule) increments the counter
value retrieved from the context by executing the following Perl one-line
function:
sub { $_[0] + 1 }
That function takes the value of its parameter (that is, %sequence_num) and
returns an incremented value which is stored back into %sequence_num. Note
that %sequence_num appears twice in the 'lcall' action, since it acts both
as an input parameter for the Perl function and as the variable where the
return value from the function is stored.

There is also an alternative way for addressing the same assignment. This
approach involves keeping the counter for each call ID not in a SEC
context, but rather in a Perl hash table. That approach is more efficient,
since the counter value is not copied back and forth between the Perl
function and the SEC context each time it needs to be incremented. Instead,
the counter value stays in a Perl data structure during these increments.
Here is a relevant rule example:

type=single
ptype=regexp
pattern=call_id (\S+)
desc=call id $1
action=lcall %sequence_num $1 -> ( sub { ++$calls{$_[0]} } ); \
       write - the new value for counter for call $1 is %sequence_num

The above example involves the use of a Perl hash (associative array)
'calls', where the call ID serves as a key into the hash. For each key
(call ID), the hash stores the counter for that call ID. Each time the
function
sub { ++$calls{$_[0]} }
gets called, the counter for the relevant call ID (the value of $1) is
incremented in the hash 'calls', and the result is stored to %sequence_num.

If you just want to retrieve the current value of the counter for some call
ID (the value of $1) without incrementing it, you can simply invoke the
following action:

lcall %sequence_num $1 -> ( sub { $calls{$_[0]} } )

With both the context-based and Perl-hash-based approach, one needs to
think about dropping stale contexts (or stale hash keys) for calls that are
no longer relevant. With contexts, you can remove them with the 'delete'
action. In the case you use the Perl hash for storing the counters, you can
use the following 'lcall' action for deleting a key from the hash 'calls':

lcall %sequence_num $1 -> ( sub { delete $calls{$_[0]} } )

I hope the above examples are useful.

kind regards,
risto








Kontakt Jim Van Meggelen (<jim...@cl...>) kirjutas
kuupäeval R, 22. september 2023 kell 19:50:

> I am using SEC for an atypical use case, where rather than using log
> events to correlate events for things like security events, I am using it
> to parse through Asterisk logs in order to capture a sequence of phone call
> events (call comes in, auto attendant answers, user selects a digit, call
> is transferred, etc). I have SEC write each to a CSV file, which can then
> be processed downstream for reporting (for example, I produced a Sankey
> chart using this data).
>
> SEC has proven fantastic for this, however one minor issue has been that
> the timestamps in the logs are not granular to smaller than a second, so
> it's possible for two or more events to occur within the same second.
> Generally this doesn't cause a problem, but when sorting the CSV file
> elsewhere, this can result in out-of-sequence lines, if they both contain
> the exact same timestamp.
>
> So, what I've been trying to figure out is how to store a counter that is
> tied to the uniqueid of the call, and increment that with each event. I
> figured I'd be able to do this by storing an integer in the event store of
> a context (tied to the uniqueid for that call). I can then increment it as
> the various log lines of the call are processed in turn.
>
> The part I think I'm not getting is due to my lack of understanding of
> Perl (and specifically perl syntax).
>
> The first rule can create the context:
>
> add call_uniqueid_seq_$4 1 (where $4 is the unique ID for that call)
>
> But then in subsequent rules I want to do something like this:
>
> call_uniqueid_seq_$4 ++
>
> However I don't expect that to be valid syntax (especially since that
> store is for strings), so I have to figure out something else.
>
> I was working my head around just 'pop'ing the value off the event store,
> and then adding 1 to it, but I realized I have no clue how to do this.
>
> So, I am hoping that I am just missing something obvious, but, failing
> that, I guess I need to understand how I would do something like:
>
> pop call_uniqueid_seq_$4 %sequence_num
> add call_uniqueid_seq_$4 %sequence_num+1
>
> I feel like I'm on the wrong track here, but I'm not finding the answers
> in the man page, or tutorial, or anywhere else, and my knowledge of Perl is
> insufficient to help make sense of this.
>
> Any thoughts or advice or clues would be greatly appreciated.
>
>
> --
> Jim Van Meggelen
> ClearlyCore Inc.
>
>
>
> +1-416-639-6001 (DID)
> +1-877-253-2716 (Canada)
> +1-866-644-7729 (USA)
> +1-416-425-6111 x6001
> jim...@cl...
> http://www.clearlycore.com
>
>
> *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:*
> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf
> _______________________________________________
> Simple-evcorr-users mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>

[Simple-evcorr-users] Storing a sequence counter in a context

[Jim V. M. <jim...@cl...>]

I am using SEC for an atypical use case, where rather than using log events to correlate events for things like security events, I am using it to parse through Asterisk logs in order to capture a sequence of phone call events (call comes in, auto attendant answers, user selects a digit, call is transferred, etc). I have SEC write each to a CSV file, which can then be processed downstream for reporting (for example, I produced a Sankey chart using this data). 

SEC has proven fantastic for this, however one minor issue has been that the timestamps in the logs are not granular to smaller than a second, so it's possible for two or more events to occur within the same second. Generally this doesn't cause a problem, but when sorting the CSV file elsewhere, this can result in out-of-sequence lines, if they both contain the exact same timestamp. 

So, what I've been trying to figure out is how to store a counter that is tied to the uniqueid of the call, and increment that with each event. I figured I'd be able to do this by storing an integer in the event store of a context (tied to the uniqueid for that call). I can then increment it as the various log lines of the call are processed in turn. 

The part I think I'm not getting is due to my lack of understanding of Perl (and specifically perl syntax). 

The first rule can create the context: 

add call_uniqueid_seq_$4 1 (where $4 is the unique ID for that call) 

But then in subsequent rules I want to do something like this: 

call_uniqueid_seq_$4 ++ 

However I don't expect that to be valid syntax (especially since that store is for strings), so I have to figure out something else. 

I was working my head around just 'pop'ing the value off the event store, and then adding 1 to it, but I realized I have no clue how to do this. 

So, I am hoping that I am just missing something obvious, but, failing that, I guess I need to understand how I would do something like: 

pop call_uniqueid_seq_$4 %sequence_num 
add call_uniqueid_seq_$4 %sequence_num+1 

I feel like I'm on the wrong track here, but I'm not finding the answers in the man page, or tutorial, or anywhere else, and my knowledge of Perl is insufficient to help make sense of this. 

Any thoughts or advice or clues would be greatly appreciated. 


-- 
Jim Van Meggelen 
ClearlyCore Inc. 



+1-416-639-6001 (DID) 
+1-877-253-2716 (Canada) 
+1-866-644-7729 (USA) 
+1-416-425-6111 x6001 
jim...@cl... 
[ http://www.clearlycore.com/ | http://www.clearlycore.com ] 

Asterisk: The Definitive Guide 
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: 
[ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] 

Re: [Simple-evcorr-users] SEC tutorial

[Risto V. <ris...@gm...>]

hi Jim,
feel free to share your remarks! Perhaps it is better to send them to my
private email address, since the mailing list has a low limit for the
maximum email size (I think it is only around 30-40KB). I will be out of
office this weekend and next week to have my vacation, but after that I can
start reading your remarks.
kind regards,
risto

Kontakt Jim Van Meggelen (<jim...@cl...>) kirjutas
kuupäeval N, 10. august 2023 kell 19:35:

> Risto,
>
> I know it's been a long time, but I have been chipping away at that
> tutorial. It has been really helpful.
>
> I'm not an academic person (I dropped out of high school), so I don't know
> the correct and formal way to comment on academic papers. As such, my notes
> might seem a bit less formal than is appropriate.
>
> I did however write a book for O'Reilly, so I do have a bit of a feel for
> book writing, and it is from that angle that most of my comments are coming.
>
> Are you interested in what I've written? I fear giving offense, but I had
> some ideas that might be worth considering.
>
> Jim
>
>
>
> --
> Jim Van Meggelen
> ClearlyCore Inc.
>
>
>
> +1-416-639-6001 (DID)
> +1-877-253-2716 (Canada)
> +1-866-644-7729 (USA)
> +1-416-425-6111 x6001
> jim...@cl...
> http://www.clearlycore.com
>
>
> *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:*
> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf
>
> ------------------------------
>
> *From: *"Risto Vaarandi" <ris...@gm...>
> *To: *"Jim Van Meggelen" <jim...@cl...>
> *Cc: *"simple-evcorr-users" <sim...@li...>
> *Sent: *Thursday, 1 December, 2022 13:07:03
> *Subject: *Re: [Simple-evcorr-users] SEC tutorial
>
> hi Jim,
> many thanks for your very kind offer! The tutorial is fairly large (around
> 50 pages), but if you have time, I would appreciate your feedback. Adding
> the remarks and recommendations into PDF works fine for me, and you can
> just send the modified PDF over to my private email address.
> kind regards,
> risto
>
> Kontakt Jim Van Meggelen (<jim...@cl...>) kirjutas
> kuupäeval N, 1. detsember 2022 kell 15:25:
>
>> Risto,
>>
>> Thanks for doing that up. I'll work through it.
>>
>> Are you interested in feedback on it? If so, do you have a preferred
>> mechanism for that? (I can just mark up the PDF as I read through and
>> upload it somewhere when I'm done). Note that I might find myself mostly
>> copyediting, rather than testing all the examples.
>>
>> Jim
>>
>>
>> --
>> Jim Van Meggelen
>> ClearlyCore Inc.
>>
>>
>>
>> +1-416-639-6001 (DID)
>> +1-877-253-2716 (Canada)
>> +1-866-644-7729 (USA)
>> +1-416-425-6111 x6001
>> jim...@cl...
>> http://www.clearlycore.com
>>
>>
>> *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:*
>> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf
>>
>> ------------------------------
>>
>> *From: *"Risto Vaarandi" <ris...@gm...>
>> *To: *"simple-evcorr-users" <sim...@li...>
>> *Sent: *Thursday, 1 December, 2022 05:23:48
>> *Subject: *[Simple-evcorr-users] SEC tutorial
>>
>> hi all,
>>
>> during the last few weeks, I have written a new SEC tutorial paper which
>> can be accessed through this web link:
>> https://raw.githubusercontent.com/simple-evcorr/tutorial/main/SEC-tutorial.pdf
>> Links to this tutorial and the relevant Github repository (
>> https://github.com/simple-evcorr/tutorial) are also available in the SEC
>> home page.
>>
>> This tutorial is primarily intended for people who haven't used SEC
>> before, and it explains the essentials of SEC in a gentle way, starting
>> from simple examples and gradually moving to more complex ones. After
>> reading this tutorial, the reader should have a good understanding of event
>> correlation rules, event correlation operations, contexts, synthetic
>> events, pattern match caching, and many other things. And it might provide
>> interesting insights even for experienced users :)
>>
>> As mentioned in the introduction, this tutorial does not intend to be a
>> replacement for the official documentation. Therefore, it does not attempt
>> to explain every possible detail, but rather provide an overview of the key
>> concepts, so that it would be easier to look up more detailed information
>> from the man page.
>>
>> Hopefully the tutorial will be useful and you will find it easy to follow
>> :)
>>
>> kind regards,
>> risto
>>
>>
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> Sim...@li...
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>>
>

Re: [Simple-evcorr-users] SEC tutorial

[Jim V. M. <jim...@cl...>]

Risto, 

I know it's been a long time, but I have been chipping away at that tutorial. It has been really helpful. 

I'm not an academic person (I dropped out of high school), so I don't know the correct and formal way to comment on academic papers. As such, my notes might seem a bit less formal than is appropriate. 

I did however write a book for O'Reilly, so I do have a bit of a feel for book writing, and it is from that angle that most of my comments are coming. 

Are you interested in what I've written? I fear giving offense, but I had some ideas that might be worth considering. 

Jim 

-- 
Jim Van Meggelen 
ClearlyCore Inc. 

+1-416-639-6001 (DID) 
+1-877-253-2716 (Canada) 
+1-866-644-7729 (USA) 
+1-416-425-6111 x6001 
jim...@cl... 
[ http://www.clearlycore.com/ | http://www.clearlycore.com ] 

Asterisk: The Definitive Guide 
FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: 
[ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] 

> From: "Risto Vaarandi" <ris...@gm...>
> To: "Jim Van Meggelen" <jim...@cl...>
> Cc: "simple-evcorr-users" <sim...@li...>
> Sent: Thursday, 1 December, 2022 13:07:03
> Subject: Re: [Simple-evcorr-users] SEC tutorial

> hi Jim,
> many thanks for your very kind offer! The tutorial is fairly large (around 50
> pages), but if you have time, I would appreciate your feedback. Adding the
> remarks and recommendations into PDF works fine for me, and you can just send
> the modified PDF over to my private email address.
> kind regards,
> risto

> Kontakt Jim Van Meggelen (< [ mailto:jim...@cl... |
> jim...@cl... ] >) kirjutas kuupäeval N, 1. detsember 2022
> kell 15:25:

>> Risto,

>> Thanks for doing that up. I'll work through it.

>> Are you interested in feedback on it? If so, do you have a preferred mechanism
>> for that? (I can just mark up the PDF as I read through and upload it somewhere
>> when I'm done). Note that I might find myself mostly copyediting, rather than
>> testing all the examples.

>> Jim

>> --
>> Jim Van Meggelen
>> ClearlyCore Inc.

>> +1-416-639-6001 (DID)
>> +1-877-253-2716 (Canada)
>> +1-866-644-7729 (USA)
>> +1-416-425-6111 x6001
>> [ mailto:jim...@cl... | jim...@cl... ]
>> [ http://www.clearlycore.com/ | http://www.clearlycore.com ]

>> Asterisk: The Definitive Guide
>> FIFTH EDITION NOW AVAILABLE TO DOWNLOAD:
>> [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf |
>> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ]

>>> From: "Risto Vaarandi" < [ mailto:ris...@gm... |
>>> ris...@gm... ] >
>>> To: "simple-evcorr-users" < [ mailto:sim...@li... |
>>> sim...@li... ] >
>>> Sent: Thursday, 1 December, 2022 05:23:48
>>> Subject: [Simple-evcorr-users] SEC tutorial

>>> hi all,

>>> during the last few weeks, I have written a new SEC tutorial paper which can be
>>> accessed through this web link: [
>>> https://raw.githubusercontent.com/simple-evcorr/tutorial/main/SEC-tutorial.pdf
>>> |
>>> https://raw.githubusercontent.com/simple-evcorr/tutorial/main/SEC-tutorial.pdf
>>> ]
>>> Links to this tutorial and the relevant Github repository ( [
>>> https://github.com/simple-evcorr/tutorial |
>>> https://github.com/simple-evcorr/tutorial ] ) are also available in the SEC
>>> home page.

>>> This tutorial is primarily intended for people who haven't used SEC before, and
>>> it explains the essentials of SEC in a gentle way, starting from simple
>>> examples and gradually moving to more complex ones. After reading this
>>> tutorial, the reader should have a good understanding of event correlation
>>> rules, event correlation operations, contexts, synthetic events, pattern match
>>> caching, and many other things. And it might provide interesting insights even
>>> for experienced users :)

>>> As mentioned in the introduction, this tutorial does not intend to be a
>>> replacement for the official documentation. Therefore, it does not attempt to
>>> explain every possible detail, but rather provide an overview of the key
>>> concepts, so that it would be easier to look up more detailed information from
>>> the man page.

>>> Hopefully the tutorial will be useful and you will find it easy to follow :)

>>> kind regards,
>>> risto

>>> _______________________________________________
>>> Simple-evcorr-users mailing list
>>> [ mailto:Sim...@li... |
>>> Sim...@li... ]
>>> [ https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users |
>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users ]

[Simple-evcorr-users] SEC-2.9.2 released

[Risto V. <ris...@gm...>]

hi all,

SEC-2.9.2 has been released which is available from the SEC home page. You
can also download it through a direct download link:
https://github.com/simple-evcorr/sec/releases/download/2.9.2/sec-2.9.2.tar.gz
.

Here is the changelog for the new version:

--- version 2.9.2

* starting from this version, list of event occurrence times that correspond
  to event group string tokens is passed to PerlFunc and NPerlFunc event
  group patterns as an additional parameter.


kind regards,
risto

[Simple-evcorr-users] R: Duplicate suppression and rearming

[Spelta E. <Edo...@be...>]

Hi,
i did the same test you did, just with an error in the pattern matching 😊

it’s beautifully working now!

Thanks a lot for the help, it was very precious!


Best regards,
M

Da: Risto Vaarandi <ris...@gm...>
Inviato: mercoledì 15 marzo 2023 19:20
A: Spelta Edoardo <Edo...@be...>
Cc: sim...@li...
Oggetto: Re: [Simple-evcorr-users] Duplicate suppression and rearming




  1.  As long as we exclude the longer window expiration case (so we just want to suppress events for 10 secs), why you suggest to use two Single rules instead of a SingleWithSuppress ?

I’m referring to these rules:
type=Single
ptype=RegExp
pattern=event_(\w+)
context=!SUPP_$1
desc=react to event $1 and set up suppression
action=write - We have observed event $1; create SUPP_$1 10

type=Single
ptype=RegExp
pattern=event_(\w+)
context=SUPP_$1
desc=do not react to event $1 and only update its suppression context
action=set SUPP_$1 10



  1.  Does SingleWithSuppress use a sliding window ? According to my latest tests it does not, so it should be equivalent to your two Single rules..shouldn’t it ?

 you are correct that the SingleWithSuppress rule does not use a sliding window. In contrast, the above two rules implement the sliding suppression window.






  1.  For the more complex scenario, when you  introduced also a MAXLIFE context of 60 secs, so i’m referring to:

type=Single
ptype=RegExp
pattern=event_(\w+)
context=!SUPP_$1
desc=react to event $1 and set up suppression
action=write - We have observed event $1; \
       create SUPP_$1_MAXLIFE 60 ( delete SUPP_$1 ); \
       create SUPP_$1 10 ( delete SUPP_$1_MAXLIFE )

type=Single
ptype=RegExp
pattern=event_(\w+)
context=SUPP_$1
desc=do not react to event $1 and only update its suppression context
action=set SUPP_$1 10



..as far as i can see, the shortest 10secs context will always expire first, thus deleting the MAXLIFE as well…so in the end this behaves just like the longer context does not exist at all. It seems equivalent to your rules in bullet 1), am i wrong ?

the lifetime of SUPP_$1 context is extended for another 10 seconds on each matching event, so the SUPP_$1 context can actually have a longer lifetime than SUPP_$1_MAXLIFE.


I’ve been generating identical events every two seconds, and with those two rules i get 1 line every 10 seconds…instead i would expect to get one every 60 seconds…because i expected the 10sec window to be constantly sliding thus suppressing everything until the 60 seconds windows expires and basically reset everything.

What am i missing ?


May I ask what is the format of the events you are generating, and do they all look the same (for example, event_A)?

I have tested the ruleset with events "event_A" that are generated after every 5 seconds with the following command line:

while true; do echo event_A >>test.log; sleep 5; done

Also, before starting the above command line, I executed sec with the following command line that monitors test.log for incoming events. In addition, this command line writes a detailed debug log to sec.log:

sec --conf=test.sec --input=test.log --log=sec.log

Here is the content from sec.log:

Wed Mar 15 19:56:28 2023: SEC (Simple Event Correlator) 2.9.1
Wed Mar 15 19:56:28 2023: Reading configuration from test.sec
Wed Mar 15 19:56:28 2023: 2 rules loaded from test.sec
Wed Mar 15 19:56:28 2023: No --bufsize command line option or --bufsize=0, setting --bufsize to 1
Wed Mar 15 19:56:28 2023: Opening input file test.log
Wed Mar 15 19:56:28 2023: Interactive process, SIGINT can't be used for changing the logging level
Wed Mar 15 19:56:31 2023: Writing event 'We have observed event A' to file '-'
Wed Mar 15 19:56:31 2023: Creating context 'SUPP_A_MAXLIFE'
Wed Mar 15 19:56:31 2023: Creating context 'SUPP_A'
Wed Mar 15 19:56:36 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:56:41 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:56:46 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:56:51 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:56:56 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:01 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:06 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:11 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:16 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:21 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:26 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:31 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:32 2023: Deleting stale context 'SUPP_A_MAXLIFE'
Wed Mar 15 19:57:32 2023: Deleting context 'SUPP_A'
Wed Mar 15 19:57:32 2023: Context 'SUPP_A' deleted
Wed Mar 15 19:57:32 2023: Stale context 'SUPP_A_MAXLIFE' deleted
Wed Mar 15 19:57:36 2023: Writing event 'We have observed event A' to file '-'
Wed Mar 15 19:57:36 2023: Creating context 'SUPP_A_MAXLIFE'
Wed Mar 15 19:57:36 2023: Creating context 'SUPP_A'

As you can see, the event "event_A" that appeared at 19:56:31 triggered a notification message, and also, suppression contexts were set up. Also, after every 5 seconds debug messages "Changing settings for context 'SUPP_A'" appear. These messages reflect the execution of the 'set SUPP_A 10' action when the event "event_A" appears. Finally, at 19:57:32 (61 seconds after 19:56:31) the context SUPP_A_MAXLIFE becomes stale, and when this context is going through the deletion process, the context SUPP_A is also deleted (see the "Context 'SUPP_A' deleted" debug message). The disappearance of both contexts ends the event suppression process, and when the next event "event_A" appears at 19:57:36, a notification is again printed to standard output.

Did you generate events with the *same* suffix in real-time and provided them to SEC in the fashion as described above? If you want to debug the ruleset interactively in a quick way, you can also use smaller timeouts than 10 and 60 seconds, and monitor events from standard input with the --input=- command line option. This allows you to type input events from the keyboard and observe what kind of SEC debug messages are produced in a terminal window.

Hope this helps,
risto

Re: [Simple-evcorr-users] Duplicate suppression and rearming

[Risto V. <ris...@gm...>]

It is great to hear that everything is working properly :)

kind regards,
risto

Kontakt Spelta Edoardo (<Edo...@be...>) kirjutas kuupäeval
N, 16. märts 2023 kell 15:39:

> Hi,
>
> i did the same test you did, just with an error in the pattern matching 😊
>
>
>
> it’s beautifully working now!
>
>
>
> Thanks a lot for the help, it was very precious!
>
>
>
>
>
> Best regards,
>
> M
>
>
>
> *Da:* Risto Vaarandi <ris...@gm...>
> *Inviato:* mercoledì 15 marzo 2023 19:20
> *A:* Spelta Edoardo <Edo...@be...>
> *Cc:* sim...@li...
> *Oggetto:* Re: [Simple-evcorr-users] Duplicate suppression and rearming
>
>
>
>
>
>
>
>    1. As long as we exclude the longer window expiration case (so we just
>    want to suppress events for 10 secs), why you suggest to use two Single
>    rules instead of a SingleWithSuppress ?
>
>
>
> I’m referring to these rules:
>
> type=Single
> ptype=RegExp
> pattern=event_(\w+)
> context=!SUPP_$1
> desc=react to event $1 and set up suppression
> action=write - We have observed event $1; create SUPP_$1 10
>
> type=Single
> ptype=RegExp
> pattern=event_(\w+)
> context=SUPP_$1
> desc=do not react to event $1 and only update its suppression context
> action=set SUPP_$1 10
>
>
>
>    1. Does SingleWithSuppress use a sliding window ? According to my
>    latest tests it does not, so it should be equivalent to your two Single
>    rules..shouldn’t it ?
>
>
>
>  you are correct that the SingleWithSuppress rule does not use a sliding
> window. In contrast, the above two rules implement the sliding suppression
> window.
>
>
>
>
>
>
>
>    1. For the more complex scenario, when you  introduced also a MAXLIFE
>    context of 60 secs, so i’m referring to:
>
>
>
> type=Single
> ptype=RegExp
> pattern=event_(\w+)
> context=!SUPP_$1
> desc=react to event $1 and set up suppression
> action=write - We have observed event $1; \
>        create SUPP_$1_MAXLIFE 60 ( delete SUPP_$1 ); \
>        create SUPP_$1 10 ( delete SUPP_$1_MAXLIFE )
>
> type=Single
> ptype=RegExp
> pattern=event_(\w+)
> context=SUPP_$1
> desc=do not react to event $1 and only update its suppression context
> action=set SUPP_$1 10
>
>
>
>
>
> ..as far as i can see, the shortest 10secs context will always expire
> first, thus deleting the MAXLIFE as well…so in the end this behaves just
> like the longer context does not exist at all. It seems equivalent to your
> rules in bullet 1), am i wrong ?
>
>
>
> the lifetime of SUPP_$1 context is extended for another 10 seconds on each
> matching event, so the SUPP_$1 context can actually have a longer lifetime
> than SUPP_$1_MAXLIFE.
>
>
>
>
>
> I’ve been generating identical events every two seconds, and with those
> two rules i get 1 line every 10 seconds…instead i would expect to get one
> every 60 seconds…because i expected the 10sec window to be constantly
> sliding thus suppressing everything until the 60 seconds windows expires
> and basically reset everything.
>
>
>
> What am i missing ?
>
>
>
>
>
> May I ask what is the format of the events you are generating, and do they
> all look the same (for example, event_A)?
>
>
>
> I have tested the ruleset with events "event_A" that are generated after
> every 5 seconds with the following command line:
>
>
>
> while true; do echo event_A >>test.log; sleep 5; done
>
>
>
> Also, before starting the above command line, I executed sec with the
> following command line that monitors test.log for incoming events. In
> addition, this command line writes a detailed debug log to sec.log:
>
>
>
> sec --conf=test.sec --input=test.log --log=sec.log
>
>
>
> Here is the content from sec.log:
>
>
>
> Wed Mar 15 19:56:28 2023: SEC (Simple Event Correlator) 2.9.1
> Wed Mar 15 19:56:28 2023: Reading configuration from test.sec
> Wed Mar 15 19:56:28 2023: 2 rules loaded from test.sec
> Wed Mar 15 19:56:28 2023: No --bufsize command line option or --bufsize=0,
> setting --bufsize to 1
> Wed Mar 15 19:56:28 2023: Opening input file test.log
> Wed Mar 15 19:56:28 2023: Interactive process, SIGINT can't be used for
> changing the logging level
> Wed Mar 15 19:56:31 2023: Writing event 'We have observed event A' to file
> '-'
> Wed Mar 15 19:56:31 2023: Creating context 'SUPP_A_MAXLIFE'
> Wed Mar 15 19:56:31 2023: Creating context 'SUPP_A'
> Wed Mar 15 19:56:36 2023: Changing settings for context 'SUPP_A'
> Wed Mar 15 19:56:41 2023: Changing settings for context 'SUPP_A'
> Wed Mar 15 19:56:46 2023: Changing settings for context 'SUPP_A'
> Wed Mar 15 19:56:51 2023: Changing settings for context 'SUPP_A'
> Wed Mar 15 19:56:56 2023: Changing settings for context 'SUPP_A'
> Wed Mar 15 19:57:01 2023: Changing settings for context 'SUPP_A'
> Wed Mar 15 19:57:06 2023: Changing settings for context 'SUPP_A'
> Wed Mar 15 19:57:11 2023: Changing settings for context 'SUPP_A'
> Wed Mar 15 19:57:16 2023: Changing settings for context 'SUPP_A'
> Wed Mar 15 19:57:21 2023: Changing settings for context 'SUPP_A'
> Wed Mar 15 19:57:26 2023: Changing settings for context 'SUPP_A'
> Wed Mar 15 19:57:31 2023: Changing settings for context 'SUPP_A'
> Wed Mar 15 19:57:32 2023: Deleting stale context 'SUPP_A_MAXLIFE'
> Wed Mar 15 19:57:32 2023: Deleting context 'SUPP_A'
> Wed Mar 15 19:57:32 2023: Context 'SUPP_A' deleted
> Wed Mar 15 19:57:32 2023: Stale context 'SUPP_A_MAXLIFE' deleted
> Wed Mar 15 19:57:36 2023: Writing event 'We have observed event A' to file
> '-'
> Wed Mar 15 19:57:36 2023: Creating context 'SUPP_A_MAXLIFE'
> Wed Mar 15 19:57:36 2023: Creating context 'SUPP_A'
>
>
>
> As you can see, the event "event_A" that appeared at 19:56:31 triggered a
> notification message, and also, suppression contexts were set up. Also,
> after every 5 seconds debug messages "Changing settings for context
> 'SUPP_A'" appear. These messages reflect the execution of the 'set SUPP_A
> 10' action when the event "event_A" appears. Finally, at 19:57:32 (61
> seconds after 19:56:31) the context SUPP_A_MAXLIFE becomes stale, and when
> this context is going through the deletion process, the context SUPP_A is
> also deleted (see the "Context 'SUPP_A' deleted" debug message). The
> disappearance of both contexts ends the event suppression process, and when
> the next event "event_A" appears at 19:57:36, a notification is again
> printed to standard output.
>
>
>
> Did you generate events with the *same* suffix in real-time and provided
> them to SEC in the fashion as described above? If you want to debug the
> ruleset interactively in a quick way, you can also use smaller timeouts
> than 10 and 60 seconds, and monitor events from standard input with the
> --input=- command line option. This allows you to type input events from
> the keyboard and observe what kind of SEC debug messages are produced in a
> terminal window.
>
>
>
> Hope this helps,
>
> risto
>

Re: [Simple-evcorr-users] Duplicate suppression and rearming

[Risto V. <ris...@gm...>]

>
>
>    1. As long as we exclude the longer window expiration case (so we just
>    want to suppress events for 10 secs), why you suggest to use two Single
>    rules instead of a SingleWithSuppress ?
>
>
>
> I’m referring to these rules:
>
> type=Single
> ptype=RegExp
> pattern=event_(\w+)
> context=!SUPP_$1
> desc=react to event $1 and set up suppression
> action=write - We have observed event $1; create SUPP_$1 10
>
> type=Single
> ptype=RegExp
> pattern=event_(\w+)
> context=SUPP_$1
> desc=do not react to event $1 and only update its suppression context
> action=set SUPP_$1 10
>
>
>
>    1. Does SingleWithSuppress use a sliding window ? According to my
>    latest tests it does not, so it should be equivalent to your two Single
>    rules..shouldn’t it ?
>
>
 you are correct that the SingleWithSuppress rule does not use a sliding
window. In contrast, the above two rules implement the sliding suppression
window.


>
>
>
>    1. For the more complex scenario, when you  introduced also a MAXLIFE
>    context of 60 secs, so i’m referring to:
>
>
>
> type=Single
> ptype=RegExp
> pattern=event_(\w+)
> context=!SUPP_$1
> desc=react to event $1 and set up suppression
> action=write - We have observed event $1; \
>        create SUPP_$1_MAXLIFE 60 ( delete SUPP_$1 ); \
>        create SUPP_$1 10 ( delete SUPP_$1_MAXLIFE )
>
> type=Single
> ptype=RegExp
> pattern=event_(\w+)
> context=SUPP_$1
> desc=do not react to event $1 and only update its suppression context
> action=set SUPP_$1 10
>
>
>
>
>
> ..as far as i can see, the shortest 10secs context will always expire
> first, thus deleting the MAXLIFE as well…so in the end this behaves just
> like the longer context does not exist at all. It seems equivalent to your
> rules in bullet 1), am i wrong ?
>

the lifetime of SUPP_$1 context is extended for another 10 seconds on each
matching event, so the SUPP_$1 context can actually have a longer lifetime
than SUPP_$1_MAXLIFE.


>
> I’ve been generating identical events every two seconds, and with those
> two rules i get 1 line every 10 seconds…instead i would expect to get one
> every 60 seconds…because i expected the 10sec window to be constantly
> sliding thus suppressing everything until the 60 seconds windows expires
> and basically reset everything.
>
>
>
> What am i missing ?
>
>
>

May I ask what is the format of the events you are generating, and do they
all look the same (for example, event_A)?

I have tested the ruleset with events "event_A" that are generated after
every 5 seconds with the following command line:

while true; do echo event_A >>test.log; sleep 5; done

Also, before starting the above command line, I executed sec with the
following command line that monitors test.log for incoming events. In
addition, this command line writes a detailed debug log to sec.log:

sec --conf=test.sec --input=test.log --log=sec.log

Here is the content from sec.log:

Wed Mar 15 19:56:28 2023: SEC (Simple Event Correlator) 2.9.1
Wed Mar 15 19:56:28 2023: Reading configuration from test.sec
Wed Mar 15 19:56:28 2023: 2 rules loaded from test.sec
Wed Mar 15 19:56:28 2023: No --bufsize command line option or --bufsize=0,
setting --bufsize to 1
Wed Mar 15 19:56:28 2023: Opening input file test.log
Wed Mar 15 19:56:28 2023: Interactive process, SIGINT can't be used for
changing the logging level
Wed Mar 15 19:56:31 2023: Writing event 'We have observed event A' to file
'-'
Wed Mar 15 19:56:31 2023: Creating context 'SUPP_A_MAXLIFE'
Wed Mar 15 19:56:31 2023: Creating context 'SUPP_A'
Wed Mar 15 19:56:36 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:56:41 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:56:46 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:56:51 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:56:56 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:01 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:06 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:11 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:16 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:21 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:26 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:31 2023: Changing settings for context 'SUPP_A'
Wed Mar 15 19:57:32 2023: Deleting stale context 'SUPP_A_MAXLIFE'
Wed Mar 15 19:57:32 2023: Deleting context 'SUPP_A'
Wed Mar 15 19:57:32 2023: Context 'SUPP_A' deleted
Wed Mar 15 19:57:32 2023: Stale context 'SUPP_A_MAXLIFE' deleted
Wed Mar 15 19:57:36 2023: Writing event 'We have observed event A' to file
'-'
Wed Mar 15 19:57:36 2023: Creating context 'SUPP_A_MAXLIFE'
Wed Mar 15 19:57:36 2023: Creating context 'SUPP_A'

As you can see, the event "event_A" that appeared at 19:56:31 triggered a
notification message, and also, suppression contexts were set up. Also,
after every 5 seconds debug messages "Changing settings for context
'SUPP_A'" appear. These messages reflect the execution of the 'set SUPP_A
10' action when the event "event_A" appears. Finally, at 19:57:32 (61
seconds after 19:56:31) the context SUPP_A_MAXLIFE becomes stale, and when
this context is going through the deletion process, the context SUPP_A is
also deleted (see the "Context 'SUPP_A' deleted" debug message). The
disappearance of both contexts ends the event suppression process, and when
the next event "event_A" appears at 19:57:36, a notification is again
printed to standard output.

Did you generate events with the *same* suffix in real-time and provided
them to SEC in the fashion as described above? If you want to debug the
ruleset interactively in a quick way, you can also use smaller timeouts
than 10 and 60 seconds, and monitor events from standard input with the
--input=- command line option. This allows you to type input events from
the keyboard and observe what kind of SEC debug messages are produced in a
terminal window.

Hope this helps,
risto