simple-evcorr-users Mailing List for Simple Event Correlator (Page 3)
Brought to you by:
ristov
You can subscribe to this list here.
2001 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(5) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2002 |
Jan
(6) |
Feb
(14) |
Mar
(8) |
Apr
(13) |
May
(6) |
Jun
(24) |
Jul
(7) |
Aug
(9) |
Sep
(20) |
Oct
(7) |
Nov
(1) |
Dec
|
2003 |
Jan
(10) |
Feb
|
Mar
(6) |
Apr
(12) |
May
(12) |
Jun
(44) |
Jul
(39) |
Aug
(10) |
Sep
(28) |
Oct
(35) |
Nov
(66) |
Dec
(51) |
2004 |
Jan
(21) |
Feb
(33) |
Mar
(32) |
Apr
(59) |
May
(59) |
Jun
(34) |
Jul
(6) |
Aug
(39) |
Sep
(13) |
Oct
(13) |
Nov
(19) |
Dec
(9) |
2005 |
Jan
(18) |
Feb
(36) |
Mar
(24) |
Apr
(18) |
May
(51) |
Jun
(34) |
Jul
(9) |
Aug
(34) |
Sep
(52) |
Oct
(20) |
Nov
(11) |
Dec
(12) |
2006 |
Jan
(20) |
Feb
(3) |
Mar
(68) |
Apr
(41) |
May
(11) |
Jun
(39) |
Jul
(17) |
Aug
(34) |
Sep
(40) |
Oct
(42) |
Nov
(25) |
Dec
(33) |
2007 |
Jan
(6) |
Feb
(28) |
Mar
(32) |
Apr
(25) |
May
(11) |
Jun
(20) |
Jul
(8) |
Aug
(12) |
Sep
(13) |
Oct
(42) |
Nov
(37) |
Dec
(16) |
2008 |
Jan
(25) |
Feb
(1) |
Mar
(28) |
Apr
(34) |
May
(16) |
Jun
(23) |
Jul
(45) |
Aug
(26) |
Sep
(5) |
Oct
(5) |
Nov
(20) |
Dec
(39) |
2009 |
Jan
(14) |
Feb
(24) |
Mar
(40) |
Apr
(47) |
May
(11) |
Jun
(19) |
Jul
(15) |
Aug
(13) |
Sep
(7) |
Oct
(34) |
Nov
(27) |
Dec
(24) |
2010 |
Jan
(14) |
Feb
(5) |
Mar
(16) |
Apr
(12) |
May
(25) |
Jun
(43) |
Jul
(13) |
Aug
(12) |
Sep
(10) |
Oct
(40) |
Nov
(23) |
Dec
(29) |
2011 |
Jan
(25) |
Feb
(7) |
Mar
(28) |
Apr
(36) |
May
(18) |
Jun
(26) |
Jul
(7) |
Aug
(16) |
Sep
(21) |
Oct
(29) |
Nov
(13) |
Dec
(36) |
2012 |
Jan
(26) |
Feb
(13) |
Mar
(12) |
Apr
(13) |
May
(12) |
Jun
(2) |
Jul
(3) |
Aug
(15) |
Sep
(34) |
Oct
(49) |
Nov
(25) |
Dec
(23) |
2013 |
Jan
(1) |
Feb
(35) |
Mar
(32) |
Apr
(6) |
May
(11) |
Jun
(68) |
Jul
(15) |
Aug
(8) |
Sep
(58) |
Oct
(27) |
Nov
(19) |
Dec
(15) |
2014 |
Jan
(40) |
Feb
(49) |
Mar
(21) |
Apr
(8) |
May
(26) |
Jun
(9) |
Jul
(33) |
Aug
(35) |
Sep
(18) |
Oct
(7) |
Nov
(13) |
Dec
(8) |
2015 |
Jan
(12) |
Feb
(2) |
Mar
(16) |
Apr
(33) |
May
(4) |
Jun
(25) |
Jul
(20) |
Aug
(9) |
Sep
(10) |
Oct
(40) |
Nov
(15) |
Dec
(17) |
2016 |
Jan
(16) |
Feb
(16) |
Mar
(4) |
Apr
(40) |
May
(9) |
Jun
(21) |
Jul
(9) |
Aug
(16) |
Sep
(13) |
Oct
(17) |
Nov
(14) |
Dec
(26) |
2017 |
Jan
(9) |
Feb
(6) |
Mar
(23) |
Apr
(7) |
May
(1) |
Jun
(6) |
Jul
(11) |
Aug
(17) |
Sep
|
Oct
(1) |
Nov
(3) |
Dec
(1) |
2018 |
Jan
(14) |
Feb
(2) |
Mar
(12) |
Apr
(10) |
May
(1) |
Jun
(12) |
Jul
(6) |
Aug
(1) |
Sep
(1) |
Oct
(9) |
Nov
(3) |
Dec
(6) |
2019 |
Jan
(1) |
Feb
|
Mar
(5) |
Apr
|
May
(3) |
Jun
(3) |
Jul
(2) |
Aug
(9) |
Sep
(11) |
Oct
(7) |
Nov
(10) |
Dec
(11) |
2020 |
Jan
(9) |
Feb
(14) |
Mar
(15) |
Apr
(26) |
May
(1) |
Jun
|
Jul
|
Aug
(4) |
Sep
|
Oct
(6) |
Nov
|
Dec
(6) |
2021 |
Jan
|
Feb
(1) |
Mar
(11) |
Apr
(1) |
May
(5) |
Jun
|
Jul
(4) |
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
|
2022 |
Jan
|
Feb
|
Mar
(6) |
Apr
(3) |
May
(2) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(6) |
Dec
(5) |
2023 |
Jan
|
Feb
|
Mar
(11) |
Apr
|
May
|
Jun
(1) |
Jul
|
Aug
(2) |
Sep
(9) |
Oct
|
Nov
|
Dec
(1) |
2024 |
Jan
|
Feb
|
Mar
|
Apr
(6) |
May
|
Jun
(1) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
|
2025 |
Jan
(1) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
From: Jim V. M. <jim...@cl...> - 2022-03-31 15:46:42
|
Risto, Absolutely. The following pattern should catch it: ^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].* pbx.c: Spawn extension \(.*\) exited non-zero on.* Here's a few lines leading up to it (interesting because here is a Read() operation where the caller does not make a choice but instead hangs up while the system is waiting for a response): [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-MainMenu:45] Read("SIP/CHANNELNAME-000138ac", "BalanceRepeatPressed,prompts/004h&prompts/004i&prompts/043,1,,,15") in new stack [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] app_read.c: Accepting a maximum of 1 digits. [2022-02-08-06:21:22:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/004h.slin' (language 'en') [2022-02-08-06:21:25:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/004i.slin' (language 'en') [2022-02-08-06:21:28:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/043.slin' (language 'en') [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] app_read.c: User disconnected [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing [h@ivr-MainMenu:1] NoOp("SIP/CHANNELNAME-000138ac", "Hangup handler GlobalCallCount=0 CHANNELNAME=0 ") in new stack [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing [h@ivr-MainMenu:2] NoOp("SIP/CHANNELNAME-000138ac", "SurveyType is ") in new stack [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Executing [h@ivr-MainMenu:3] ExecIf("SIP/CHANNELNAME-000138ac", "1?Hangup()") in new stack [2022-02-08-06:21:33:] VERBOSE[24858][C-00004037] pbx.c: Spawn extension (ivr-MainMenu, h, 3) exited non-zero on 'SIP/CHANNELNAME-000138ac' -- Jim Van Meggelen ClearlyCore Inc. +1-416-639-6001 (DID) +1-877-253-2716 (Canada) +1-866-644-7729 (USA) +1-416-425-6111 x6001 jim...@cl... [ http://www.clearlycore.com/ | http://www.clearlycore.com ] Asterisk: The Definitive Guide FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] > From: "Risto Vaarandi" <ris...@gm...> > To: "Jim Van Meggelen" <jim...@cl...> > Cc: "simple-evcorr-users" <sim...@li...> > Sent: Thursday, 31 March, 2022 10:35:49 > Subject: Re: [Simple-evcorr-users] Parsing Asterisk log files for downstream > reporting - so far so good! > hi Jim, > can you provide an example of the log event that indicates the end of the call? > kind regards, > risto > Kontakt Jim Van Meggelen (< [ mailto:jim...@cl... | > jim...@cl... ] >) kirjutas kuupäeval N, 31. märts 2022 kell > 15:45: >> Risto, >> Thank you for the reply. >> You are correct that the C-[0-9a-f]{8} string uniquely identifies each call, and >> is present on all log lines. >> We cannot safely assume the maximum call length; some calls could reasonably be >> 30 minutes or even more. We will be able to identify the end of call, though, >> so a pattern/rule can look for that. >> I'm very interested in your thoughts. Thanks again. >> -- >> Jim Van Meggelen >> ClearlyCore Inc. >> +1-416-639-6001 (DID) >> +1-877-253-2716 (Canada) >> +1-866-644-7729 (USA) >> +1-416-425-6111 x6001 >> [ mailto:jim...@cl... | jim...@cl... ] >> [ http://www.clearlycore.com/ | http://www.clearlycore.com ] >> Asterisk: The Definitive Guide >> FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: >> [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | >> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] >>> From: "Risto Vaarandi" < [ mailto:ris...@gm... | >>> ris...@gm... ] > >>> To: "Jim Van Meggelen" < [ mailto:jim...@cl... | >>> jim...@cl... ] > >>> Cc: [ mailto:sim...@li... | >>> sim...@li... ] >>> Sent: Thursday, 31 March, 2022 05:48:44 >>> Subject: Re: [Simple-evcorr-users] Parsing Asterisk log files for downstream >>> reporting - so far so good! >>> hi Jim, >>> I do have couple of things in mind that might help addressing this issue, but >>> before coming up with any suggestions, may I ask some questions? As I >>> understand, the phone call is uniquely identified by the numeral that follows >>> the C character (C-00004037 in your example) which is present in all log >>> messages for that particular call. >>> However, is there also a specific log message that denotes the end of the call? >>> If there is no message indicating the end-of-call, can we assume that the call >>> has ended if there have been no messages for that call ID for the last 5 or 10 >>> minutes? >>> kind regards, >>> risto >>> Kontakt Jim Van Meggelen (< [ mailto:jim...@cl... | >>> jim...@cl... ] >) kirjutas kuupäeval N, 31. märts 2022 kell >>> 07:41: >>>> I have an atypical use for SEC, in which I'm parsing Asterisk log files to >>>> produce CSV peg counts for downstream usage reporting. >>>> This is working well and I'm loving this tool! >>>> However, I've run into a problem that I cannot seem to find a way around. >>>> The log file produces events that are tied to a unique ID for each channel/call >>>> ( C-00000000 ), and using that, plus the timestamp, I can produce output that >>>> is correlated with a call, and chronological. So far so good. >>>> However, there's one event that is difficult to work with, because it can happen >>>> more than once on a typical phone call, needs to correlate with an event that >>>> happens a few lines prior, but carries no identifying correlation. >>>> Here's an example of the output I'm working with (curated for brevity and >>>> relevance, and specific to a single unique ID; there are actually hundreds of >>>> lines of output per-call): >>>> [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] pbx.c: Executing >>>> [8005551234@ivr-welcome:9] Read("SIP/CHANNELNAME-000138ac", >>>> "EarlyChoice,prompts/001,1,,0,0.01") in new stack >>>> [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >>>> maximum of 1 digits. >>>> [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/001.slin' (language 'en') >>>> [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] app_read.c: User entered >>>> nothing. >>>> -- >>>> [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] pbx.c: Executing >>>> [8005551234@ivr-welcome:13] Read("SIP/CHANNELNAME-000138ac", >>>> "EarlyChoice,prompts/001,1,,0,0.01") in new stack >>>> [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >>>> maximum of 1 digits. >>>> [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/001.slin' (language 'fr') >>>> [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] app_read.c: User entered >>>> nothing. >>>> -- >>>> [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] pbx.c: Executing >>>> [8005551234@ivr-welcome:17] Read("SIP/CHANNELNAME-000138ac", >>>> "LanguageChoice,prompts/002,1,,,0.01") in new stack >>>> [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >>>> maximum of 1 digits. >>>> [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/002.slin' (language 'en') >>>> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: User entered >>>> nothing. >>>> -- >>>> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing >>>> [8005551234@ivr-welcome:21] Read("SIP/CHANNELNAME-000138ac", >>>> "LanguageChoice,prompts/002,1,,1,15") in new stack >>>> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >>>> maximum of 1 digits. >>>> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/002.slin' (language 'fr') >>>> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: User entered '1' >>>> -- >>>> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing >>>> [8005551234@ivr-welcome:34] Read("SIP/CHANNELNAME-000138ac", >>>> "ActivationHelp,prompts/902&prompts/063,1,,,15,") in new stack >>>> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/902.slin' (language 'en') >>>> [2022-02-08-06:18:16:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/063.slin' (language 'en') >>>> [2022-02-08-06:18:32:] VERBOSE[24858][C-00004037] app_read.c: User entered '3' >>>> -- >>>> [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] pbx.c: Executing >>>> [8005551234@ivr-MainMenu:2] Read("SIP/CHANNELNAME-000138ac", >>>> "ListenForZero,prompts/044,1,,,0.01") in new stack >>>> [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >>>> maximum of 1 digits. >>>> [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/044.slin' (language 'en') >>>> [2022-02-08-06:19:46:] VERBOSE[24858][C-00004037] app_read.c: User entered >>>> nothing. >>>> -- >>>> [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] pbx.c: Executing >>>> [8005551234@ivr-MainMenu:8] Read("SIP/CHANNELNAME-000138ac", >>>> "ListenForZero,prompts/045,1,,,0.01") in new stack >>>> [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >>>> maximum of 1 digits. >>>> [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/045.slin' (language 'en') >>>> [2022-02-08-06:19:57:] VERBOSE[24858][C-00004037] app_read.c: User entered >>>> nothing. >>>> -- >>>> [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] pbx.c: Executing >>>> [8005551234@ivr-MainMenu:11] Read("SIP/CHANNELNAME-000138ac", >>>> "ListenForZero,prompts/045a,1,,,0.01") in new stack >>>> [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >>>> maximum of 1 digits. >>>> [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/045a.slin' (language 'en') >>>> [2022-02-08-06:20:04:] VERBOSE[24858][C-00004037] app_read.c: User entered >>>> nothing. >>>> -- >>>> [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] pbx.c: Executing >>>> [8005551234@ivr-MainMenu:15] Read("SIP/CHANNELNAME-000138ac", >>>> "ListenForZero,prompts/046,1,,,0.01") in new stack >>>> [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >>>> maximum of 1 digits. >>>> [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/046.slin' (language 'en') >>>> [2022-02-08-06:20:09:] VERBOSE[24858][C-00004037] app_read.c: User entered >>>> nothing. >>>> -- >>>> [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] pbx.c: Executing >>>> [8005551234@ivr-MainMenu:39] Read("SIP/CHANNELNAME-000138ac", >>>> "ListenForZero,prompts/049,1,,,0.01") in new stack >>>> [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >>>> maximum of 1 digits. >>>> [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/049.slin' (language 'en') >>>> [2022-02-08-06:20:21:] VERBOSE[24858][C-00004037] app_read.c: User entered >>>> nothing. >>>> -- >>>> [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] pbx.c: Executing >>>> [8005551234@ivr-MainMenu:45] Read("SIP/CHANNELNAME-000138ac", >>>> "BalanceRepeatPressed,prompts/004h&prompts/004i&prompts/043,1,,,15") in new >>>> stack >>>> [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >>>> maximum of 1 digits. >>>> [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004h.slin' (language 'en') >>>> [2022-02-08-06:20:29:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004i.slin' (language 'en') >>>> [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] app_read.c: User entered '1' >>>> -- >>>> [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] pbx.c: Executing >>>> [8005551234@ivr-MainMenu:2] Read("SIP/CHANNELNAME-000138ac", >>>> "ListenForZero,prompts/044,1,,,0.01") in new stack >>>> [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >>>> maximum of 1 digits. >>>> [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/044.slin' (language 'en') >>>> [2022-02-08-06:20:42:] VERBOSE[24858][C-00004037] app_read.c: User entered >>>> nothing. >>>> -- >>>> [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] pbx.c: Executing >>>> [8005551234@ivr-MainMenu:8] Read("SIP/CHANNELNAME-000138ac", >>>> "ListenForZero,prompts/045,1,,,0.01") in new stack >>>> [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >>>> maximum of 1 digits. >>>> [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] file.c: >>>> <SIP/CHANNELNAME-000138ac> Playing 'prompts/045.slin' (language 'en') >>>> [2022-02-08-06:20:53:] VERBOSE[24858][C-00004037] app_read.c: User entered >>>> nothing. >>>> The event I want to peg is each of the ' app_read.c: User entered ' events, but >>>> they each need to correlate with the Read("SIP/CHANNELNAME ... event that >>>> precedes them. >>>> Here is a rule I've written specifically to attempt to catch a line containing >>>> Read("SIP/CHANNELNAME-000138ac", >>>> "ActivationHelp,prompts/902&prompts/063,1,,,15,") : >>>> (it's a bit of a mess because I've been trying differnt things to get it to work >>>> the way I need it to) >>>> (the regexp is probably overly elaborate, but [ http://regex101.com/ | >>>> regex101.com ] makes it too fun not to!) >>>> type=PAIR >>>> desc=IVR caller offered activation or statement inquiry >>>> ptype=RegExp >>>> action=create IVR_activation_offered_$4 ; \ >>>> write - $4 activation or statement inquiry offered >>>> pattern=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*Executing >>>> \[(.+)\@(flexiti-ivr-welcome:[0-9]{1,3})\] (.+)\(\"([A-Z]{3,10})\/(.+)\", >>>> \"ActivationHelp,prompts.* >>>> ptype2=regexp >>>> pattern2=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].* >>>> app_read.c: User entered '?(nothing|[0-9]*)'?.* >>>> desc2=Flexiti IVR - caller selection at activation or statement inquiry >>>> action2=assign %i -ivr ; \ >>>> write %p%i.csv $1,$2,$4,,,,,,,,,$5 ; closef %p%i.csv ; \ >>>> write - $4 ACTIVATION_STATEMENT %p %i $1 User entered: $5 >>>> This works, but then keeps matching the pattern2 throughout the call, whereas >>>> what I want is to match the ActivationHelp, then match the very first 'User >>>> entered' following that, and then stop matching for this rule. There will be >>>> other similar matches too (but not ' ActivationHelp '), and those too need to >>>> correlate with one and only the first ' User entered ' that follows them. >>>> I'm sure it's something simple I'm missing, but I've been banging my head >>>> against the wall on this and I just can't see where my error is. >>>> I'm still learning SEC, so there's a lot I just don't get, and this is right at >>>> the limits of my skills. >>>> Any and all advice would be gratefully appreciated. >>>> Jim >>>> -- >>>> Jim Van Meggelen >>>> ClearlyCore Inc. >>>> +1-416-639-6001 (DID) >>>> +1-877-253-2716 (Canada) >>>> +1-866-644-7729 (USA) >>>> +1-416-425-6111 x6001 >>>> [ mailto:jim...@cl... | jim...@cl... ] >>>> [ http://www.clearlycore.com/ | http://www.clearlycore.com ] >>>> Asterisk: The Definitive Guide >>>> FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: >>>> [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | >>>> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] >>>> _______________________________________________ >>>> Simple-evcorr-users mailing list >>>> [ mailto:Sim...@li... | >>>> Sim...@li... ] >>>> [ https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users | >>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users ] |
From: Risto V. <ris...@gm...> - 2022-03-31 14:36:09
|
hi Jim, can you provide an example of the log event that indicates the end of the call? kind regards, risto Kontakt Jim Van Meggelen (<jim...@cl...>) kirjutas kuupäeval N, 31. märts 2022 kell 15:45: > Risto, > > Thank you for the reply. > > You are correct that the C-[0-9a-f]{8} string uniquely identifies each > call, and is present on all log lines. > > We cannot safely assume the maximum call length; some calls could > reasonably be 30 minutes or even more. We will be able to identify the end > of call, though, so a pattern/rule can look for that. > > I'm very interested in your thoughts. Thanks again. > > > -- > Jim Van Meggelen > ClearlyCore Inc. > > > > +1-416-639-6001 (DID) > +1-877-253-2716 (Canada) > +1-866-644-7729 (USA) > +1-416-425-6111 x6001 > jim...@cl... > http://www.clearlycore.com > > > *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:* > https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf > > ------------------------------ > > *From: *"Risto Vaarandi" <ris...@gm...> > *To: *"Jim Van Meggelen" <jim...@cl...> > *Cc: *sim...@li... > *Sent: *Thursday, 31 March, 2022 05:48:44 > *Subject: *Re: [Simple-evcorr-users] Parsing Asterisk log files for > downstream reporting - so far so good! > > hi Jim, > > I do have couple of things in mind that might help addressing this issue, > but before coming up with any suggestions, may I ask some questions? As I > understand, the phone call is uniquely identified by the numeral that > follows the C character (C-00004037 in your example) which is present in > all log messages for that particular call. > > However, is there also a specific log message that denotes the end of the > call? If there is no message indicating the end-of-call, can we assume that > the call has ended if there have been no messages for that call ID for the > last 5 or 10 minutes? > > kind regards, > risto > > Kontakt Jim Van Meggelen (<jim...@cl...>) kirjutas > kuupäeval N, 31. märts 2022 kell 07:41: > >> I have an atypical use for SEC, in which I'm parsing Asterisk log files >> to produce CSV peg counts for downstream usage reporting. >> >> This is working well and I'm loving this tool! >> >> However, I've run into a problem that I cannot seem to find a way around. >> >> The log file produces events that are tied to a unique ID for each >> channel/call (C-00000000), and using that, plus the timestamp, I can >> produce output that is correlated with a call, and chronological. So far so >> good. >> >> However, there's one event that is difficult to work with, because it can >> happen more than once on a typical phone call, needs to correlate with an >> event that happens a few lines prior, but carries no identifying >> correlation. >> >> Here's an example of the output I'm working with (curated for brevity and >> relevance, and specific to a single unique ID; there are actually hundreds >> of lines of output per-call): >> >> [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-welcome:9] Read("SIP/CHANNELNAME-000138ac", >> "EarlyChoice,prompts/001,1,,0,0.01") in new stack >> [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/001.slin' (language 'en') >> [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] app_read.c: User >> entered nothing. >> -- >> [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-welcome:13] Read("SIP/CHANNELNAME-000138ac", >> "EarlyChoice,prompts/001,1,,0,0.01") in new stack >> [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/001.slin' (language 'fr') >> [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] app_read.c: User >> entered nothing. >> -- >> [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-welcome:17] Read("SIP/CHANNELNAME-000138ac", >> "LanguageChoice,prompts/002,1,,,0.01") in new stack >> [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/002.slin' (language 'en') >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: User >> entered nothing. >> -- >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-welcome:21] Read("SIP/CHANNELNAME-000138ac", >> "LanguageChoice,prompts/002,1,,1,15") in new stack >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/002.slin' (language 'fr') >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: User >> entered '1' >> -- >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-welcome:34] Read("SIP/CHANNELNAME-000138ac", >> "ActivationHelp,prompts/902&prompts/063,1,,,15,") in new stack >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/902.slin' (language 'en') >> [2022-02-08-06:18:16:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/063.slin' (language 'en') >> [2022-02-08-06:18:32:] VERBOSE[24858][C-00004037] app_read.c: User >> entered '3' >> -- >> [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:2] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/044,1,,,0.01") in new stack >> [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/044.slin' (language 'en') >> [2022-02-08-06:19:46:] VERBOSE[24858][C-00004037] app_read.c: User >> entered nothing. >> -- >> [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:8] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/045,1,,,0.01") in new stack >> [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/045.slin' (language 'en') >> [2022-02-08-06:19:57:] VERBOSE[24858][C-00004037] app_read.c: User >> entered nothing. >> -- >> [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:11] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/045a,1,,,0.01") in new stack >> [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/045a.slin' (language 'en') >> [2022-02-08-06:20:04:] VERBOSE[24858][C-00004037] app_read.c: User >> entered nothing. >> -- >> [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:15] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/046,1,,,0.01") in new stack >> [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/046.slin' (language 'en') >> [2022-02-08-06:20:09:] VERBOSE[24858][C-00004037] app_read.c: User >> entered nothing. >> -- >> [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:39] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/049,1,,,0.01") in new stack >> [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/049.slin' (language 'en') >> [2022-02-08-06:20:21:] VERBOSE[24858][C-00004037] app_read.c: User >> entered nothing. >> -- >> [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:45] Read("SIP/CHANNELNAME-000138ac", >> "BalanceRepeatPressed,prompts/004h&prompts/004i&prompts/043,1,,,15") in new >> stack >> [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004h.slin' (language 'en') >> [2022-02-08-06:20:29:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004i.slin' (language 'en') >> [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] app_read.c: User >> entered '1' >> -- >> [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:2] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/044,1,,,0.01") in new stack >> [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/044.slin' (language 'en') >> [2022-02-08-06:20:42:] VERBOSE[24858][C-00004037] app_read.c: User >> entered nothing. >> -- >> [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:8] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/045,1,,,0.01") in new stack >> [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/045.slin' (language 'en') >> [2022-02-08-06:20:53:] VERBOSE[24858][C-00004037] app_read.c: User >> entered nothing. >> >> >> The event I want to peg is each of the 'app_read.c: User entered' >> events, but they each need to correlate with the Read("SIP/CHANNELNAME >> ... event that precedes them. >> >> Here is a rule I've written specifically to attempt to catch a line >> containing >> Read("SIP/CHANNELNAME-000138ac", >> "ActivationHelp,prompts/902&prompts/063,1,,,15,"): >> (it's a bit of a mess because I've been trying differnt things to get it >> to work the way I need it to) >> (the regexp is probably overly elaborate, but regex101.com makes it too >> fun not to!) >> >> type=PAIR >> desc=IVR caller offered activation or statement inquiry >> ptype=RegExp >> action=create IVR_activation_offered_$4 ; \ >> write - $4 activation or statement inquiry offered >> pattern=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*Executing >> \[(.+)\@(flexiti-ivr-welcome:[0-9]{1,3})\] (.+)\(\"([A-Z]{3,10})\/(.+)\", >> \"ActivationHelp,prompts.* >> ptype2=regexp >> pattern2=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].* >> app_read.c: User entered '?(nothing|[0-9]*)'?.* >> desc2=Flexiti IVR - caller selection at activation or statement inquiry >> action2=assign %i -ivr ; \ >> write %p%i.csv $1,$2,$4,,,,,,,,,$5 ; closef %p%i.csv ; \ >> write - $4 ACTIVATION_STATEMENT %p %i $1 User entered: $5 >> >> This works, but then keeps matching the pattern2 throughout the call, >> whereas what I want is to match the ActivationHelp, then match the very >> first 'User entered' following that, and then stop matching for this rule. >> There will be other similar matches too (but not 'ActivationHelp'), and >> those too need to correlate with one and only the first 'User entered' >> that follows them. >> >> I'm sure it's something simple I'm missing, but I've been banging my head >> against the wall on this and I just can't see where my error is. >> >> I'm still learning SEC, so there's a lot I just don't get, and this is >> right at the limits of my skills. >> >> Any and all advice would be gratefully appreciated. >> >> Jim >> >> >> >> >> -- >> Jim Van Meggelen >> ClearlyCore Inc. >> >> >> >> +1-416-639-6001 (DID) >> +1-877-253-2716 (Canada) >> +1-866-644-7729 (USA) >> +1-416-425-6111 x6001 >> jim...@cl... >> http://www.clearlycore.com >> >> >> *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:* >> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf >> _______________________________________________ >> Simple-evcorr-users mailing list >> Sim...@li... >> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users >> > > |
From: Jim V. M. <jim...@cl...> - 2022-03-31 12:46:12
|
Risto, Thank you for the reply. You are correct that the C-[0-9a-f]{8} string uniquely identifies each call, and is present on all log lines. We cannot safely assume the maximum call length; some calls could reasonably be 30 minutes or even more. We will be able to identify the end of call, though, so a pattern/rule can look for that. I'm very interested in your thoughts. Thanks again. -- Jim Van Meggelen ClearlyCore Inc. +1-416-639-6001 (DID) +1-877-253-2716 (Canada) +1-866-644-7729 (USA) +1-416-425-6111 x6001 jim...@cl... [ http://www.clearlycore.com/ | http://www.clearlycore.com ] Asterisk: The Definitive Guide FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] > From: "Risto Vaarandi" <ris...@gm...> > To: "Jim Van Meggelen" <jim...@cl...> > Cc: sim...@li... > Sent: Thursday, 31 March, 2022 05:48:44 > Subject: Re: [Simple-evcorr-users] Parsing Asterisk log files for downstream > reporting - so far so good! > hi Jim, > I do have couple of things in mind that might help addressing this issue, but > before coming up with any suggestions, may I ask some questions? As I > understand, the phone call is uniquely identified by the numeral that follows > the C character (C-00004037 in your example) which is present in all log > messages for that particular call. > However, is there also a specific log message that denotes the end of the call? > If there is no message indicating the end-of-call, can we assume that the call > has ended if there have been no messages for that call ID for the last 5 or 10 > minutes? > kind regards, > risto > Kontakt Jim Van Meggelen (< [ mailto:jim...@cl... | > jim...@cl... ] >) kirjutas kuupäeval N, 31. märts 2022 kell > 07:41: >> I have an atypical use for SEC, in which I'm parsing Asterisk log files to >> produce CSV peg counts for downstream usage reporting. >> This is working well and I'm loving this tool! >> However, I've run into a problem that I cannot seem to find a way around. >> The log file produces events that are tied to a unique ID for each channel/call >> ( C-00000000 ), and using that, plus the timestamp, I can produce output that >> is correlated with a call, and chronological. So far so good. >> However, there's one event that is difficult to work with, because it can happen >> more than once on a typical phone call, needs to correlate with an event that >> happens a few lines prior, but carries no identifying correlation. >> Here's an example of the output I'm working with (curated for brevity and >> relevance, and specific to a single unique ID; there are actually hundreds of >> lines of output per-call): >> [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-welcome:9] Read("SIP/CHANNELNAME-000138ac", >> "EarlyChoice,prompts/001,1,,0,0.01") in new stack >> [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/001.slin' (language 'en') >> [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] app_read.c: User entered >> nothing. >> -- >> [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-welcome:13] Read("SIP/CHANNELNAME-000138ac", >> "EarlyChoice,prompts/001,1,,0,0.01") in new stack >> [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/001.slin' (language 'fr') >> [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] app_read.c: User entered >> nothing. >> -- >> [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-welcome:17] Read("SIP/CHANNELNAME-000138ac", >> "LanguageChoice,prompts/002,1,,,0.01") in new stack >> [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/002.slin' (language 'en') >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: User entered >> nothing. >> -- >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-welcome:21] Read("SIP/CHANNELNAME-000138ac", >> "LanguageChoice,prompts/002,1,,1,15") in new stack >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/002.slin' (language 'fr') >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: User entered '1' >> -- >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-welcome:34] Read("SIP/CHANNELNAME-000138ac", >> "ActivationHelp,prompts/902&prompts/063,1,,,15,") in new stack >> [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/902.slin' (language 'en') >> [2022-02-08-06:18:16:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/063.slin' (language 'en') >> [2022-02-08-06:18:32:] VERBOSE[24858][C-00004037] app_read.c: User entered '3' >> -- >> [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:2] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/044,1,,,0.01") in new stack >> [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/044.slin' (language 'en') >> [2022-02-08-06:19:46:] VERBOSE[24858][C-00004037] app_read.c: User entered >> nothing. >> -- >> [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:8] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/045,1,,,0.01") in new stack >> [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/045.slin' (language 'en') >> [2022-02-08-06:19:57:] VERBOSE[24858][C-00004037] app_read.c: User entered >> nothing. >> -- >> [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:11] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/045a,1,,,0.01") in new stack >> [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/045a.slin' (language 'en') >> [2022-02-08-06:20:04:] VERBOSE[24858][C-00004037] app_read.c: User entered >> nothing. >> -- >> [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:15] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/046,1,,,0.01") in new stack >> [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/046.slin' (language 'en') >> [2022-02-08-06:20:09:] VERBOSE[24858][C-00004037] app_read.c: User entered >> nothing. >> -- >> [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:39] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/049,1,,,0.01") in new stack >> [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/049.slin' (language 'en') >> [2022-02-08-06:20:21:] VERBOSE[24858][C-00004037] app_read.c: User entered >> nothing. >> -- >> [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:45] Read("SIP/CHANNELNAME-000138ac", >> "BalanceRepeatPressed,prompts/004h&prompts/004i&prompts/043,1,,,15") in new >> stack >> [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004h.slin' (language 'en') >> [2022-02-08-06:20:29:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/004i.slin' (language 'en') >> [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] app_read.c: User entered '1' >> -- >> [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:2] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/044,1,,,0.01") in new stack >> [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/044.slin' (language 'en') >> [2022-02-08-06:20:42:] VERBOSE[24858][C-00004037] app_read.c: User entered >> nothing. >> -- >> [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] pbx.c: Executing >> [8005551234@ivr-MainMenu:8] Read("SIP/CHANNELNAME-000138ac", >> "ListenForZero,prompts/045,1,,,0.01") in new stack >> [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] app_read.c: Accepting a >> maximum of 1 digits. >> [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] file.c: >> <SIP/CHANNELNAME-000138ac> Playing 'prompts/045.slin' (language 'en') >> [2022-02-08-06:20:53:] VERBOSE[24858][C-00004037] app_read.c: User entered >> nothing. >> The event I want to peg is each of the ' app_read.c: User entered ' events, but >> they each need to correlate with the Read("SIP/CHANNELNAME ... event that >> precedes them. >> Here is a rule I've written specifically to attempt to catch a line containing >> Read("SIP/CHANNELNAME-000138ac", >> "ActivationHelp,prompts/902&prompts/063,1,,,15,") : >> (it's a bit of a mess because I've been trying differnt things to get it to work >> the way I need it to) >> (the regexp is probably overly elaborate, but [ http://regex101.com/ | >> regex101.com ] makes it too fun not to!) >> type=PAIR >> desc=IVR caller offered activation or statement inquiry >> ptype=RegExp >> action=create IVR_activation_offered_$4 ; \ >> write - $4 activation or statement inquiry offered >> pattern=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*Executing >> \[(.+)\@(flexiti-ivr-welcome:[0-9]{1,3})\] (.+)\(\"([A-Z]{3,10})\/(.+)\", >> \"ActivationHelp,prompts.* >> ptype2=regexp >> pattern2=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].* >> app_read.c: User entered '?(nothing|[0-9]*)'?.* >> desc2=Flexiti IVR - caller selection at activation or statement inquiry >> action2=assign %i -ivr ; \ >> write %p%i.csv $1,$2,$4,,,,,,,,,$5 ; closef %p%i.csv ; \ >> write - $4 ACTIVATION_STATEMENT %p %i $1 User entered: $5 >> This works, but then keeps matching the pattern2 throughout the call, whereas >> what I want is to match the ActivationHelp, then match the very first 'User >> entered' following that, and then stop matching for this rule. There will be >> other similar matches too (but not ' ActivationHelp '), and those too need to >> correlate with one and only the first ' User entered ' that follows them. >> I'm sure it's something simple I'm missing, but I've been banging my head >> against the wall on this and I just can't see where my error is. >> I'm still learning SEC, so there's a lot I just don't get, and this is right at >> the limits of my skills. >> Any and all advice would be gratefully appreciated. >> Jim >> -- >> Jim Van Meggelen >> ClearlyCore Inc. >> +1-416-639-6001 (DID) >> +1-877-253-2716 (Canada) >> +1-866-644-7729 (USA) >> +1-416-425-6111 x6001 >> [ mailto:jim...@cl... | jim...@cl... ] >> [ http://www.clearlycore.com/ | http://www.clearlycore.com ] >> Asterisk: The Definitive Guide >> FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: >> [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | >> https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] >> _______________________________________________ >> Simple-evcorr-users mailing list >> [ mailto:Sim...@li... | >> Sim...@li... ] >> [ https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users | >> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users ] |
From: Risto V. <ris...@gm...> - 2022-03-31 09:49:03
|
hi Jim, I do have couple of things in mind that might help addressing this issue, but before coming up with any suggestions, may I ask some questions? As I understand, the phone call is uniquely identified by the numeral that follows the C character (C-00004037 in your example) which is present in all log messages for that particular call. However, is there also a specific log message that denotes the end of the call? If there is no message indicating the end-of-call, can we assume that the call has ended if there have been no messages for that call ID for the last 5 or 10 minutes? kind regards, risto Kontakt Jim Van Meggelen (<jim...@cl...>) kirjutas kuupäeval N, 31. märts 2022 kell 07:41: > I have an atypical use for SEC, in which I'm parsing Asterisk log files to > produce CSV peg counts for downstream usage reporting. > > This is working well and I'm loving this tool! > > However, I've run into a problem that I cannot seem to find a way around. > > The log file produces events that are tied to a unique ID for each > channel/call (C-00000000), and using that, plus the timestamp, I can > produce output that is correlated with a call, and chronological. So far so > good. > > However, there's one event that is difficult to work with, because it can > happen more than once on a typical phone call, needs to correlate with an > event that happens a few lines prior, but carries no identifying > correlation. > > Here's an example of the output I'm working with (curated for brevity and > relevance, and specific to a single unique ID; there are actually hundreds > of lines of output per-call): > > [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] pbx.c: Executing > [8005551234@ivr-welcome:9] Read("SIP/CHANNELNAME-000138ac", > "EarlyChoice,prompts/001,1,,0,0.01") in new stack > [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] app_read.c: Accepting a > maximum of 1 digits. > [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/001.slin' (language 'en') > [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] app_read.c: User entered > nothing. > -- > [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] pbx.c: Executing > [8005551234@ivr-welcome:13] Read("SIP/CHANNELNAME-000138ac", > "EarlyChoice,prompts/001,1,,0,0.01") in new stack > [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] app_read.c: Accepting a > maximum of 1 digits. > [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/001.slin' (language 'fr') > [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] app_read.c: User entered > nothing. > -- > [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] pbx.c: Executing > [8005551234@ivr-welcome:17] Read("SIP/CHANNELNAME-000138ac", > "LanguageChoice,prompts/002,1,,,0.01") in new stack > [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] app_read.c: Accepting a > maximum of 1 digits. > [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/002.slin' (language 'en') > [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: User entered > nothing. > -- > [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing > [8005551234@ivr-welcome:21] Read("SIP/CHANNELNAME-000138ac", > "LanguageChoice,prompts/002,1,,1,15") in new stack > [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: Accepting a > maximum of 1 digits. > [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/002.slin' (language 'fr') > [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: User entered > '1' > -- > [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing > [8005551234@ivr-welcome:34] Read("SIP/CHANNELNAME-000138ac", > "ActivationHelp,prompts/902&prompts/063,1,,,15,") in new stack > [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/902.slin' (language 'en') > [2022-02-08-06:18:16:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/063.slin' (language 'en') > [2022-02-08-06:18:32:] VERBOSE[24858][C-00004037] app_read.c: User entered > '3' > -- > [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] pbx.c: Executing > [8005551234@ivr-MainMenu:2] Read("SIP/CHANNELNAME-000138ac", > "ListenForZero,prompts/044,1,,,0.01") in new stack > [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] app_read.c: Accepting a > maximum of 1 digits. > [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/044.slin' (language 'en') > [2022-02-08-06:19:46:] VERBOSE[24858][C-00004037] app_read.c: User entered > nothing. > -- > [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] pbx.c: Executing > [8005551234@ivr-MainMenu:8] Read("SIP/CHANNELNAME-000138ac", > "ListenForZero,prompts/045,1,,,0.01") in new stack > [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] app_read.c: Accepting a > maximum of 1 digits. > [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/045.slin' (language 'en') > [2022-02-08-06:19:57:] VERBOSE[24858][C-00004037] app_read.c: User entered > nothing. > -- > [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] pbx.c: Executing > [8005551234@ivr-MainMenu:11] Read("SIP/CHANNELNAME-000138ac", > "ListenForZero,prompts/045a,1,,,0.01") in new stack > [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] app_read.c: Accepting a > maximum of 1 digits. > [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/045a.slin' (language 'en') > [2022-02-08-06:20:04:] VERBOSE[24858][C-00004037] app_read.c: User entered > nothing. > -- > [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] pbx.c: Executing > [8005551234@ivr-MainMenu:15] Read("SIP/CHANNELNAME-000138ac", > "ListenForZero,prompts/046,1,,,0.01") in new stack > [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] app_read.c: Accepting a > maximum of 1 digits. > [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/046.slin' (language 'en') > [2022-02-08-06:20:09:] VERBOSE[24858][C-00004037] app_read.c: User entered > nothing. > -- > [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] pbx.c: Executing > [8005551234@ivr-MainMenu:39] Read("SIP/CHANNELNAME-000138ac", > "ListenForZero,prompts/049,1,,,0.01") in new stack > [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] app_read.c: Accepting a > maximum of 1 digits. > [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/049.slin' (language 'en') > [2022-02-08-06:20:21:] VERBOSE[24858][C-00004037] app_read.c: User entered > nothing. > -- > [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] pbx.c: Executing > [8005551234@ivr-MainMenu:45] Read("SIP/CHANNELNAME-000138ac", > "BalanceRepeatPressed,prompts/004h&prompts/004i&prompts/043,1,,,15") in new > stack > [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] app_read.c: Accepting a > maximum of 1 digits. > [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/004h.slin' (language 'en') > [2022-02-08-06:20:29:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/004i.slin' (language 'en') > [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] app_read.c: User entered > '1' > -- > [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] pbx.c: Executing > [8005551234@ivr-MainMenu:2] Read("SIP/CHANNELNAME-000138ac", > "ListenForZero,prompts/044,1,,,0.01") in new stack > [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] app_read.c: Accepting a > maximum of 1 digits. > [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/044.slin' (language 'en') > [2022-02-08-06:20:42:] VERBOSE[24858][C-00004037] app_read.c: User entered > nothing. > -- > [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] pbx.c: Executing > [8005551234@ivr-MainMenu:8] Read("SIP/CHANNELNAME-000138ac", > "ListenForZero,prompts/045,1,,,0.01") in new stack > [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] app_read.c: Accepting a > maximum of 1 digits. > [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] file.c: > <SIP/CHANNELNAME-000138ac> Playing 'prompts/045.slin' (language 'en') > [2022-02-08-06:20:53:] VERBOSE[24858][C-00004037] app_read.c: User entered > nothing. > > > The event I want to peg is each of the 'app_read.c: User entered' events, > but they each need to correlate with the Read("SIP/CHANNELNAME ... event > that precedes them. > > Here is a rule I've written specifically to attempt to catch a line > containing > Read("SIP/CHANNELNAME-000138ac", > "ActivationHelp,prompts/902&prompts/063,1,,,15,"): > (it's a bit of a mess because I've been trying differnt things to get it > to work the way I need it to) > (the regexp is probably overly elaborate, but regex101.com makes it too > fun not to!) > > type=PAIR > desc=IVR caller offered activation or statement inquiry > ptype=RegExp > action=create IVR_activation_offered_$4 ; \ > write - $4 activation or statement inquiry offered > pattern=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*Executing > \[(.+)\@(flexiti-ivr-welcome:[0-9]{1,3})\] (.+)\(\"([A-Z]{3,10})\/(.+)\", > \"ActivationHelp,prompts.* > ptype2=regexp > pattern2=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].* > app_read.c: User entered '?(nothing|[0-9]*)'?.* > desc2=Flexiti IVR - caller selection at activation or statement inquiry > action2=assign %i -ivr ; \ > write %p%i.csv $1,$2,$4,,,,,,,,,$5 ; closef %p%i.csv ; \ > write - $4 ACTIVATION_STATEMENT %p %i $1 User entered: $5 > > This works, but then keeps matching the pattern2 throughout the call, > whereas what I want is to match the ActivationHelp, then match the very > first 'User entered' following that, and then stop matching for this rule. > There will be other similar matches too (but not 'ActivationHelp'), and > those too need to correlate with one and only the first 'User entered' > that follows them. > > I'm sure it's something simple I'm missing, but I've been banging my head > against the wall on this and I just can't see where my error is. > > I'm still learning SEC, so there's a lot I just don't get, and this is > right at the limits of my skills. > > Any and all advice would be gratefully appreciated. > > Jim > > > > > -- > Jim Van Meggelen > ClearlyCore Inc. > > > > +1-416-639-6001 (DID) > +1-877-253-2716 (Canada) > +1-866-644-7729 (USA) > +1-416-425-6111 x6001 > jim...@cl... > http://www.clearlycore.com > > > *Asterisk: The Definitive GuideFIFTH EDITION NOW AVAILABLE TO DOWNLOAD:* > https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf > _______________________________________________ > Simple-evcorr-users mailing list > Sim...@li... > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users > |
From: Jim V. M. <jim...@cl...> - 2022-03-30 22:30:19
|
I have an atypical use for SEC, in which I'm parsing Asterisk log files to produce CSV peg counts for downstream usage reporting. This is working well and I'm loving this tool! However, I've run into a problem that I cannot seem to find a way around. The log file produces events that are tied to a unique ID for each channel/call ( C-00000000 ), and using that, plus the timestamp, I can produce output that is correlated with a call, and chronological. So far so good. However, there's one event that is difficult to work with, because it can happen more than once on a typical phone call, needs to correlate with an event that happens a few lines prior, but carries no identifying correlation. Here's an example of the output I'm working with (curated for brevity and relevance, and specific to a single unique ID; there are actually hundreds of lines of output per-call): [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-welcome:9] Read("SIP/CHANNELNAME-000138ac", "EarlyChoice,prompts/001,1,,0,0.01") in new stack [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] app_read.c: Accepting a maximum of 1 digits. [2022-02-08-06:17:53:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/001.slin' (language 'en') [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] app_read.c: User entered nothing. -- [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-welcome:13] Read("SIP/CHANNELNAME-000138ac", "EarlyChoice,prompts/001,1,,0,0.01") in new stack [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] app_read.c: Accepting a maximum of 1 digits. [2022-02-08-06:17:55:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/001.slin' (language 'fr') [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] app_read.c: User entered nothing. -- [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-welcome:17] Read("SIP/CHANNELNAME-000138ac", "LanguageChoice,prompts/002,1,,,0.01") in new stack [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] app_read.c: Accepting a maximum of 1 digits. [2022-02-08-06:17:58:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/002.slin' (language 'en') [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: User entered nothing. -- [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-welcome:21] Read("SIP/CHANNELNAME-000138ac", "LanguageChoice,prompts/002,1,,1,15") in new stack [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: Accepting a maximum of 1 digits. [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/002.slin' (language 'fr') [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] app_read.c: User entered '1' -- [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-welcome:34] Read("SIP/CHANNELNAME-000138ac", "ActivationHelp,prompts/902&prompts/063,1,,,15,") in new stack [2022-02-08-06:18:01:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/902.slin' (language 'en') [2022-02-08-06:18:16:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/063.slin' (language 'en') [2022-02-08-06:18:32:] VERBOSE[24858][C-00004037] app_read.c: User entered '3' -- [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-MainMenu:2] Read("SIP/CHANNELNAME-000138ac", "ListenForZero,prompts/044,1,,,0.01") in new stack [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] app_read.c: Accepting a maximum of 1 digits. [2022-02-08-06:19:36:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/044.slin' (language 'en') [2022-02-08-06:19:46:] VERBOSE[24858][C-00004037] app_read.c: User entered nothing. -- [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-MainMenu:8] Read("SIP/CHANNELNAME-000138ac", "ListenForZero,prompts/045,1,,,0.01") in new stack [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] app_read.c: Accepting a maximum of 1 digits. [2022-02-08-06:19:55:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/045.slin' (language 'en') [2022-02-08-06:19:57:] VERBOSE[24858][C-00004037] app_read.c: User entered nothing. -- [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-MainMenu:11] Read("SIP/CHANNELNAME-000138ac", "ListenForZero,prompts/045a,1,,,0.01") in new stack [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] app_read.c: Accepting a maximum of 1 digits. [2022-02-08-06:20:02:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/045a.slin' (language 'en') [2022-02-08-06:20:04:] VERBOSE[24858][C-00004037] app_read.c: User entered nothing. -- [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-MainMenu:15] Read("SIP/CHANNELNAME-000138ac", "ListenForZero,prompts/046,1,,,0.01") in new stack [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] app_read.c: Accepting a maximum of 1 digits. [2022-02-08-06:20:07:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/046.slin' (language 'en') [2022-02-08-06:20:09:] VERBOSE[24858][C-00004037] app_read.c: User entered nothing. -- [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-MainMenu:39] Read("SIP/CHANNELNAME-000138ac", "ListenForZero,prompts/049,1,,,0.01") in new stack [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] app_read.c: Accepting a maximum of 1 digits. [2022-02-08-06:20:18:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/049.slin' (language 'en') [2022-02-08-06:20:21:] VERBOSE[24858][C-00004037] app_read.c: User entered nothing. -- [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-MainMenu:45] Read("SIP/CHANNELNAME-000138ac", "BalanceRepeatPressed,prompts/004h&prompts/004i&prompts/043,1,,,15") in new stack [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] app_read.c: Accepting a maximum of 1 digits. [2022-02-08-06:20:26:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/004h.slin' (language 'en') [2022-02-08-06:20:29:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/004i.slin' (language 'en') [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] app_read.c: User entered '1' -- [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-MainMenu:2] Read("SIP/CHANNELNAME-000138ac", "ListenForZero,prompts/044,1,,,0.01") in new stack [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] app_read.c: Accepting a maximum of 1 digits. [2022-02-08-06:20:32:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/044.slin' (language 'en') [2022-02-08-06:20:42:] VERBOSE[24858][C-00004037] app_read.c: User entered nothing. -- [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] pbx.c: Executing [8005551234@ivr-MainMenu:8] Read("SIP/CHANNELNAME-000138ac", "ListenForZero,prompts/045,1,,,0.01") in new stack [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] app_read.c: Accepting a maximum of 1 digits. [2022-02-08-06:20:51:] VERBOSE[24858][C-00004037] file.c: <SIP/CHANNELNAME-000138ac> Playing 'prompts/045.slin' (language 'en') [2022-02-08-06:20:53:] VERBOSE[24858][C-00004037] app_read.c: User entered nothing. The event I want to peg is each of the ' app_read.c: User entered ' events, but they each need to correlate with the Read("SIP/CHANNELNAME ... event that precedes them. Here is a rule I've written specifically to attempt to catch a line containing Read("SIP/CHANNELNAME-000138ac", "ActivationHelp,prompts/902&prompts/063,1,,,15,") : (it's a bit of a mess because I've been trying differnt things to get it to work the way I need it to) (the regexp is probably overly elaborate, but regex101.com makes it too fun not to!) type=PAIR desc=IVR caller offered activation or statement inquiry ptype=RegExp action=create IVR_activation_offered_$4 ; \ write - $4 activation or statement inquiry offered pattern=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].*Executing \[(.+)\@(flexiti-ivr-welcome:[0-9]{1,3})\] (.+)\(\"([A-Z]{3,10})\/(.+)\", \"ActivationHelp,prompts.* ptype2=regexp pattern2=^.*\[(20[234][0-9]-[0-9]{2}-[0-3][0-9])-([0-2][0-9]:[0-5][0-9]:[0-5][0-9]).*VERBOSE\[([0-9]{3,6})\]\[(C-[0-9a-f]{8})\].* app_read.c: User entered '?(nothing|[0-9]*)'?.* desc2=Flexiti IVR - caller selection at activation or statement inquiry action2=assign %i -ivr ; \ write %p%i.csv $1,$2,$4,,,,,,,,,$5 ; closef %p%i.csv ; \ write - $4 ACTIVATION_STATEMENT %p %i $1 User entered: $5 This works, but then keeps matching the pattern2 throughout the call, whereas what I want is to match the ActivationHelp, then match the very first 'User entered' following that, and then stop matching for this rule. There will be other similar matches too (but not ' ActivationHelp '), and those too need to correlate with one and only the first ' User entered ' that follows them. I'm sure it's something simple I'm missing, but I've been banging my head against the wall on this and I just can't see where my error is. I'm still learning SEC, so there's a lot I just don't get, and this is right at the limits of my skills. Any and all advice would be gratefully appreciated. Jim -- Jim Van Meggelen ClearlyCore Inc. +1-416-639-6001 (DID) +1-877-253-2716 (Canada) +1-866-644-7729 (USA) +1-416-425-6111 x6001 jim...@cl... [ http://www.clearlycore.com/ | http://www.clearlycore.com ] Asterisk: The Definitive Guide FIFTH EDITION NOW AVAILABLE TO DOWNLOAD: [ https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf | https://cdn.oreillystatic.com/pdf/Asterisk_The_Definitive_Guide.pdf ] |
From: Risto V. <ris...@gm...> - 2021-11-20 11:21:01
|
hi all, just a small note -- the JSON parsing example in the sec rule repository has been updated: https://github.com/simple-evcorr/rulesets/tree/master/parsing-json Instead of the perl JSON wrapper module which will call some backend module, the updated example directly employs fast and efficient Cpanel::JSON::XS module (the module has been packaged for major Linux and BSD distributions and can thus be easily installed). Due to this update, a new ruleset tarball has also been released: https://github.com/simple-evcorr/rulesets/releases kind regards, risto |
From: Risto V. <ris...@gm...> - 2021-07-08 15:29:00
|
> > Beautiful..thank you SO much Risto. I love the fact that you not only provide something I can use right away, but took the time to explain how it works. I learn more about SEC every time I email the list. > > Thanks again! > > James > It is great to hear that the examples from my post were helpful :) risto |
From: James L. <jl...@sl...> - 2021-07-08 15:20:20
|
Beautiful..thank you SO much Risto. I love the fact that you not only provide something I can use right away, but took the time to explain how it works. I learn more about SEC every time I email the list. Thanks again! James On Thu, 2021-07-08 at 13:08 +0300, Risto Vaarandi wrote: > hi James, > yes, you can employ the EventGroup rule for addressing this task, > andlet me provide two slightly different solutions below. The first > andsomewhat simpler solution looks like this: > type=EventGroupptype=RegExppattern=^\d+\.\d+ \S+ ([\d.]+) \d+ > ([\d.]+) 445 \d+\.\d+\\\\pipe\\\\lsass netlogon > DsrEnumerateDomainTrustscontext=!ATTACKER_$1_DC_$2count=alias > ATTACKER_$1 ATTACKER_$1_DC_$2init=create ATTACKER_$1end=delete > ATTACKER_$1desc=Attacker $1 has enumerated domain trust against > several domain controllersaction=write - %sthresh=3window=30 > The regular expression pattern of the rule has been designed to > matchevents in the format you have described, except the expression > assumesthat instead of SameIP and DifferentIP there are IPv4 > addresses in theevent. For example: > 1625677087.655857 CAY8g9D3jAKFwhD02 10.1.1.1 59143 192.168.1.1 > 4450.017172 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts > Also, it is assumed that the first IP address (10.1.1.1 in the > aboveevent) belongs to the attacker, while the second IP > address(192.168.1.1) belongs to the domain controller. The above > EventGrouprule runs event correlation operations that write alerts to > standardoutput if the same attacker accesses three different > domaincontrollers within 30 seconds. Since the 'desc' field of the > rule hasa value with $1 variable (IP address of the attacker), the > rule runs aseparate event correlation operation for each attacker, > with eachoperation having its own event counter. > In order to make sure that access to each domain controller from > thesame attacker is counted just once by the corresponding operation, > theabove rule employs context aliases which refer to the same > contextdata structure. The use of just one context with multiple > alias namesfor each attacker eases the garbage collection procedure > when eventcorrelation operation for the attacker is complete. Each > time the ruleobserves an event for a new attacker, it creates a new > eventcorrelation operation for this attacker, and the initialization > action'create ATTACKER_$1' provided with the 'init' field is > executed. Thisaction will create the context ATTACKER_<attackerip> > that all futurealias names for this attacker will refer to. > When an event for attacker <attackerip> and domain controller > <dcip>is observed, the rule will check if the context > aliasATTACKER_<attackerip>_DC_<dcip> exists (that check is > implemented bythe 'context' field of the rule). If that alias is not > present, theevent gets processed, and the event correlation operation > that isexecuting for attacker <attackerip> will create a new alias > with thatname, with the alias pointing to context > ATTACKER_<attackerip> (therelevant action is defined in the 'count' > field of the rule). Forexample, when the event > 1625677087.655857 CAY8g9D3jAKFwhD02 10.1.1.1 59143 192.168.1.1 > 4450.017172 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts > is observed, the alias ATTACKER_10.1.1.1_DC_192.168.1.1 gets > createdwhich points to context ATTACKER_10.1.1.1. When the event for > theattacker 10.1.1.1 and domain controller 192.168.1.1 appears again, > theboolean expression !ATTACKER_10.1.1.1_DC_192.168.1.1 provided with > the'context' field evaluates false, and the event is no longer > matchingthe rule. This ensures that when multiple events for the same > domaincontroller and attacker combination appear, only the first > event iscounted. Therefore, if an event correlation operation is > executing forattacker 10.1.1.1 and its event counter reaches the > value of 3, threeevents have been observed for attacker 10.1.1.1 for > different domaincontroller IP addresses. When the event correlation > operation for someattacker IP terminates, the garbage collection > action given with the'end' field is executed, and 'delete > ATTACKER_$1' action will removethe context ATTACKER_<attackerip> and > all its aliases. > In order to illustrate the work of this rule, consider the > followingfour events for attacker 10.1.1.1 and three different > domaincontrollers: > 1625677087.655857 CAY8g9D3jAKFwhD02 10.1.1.1 59143 192.168.1.1 > 4450.017172 \\pipe\\lsass netlogon > DsrEnumerateDomainTrusts1625677087.655860 CAY8g9D3jAKFwhD02 10.1.1.1 > 59143 192.168.1.1 4450.017172 \\pipe\\lsass netlogon > DsrEnumerateDomainTrusts1625677087.744290 C6mmd42niEVcW1djY5 10.1.1.1 > 59144 192.168.1.2 4450.003689 \\pipe\\lsass netlogon > DsrEnumerateDomainTrusts1625677087.776183 CamRzj1Y20KA9cv9Mi 10.1.1.1 > 59145 192.168.1.3 4450.003968 \\pipe\\lsass netlogon > DsrEnumerateDomainTrusts > Here is the debug log of SEC which illustrates the event correlation > process: > 1625677087.655857 CAY8g9D3jAKFwhD02 10.1.1.1 59143 192.168.1.1 > 4450.017172 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts <- > firstinput eventCreating context 'ATTACKER_10.1.1.1' <- an event > correlationoperation starts for attacker 10.1.1.1 and context > ATTACKER_10.1.1.1is createdCreating alias > 'ATTACKER_10.1.1.1_DC_192.168.1.1' for > context'ATTACKER_10.1.1.1' <- a context alias for attacker > 10.1.1.1 andDC 192.168.1.1 is created and event counter of the > operation for10.1.1.1 is incremented to 1 > 1625677087.655860 CAY8g9D3jAKFwhD02 10.1.1.1 59143 192.168.1.1 > 4450.017172 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts <- > secondinput event: since the alias ATTACKER_10.1.1.1_DC_192.168.1.1 > alreadyexists, the input event is not processed > 1625677087.744290 C6mmd42niEVcW1djY5 10.1.1.1 59144 192.168.1.2 > 4450.003689 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts <- > thirdinput eventCreating alias 'ATTACKER_10.1.1.1_DC_192.168.1.2' for > context'ATTACKER_10.1.1.1' <- a context alias for attacker > 10.1.1.1 andDC 192.168.1.2 is created and event counter of the > operation for10.1.1.1 is incremented to 2 > 1625677087.776183 CamRzj1Y20KA9cv9Mi 10.1.1.1 59145 192.168.1.3 > 4450.003968 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts <- > fourthinput eventCreating alias 'ATTACKER_10.1.1.1_DC_192.168.1.3' > for context'ATTACKER_10.1.1.1' <- a context alias for attacker > 10.1.1.1 andDC 192.168.1.3 is created and event counter of the > operation for10.1.1.1 is incremented to 3 > Writing event 'Attacker 10.1.1.1 has enumerated domain trust > againstseveral domain controllers' to file '-'Attacker 10.1.1.1 has > enumerated domain trust against several domaincontrollers <- > since the counter of the operation for 10.1.1.1 hasreached the value > of 3, write an alert message to standard output > Deleting context 'ATTACKER_10.1.1.1' <- the event > correlationoperation for attacker 10.1.1.1 is terminating and > dropping thecontext ATTACKER_10.1.1.1 and its aliasesContext > 'ATTACKER_10.1.1.1' deletedContext 'ATTACKER_10.1.1.1_DC_192.168.1.1' > deletedContext 'ATTACKER_10.1.1.1_DC_192.168.1.3' deletedContext > 'ATTACKER_10.1.1.1_DC_192.168.1.2' deleted > The above EventGroup rule has one drawback -- if the threshold of > 3events is not reached during 30 seconds for an event > correlationoperation, the 30 second event correlation window of the > operationwill start to slide, and the operation could potentially > exist for alonger period of time. However, since aliases do not have > any fixedlifetime, they will continue to suppress events for a > givencombination of attacker and domain controller, even if the first > eventfor this combination is already outside of the 30 second > eventcorrelation window. If this is an issue in your environment and > youwant to address it, you could enhance the 'count' field of the > aboverule as follows:alias ATTACKER_$1 ATTACKER_$1_DC_$2; create > ATTACKER_$1_DC_$2_LIFETIME30 ( unalias ATTACKER_$1_DC_$2 )Here is the > full rule definition with the modified 'count' field: > type=EventGroupptype=RegExppattern=^\d+\.\d+ \S+ ([\d.]+) \d+ > ([\d.]+) 445 \d+\.\d+\\\\pipe\\\\lsass netlogon > DsrEnumerateDomainTrustscontext=!ATTACKER_$1_DC_$2count=alias > ATTACKER_$1 ATTACKER_$1_DC_$2; \ create > ATTACKER_$1_DC_$2_LIFETIME 30 ( unalias ATTACKER_$1_DC_$2 > )init=create ATTACKER_$1end=delete ATTACKER_$1desc=Attacker $1 has > enumerated domain trust against several domain > controllersaction=write - %sthresh=3window=30 > With this change, a separate > contextATTACKER_<attackerip>_DC_<dcip>_LIFETIME will be set up for > each aliasafter the creation of the alias. The context will exist for > 30 seconds(the size of the event correlation window) and will drop > the aliaswith 'unalias' action when context lifetime expires. > Therefore, eachalias will exist for 30 seconds only, and when the > event thattriggered its creation moves outside the event correlation > window, theoperation is able to count the event for the given > attacker and domaincontroller combination again. Note that although > the *_LIFETIMEcontexts are not removed when the operation terminates, > they will timeout quite quickly, and executing 'unalias' action for a > non-existingalias name is no-op (you will only get a debug-level > message into SEClog that the alias does not exist). > One final note -- you can find a similar solution for this scenario > ina paper that is available from the SEC home page: > https://ristov.github.io/publications/cogsima15-sec-web.pdf > > Hope this helps,risto > > > Kontakt James Lay (<jl...@sl...>) kirjutas kuupäeval > K,7. juuli 2021 kell 22:23: > > Hey all. > > So....been looking at EventGroups thinking this might fit the bill, > > but I'm not sure so I'm coming to you folks. I have the below: > > 1625677087.655857 CAY8g9D3jAKFwhD02 SameIP 59143 DifferentIP 445 > > 0.017172 \\pipe\\lsass netlogon > > DsrEnumerateDomainTrusts1625677087.744290 C6mmd42niEVcW1djY5 SameIP > > 59144 DifferentIP 445 0.003689 \\pipe\\lsass netlogon > > DsrEnumerateDomainTrusts1625677087.776183 CamRzj1Y20KA9cv9Mi SameIP > > 59145 DifferentIP 445 0.003968 \\pipe\\lsass netlogon > > DsrEnumerateDomainTrusts1625677203.779593 CF4JvS21lo6Beh3jUb SameIP > > 59149 DifferentIP 445 0.001475 \\pipe\\lsass netlogon > > DsrEnumerateDomainTrusts1625677203.892484 C6LZI2ooakJbNI6vj SameIP > > 59150 DifferentIP 445 0.023818 \\pipe\\lsass netlogon > > DsrEnumerateDomainTrusts > > My goal is to flag on this type of behaviour (a single IP > > enumerating domain trust against several domain controllers in a > > small space of time). Anyone have any tips on how to implement > > something like this? Is EventGroup not what I need? Thank you. > > James_______________________________________________Simple-evcorr- > > users mailing lis...@li... > > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users |
From: Risto V. <ris...@gm...> - 2021-07-08 10:08:27
|
hi James, yes, you can employ the EventGroup rule for addressing this task, and let me provide two slightly different solutions below. The first and somewhat simpler solution looks like this: type=EventGroup ptype=RegExp pattern=^\d+\.\d+ \S+ ([\d.]+) \d+ ([\d.]+) 445 \d+\.\d+ \\\\pipe\\\\lsass netlogon DsrEnumerateDomainTrusts context=!ATTACKER_$1_DC_$2 count=alias ATTACKER_$1 ATTACKER_$1_DC_$2 init=create ATTACKER_$1 end=delete ATTACKER_$1 desc=Attacker $1 has enumerated domain trust against several domain controllers action=write - %s thresh=3 window=30 The regular expression pattern of the rule has been designed to match events in the format you have described, except the expression assumes that instead of SameIP and DifferentIP there are IPv4 addresses in the event. For example: 1625677087.655857 CAY8g9D3jAKFwhD02 10.1.1.1 59143 192.168.1.1 445 0.017172 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts Also, it is assumed that the first IP address (10.1.1.1 in the above event) belongs to the attacker, while the second IP address (192.168.1.1) belongs to the domain controller. The above EventGroup rule runs event correlation operations that write alerts to standard output if the same attacker accesses three different domain controllers within 30 seconds. Since the 'desc' field of the rule has a value with $1 variable (IP address of the attacker), the rule runs a separate event correlation operation for each attacker, with each operation having its own event counter. In order to make sure that access to each domain controller from the same attacker is counted just once by the corresponding operation, the above rule employs context aliases which refer to the same context data structure. The use of just one context with multiple alias names for each attacker eases the garbage collection procedure when event correlation operation for the attacker is complete. Each time the rule observes an event for a new attacker, it creates a new event correlation operation for this attacker, and the initialization action 'create ATTACKER_$1' provided with the 'init' field is executed. This action will create the context ATTACKER_<attackerip> that all future alias names for this attacker will refer to. When an event for attacker <attackerip> and domain controller <dcip> is observed, the rule will check if the context alias ATTACKER_<attackerip>_DC_<dcip> exists (that check is implemented by the 'context' field of the rule). If that alias is not present, the event gets processed, and the event correlation operation that is executing for attacker <attackerip> will create a new alias with that name, with the alias pointing to context ATTACKER_<attackerip> (the relevant action is defined in the 'count' field of the rule). For example, when the event 1625677087.655857 CAY8g9D3jAKFwhD02 10.1.1.1 59143 192.168.1.1 445 0.017172 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts is observed, the alias ATTACKER_10.1.1.1_DC_192.168.1.1 gets created which points to context ATTACKER_10.1.1.1. When the event for the attacker 10.1.1.1 and domain controller 192.168.1.1 appears again, the boolean expression !ATTACKER_10.1.1.1_DC_192.168.1.1 provided with the 'context' field evaluates false, and the event is no longer matching the rule. This ensures that when multiple events for the same domain controller and attacker combination appear, only the first event is counted. Therefore, if an event correlation operation is executing for attacker 10.1.1.1 and its event counter reaches the value of 3, three events have been observed for attacker 10.1.1.1 for different domain controller IP addresses. When the event correlation operation for some attacker IP terminates, the garbage collection action given with the 'end' field is executed, and 'delete ATTACKER_$1' action will remove the context ATTACKER_<attackerip> and all its aliases. In order to illustrate the work of this rule, consider the following four events for attacker 10.1.1.1 and three different domain controllers: 1625677087.655857 CAY8g9D3jAKFwhD02 10.1.1.1 59143 192.168.1.1 445 0.017172 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts 1625677087.655860 CAY8g9D3jAKFwhD02 10.1.1.1 59143 192.168.1.1 445 0.017172 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts 1625677087.744290 C6mmd42niEVcW1djY5 10.1.1.1 59144 192.168.1.2 445 0.003689 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts 1625677087.776183 CamRzj1Y20KA9cv9Mi 10.1.1.1 59145 192.168.1.3 445 0.003968 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts Here is the debug log of SEC which illustrates the event correlation process: 1625677087.655857 CAY8g9D3jAKFwhD02 10.1.1.1 59143 192.168.1.1 445 0.017172 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts <- first input event Creating context 'ATTACKER_10.1.1.1' <- an event correlation operation starts for attacker 10.1.1.1 and context ATTACKER_10.1.1.1 is created Creating alias 'ATTACKER_10.1.1.1_DC_192.168.1.1' for context 'ATTACKER_10.1.1.1' <- a context alias for attacker 10.1.1.1 and DC 192.168.1.1 is created and event counter of the operation for 10.1.1.1 is incremented to 1 1625677087.655860 CAY8g9D3jAKFwhD02 10.1.1.1 59143 192.168.1.1 445 0.017172 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts <- second input event: since the alias ATTACKER_10.1.1.1_DC_192.168.1.1 already exists, the input event is not processed 1625677087.744290 C6mmd42niEVcW1djY5 10.1.1.1 59144 192.168.1.2 445 0.003689 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts <- third input event Creating alias 'ATTACKER_10.1.1.1_DC_192.168.1.2' for context 'ATTACKER_10.1.1.1' <- a context alias for attacker 10.1.1.1 and DC 192.168.1.2 is created and event counter of the operation for 10.1.1.1 is incremented to 2 1625677087.776183 CamRzj1Y20KA9cv9Mi 10.1.1.1 59145 192.168.1.3 445 0.003968 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts <- fourth input event Creating alias 'ATTACKER_10.1.1.1_DC_192.168.1.3' for context 'ATTACKER_10.1.1.1' <- a context alias for attacker 10.1.1.1 and DC 192.168.1.3 is created and event counter of the operation for 10.1.1.1 is incremented to 3 Writing event 'Attacker 10.1.1.1 has enumerated domain trust against several domain controllers' to file '-' Attacker 10.1.1.1 has enumerated domain trust against several domain controllers <- since the counter of the operation for 10.1.1.1 has reached the value of 3, write an alert message to standard output Deleting context 'ATTACKER_10.1.1.1' <- the event correlation operation for attacker 10.1.1.1 is terminating and dropping the context ATTACKER_10.1.1.1 and its aliases Context 'ATTACKER_10.1.1.1' deleted Context 'ATTACKER_10.1.1.1_DC_192.168.1.1' deleted Context 'ATTACKER_10.1.1.1_DC_192.168.1.3' deleted Context 'ATTACKER_10.1.1.1_DC_192.168.1.2' deleted The above EventGroup rule has one drawback -- if the threshold of 3 events is not reached during 30 seconds for an event correlation operation, the 30 second event correlation window of the operation will start to slide, and the operation could potentially exist for a longer period of time. However, since aliases do not have any fixed lifetime, they will continue to suppress events for a given combination of attacker and domain controller, even if the first event for this combination is already outside of the 30 second event correlation window. If this is an issue in your environment and you want to address it, you could enhance the 'count' field of the above rule as follows: alias ATTACKER_$1 ATTACKER_$1_DC_$2; create ATTACKER_$1_DC_$2_LIFETIME 30 ( unalias ATTACKER_$1_DC_$2 ) Here is the full rule definition with the modified 'count' field: type=EventGroup ptype=RegExp pattern=^\d+\.\d+ \S+ ([\d.]+) \d+ ([\d.]+) 445 \d+\.\d+ \\\\pipe\\\\lsass netlogon DsrEnumerateDomainTrusts context=!ATTACKER_$1_DC_$2 count=alias ATTACKER_$1 ATTACKER_$1_DC_$2; \ create ATTACKER_$1_DC_$2_LIFETIME 30 ( unalias ATTACKER_$1_DC_$2 ) init=create ATTACKER_$1 end=delete ATTACKER_$1 desc=Attacker $1 has enumerated domain trust against several domain controllers action=write - %s thresh=3 window=30 With this change, a separate context ATTACKER_<attackerip>_DC_<dcip>_LIFETIME will be set up for each alias after the creation of the alias. The context will exist for 30 seconds (the size of the event correlation window) and will drop the alias with 'unalias' action when context lifetime expires. Therefore, each alias will exist for 30 seconds only, and when the event that triggered its creation moves outside the event correlation window, the operation is able to count the event for the given attacker and domain controller combination again. Note that although the *_LIFETIME contexts are not removed when the operation terminates, they will time out quite quickly, and executing 'unalias' action for a non-existing alias name is no-op (you will only get a debug-level message into SEC log that the alias does not exist). One final note -- you can find a similar solution for this scenario in a paper that is available from the SEC home page: https://ristov.github.io/publications/cogsima15-sec-web.pdf Hope this helps, risto Kontakt James Lay (<jl...@sl...>) kirjutas kuupäeval K, 7. juuli 2021 kell 22:23: > > Hey all. > > So....been looking at EventGroups thinking this might fit the bill, but I'm not sure so I'm coming to you folks. I have the below: > > 1625677087.655857 CAY8g9D3jAKFwhD02 SameIP 59143 DifferentIP 445 0.017172 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts > 1625677087.744290 C6mmd42niEVcW1djY5 SameIP 59144 DifferentIP 445 0.003689 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts > 1625677087.776183 CamRzj1Y20KA9cv9Mi SameIP 59145 DifferentIP 445 0.003968 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts > 1625677203.779593 CF4JvS21lo6Beh3jUb SameIP 59149 DifferentIP 445 0.001475 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts > 1625677203.892484 C6LZI2ooakJbNI6vj SameIP 59150 DifferentIP 445 0.023818 \\pipe\\lsass netlogon DsrEnumerateDomainTrusts > > My goal is to flag on this type of behaviour (a single IP enumerating domain trust against several domain controllers in a small space of time). Anyone have any tips on how to implement something like this? Is EventGroup not what I need? Thank you. > > James > _______________________________________________ > Simple-evcorr-users mailing list > Sim...@li... > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users |
From: James L. <jl...@sl...> - 2021-07-07 19:22:30
|
Hey all. So....been looking at EventGroups thinking this might fit the bill, but I'm not sure so I'm coming to you folks. I have the below: 1625677087.655857 CAY8g9D3jAKFwhD02 SameIP 59143 Differe ntIP 445 0.017172 \\pipe\\lsass netlogon DsrEnum erateDomainTrusts 1625677087.744290 C6mmd42niEVcW1djY5 SameIP 59144 Differe ntIP 445 0.003689 \\pipe\\lsass netlogon DsrEnum erateDomainTrusts 1625677087.776183 CamRzj1Y20KA9cv9Mi SameIP 59145 Differe ntIP 445 0.003968 \\pipe\\lsass netlogon DsrEnum erateDomainTrusts 1625677203.779593 CF4JvS21lo6Beh3jUb SameIP 59149 Differe ntIP 445 0.001475 \\pipe\\lsass netlogon DsrEnum erateDomainTrusts 1625677203.892484 C6LZI2ooakJbNI6vj SameIP 59150 Differe ntIP 445 0.023818 \\pipe\\lsass netlogon DsrEnum erateDomainTrusts My goal is to flag on this type of behaviour (a single IP enumerating domain trust against several domain controllers in a small space of time). Anyone have any tips on how to implement something like this? Is EventGroup not what I need? Thank you. James |
From: Risto V. <ris...@gm...> - 2021-05-26 08:23:18
|
hi Brian, let me provide my comments below in inline fashion: > > I've seen a log rotation where the input file did not get re-opened, and am working on troubleshooting that. > > For the SEC process that failed, sending a SIGUSR2 failed, but sending a SIGABRT worked. > (both sent as the same user as the process owner) To clarify the purpose of SIGUSR2 a bit -- this signal has been indeed designed for handling file rotations, but it only works for SEC outputs (for example, files created with 'write' action) and SEC log file (specified with --log command line option). As for handling input file rotations, it happens automatically and there is no need for sending a specific signal to SEC. For each input file, its inode number is monitored and whenever it changes, the input file has been rotated and will be reopened. As for the SIGABRT signal, it forces SEC to reopen all input files if --nokeepopen command line option has been provided, but by default SEC only attempts to open those input files which are currently in the closed state. > > The input file for that process is an NFS mounted read-only backed file system, to which I have no real access for experimentation. > > I created a baby SEC config file for testing, and specified a local filesystem, and was unable to recreate the failure. > I tried using "mv input_orig input_new; touch input_new", and "cp /dev/null >> input_orig". > I used non-detached mode, as opposed to detached mode for the failing config, though I wouldn't expect that to make a difference. > > I'm currently thinking that it might have to do with the NFS mount options, perhaps specifically the locking methods, or maybe the soft vs. hard mount. Since handling log rotations depends on file inode numbers, I suspect the file inode number occasionally stays the same after rotation on an NFS mounted file system. To find out what actually happened, it is best to look into the SEC log file. Whenever a rotated input file has been detected, the message "Input file <file> has been recreated" will be written into the log file before input file is reopened (if the input file is truncated without inode number change, the message "Input file <file> has been truncated" is logged). If you are currently not collecting SEC log messages, I'd recommend activating logging with the --log command line option and in order to keep the log file smaller, you can exclude debug-level messages with --debug=5 command line option. Once the issue surfaces again, log messages will provide a clue what actually happened. For finding out what information SEC currently has about input files, you can let SEC generate a dump file with SIGUSR1 signal and then look into the "Input sources" section in the dump file. Here is an example fragment from this section: /var/log/sshd.log (status: Open, type: regular file, read offset: 256, file size: 256, device/inode: 2065/6449643528, received data: 8123 lines, context: _FILE_EVENT_SSHD) >From the above information, you can see the device and inode numbers of the input file. With /usr/bin/stat tool you can find out what device and inode numbers are reported for that file by the NFS server and if these numbers are the same you can see in the SEC dump file (these numbers should always be the same, apart from a very small time frame before SEC handles the rotated file). > > The mtab entry (redhat 7.9) for this includes the following options: > > foo.ucsd.edu:/remotefilesystem /localfilemountdir nfs ro,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,soft,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=AAA.BBB.CCC.DDD,mountvers=3,mountport=4002,mountproto=udp,local_lock=all,addr=AAA.BBB.CCC.DDD 0 0 > > > The same SEC config running on a Solaris 11 box with the same NFS mounted filesystem, doesn't have this problem. The mnttab file there has these options: > > foo.ucsd.edu:/remotefilesystem /localfilenountdir nfs ro,nodevices,noquota,vers=3,proto=tcp,xattr,zone=ratbert2,sharezone=1,dev=9540001 1613782895 > > Ideas anyone? >From different forum posts I found a discussion on 'actimeo' file system option which can be used for setting the caching time for file attributes. If the current value is too large, the NFS client might cache file attributes for too long and not detect the inode number change in a timely fashion. But it is just a guess and for investigating this issue more closely, some experiments with a test NFS server are needed. Hope this helps, risto > > -- > Brian Parent > Information Technology Services Department > ITS Computing Infrastructure Operations Group > its...@uc... (team email address for Service Now) > UC San Diego > (858) 534-6090 > > > _______________________________________________ > Simple-evcorr-users mailing list > Sim...@li... > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users |
From: Brian P. <bp...@uc...> - 2021-05-25 20:25:31
|
I've seen a log rotation where the input file did not get re-opened, and am working on troubleshooting that. For the SEC process that failed, sending a SIGUSR2 failed, but sending a SIGABRT worked. (both sent as the same user as the process owner) The input file for that process is an NFS mounted read-only backed file system, to which I have no real access for experimentation. I created a baby SEC config file for testing, and specified a local filesystem, and was unable to recreate the failure. I tried using "mv input_orig input_new; touch input_new", and "cp /dev/null >> input_orig". I used non-detached mode, as opposed to detached mode for the failing config, though I wouldn't expect that to make a difference. I'm currently thinking that it might have to do with the NFS mount options, perhaps specifically the locking methods, or maybe the soft vs. hard mount. The mtab entry (redhat 7.9) for this includes the following options: foo.ucsd.edu:/remotefilesystem /localfilemountdir nfs ro,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,soft,nolock,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=AAA.BBB.CCC.DDD,mountvers=3,mountport=4002,mountproto=udp,local_lock=all,addr=AAA.BBB.CCC.DDD 0 0 The same SEC config running on a Solaris 11 box with the same NFS mounted filesystem, doesn't have this problem. The mnttab file there has these options: foo.ucsd.edu:/remotefilesystem /localfilenountdir nfs ro,nodevices,noquota,vers=3,proto=tcp,xattr,zone=ratbert2,sharezone=1,dev=9540001 1613782895 Ideas anyone? -- Brian Parent Information Technology Services Department ITS Computing Infrastructure Operations Group its...@uc... (team email address for Service Now) UC San Diego (858) 534-6090 |
From: Risto V. <ris...@gm...> - 2021-05-24 06:19:11
|
hi Jaakko, thanks a lot for packaging the new version for Debian!! kind regards, risto > > Hi, > > Debian is currently pretty frozen and in process of releasing new > version, so I made the package temporarily available at: > > https://liiwi.idle.fi/sec/ > > I'll remove it from there when it's uploaded to Debian. The package > also installs cleanly to current stable. > > -Jaakko > On Wed, May 12, 2021 at 8:41 PM Risto Vaarandi <ris...@gm...> wrote: > > > > hi all, > > > > SEC-2.9.0 has been released which is available from the SEC home page > > (use the following link for direct download: > > https://github.com/simple-evcorr/sec/releases/download/2.9.0/sec-2.9.0.tar.gz). > > > > Here is the changelog for the new version: > > > > * added support for 'cmdexec', 'spawnexec', 'cspawnexec', 'pipeexec' > > and 'reportexec' actions. > > * added support for 'shell' field in SingleWithScript rules. > > * added support for 'egptype' and 'egpattern' fields in EventGroup rules. > > * added support for %.sp built-in action list variable. > > * added ipv6 support for 'tcpsock' and 'udpsock' actions. > > * bugfixes for 'write', 'writen', 'owritecl', 'udgram', 'ustream', > > 'udpsock' and 'tcpsock' actions (exceptions from syswrite() and send() > > are now handled, and 'ustream' action no longer blocks on Linux when > > peer backlog queue is full). > > * improved socket handling routines. > > * improved error reporting for invalid command line arguments. > > * starting from this version, a program provided with --timeout-script > > command line option is executed without shell interpretation. > > * starting from this version, SEC uses Perl JSON::PP module instead of > > JSON module (JSON::PP is included in the standard Perl installation). > > > > > > kind regards, > > risto > > > > > > _______________________________________________ > > Simple-evcorr-users mailing list > > Sim...@li... > > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users |
From: Jaakko N. <jaa...@ik...> - 2021-05-24 03:12:11
|
Hi, Debian is currently pretty frozen and in process of releasing new version, so I made the package temporarily available at: https://liiwi.idle.fi/sec/ I'll remove it from there when it's uploaded to Debian. The package also installs cleanly to current stable. -Jaakko On Wed, May 12, 2021 at 8:41 PM Risto Vaarandi <ris...@gm...> wrote: > > hi all, > > SEC-2.9.0 has been released which is available from the SEC home page > (use the following link for direct download: > https://github.com/simple-evcorr/sec/releases/download/2.9.0/sec-2.9.0.tar.gz). > > Here is the changelog for the new version: > > * added support for 'cmdexec', 'spawnexec', 'cspawnexec', 'pipeexec' > and 'reportexec' actions. > * added support for 'shell' field in SingleWithScript rules. > * added support for 'egptype' and 'egpattern' fields in EventGroup rules. > * added support for %.sp built-in action list variable. > * added ipv6 support for 'tcpsock' and 'udpsock' actions. > * bugfixes for 'write', 'writen', 'owritecl', 'udgram', 'ustream', > 'udpsock' and 'tcpsock' actions (exceptions from syswrite() and send() > are now handled, and 'ustream' action no longer blocks on Linux when > peer backlog queue is full). > * improved socket handling routines. > * improved error reporting for invalid command line arguments. > * starting from this version, a program provided with --timeout-script > command line option is executed without shell interpretation. > * starting from this version, SEC uses Perl JSON::PP module instead of > JSON module (JSON::PP is included in the standard Perl installation). > > > kind regards, > risto > > > _______________________________________________ > Simple-evcorr-users mailing list > Sim...@li... > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users |
From: Risto V. <ris...@gm...> - 2021-05-12 17:41:22
|
hi all, SEC-2.9.0 has been released which is available from the SEC home page (use the following link for direct download: https://github.com/simple-evcorr/sec/releases/download/2.9.0/sec-2.9.0.tar.gz). Here is the changelog for the new version: * added support for 'cmdexec', 'spawnexec', 'cspawnexec', 'pipeexec' and 'reportexec' actions. * added support for 'shell' field in SingleWithScript rules. * added support for 'egptype' and 'egpattern' fields in EventGroup rules. * added support for %.sp built-in action list variable. * added ipv6 support for 'tcpsock' and 'udpsock' actions. * bugfixes for 'write', 'writen', 'owritecl', 'udgram', 'ustream', 'udpsock' and 'tcpsock' actions (exceptions from syswrite() and send() are now handled, and 'ustream' action no longer blocks on Linux when peer backlog queue is full). * improved socket handling routines. * improved error reporting for invalid command line arguments. * starting from this version, a program provided with --timeout-script command line option is executed without shell interpretation. * starting from this version, SEC uses Perl JSON::PP module instead of JSON module (JSON::PP is included in the standard Perl installation). kind regards, risto |
From: Risto V. <ris...@gm...> - 2021-04-06 10:27:57
|
hi all, SEC-2.9.alpha2 has been released which can be downloaded from: https://github.com/simple-evcorr/sec/releases/download/2.9.alpha2/sec-2.9.alpha2.tar.gz The download link has also been provided in the SEC home page. Compared to the 2.9.alpha1 version, this version introduces a number of improvements to socket handling routines. kind regards, risto |
From: MILLS, R. <rx...@at...> - 2021-03-26 17:58:14
|
Wow! 20 years of SEC! Superb lightweight tool suitable for many applications and excellent support via this user group. And each year SEC continues to evolve with new capabilities. Thank you very much Risto! And many thanks too for those that participate in this SEC user group across the years. High Regards, Rocky -----Original Message----- From: Brian Parent via Simple-evcorr-users <sim...@li...> Sent: Tuesday, March 23, 2021 11:38 AM To: Frazier, Jon <Jon...@gm...> Cc: sim...@li... Subject: Re: [Simple-evcorr-users] 20th birthday of SEC Yes, thanks Risto, and everyone for helping make/keep this super valuable and powerful software available. I've been using it at UCSD for nearly 20 years to detect brute force ssh scanning, and feed that info to the security folks to facilitate automated campus edge blocking. Re: > From: "Frazier, Jon" <Jon...@gm...> > Date: Tue, 23 Mar 2021 14:34:52 +0000 > Subject: Re: [Simple-evcorr-users] 20th birthday of SEC > To: Risto Vaarandi <ris...@gm...>, > "sim...@li..." > <sim...@li...> > > Thank you Risto as without you this tool would not be available. > I know it personally helped me in two different shops to more easily perform certain job requirements. > Back in the early days there were more discussions on how to use it as well as some benchmarking on number of events over time. > > Regards, > Jon Frazier > > > -----Original Message----- > From: Risto Vaarandi <ris...@gm...> > Sent: Tuesday, March 23, 2021 9:28 AM > To: sim...@li... > Subject: [External] [Simple-evcorr-users] 20th birthday of SEC > > _______________________________________________________________________________ > Caution: This email originated from outside of GM Financial and may contain unsafe content. > _______________________________________________________________________________ > > hi all, > > on March 23 2001, SEC version 1.0 was released into public domain. I would like to take the opportunity to thank all SEC users for creative discussions in this mailing list during the last two decades. I would also like to thank all people who have suggested new features or supplied software and documentation fixes. I am especially grateful to John P. Rouillard for many design proposals and new ideas that are now part of the SEC code. Finally, my thanks will also go to long term SEC package maintainers for their continuous work during more than a decade -- Jaakko Niemi (Debian and Ubuntu), Stefan Schulze Frielinghaus (RHEL, CentOS and Fedora), Malcolm Lewis (SLE and openSUSE), Okan Demirmen (OpenBSD), and all other package maintainers for platforms I might not be aware of. > > Thank you all! > > risto > > > _______________________________________________ > Simple-evcorr-users mailing list > Sim...@li... > https://urldefense.com/v3/__https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users__;!!Mih3wA!SbQ4ewMYzRZDhxnV511cyWjrrcQuNzENaHvgw74cGabUQkyo8i0Bpo7LzMv7yRA$ > > > ________________________________ > > Notice to all users The information contained in this email, including any attachment(s) is confidential and intended solely for the addressee and may contain privileged, confidential or restricted information. If you are not the intended recipient or responsible to deliver to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you received this message in error please notify the originator and then delete. Neither, the sender or GMF's network will be liable for direct, indirect or consequential infection by viruses associated with this email. > > > _______________________________________________ > Simple-evcorr-users mailing list > Sim...@li... > https://urldefense.com/v3/__https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users__;!!Mih3wA!SbQ4ewMYzRZDhxnV511cyWjrrcQuNzENaHvgw74cGabUQkyo8i0Bpo7LzMv7yRA$ -- Brian Parent Information Technology Services Department ITS Computing Infrastructure Operations Group its...@uc... (team email address for Service Now) UC San Diego (858) 534-6090 _______________________________________________ Simple-evcorr-users mailing list Sim...@li... https://urldefense.com/v3/__https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users__;!!BhdT!1_XcnKKZAir-6VcnHSqUnD3Dy0V8kUmh9Uqks0Bt5aLhqrgG29MJpZYxigT1Wn0$ |
From: Brian P. <bp...@uc...> - 2021-03-23 15:38:33
|
Yes, thanks Risto, and everyone for helping make/keep this super valuable and powerful software available. I've been using it at UCSD for nearly 20 years to detect brute force ssh scanning, and feed that info to the security folks to facilitate automated campus edge blocking. Re: > From: "Frazier, Jon" <Jon...@gm...> > Date: Tue, 23 Mar 2021 14:34:52 +0000 > Subject: Re: [Simple-evcorr-users] 20th birthday of SEC > To: Risto Vaarandi <ris...@gm...>, > "sim...@li..." > <sim...@li...> > > Thank you Risto as without you this tool would not be available. > I know it personally helped me in two different shops to more easily perform certain job requirements. > Back in the early days there were more discussions on how to use it as well as some benchmarking on number of events over time. > > Regards, > Jon Frazier > > > -----Original Message----- > From: Risto Vaarandi <ris...@gm...> > Sent: Tuesday, March 23, 2021 9:28 AM > To: sim...@li... > Subject: [External] [Simple-evcorr-users] 20th birthday of SEC > > _______________________________________________________________________________ > Caution: This email originated from outside of GM Financial and may contain unsafe content. > _______________________________________________________________________________ > > hi all, > > on March 23 2001, SEC version 1.0 was released into public domain. I would like to take the opportunity to thank all SEC users for creative discussions in this mailing list during the last two decades. I would also like to thank all people who have suggested new features or supplied software and documentation fixes. I am especially grateful to John P. Rouillard for many design proposals and new ideas that are now part of the SEC code. Finally, my thanks will also go to long term SEC package maintainers for their continuous work during more than a decade -- Jaakko Niemi (Debian and Ubuntu), Stefan Schulze Frielinghaus (RHEL, CentOS and Fedora), Malcolm Lewis (SLE and openSUSE), Okan Demirmen (OpenBSD), and all other package maintainers for platforms I might not be aware of. > > Thank you all! > > risto > > > _______________________________________________ > Simple-evcorr-users mailing list > Sim...@li... > https://urldefense.com/v3/__https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users__;!!Mih3wA!SbQ4ewMYzRZDhxnV511cyWjrrcQuNzENaHvgw74cGabUQkyo8i0Bpo7LzMv7yRA$ > > > ________________________________ > > Notice to all users The information contained in this email, including any attachment(s) is confidential and intended solely for the addressee and may contain privileged, confidential or restricted information. If you are not the intended recipient or responsible to deliver to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you received this message in error please notify the originator and then delete. Neither, the sender or GMF's network will be liable for direct, indirect or consequential infection by viruses associated with this email. > > > _______________________________________________ > Simple-evcorr-users mailing list > Sim...@li... > https://urldefense.com/v3/__https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users__;!!Mih3wA!SbQ4ewMYzRZDhxnV511cyWjrrcQuNzENaHvgw74cGabUQkyo8i0Bpo7LzMv7yRA$ -- Brian Parent Information Technology Services Department ITS Computing Infrastructure Operations Group its...@uc... (team email address for Service Now) UC San Diego (858) 534-6090 |
From: Frazier, J. <Jon...@gm...> - 2021-03-23 14:50:55
|
Thank you Risto as without you this tool would not be available. I know it personally helped me in two different shops to more easily perform certain job requirements. Back in the early days there were more discussions on how to use it as well as some benchmarking on number of events over time. Regards, Jon Frazier -----Original Message----- From: Risto Vaarandi <ris...@gm...> Sent: Tuesday, March 23, 2021 9:28 AM To: sim...@li... Subject: [External] [Simple-evcorr-users] 20th birthday of SEC _______________________________________________________________________________ Caution: This email originated from outside of GM Financial and may contain unsafe content. _______________________________________________________________________________ hi all, on March 23 2001, SEC version 1.0 was released into public domain. I would like to take the opportunity to thank all SEC users for creative discussions in this mailing list during the last two decades. I would also like to thank all people who have suggested new features or supplied software and documentation fixes. I am especially grateful to John P. Rouillard for many design proposals and new ideas that are now part of the SEC code. Finally, my thanks will also go to long term SEC package maintainers for their continuous work during more than a decade -- Jaakko Niemi (Debian and Ubuntu), Stefan Schulze Frielinghaus (RHEL, CentOS and Fedora), Malcolm Lewis (SLE and openSUSE), Okan Demirmen (OpenBSD), and all other package maintainers for platforms I might not be aware of. Thank you all! risto _______________________________________________ Simple-evcorr-users mailing list Sim...@li... https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users ________________________________ Notice to all users The information contained in this email, including any attachment(s) is confidential and intended solely for the addressee and may contain privileged, confidential or restricted information. If you are not the intended recipient or responsible to deliver to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you received this message in error please notify the originator and then delete. Neither, the sender or GMF's network will be liable for direct, indirect or consequential infection by viruses associated with this email. |
From: Risto V. <ris...@gm...> - 2021-03-23 14:28:13
|
hi all, on March 23 2001, SEC version 1.0 was released into public domain. I would like to take the opportunity to thank all SEC users for creative discussions in this mailing list during the last two decades. I would also like to thank all people who have suggested new features or supplied software and documentation fixes. I am especially grateful to John P. Rouillard for many design proposals and new ideas that are now part of the SEC code. Finally, my thanks will also go to long term SEC package maintainers for their continuous work during more than a decade -- Jaakko Niemi (Debian and Ubuntu), Stefan Schulze Frielinghaus (RHEL, CentOS and Fedora), Malcolm Lewis (SLE and openSUSE), Okan Demirmen (OpenBSD), and all other package maintainers for platforms I might not be aware of. Thank you all! risto |
From: Risto V. <ris...@gm...> - 2021-03-16 23:25:39
|
hi Stuart, I just saw a post with almost the same question as the previous one (perhaps it was posted before my answer reached your mailbox), and my apologies if information in this email is redundant. > > Correction -- this also produces the same error > > But this does not: > # ----- Radius Auth Failure ----- > # > type=SingleWithSuppress > ptype=regexp > pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request against (.*?) failed > desc=$3 Radius auth request failed > action=write /home/tocops/.tocpipe ops $1 Radius on $3 failed; action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk In the above line, the 'action' keyword appears twice, and that's the reason for the syntax error. For fixing the problem, the 'action' keyword in front of the second action (shellcmd) should be removed, and the line should be rewritten as: action=write /home/tocops/.tocpipe ops $1 Radius on $3 failed; shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk > window=60 > > --sk > kind regards, risto |
From: Stuart K. <st...@al...> - 2021-03-16 23:00:54
|
Correction -- this also produces the same error But this does not: # ----- Radius Auth Failure ----- # type=SingleWithSuppress ptype=regexp pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request against (.*?) failed desc=$3 Radius auth request failed action=write /home/tocops/.tocpipe ops $1 Radius on $3 failed; action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk window=60 --sk -----Original Message----- From: Stuart Kendrick Sent: Tuesday, March 16, 2021 3:14 PM To: sim...@li... Subject: executing multiple actions I am struggling to execute multiple actions. I don't see mention of how to execute multiple actions in the sec man page http://simple-evcorr.github.io/man.html ... but from this page: http://simple-evcorr.sourceforge.net/SEC-tutorial/article.html I believed that separating actions with semi-colons would be sufficient But perhaps not This works: # ----- Radius Auth Failure ----- # type=SingleWithSuppress ptype=regexp pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request against (.*?) failed desc=$3 Radius auth request failed action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk window=60 As does this: # ----- Radius Auth Failure ----- # type=SingleWithSuppress ptype=regexp pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request against (.*?) failed desc=$3 Radius auth request failed action=write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5 window=60 But this does not: # ----- Radius Auth Failure ----- # type=SingleWithSuppress ptype=regexp pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request against (.*?) failed desc=$3 Radius auth request failed action=write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5; action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk window=60 2021-03-16T15:09:56.146094-07:00 vishnu sec[7941]: Reading configuration from /opt/local/etc/sec/service.conf 2021-03-16T15:09:56.146198-07:00 vishnu sec[7941]: Rule in /opt/local/etc/sec/service.conf at line 8: Invalid action 'action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk' 2021-03-16T15:09:56.146289-07:00 vishnu sec[7941]: Rule in /opt/local/etc/sec/service.conf at line 8: Invalid action list ' write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5; action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk ' 2021-03-16T15:09:56.146363-07:00 vishnu sec[7941]: No valid rules found in configuration file /opt/local/etc/sec/service.conf Is executing multiple actions supported? If so, do I need more than a semi-colon in terms of syntax? --sk Stuart Kendrick Allen Institute |
From: Stuart K. <st...@al...> - 2021-03-16 22:58:31
|
Thank you Risto --sk -----Original Message----- From: Risto Vaarandi <ris...@gm...> Sent: Tuesday, March 16, 2021 3:57 PM To: Stuart Kendrick <st...@al...> Cc: sim...@li... Subject: Re: [Simple-evcorr-users] executing multiple actions CAUTION: This email originated from outside the Allen Institute. Please do not click links or open attachments unless you've validated the sender and know the content is safe. ________________________________ hi Stuart, if you want to specify multiple actions for the 'action' field of the rule, semicolon should indeed be used as a separator. However, the 'action' keyword with an equal sign should appear just once in the beginning of the rule field definition. Therefore, the example rule from your post would need one small modification: type=SingleWithSuppress ptype=regexp pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request against (.*?) failed desc=$3 Radius auth request failed action=write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5; shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk window=60 hope this helps, risto Kontakt Stuart Kendrick (<st...@al...>) kirjutas kuupäeval K, 17. märts 2021 kell 00:48: > > I am struggling to execute multiple actions. I don't see mention of how to execute multiple actions in the sec man page https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsimple-evcorr.github.io%2Fman.html&data=04%7C01%7C%7Cf2c4056b50c5440003a308d8e8cedf5a%7C32669cd6737f4b398bddd6951120d3fc%7C0%7C0%7C637515322513706559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=zRfg%2Fr740PVdtIRdLCamnqdFF8n8DF2czNefGDJt0wo%3D&reserved=0 ... but from this page: > https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsimpl > e-evcorr.sourceforge.net%2FSEC-tutorial%2Farticle.html&data=04%7C0 > 1%7C%7Cf2c4056b50c5440003a308d8e8cedf5a%7C32669cd6737f4b398bddd6951120 > d3fc%7C0%7C0%7C637515322513706559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w > LjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdat > a=xK2HBnDZNYGw2HngfmDgHmKVvZcXriYbdXt1ohgTIZA%3D&reserved=0 > I believed that separating actions with semi-colons would be > sufficient > > > But perhaps not > > > This works: > # ----- Radius Auth Failure ----- > # > type=SingleWithSuppress > ptype=regexp > pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request > against (.*?) failed > desc=$3 Radius auth request failed > action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk > window=60 > > As does this: > # ----- Radius Auth Failure ----- > # > type=SingleWithSuppress > ptype=regexp > pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request > against (.*?) failed > desc=$3 Radius auth request failed > action=write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5 > window=60 > > But this does not: > # ----- Radius Auth Failure ----- > # > type=SingleWithSuppress > ptype=regexp > pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request > against (.*?) failed > desc=$3 Radius auth request failed > action=write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5; > action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk > window=60 > > > 2021-03-16T15:09:56.146094-07:00 vishnu sec[7941]: Reading > configuration from /opt/local/etc/sec/service.conf > 2021-03-16T15:09:56.146198-07:00 vishnu sec[7941]: Rule in /opt/local/etc/sec/service.conf at line 8: Invalid action 'action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk' > 2021-03-16T15:09:56.146289-07:00 vishnu sec[7941]: Rule in /opt/local/etc/sec/service.conf at line 8: Invalid action list ' write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5; action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk ' > 2021-03-16T15:09:56.146363-07:00 vishnu sec[7941]: No valid rules > found in configuration file /opt/local/etc/sec/service.conf > > Is executing multiple actions supported? If so, do I need more than a semi-colon in terms of syntax? > > --sk > > Stuart Kendrick > Allen Institute > > > _______________________________________________ > Simple-evcorr-users mailing list > Sim...@li... > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist > s.sourceforge.net%2Flists%2Flistinfo%2Fsimple-evcorr-users&data=04 > %7C01%7C%7Cf2c4056b50c5440003a308d8e8cedf5a%7C32669cd6737f4b398bddd695 > 1120d3fc%7C0%7C0%7C637515322513706559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi > MC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000& > sdata=HyeFhhdti9w3Lo7Wx9zJkFqKYrympC8DQg6EfUX0ChQ%3D&reserved=0 |
From: Risto V. <ris...@gm...> - 2021-03-16 22:57:37
|
hi Stuart, if you want to specify multiple actions for the 'action' field of the rule, semicolon should indeed be used as a separator. However, the 'action' keyword with an equal sign should appear just once in the beginning of the rule field definition. Therefore, the example rule from your post would need one small modification: type=SingleWithSuppress ptype=regexp pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request against (.*?) failed desc=$3 Radius auth request failed action=write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5; shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk window=60 hope this helps, risto Kontakt Stuart Kendrick (<st...@al...>) kirjutas kuupäeval K, 17. märts 2021 kell 00:48: > > I am struggling to execute multiple actions. I don't see mention of how to execute multiple actions in the sec man page http://simple-evcorr.github.io/man.html ... but from this page: > http://simple-evcorr.sourceforge.net/SEC-tutorial/article.html > I believed that separating actions with semi-colons would be sufficient > > > But perhaps not > > > This works: > # ----- Radius Auth Failure ----- > # > type=SingleWithSuppress > ptype=regexp > pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request against (.*?) failed > desc=$3 Radius auth request failed > action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk > window=60 > > As does this: > # ----- Radius Auth Failure ----- > # > type=SingleWithSuppress > ptype=regexp > pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request against (.*?) failed > desc=$3 Radius auth request failed > action=write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5 > window=60 > > But this does not: > # ----- Radius Auth Failure ----- > # > type=SingleWithSuppress > ptype=regexp > pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request against (.*?) failed > desc=$3 Radius auth request failed > action=write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5; action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk > window=60 > > > 2021-03-16T15:09:56.146094-07:00 vishnu sec[7941]: Reading configuration from /opt/local/etc/sec/service.conf > 2021-03-16T15:09:56.146198-07:00 vishnu sec[7941]: Rule in /opt/local/etc/sec/service.conf at line 8: Invalid action 'action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk' > 2021-03-16T15:09:56.146289-07:00 vishnu sec[7941]: Rule in /opt/local/etc/sec/service.conf at line 8: Invalid action list ' write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5; action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk ' > 2021-03-16T15:09:56.146363-07:00 vishnu sec[7941]: No valid rules found in configuration file /opt/local/etc/sec/service.conf > > Is executing multiple actions supported? If so, do I need more than a semi-colon in terms of syntax? > > --sk > > Stuart Kendrick > Allen Institute > > > _______________________________________________ > Simple-evcorr-users mailing list > Sim...@li... > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users |
From: Stuart K. <st...@al...> - 2021-03-16 22:47:47
|
I am struggling to execute multiple actions. I don't see mention of how to execute multiple actions in the sec man page http://simple-evcorr.github.io/man.html ... but from this page: http://simple-evcorr.sourceforge.net/SEC-tutorial/article.html I believed that separating actions with semi-colons would be sufficient But perhaps not This works: # ----- Radius Auth Failure ----- # type=SingleWithSuppress ptype=regexp pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request against (.*?) failed desc=$3 Radius auth request failed action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk window=60 As does this: # ----- Radius Auth Failure ----- # type=SingleWithSuppress ptype=regexp pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request against (.*?) failed desc=$3 Radius auth request failed action=write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5 window=60 But this does not: # ----- Radius Auth Failure ----- # type=SingleWithSuppress ptype=regexp pattern=T(\d\d:\d\d:\d\d).*? (.*?) poll-radius.*?Radius auth request against (.*?) failed desc=$3 Radius auth request failed action=write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5; action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk window=60 2021-03-16T15:09:56.146094-07:00 vishnu sec[7941]: Reading configuration from /opt/local/etc/sec/service.conf 2021-03-16T15:09:56.146198-07:00 vishnu sec[7941]: Rule in /opt/local/etc/sec/service.conf at line 8: Invalid action 'action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk' 2021-03-16T15:09:56.146289-07:00 vishnu sec[7941]: Rule in /opt/local/etc/sec/service.conf at line 8: Invalid action list ' write /home/tocops/.tocpipe ops $1 Radius on $3 failedwindow=5; action=shellcmd /opt/local/script/send-sms -m %s -s sec -r stuartk ' 2021-03-16T15:09:56.146363-07:00 vishnu sec[7941]: No valid rules found in configuration file /opt/local/etc/sec/service.conf Is executing multiple actions supported? If so, do I need more than a semi-colon in terms of syntax? --sk Stuart Kendrick Allen Institute |