Re: [Simple-evcorr-users] using variables learned in rule A in rule B's perlfunc: possible?

[Michael H. <mic...@wi...>]

Risto-

Thank you for taking time to respond so thoroughly.  Your examples were quite clear.  I would not have thought to try to use global variables.  Ultimately I chose your cached/context approach, which worked great.

I ended up taking a monolithic config file of nearly 400 hard to maintain/organize rules and spread them more logically across 29 (and counting) configuration files.  The impetus was needing to re-use some (but not all) of the rules in a second network that I admin.

In an attempt to give back, I included below my general approach in case others find it useful.

Thanks again,
-Michael

==============/=============

# 000-main.config
# this file will be the same on each network

type=Single
ptype=SubStr
pattern=SEC_STARTUP
desc=Started SEC
action=assign %a /destinationfile.log

type=Single
ptype=RegExp
pattern=^\w+\s+[0-9]+ [0-9]+:[0-9]+:[0-9]+ (.+?) .+
varmap=SYSLOG; hostname=1
desc=hostname
action=none
continue=TakeNext

type=Jump
ptype=cached
pattern=SYSLOG
context=$+{hostname} -> ( sub { return 1 if $_[0]; return 0 } )
cfset=001-all
continue=EndMatch

# This is a catch-all rule to dump to the logfile anything that didn't match above..
type=Single
ptype=RegExp
pattern=.*
desc=$0
action=write %a $0

---------/--------

#001-all.config
# this file will be different between networks due to differences in vendors, device naming standards, etc

type=Options
joincfset=001-all
procallin=no

type=Jump
ptype=cached
pattern=SYSLOG
context=$+{hostname} -> ( sub { return 1 if $_[0] =~ m/^r-/ } )
cfset=050-juniper
continue=EndMatch

type=Jump
ptype=cached
pattern=SYSLOG
context=$+{hostname} -> ( sub { return 1 if $_[0] =~ m/^[ts]-/ } )

# This is a catch-all rule to dump to the logfile anything that doesn't match above..
type=Single
ptype=RegExp
pattern=.*
desc=$0
action=write %a $0

---------/--------

# 050-juniper.config
# these files will be the same between networks, and where most of the rules and re-use will come in.

type=Options
joincfset=050-juniper
procallin=no

# near top due to frequency..
type=Jump
ptype=RegExp
pattern=.+
cfset=110-juniper-mx104
continue=TakeNext

# two examples, I have many stanzas like this for individual JunOS daemons [hence many files]
type=Jump
ptype=RegExp
pattern= mgd\[[0-9]+\]:
cfset=150-juniper-mgd
continue=EndMatch

type=Jump
ptype=RegExp
pattern= rpd\[[0-9]+\]:
cfset=150-juniper-rpd
continue=EndMatch

# things that don’t match specific daemons
type=Jump
ptype=RegExp
pattern=.+
cfset=100-juniper-nodaemon
continue=TakeNext

# This is a catch-all rule to dump to the logfile anything that survived
type=Single
ptype=RegExp
pattern=.*
desc=$0
action=write %a $0

[FIN]

From: ris...@gm... <ris...@gm...>
Sent: Saturday, October 17, 2020 5:13 PM
To: Michael Hare <mic...@wi...>
Cc: sim...@li...
Subject: Re: [Simple-evcorr-users] using variables learned in rule A in rule B's perlfunc: possible?

hi Michael,

there are a couple of ways to address this problem. Firstly, instead of using sec match variables, one can set up Perl's native variables for sharing data between rules. For example, the regular expression pattern of the first rule can be easily converted into perlfunc pattern, so that the pattern would assign the hostname to Perl global variable $hostname. This global variable can then be accessed in perfunc patterns of other rules. Here is an example that illustrates the idea:

type=Single
ptype=perlfunc
pattern=sub { if ($_[0] =~ /^\w+\s+[0-9]+ [0-9]+:[0-9]+:[0-9]+ (.+?) /) \
              { $hostname = $1; } else { $hostname = ""; } return 1; }
desc=hostname
action=none
continue=TakeNext

type=Jump
ptype=perlfunc
pattern=sub { return 1 if $hostname =~ m/^first-use-case/ }
cfset=rules-for-this-match-1

type=Jump
ptype=perlfunc
pattern=sub { return 1 if $hostname =~ m/^second-use-case/ }
cfset=rules-for-this-match-2

Since SEC supports caching the results of pattern matching, one could also store the matching result from the first rule into cache, and then retrieve the result from cache with the 'cached' pattern type. Since this pattern type assumes that the name of the cached entry is provided in the 'pattern' field, the hostname check with a perl function has to be implemented in a context expression (the expression is evaluated after the pattern match).

Here is an example which creates an entry named SYSLOG in a pattern match cache, so that all match variables created in the first rule can be retrieved in later rules. Note that the entry is created in the 'varmap' field which also sets up named match variable $+{hostname}. In further rules, the 'cached' pattern type is used for retrieving the SYSLOG entry from cache, and creating all match variables from this entry. In order to check the hostname, the $+{hostname} variable that was originally set in the first rule is passed into perlfunc patterns in the second and third rule. Also, if you need to check more than just few match variables in perlfunc pattern, it is more efficient to pass a reference to the whole cache entry into the Perl function, so that individual cached match variables can be accessed through the reference (I have added an additional fourth rule into the example which illustrates this idea):

type=Single
ptype=RegExp
pattern=^\w+\s+[0-9]+ [0-9]+:[0-9]+:[0-9]+ (.+?) .+
varmap=SYSLOG; hostname=1
desc=hostname
action=none
continue=TakeNext

type=Jump
ptype=cached
pattern=SYSLOG
context=$+{hostname} -> ( sub { return 1 if $_[0] =~ m/^first-use-case/ } )
cfset=rules-for-this-match-1

type=Jump
ptype=cached
pattern=SYSLOG
context=$+{hostname} -> ( sub { return 1 if $_[0] =~ m/^second-use-case/ } )
cfset=rules-for-this-match-2

type=Jump
ptype=cached
pattern=SYSLOG
context=SYSLOG :> ( sub { return 1 if $_[0]->{"hostname"} =~ m/^third-use-case/ } )
cfset=rules-for-this-match-3

A small side-note about the first rule -- you can also create the $+{hostname} variable in the regular expression itself by rewriting it as follows:
^\w+\s+[0-9]+ [0-9]+:[0-9]+:[0-9]+ (?<hostname>.+?) .+
And in that case, you can rewrite the 'varmap=SYSLOG; hostname=1' statement in the first rule simply as 'varmap=SYSLOG'.

Hopefully these examples are useful and help you to tackle the problem.

kind regards,
risto



Hi-

I'm sorry to ask what is probably very basic question, but I have struggling with this for awhile (I have perused the manual a lot and the mailing list a bit) and could use some guidance.

The short version is: is there a way to take the results of a pattern match in one rule and use that value in a perlfunc in another?

More verbosely, at this time I use SEC for network syslog exclusion; nothing fancy.  I would like to start using Jump rules based on hostname.  Hostname is derived from the incoming log line.

I thought I would be clever and use a single rule to determine if there was a hostname or not, save it somewhere reusable, and then launch jump rules based on that.

something like

type=Single
ptype=RegExp
pattern=^\w+\s+[0-9]+ [0-9]+:[0-9]+:[0-9]+ (.+?) .+
varmap= hostname=1
desc=hostname
action=assign %r $+{hostname}
continue=TakeNext

type=Jump
ptype=perlfunc
pattern=sub { return 1 if $+{hostname} =~ m/^first-use-case/ }
cfset=rules-for-this-match-1

type=Jump
ptype=perlfunc
pattern=sub { return 1 if $+{hostname} =~ m/^second-use-case/ }
cfset=rules-for-this-match-2

I know this doesn't work.  I understand that '%r' is not a perl hash, and is an action list variable, and that $+{hostname} is undef inside the type=Jump rule perlfunc.  I also know that %r is being set correctly, I see it in "variables -> r" if I do SIGUSR1 dump.

So is it possible stash away a variable from one rule and use it in a Jump rule like above?  I can work around this easily by using a single rule like below, but if I have for example 20 jump permutations, it seems quite redundant to keep recalculating the hostname for comparison.

type=Jump
ptype=perlfunc
pattern=sub { return 0 unless (defined($_[1]) && $_[0] =~ /^\w+\s+[0-9]+ [0-9]+:[0-9]+:[0-9]+ (.+?) .+/); return 1 if $1 =~ m/^first-use-case/}
cfset=all-rules

Thanks in advance,
-Michael


_______________________________________________
Simple-evcorr-users mailing list
Sim...@li...<mailto:Sim...@li...>
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users