Re: [Bigsister-devel] idea for "simple" escalation [SOLVED!]

[Craig C. <cr...@bl...>]

Escalation CAN be done out-of-the-box!

I'm answering my own question for the list archive in the hope that this
will save someone time some day.

--GOAL:

When an alarm first occurs, notify someone (eg, send an email message).
After some period of time, if the alarm is still active, notify
*additional* people. (As in "Bob didn't fix it after a half hour, better
notify Mary too!")

--SOLUTION:

A little bit of Perl eval() gymnastics in PAGER{} rules within your
bb_event_generator.cfg file. We'll make the first notification of an alarm
go to one recipient (or a set of recipients.) Then we'll make subsequent
*reminders* for the alarm go to more recipients (a light weight sort of
escalation).

The trick is to use:

  (time()-$time)

in your PAGER{} preconditions. That's it, that's all the magic.

time() of course eval()'s to the current wallclock-seconds, and $time evals
to the wallclock-seconds when the alarm was first raised. Aside: in my
first post, I pined for an $age variable in the preconditions evaluation.
An $age variable could easily be defined in
share/bigsister/bin/bs_evgen.pm, but this eval() hack works without having
to edit any code.

--EXAMPLE:

# there should be a TAB after the *.* (all hosts, all tests)
# and bear in mind that this rule will generate *reminders* of
# this alarm every 10 minutes (repeate=10) unless some other rule
# changes that.
*.*    delay=0 repeat=10 mail=my_escalation_group

# Now use PAGER{} to direct alerts that are to "my_escalation_group"
# to where they need to go based on how long this alarm has been
# raised.
#
# Note that the (time()-$time) yields units of seconds, while the
# repeat=10 equate is in minutes. So <=1800 is less-than-or-equal
# 30min. So I'm using <=1850 (50 seconds of elbow room) to make
# sure the initial alert, and the 10min, 20min and 30min reminders
# go to one set of mail= recipients. And the >1850 makes the
# 40min, etc reminders go to the second set of recipients.
#
# Notice that the initial (<30min) recipients are also in the
# second (>30min) set so they get the "up" message too when
# the alarm clears.
#
# You could also add more preconditions; eg, use weekday and daytime
# preconditions to make schedules if you have different people on
# call at different times/days.

PAGER{ $mail eq 'my_escalation_group' and (time()-$time)<=1850 }
mai...@do...
PAGER{ $mail eq 'my_escalation_group' and (time()-$time)>1850 }
mai...@do...,us...@do...

--PITFALLS:

If you're going to use my idea of "fake" recipients, and then expand them
in PAGER{} rules, be *certain* you have a matching PAGER{} rule for all
wallclock times. If you initial recipient "set" is just one email address,
just set that in the mail= equate on the alarm rule. Then user a PAGER{}
precondition with timeframe to add more recipients by setting the mail=
equate to fir...@do...,add...@do... . The "fake" recipients just
gives you fine grain control to break out handling of schedules and such.