I've had a couple of instances where swatch have flooded our email infrastructure (swatch sends email via postfix), and I'm looking for a way to avoid this from happening again. Using swatch's throttle feature should do the trick, but it doesn't seem to be working. To test this feature, I've used this entry in my swatch config file (both throttle lines are commented out now, but only one is commented out when running):
code star -
watchfor /throttleme/
mail=admin@example.com,subject=throttle test
#throttle 00:02:00,use=regex
#throttle 00:02:00
code end -
Neihter if the throttle lines above make swatch reduce the number of syslog messages containing "throttleme" that swatch trigger an email for.
Can someone spot errors in my configuration, or is this an issue with swatch itself? I'm running version 3.1.1.
Best regards,
Kenneth Holter
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I haven't used throttle for a while now; I've migrated to the newer 'threshold track_by' syntax (I'm running v3.2.3).
watchfor /throttleme/
mail=admin@example.com, subject=throttle test
threshold_track by=throttleme, type=limit, count=1, seconds=3600
In this case, Swatch would see 'throttleme', send e-mail to admin, and then ignore all subsequent lines containing 'throttleme' for 3600 seconds
We tend to throttle based on hostname … and we use the older awk-syntax
watchfor=/Read-only file system/
mail=admin@example,com, subject=Read-only file system on $4
threshold track_by=$4, type=limit, count = 1, seconds=3600
where, using the older awk-style syntax, '$4' captures the fourth field in the line … which, in syslog format, is the hostname of the box sending the message. This way, if 'server1' logs this Read-only message, we hear about it, then Swatch suppresses subsequent 'Read-only file system' actions for 'server1' for an hour … but … if 'server2' starts logging 'Read-only file system' messages, then swatch performs an action for that message … i.e. we want to hear (once/hour) about *all* hosts reporting 'Read-only file system', not just the first host.
The newer (non-awk) syntax … I haven't used. But it should work something like this:
# Use this variable to stuff the hostname into $1
perlcode my $grab_host = '^\w+\s+\d+\s+\d\d:\d\d:\d\d\s+(\w+)';
watchfor=/$grab_host.*Read-only file system/
mail=admin@example,com, subject=Read-only file system on $4
threshold track_by=$4, type=limit, count = 1, seconds=3600
Or, more simply,
watchfor=/^\w+\s+\d+\s+\d\d:\d\d:\d\d\s+(\w+).*Read-only file system/
mail=admin@example,com, subject=Read-only file system on $1
threshold track_by=$1, type=limit, count = 1, seconds=3600
Where the parens in the 'watchfor' line grab the hostname and stuff it into $1 (subsequent parens would populate $2 and so forth).
hth,
-sk
Stuart Kendrick
FHCRC
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'll look into upgrading to the newest version and start using that threshold track_by feature, instead of debugging the throttle issue I'm having with the version I'm currently running.
Greetings,
Kenneth
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've tried to implement the awk style syntax you outlined above, but can't seem to get it working.
First of all, this is the swatch entry:
- code start -
watchfor /throttleme/
mail=linuxadmin@example.com,subject=swatch throttle test by server $2
threshold track_by=$2, type=limit, count=1, seconds=30
- code end -
The reason I use $2 instead of $4 is that the hostname seem to be located there:
The email that is triggered by this event has these values:
Mail subject: swatch throttle test by server
Mail body: 2011-04-05T10:14:49+02:00 client1 root: throttleme client1
To me it looks like it should work, but it doesn't. The issue seems to be that there is no value in $2 (i.e. it's empty). Is there anything I have to set up for this to work?
- Kenneth
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I would agree that in your log file, the second field contains the host name ('client1' in your example) … dang, that looks like it would work, to me.
(a) You're loading swatch with the command-line parameter "-awk-field-syntax" ?
Our full invocation:
/opt/local/script/swatch -c /opt/local/etc/swatch/swatch.conf -t /var/log/syslog -tail-args -F-awk-field-syntax -script-dir=/home/swatch
If you are, then I don't have a story to tell. But if you aren't, then of course $2 is empty … because you have no second parens in the watchfor line (you have no first set of parens either, so $1 would be empty as well).
(b) Of course, if you can get away from awk syntax (deprecated in swatch), that would be even better. What if you tried /without/ the awk-field-syntax parameter, something like:
/opt/local/script/swatch -c /opt/local/etc/swatch/swatch.conf -t /var/log/syslog -tail-args -F -script-dir=/home/swatch
Then your stanza would look something like this:
watchfor=/^\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d\+\d\d:\d\d\s(.*?)\s.*throttleme/
mail=linuxadmin@example.com,subject=swatch throttle test by server $1
threshold track_by=$1, type=limit, count=1, seconds=30
Where the first parens in the 'watchfor' line capture the hostname and stuffs the hostname into $1
This is where I would like to go. Except that I'm rather attached to the use of $*, which, under -awk-field-syntax, matches the entire line. I like to do things like:
Where 'page_em' is an in-house script which simplifies the line in various ways, before forwarding it to qpage. And I don't see what token to use to represent the entire line, when I quite using -awk-field-syntax … '$*' doesn't work
Let me know what happens next,
-sk
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've gotten this more or less to work now. In my config file I have this, which seems to work fine with regards to extracting the client hostname
config file start -
perlcode my $grab_host = '(\s\w+\s)';
watchfor /$grab_host.*throttleme/
mail=linuxadmin@example.com,subject=Field 1 is $1
threshold track_by=$1, type=limit, count=1, seconds=30
- config file end -
The emails I receive now contain the hostname in field 1 such as this: "Field 1 is clienthostname"
The one issue that remains is that it's pretty annoying having to include "$grab_host.*" in every stanza. Is there a way to omit this, and have only the syslog message to grep for?
Regards,
Kenneth
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I don't know of a way to simplify the config file in the manner you are describing … it would be convenient … but I don't see how to tell Swatch to automagically insert such a phrase into every single watchfor stanza.
-sk
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've had a couple of instances where swatch have flooded our email infrastructure (swatch sends email via postfix), and I'm looking for a way to avoid this from happening again. Using swatch's throttle feature should do the trick, but it doesn't seem to be working. To test this feature, I've used this entry in my swatch config file (both throttle lines are commented out now, but only one is commented out when running):
watchfor /throttleme/
mail=admin@example.com,subject=throttle test
#throttle 00:02:00,use=regex
#throttle 00:02:00
Neihter if the throttle lines above make swatch reduce the number of syslog messages containing "throttleme" that swatch trigger an email for.
Can someone spot errors in my configuration, or is this an issue with swatch itself? I'm running version 3.1.1.
Best regards,
Kenneth Holter
Hi Kenneth,
I haven't used throttle for a while now; I've migrated to the newer 'threshold track_by' syntax (I'm running v3.2.3).
watchfor /throttleme/
mail=admin@example.com, subject=throttle test
threshold_track by=throttleme, type=limit, count=1, seconds=3600
In this case, Swatch would see 'throttleme', send e-mail to admin, and then ignore all subsequent lines containing 'throttleme' for 3600 seconds
We tend to throttle based on hostname … and we use the older awk-syntax
watchfor=/Read-only file system/
mail=admin@example,com, subject=Read-only file system on $4
threshold track_by=$4, type=limit, count = 1, seconds=3600
where, using the older awk-style syntax, '$4' captures the fourth field in the line … which, in syslog format, is the hostname of the box sending the message. This way, if 'server1' logs this Read-only message, we hear about it, then Swatch suppresses subsequent 'Read-only file system' actions for 'server1' for an hour … but … if 'server2' starts logging 'Read-only file system' messages, then swatch performs an action for that message … i.e. we want to hear (once/hour) about *all* hosts reporting 'Read-only file system', not just the first host.
The newer (non-awk) syntax … I haven't used. But it should work something like this:
# Use this variable to stuff the hostname into $1
perlcode my $grab_host = '^\w+\s+\d+\s+\d\d:\d\d:\d\d\s+(\w+)';
watchfor=/$grab_host.*Read-only file system/
mail=admin@example,com, subject=Read-only file system on $4
threshold track_by=$4, type=limit, count = 1, seconds=3600
Or, more simply,
watchfor=/^\w+\s+\d+\s+\d\d:\d\d:\d\d\s+(\w+).*Read-only file system/
mail=admin@example,com, subject=Read-only file system on $1
threshold track_by=$1, type=limit, count = 1, seconds=3600
Where the parens in the 'watchfor' line grab the hostname and stuff it into $1 (subsequent parens would populate $2 and so forth).
hth,
-sk
Stuart Kendrick
FHCRC
Thanks for the excellent reply!
I'll look into upgrading to the newest version and start using that threshold track_by feature, instead of debugging the throttle issue I'm having with the version I'm currently running.
Greetings,
Kenneth
I've tried to implement the awk style syntax you outlined above, but can't seem to get it working.
First of all, this is the swatch entry:
- code start -
watchfor /throttleme/
mail=linuxadmin@example.com,subject=swatch throttle test by server $2
threshold track_by=$2, type=limit, count=1, seconds=30
- code end -
The reason I use $2 instead of $4 is that the hostname seem to be located there:
# grep throttleme /var/log/everything.log|tail -1
2011-04-05T10:08:26+02:00 client1 root: throttleme client1
# grep throttleme /var/log/everything.log|tail -1|awk '{ print $2 }'
client1
The email that is triggered by this event has these values:
Mail subject: swatch throttle test by server
Mail body: 2011-04-05T10:14:49+02:00 client1 root: throttleme client1
To me it looks like it should work, but it doesn't. The issue seems to be that there is no value in $2 (i.e. it's empty). Is there anything I have to set up for this to work?
- Kenneth
Hi Kenneth,
I would agree that in your log file, the second field contains the host name ('client1' in your example) … dang, that looks like it would work, to me.
(a) You're loading swatch with the command-line parameter "-awk-field-syntax" ?
Our full invocation:
/opt/local/script/swatch -c /opt/local/etc/swatch/swatch.conf -t /var/log/syslog -tail-args -F-awk-field-syntax -script-dir=/home/swatch
If you are, then I don't have a story to tell. But if you aren't, then of course $2 is empty … because you have no second parens in the watchfor line (you have no first set of parens either, so $1 would be empty as well).
(b) Of course, if you can get away from awk syntax (deprecated in swatch), that would be even better. What if you tried /without/ the awk-field-syntax parameter, something like:
/opt/local/script/swatch -c /opt/local/etc/swatch/swatch.conf -t /var/log/syslog -tail-args -F -script-dir=/home/swatch
Then your stanza would look something like this:
watchfor=/^\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d\+\d\d:\d\d\s(.*?)\s.*throttleme/
mail=linuxadmin@example.com,subject=swatch throttle test by server $1
threshold track_by=$1, type=limit, count=1, seconds=30
Where the first parens in the 'watchfor' line capture the hostname and stuffs the hostname into $1
This is where I would like to go. Except that I'm rather attached to the use of $*, which, under -awk-field-syntax, matches the entire line. I like to do things like:
$page_me=/opt/local/script/page_em
watchfor=/throttleme/
exec=$page_em skendric $*
Where 'page_em' is an in-house script which simplifies the line in various ways, before forwarding it to qpage. And I don't see what token to use to represent the entire line, when I quite using -awk-field-syntax … '$*' doesn't work
Let me know what happens next,
-sk
I've gotten this more or less to work now. In my config file I have this, which seems to work fine with regards to extracting the client hostname
perlcode my $grab_host = '(\s\w+\s)';
watchfor /$grab_host.*throttleme/
mail=linuxadmin@example.com,subject=Field 1 is $1
threshold track_by=$1, type=limit, count=1, seconds=30
- config file end -
The emails I receive now contain the hostname in field 1 such as this: "Field 1 is clienthostname"
The one issue that remains is that it's pretty annoying having to include "$grab_host.*" in every stanza. Is there a way to omit this, and have only the syslog message to grep for?
Regards,
Kenneth
Hi Kenneth,
I don't know of a way to simplify the config file in the manner you are describing … it would be convenient … but I don't see how to tell Swatch to automagically insert such a phrase into every single watchfor stanza.
-sk
Ok no problem. Thanks so much for the help!
- Kenneth