[Sqlgrey-users] Some thoughts on new greylisting tests & changes to sqlgrey

[Bpb A. <apt...@cy...>]

Hi,

I've been using policyd (http://policyd.sourceforge.net/) as my
greylisting policy daemon for the past few years and it works pretty well.

I like policyd because it's effective, actively developed, and
reasonably stable, it has a pretty decent feature set, and the developer
is easy to get along with. Its major disadvantage is that it's written
in C which limits how much I can extend it.

An example of the kind of extension that I'd like to make is the ability
to generate a 'reputation' for a network block or autonomous system (AS;
a collection of networks managed by the same entity; these are important
for routing.)

To generate a reputation for an AS, we need a way to derive AS from IP
address, and the ability to track the unreturned connections, successful
deliveries and attempted spamtrap deliveries from IP addresses.
Conveniently, policy daemons can gather connection and delivery data and
the Net::Abuse::Utils module makes IP->ASN mapping easy. With a bit of
logging and some statistics we should be able to come up with a
reasonable metric for the spamminess of an AS and adjust our greylisting
appropriately. Or grab SpamAssassin scores from the mail logs and track
average score per AS.

Another case: suppose I trust SenderBase ratings and want to greylist
networks that deliver a lot of mail and don't have matching rDNS, etc.
Or even better - tail the outbound mail logs and whitelist
sender/recipient pairs, but reverse them so messages from people
responding to sent mail are not delayed.

See the code below for an example of how easy it is to get this sort of
external data. Note that like DNSBL lookups, it may not be effective for
high-volume sites to run these sort of checks on all inbound mail. You
can reduce the load of lookups by sampling a subset of inbound mail,
caching and memoizing, otherwise you can change the code to only do
lookups if a parameter is set in the config file.

I've spent the past week or so digging into the sqlgrey code and making
a bunch of pendantic changes based on perltidy and perlcritic[1],
migrating configuration info into Config::Tiny while providing a
migration path and backwards compatibility with the existing
configuration file[2], adding filesystem independence with File::Spec,
adding some taint checks, and slowly trying to refactor the code into a
module (Mail::Postfix::Policy::Sqlgrey?) and a small driver program.

My current changes are posted at
http://apthorpe.cynistar.net/code/sqlgrey/sqlgreyng for review &
comment. Ignore the 'ng' bit - that's just to differentiate it from the
existing code, not to denote a fork or anything like that. I believe
I've preserved all of the application's functionality but I can't
guarantee it works. One reason for migrating the code into a formal
module is to add a test suite to make sure everything gets tested before
adding new functionality.

Anyway, take a look at policyd and at my mods and let me know what you
think.

Thanks,

-- 
Bob

[1] Compared to a lot of perl scripts, sqlgrey is pretty sensible so
nobody should get insulted that I spent a few days cleaning it up with
perltidy & perlcritic. I find that I learn much faster by reading code
and the tedious edits needed to satisfy perlcritic make you read a lot.
Besides, "Perl Best Practices" has a lot of good advice; I've used it
(by way of perlcritic) to clean up a bunch of my own scripts.

[2] Why Config::Tiny? The module is small and fast, the file format is
very similar to the existing sqlgrey.conf file, it allows sections so
you don't have to encode lists in scalars ('log' and 'log_override'.)
Who wants to write their own config file parsing routines anyway? :)

---- Sample code ----
#!/usr/bin/perl
use strict;
use warnings;

use Carp;
use English qw(-no_match_vars);
use Net::Abuse::Utils qw(:all);
use Net::SenderBase;

if ( !@ARGV ) {
    croak "Usage: $PROGRAM_NAME #.#.#.#\n";
}

my $addr = $ARGV[0];

my $report = "Looking up $addr:\n";

# Get ASN (routing) info
$report .= "ASN info>\n" . join( q{:}, get_asn_info($addr) ) . "\n";

# Get SenderBase mail stats
my $q = Net::SenderBase::Query->new(
    Transport => 'dns',
    Address   => $addr
);

my $r = $q->results();

$report .= "SenderBase info>\n" . $r->raw_data . "\n";

print $report;

exit 0;

---- Sample Output ----

Looking up 80.98.117.76:
ASN info>
6830:80.98.0.0/17:HU:ripencc:2001-07-16
SenderBase info>
0=1|1=UPC Magyarorszag
Kft.|2=7.1|3=6.6|4=482313|6=1057244176|7=106|8=221184|9=3650|45=N|46=15|48=24|49=1.00|50=Budapest|51=05|53=HU|54=19.0833|55=47.5