Using ASSP to prevent spam on your mail server
I recently moved the greghughes.net domain (web site, mail and everything else) to a godaddy.com virtual dedicated server. In doing so, I lost the anti-spam services that were previously provided by my old web host. Needless to say, the resulting load of spam was fairly overwhelming. My prior host had an appliance out front that caught the better part of the junk email headed for my email server, but a fair amount still got through. At any rate, the move and resulting lack of junk mail protection necessitated a thoughtful look at the options out there.
My criteria were as follows:
Needs to be software I can run myself. I've had my fun (yeah, that's sarcasm) with expensive services that are not overly effective. Complicated billing, archaic payment systems (invoices without a dollar amount? what?) and a couple hundred bucks or more a year was not for me. Preferably open-source. Nothing solves problems that plague the community like the members of the community, so I figured there must be something out there that the afflicted masses build and maintain. It had to stop spam, not just identify and tag it. My email server (MailEnable) is already capable of detecting and "flagging" emails as spam, but that doesn't stop it from getting to my mail server in the first place. The goal was to prevent, not react. So I was looking for a gateway-like solution - something that receives all the inbound email, checks it, and forwards on only the good stuff. It needs to learn how to act. Static rules don't work. We see it in the fraud world, and it certainly applies to spam battles, as well. The system has to be able to learn and adapt and operate in the context of my email accounts. It needs to be kept current. An open source project that no one has worked on for six months or more is likely a dead project, and that won't get you anywhere in a world where the landscape changes constantly. Spammers change tactics a lot, and the tools to prevent spam have to evolve to keep pace. I did a bit of research, and frankly I came up with very little that met all my criteria. Sure, there are a whole slew of commercial products out there, but as I said before, I was looking for open source and free (or very close to it). I'm not looking to buy.
The one thing I found that truly seemed to fit the bill was ASSP, which stands for Anti-Spam SMTP Proxy. It's an open source, Perl-based gateway application that you can run on any operating system that supports the Perl interpreted language (which is pretty much all of them). It requires Perl v5.8 and a specific set of Perl modules, and it can be run as a daemon/service. ASSP has been updated about every two months in the recent past, with the most recent update having been in December (as of the time of this writing).
"The ASSP server project is an Open Source platform-independent transparent SMTP proxy server that leverages numerous methodologies and technologies to both rigidly and adaptively identify spam."
I quickly downloaded the ASSP files, installed the necessary Perl modules and was on my way. I had the ASSP service up and running within just about 15 or 20 minutes. Note that to get the app to run as a service, you will need to manually edit the config file and set the flag in there to specify that you want to run it as a service, or else the only way you'll be able to get it to start is on the command line. Alternatively, you can start ASSP from the command line, access the web admin interface, and change the setting there. Once you do so, you'll be able to start the Windows service or run the daemon in Linux or whatever OS you're working with.
The first thing I did after getting the service set up was to access the web administrative interface and change the default admin password. Do that first. Please. Then I put all of the anti-spam options into "training" mode and I specified a few of the basic server settings (like my domain and email account). I set it up to accept all inbound connections for email (SMTP) from the Internet on port 25, and to forward all emails that are determined not to be spam to the MailEnable server on another (unused) port. Since the MailEnable SMTP server is on the same host, the configuration and security setup was pretty simple. Of course, I them spent some considerable time looking through the many, many settings available. It's cool stuff, but you don't have to tackle it all right up front.
It's worth mentioning here that the ASSP wiki has a lot of good information about setting you system up. Be sure to refer to that resource. If you do, you can be up and running in no time. If you don't, you might just wish you had. Remember, always read the freakin' manual before you ask questions. Heh.
The training mode actually results in all email being delivered (not blocked), but it adds some header information to the email which you can read if you like in order to determine whether or not the ASSP system is flagging it as spam. I actually set up my Thunderbird client with a rule to look for the ASSP header and if the spam flag was true, to move the email off to another folder.
What you are supposed to do during this training period is to categorize the good and bad email, and in doing so tell the ASSP service how to treat the email it sees coming in. I used the email interface for submitting spam and good mail to ASSP for about a week before I turned training mode off. Reporting is very easy. I specified two email aliases in the ASSP system, such as email@example.com and firstname.lastname@example.org (those are not the actual addresses of course) and on a regular basis forwarded groups of email back to the ASSP service that fit into each category. In fact, I even went back into my archive of valid email from before installing ASSP and forwarded a bunch of it to the system, so it could quickly learn what valid email looks like in my world. Your learning period will probably be about a week or so, or however long it takes you to gather 400 or more spam emails along with some some good, valid email.
Once you've provided the system with a corpus of good and bad email, you run a little Perl script on the server to update the Bayesian spam detection database, which is the adaptive learning part of the system. I did this a few times - about daily - throughout the first week. With each update the system got smarter and smarter. Once spam email was being very effectively categorized by ASSP, I switched the system from learning mode into normal operating mode and also configured ASSP to forward a copy of all spam emails it receives to a separate email account (say something like email@example.com). In doing so I have created a place for the system to provide me with all the spam email so that I can continue to peruse it when I feel like it in order to make sure nothing gets trapped in there as a false positive. But my main email account is spam-free. Initially I found a few valid emails were ending up being categorized as spam, but all I had to do was to forward those to the email error reporting interface mentioned above and then rebuild the database, and now for the past few days I have seen zero false positives. I intend to continue to check that account now and then, just to ensure I don't miss any critical email. It's a quick and easy process, especially since all the spam that is blocked by the system as a result of coming from known spammer sources (RBL lists) never even makes it into the system. So, I'm just weeding through the small remainder of the stuff that the system analyzes and weeds out in the second phase of its analysis.
Here is what the service has done for my email account since I turned it on about 12 days ago:
General Runtime Information
ASSP Proxy Uptime: 12.232 days
Messages Processed: 2297 (187.8 per day)
Non-Local Mail Blocked (percentage of email that is spam): 87.5%
CPU Usage: 0.27% avg
That's 288 valid emails and 2009 blocked as spam. As I said at the beginning, a bit overwhelming for only one email account in the mix, and obviously quite necessary to do something about it.
I still need to do some small amount of work to make sure the service stays up and running from a high-availability standpoint, and in fact I have that minor issue with not only the ASSP service but also a couple other email services and even the IIS service. Resource constraints seem to play havoc now and then on my virtual server, but I think I have managed to get a handle on that.
For anyone that's looking to put an anti-spam proxy in place for your own mail server, I most definitely recommend checking out ASSP and giving it a try. Download it here (use the most recent stable version). Or check out the ASSP Wiki, which contains documentation, the FAQ, and everything else you can think of. A high-level list of features can also be found on the ASSP home page at SourceForge.
Maintaining your ASSP box
Written by Red Paranoid Friday, 23 November 2007
In my first article online, I mentioned the method used for installing zimbra and ASSP. In this article, I'd like to focus on the maintenance of ASSP alone. If you've implemented ASSP on your network, you've undoubtedly found out that the system does require some attention, even if significant. Indeed, your hardest task at hand is explaining to the users how it works (sometimes more than once -- but this is a good thing!) as well getting users to report back to ASSP via the e-mail interface. This often distracts you from the more technical aspects of your maintenance, which is fine provided that you do get to it eventually.
In essence, there are three tasks that you need to perform on a regular basis, particularly in the first few weeks of ASSP's implementation (after which you can simply automate everything). These are as follows:
1) Assessing how accurate ASSP in its evaluation
There are a *ton* of methods used to evaluate e-mail, and ASSP implements most of 'em. Because of this, you'll find that many of your users' e-mails are flagged as spam (particularly if the company you work for has the misforture of working in an industry such as pharmaceuticals). As the admin, it's YOUR job to determine whether ASSP is incorrectly identifying messages as spam and if so, what is the mechanism that is throwing it off. ASSP has a great interface for such a task: the analyzer. Simply plug the header and body of a false positive into the analyzer, submit it and check out the eval criteria. Look out in particular for criteria marked in red, they're what tip the scales (but obviously, do take notice of orange messages especially if they're recurring).
"Hey wait a minute," you say after a few tries, "after removing an entry from the bombre, redre or scriptre files, the analyzer still picks up the words I removed! What gives?" You need to restart the ASSP service. I usually do that from the command line, i.e. 'sudo /etc/init.d/assp restart'. I make sure that there are no ongoing SMTP sessions via the web interface though. That should do the trick. And, before I get any comments like "doesn't do the trick" -- if it doesn't, make sure your files ***really*** are clean of entries that are triggering the criteria, will you? ASSP 126.96.36.199 has improved its analyzer so it tells you exactly which entries are triggering BombRe et al. If you haven't updated to that version and are desperate, DO THE FRIGGIN' UPDATE! :)
If your problematic criteria is the Bayesian filter, it's time to rebuild your spam database -- and speaking of which...
2) Rebuilding your spam database
You'll find a lot of docs online about how to rebuild your spam database on a regular basis using cron. In case you're wondering -- yes, it's generally a good idea to rebuild your spam database and yes, it's ok to do it via cron script. For that first rebuild, though, you're going to want to make sure that your users have been submitting mail to ASSP so that it can accurately evaluate what's been going on. The wiki recommends at least 400 mails -- I recommend waiting till a good week after you've sent that mail explaining to the users how they're supposed to use the system and then going from office to office regurgitating it again and again to make sure everyone understands (that is, if you've got few enough people to be able to actually do that). Remember that, with few exceptions, your users just want the system to work with as little discomfort (read: input) as possible... So you need to motivate them -- casually informing them that all those messages whose subjects are prepended with the spam tag will one day never reach their mailbox usually helps drive the idea home.
3) Making sure clamav is updating itself correctly
If you don't have one of those great but expensive anti-virus gateway jobs, you're definitely going to want to make sure that Clam is doing its job. No responsible e-mail administrator should leave it up to the workstation's anti-virus to catch anti-viruses for obvious reasons... Just in case those reasons *aren't* obvious, though, allow me to make mention of three of them:
- If you're letting e-mails with potential viruses get all the way to your user, you're receiving entirely TOO MUCH MAIL! - A gateway anti-virus is easier to maintain than, say a laptop's anti-virus. Namely because you're not going to unplug that sucker for long periods of time, or pack it up and take it home where your teenager will disable all your protection so that he/she can download porn or illicitly acquired media using craooy P2P software. - The longer that potentially compromising data stays on your LAN segment, the higher the probability it ends up screwing you over.
When you install Clam support for ASSP using the methodology I described in my previous article, freshclam (the update service) is automatically installed and scheduled. HOWEVER: if a new version of Clam comes out, you'd better make sure you install it -- because otherwise, freshclam will cease to function. Clam's policy is essentially that if you can't keep up, don't step up: they provide you with a free open-source application, the least you can do is make sure that you know when a new version's coming out! Heck, they've even provided a mailing list (http://lists.clamav.net/mailman/listinfo/clamav-announce) so you've no excuse if you get caught with your pants down... Last Updated ( Sunday, 25 November 2007 )
Anti-SPAM SMTP Proxy (ASSP) One Month Later
I’ve been running ASSP at a client site for Just over a month and I’m really impressed. It was easy to configure and has been quite effective at blocking SPAM.
The nicest feature of ASSP is the way that the Bayes filter trains itself without requiring much in the way of human intervention. I have had to spend some time manually combing the logs for false positives (right at the beginning), but it really hasn’t been too bad.
Auto-Whitelisting All recipients on outbound emails are automatically whitelisted (nice feature!) and a copy of every email is placed in the “HAM” folder for training the Bayes filter. It really hasn’t taken long for the filter to stockpile a large number of SPAM and HAM and this has really improved the situation for the users.
Conclusion If you are evaluating anti-SPAM solutions for your network then I suggest you evaluate ASSP before dropping big bucks on a proprietary solution. There really isn’t much point in spending thousands of dollars on a commercial product when ASSP is free, easy to configure and effective.
ASSP Spam Protection Hosting
A very effective built-in spam protection system called ASSP is located in your web hosting control panel and can be turned on or off at the click of the mouse.
ASSP is often termed the most powerful spam protection system available in the industry, and for good reason.
The ASSP control panel interface provides a variety of modules that allows you to pick and choose the types of filters you wish to enable for every single one of your hosted domain names! A good number of modules are included, such as RBL, HELO, PTR, and MX/A filters.
At the same time, ASSP requires very little configuration; you do not need to continually update it with details of your email accounts, mailing list memberships, etc. ASSP does this for you when you go about your daily email routines!
You can use this spam filter in conjunction with any other anti-spam tool you already use at your computer, if you wish. However, if you're like most users, you'll find that this is entire unnecessary. With ASSP alone, you'll find the amount of spam you receive in your inbox reduced by 90% or more!