File Release Notes and Changelog
Release Name: 1.4a
Notes:
SpamProbe v1.4 README
LEGALESE
Burton Computer Corporation
http://www.burton-computer.com
Copyright (C) 2002,2003,2004,2005,2006 Burton Computer Corporation ALL
RIGHTS RESERVED
This program is open source software; you can redistribute it and/or
modify it under the terms of the Q Public License (QPL) version
1.0. Use of this software in whole or in part, including linking it
(modified or unmodified) into other programs is subject to the terms
of the QPL.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the Q Public
License for more details.
You should have received a copy of the Q Public License along with
this program; see the file LICENSE.txt. If not, visit the Burton
Computer Corporation or CoolDevTools web site QPL pages at:
http://www.burton-computer.com/qpl.html
INTRODUCTION
Welcome to SpamProbe! Are you tired of the constant bombardment of
your inbox by unwanted email pushing everything from porn to get rich
quick schemes? Have you tried other spam filters but become
disenchanted with them when you realized that their manually generated
rule sets weren't updated fast enough to keep up with spammers wording
changes? Or that they generated unwanted false positive scores?
SpamProbe operates on a different basis entirely. Instead of using
pattern matching and a set of human generated rules SpamProbe relies
on a Bayesian analysis of the frequency of words used in spam and
non-spam emails received by an individual person. The process is
completely automatic and tailors itself to the kinds of emails that
each person receives.
My work on SpamProbe was inspired by an excellent article by Paul
Graham. He describes the basic idea and his results. You can read
his article here:
http://www.paulgraham.com/spam.html
I highly recommend reading the article and the other spam related
links on his site for excellent insights into why spam is a problem
and how you can defeat it. Of course running SpamProbe is an
excellent step! :-)
FEATURES
* Spam detection using Bayesian analysis of terms contained in each
email. Words used often in spams but not in good email tend to
indicate that a message is spam.
* Written in C++ for good performance. Database access using PBL or
BerkeleyDB for quick startup and fast term count retrieval.
* Recognition and decoding of MIME attachments in quoted-printable
and base64 encoding. Automatically skips non-text attachments.
MIME decoding enables SpamProbe to make decisions based on words in
the emails rather than base64 gobbledigook.
* Counts two word phrases as well as single words for higher
precision.
* Counts words in from, to, cc, received, and subject headers
separately for improved accuracy since some words are better
indicators of spam when they appear in the subject than in the
body. Received headers after the first are counted separately from
first Received header since they are likely to be forged in spams.
* Ignores HTML tags in emails for scoring purposes unless the -h
command line option is used. Many spams use HTML and few humans do
so HTML tends to become a powerful recognizer of spams. However in
the author's opinion this also substantially increases the
likelihood of false positives if someone does send a non-spam email
containing HTML tags. SpamProbe does pull urls from inside of html
tags however since those tend to be spammer specific.
* Supports MBOX, MBX and Maildir mailbox formats.
* Analyzes image attachments to extract useful terms for scoring.
This catches the majority of spams with only images and no words.
* Locks mboxes and databases using fcntl file locking to avoid
problems when multiple emails arrive simultaneously. Can
optionally be compiled to use Berkeley DB's environment functions
to provide better interaction with db_dump, db_verify, etc.
* Scores only the Received, Subject, To, From, and Cc headers. All
other headers are ignored to make it hard for spammers to hide
non-spammy words in X- headers to fool the filter. The -H command
line option can be used to override this.
* Supports Content-Length: field in mbox headers. This can be
disabled using -Y option to use only From_ to recognize new
messages.
* Uses MD5 digest of emails to recognize reclassification of an
already classified spam to avoid distortion of the word counts if
emails are reclassified. This way emails can be kept in an mbox
that is repeatedly scanned by spamprobe without counting them more
than once. Digest printed along with score can be stored in emails
so that binary attachments can be removed by MUA after scoring
without breaking SpamProbe's ability to reclassify the email later.
* Provides a time stamp based database cleanup command to remove
terms from the database if their counts never rise above a certain
threshold value (normally 2). Tends to limit the otherwise
unbounded growth of the database as each message adds some unique
terms. Also provides a purge command to remove all terms with
counts below a specified minimum no matter what their age.
* The -X option uses a more aggressive scoring algorithm which uses
all significantly good or spammy terms when scoring a message and
also gives more weight to within message frequency of terms. This
method improves recall and works best when the database contains a
few thousand messages.
* edit-term command allows users to directly modify the counts of
individual terms. For example to force a particular term to be
considered spammy or good.
* train mode can be used in place of receive to minimize database
writes by only updating database for difficult to classify
messages.
* auto-train mode can be used to build a database efficiently from a
collection of email stored in multiple files.
* Multiple database options including BerkeleyDB, PBL, and fixed size
hash file.
INSTALLATION
SpamProbe is distributed in source code form. The current version
uses autoconf to handle generating an appropriate Makefile for your
system. Generally you should be able to install SpamProbe like this:
./configure
make
make install
The configure script accepts some useful command line options
including:
--prefix=/path/to/install
Use this to specify a directory tree to install SpamProbe and its
man page into. The default is /usr/local.
--with-pbl=/path/to/pbl
Use this to specify the directory where you installed PBL (Peter
Graf's Program Base Library) if the configure script is not able to
find it on its own. See notes below for more information about the
relative merits of PBL and Berkeley DB.
--with-db=/path/to/db
If you would rather use Berkeley DB than PBL use this to specify
the directory where you installed Berkeley DB if the configure
script is not able to find it on its own.
--enable-cdb
Use this to enable use of Berkeley DB's CDB mode. In this mode SP
will create and use a Berkeley DB "environment" and take advantage
of the "concurrent data store" feature of Berkeley DB.
Theoretically this allows better interaction with Berkeley DB's
utilities (db_verify, db_recover, etc) but some people have
reported problems with this mode so it is now optional.
--enable-default-8bit
Use this if you want to change SP's default handling of eight bit
characters. Normally 8 bit characters are converted to the letter
z in the database to save space (this can be changed at runtime
using the -8 command line option). If you normally receive 8 bit
characters in email then you might want to compile with this flag
to avoid the need to remember to type -8 all of the time.
--enable-assert
Use this if you want to enable the assertions in the source code
for debugging purposes. Generally this is not desirable for
production use since the assertions reduce performance.
--enable-big-endian
Causes all database reads and writes to use big-endian byte order
even on little-endian platforms. Use this if you need to copy a
database across different platforms and want to guarantee byte
level data portability. This option slows down database I/O on
little endian systems.
If you want to statically link the BerkeleyDB library run configure
with the appropriate command line option in the LDFLAGS variable. For
gcc you do could something like this:
LDFLAGS=-static ./configure
make
make install
If you want to use Berkeley DB and configure does not find your
BerkeleyDB library and you know it is installed but has a non-standard
name then you can explicitly define the library to use. For example
some RedHat 6.2 systems might require:
CPPFLAGS=-I/usr/include/db3 ./configure
You need GNU G++ or some other C++ compiler to compile SpamProbe. You
also need either PBL or Berkeley DB installed on your system. PBL
uses more disk space and may require you to download and install it on
your system (see below) but is preferred because it supports
transactions and is faster under heavy loads.
Berkeley DB comes preinstalled on some operating systems so using it
may be easier. It also creates smaller data files than PBL. However
some users have reported data corruption issues with Berkeley DB in
the past so use of PBL is recommended. That having been said the
author of SpamProbe used Berkeley DB for almost a year without any
problems in production so the reported issues seem to be OS or system
specific.
BUILD NOTES USING PBL
To obtain PBL download it from Peter Graf's web site:
http://mission.base.com/peter/source/
PBL does not use configure for its build process however its makefile
seems to be appropriate out of the box for Red Hat Linux systems and
may work without changes on other posix-like systems as well. To
compile SpamProbe with PBL I did the following:
<downloaded pbl_1_02.tar.gz to the current directory>
tar xzvf pbl_1_02.tar.gz
cd pbl_1_02
make
cd ..
<downloaded spamprobe-1.0.tar.gz to the current directory>
tar xzvf spamprobe-1.0.tar.gz
cd spamprobe-1.0
./configure --with-pbl=../pbl_1_02
make
make install
BUILD NOTES USING BERKELEY DB
To obtain BerkeleyDB if you do not already have it installed (many
Linux distributions come with DB preinstalled) visit the SleepCat
website to download a copy. BerkeleyDB is open source and you can use
it for free.
http://www.sleepycat.com
Installation instructions for BerkeleyDB can be found here:
http://www.sleepycat.com/docs/ref/build_unix/intro.html
NOTES USING HASH DATABASE
SpamProbe can use a simple, fixed size hash data file as an
alternative to PBL or BDB. There are two advantages to the hash
format. The first is speed. In my experiments the hash file format
is around 2x the speed of PBL (ranged from 1.8x to 3.5x). The second
advantage is that the hash data file size is fixed. You choose a size
when you create the file and it never changes. File size can be
anywhere from 1-100 MB. You need to choose a size large enough to hold
your terms with room to spare. More on that later.
The hash file format also has significant disadvantages. Becuase the
file size is fixed you must monitor the file to ensure that it does
not become overly full. When the file becomes more than half full
performance will suffer. Also the hash format does not store original
terms so you cannot use the dump command to learn what terms are
spammy or hammy in your database. Finally, the hash format is
imprecise. Hash collisions can cause the counts from different terms
to be mixed together which can reduce accuracy.
To create a hash data file you add a prefix to the directory name in
the -d command line option. You can specify just the directory like
this:
spamprobe -d hash:$HOME/.spamprobe
or you can add a size in megabytes for the file like this:
spamprobe -d hash:42:$HOME/.spamprobe
The size is only used when a file is first created. SP auto detects
the size of an existing hash file. You need to allow enough space for
twice as many terms as you are likely to have in your file. In my
database I have 2.2 million terms. That required a database of are 53
MB. SP uses 12 bytes per term in the hash file so you can estimate
the file size you'll need by multiplying the number of terms by 24.
The hash format does not store the original terms. Instead it stores
the 32 bit hash code for each term. You can do just about anything
with a hash file that you could with a PBL file including
import/export, edit-term, cleanup, purge, etc. You can use export
your PBL database and import it to build a hash file (note that you
cannot go the other direction) and you can export one hash file and
import into a new one to enlarge your file.
SETUP OF SPAMPROBE FOR USERS
Once you have a spamprobe executable copy it to someplace in your PATH
so that procmail can find it. Then create a directory for SpamProbe
to store its databases in. By default SpamProbe wants to use the
directory ~/.spamprobe. You must create this directory manually in
order to run SpamProbe or else specify some other directory using the
-d option. Something like this should suffice:
spamprobe create-db
or to use a different directory add the -d command line option:
spamprobe -d somedirectory create-db
SpamProbe can use either the PBL or Berkeley DB library for its
databases. Both are fast on local file systems but very slow over
NFS. Please ensure that your spamprobe directory is on a local file
system to ensure good performance.
MAILDIR FORMAT
SpamProbe will accept a maildir directory name anywhere that an Mbox
or MBX file name can be specified. When SpamProbe encounters a
Maildir mailbox (directory) name it will automatically process all of
the non-hidden files in the cur and new subdirectories of the mailbox.
There is no need to individually specify these subdirectories.
GETTING STARTED
SpamProbe is not a stand alone mail filter. It doesn't sort your mail
or split it into different mailboxes. Instead it relies on some other
program such as procmail to actually file your mail for you. What
SpamProbe does do is track the word counts in good and spam emails and
generate a score for each email that indicates whether or not it is
likely to be spam. Scores range from 0 to 1 with any score of 0.9 or
higher indicating a probable spam.
Personally I use SpamProbe with procmail to filter my incoming email
into mail boxes. I have procmail score each inbound email using
SpamProbe and insert a special header into each email containing its
score. Then I have procmail move spams into a special mailbox.
No spam filter is perfect and SpamProbe sometimes makes mistakes. To
correct those mistakes I have a special mailbox that I put undetected
spams into. I run SpamProbe periodically and have it reclassify any
emails in that mailbox as spam so that it will make a better guess the
next time around.
This is not a procmail primer. You will need to ensure that you have
procmail and formail installed before you can use this technique.
Also I recommend that you read the procmail documentation so that you
can fully understand this example and adapt it to your own needs.
That having been said, my .procmailrc file looks like this:
MAILDIR=$HOME/IMAP
:0 c
saved
:0
SCORE=| /home/brian/bin/spamprobe train
:0 wf
| formail -I "X-SpamProbe: $SCORE"
:0 a:
*^X-SpamProbe: SPAM
spamprobe
I use IMAP to fetch my email so my mailboxes all live in a directory
named IMAP on my mail server.
NOTE: The first stanza copies all incoming emails into a special mbox
called saved. SpamProbe IS OPEN SOURCE SOFTWARE and though it works
well for me it is possible that it could somehow lose emails. Caution
is always a good idea. That having been said, with the procmailrc
file as shown above the worst that could happen if SpamProbe crashes
is that the email would not be scored properly and procmail would
deliver it to your inbox. Of course if procmail crashes all bets are
off.
The second stanza runs spamprobe in "train" mode to score the email,
classify it as either spam or good, and possibly update the database.
The train command tries to minimize the number of database updates by
only updating the database with terms from an incoming message if
there was insufficient confidence in the message's score. The train
command always updates the database on the first 1500 of each type
received. This ensures that sufficient email is classified to allow
the filter to operate reliably.
The next stanza runs formail to add a custom header to the email
containing the SpamProbe score. The final stanza uses the contents of
the custom header to file detected spams into a special mbox named
spamprobe.
As an alternative to using the train command, you can run spamprobe in
"receive" mode. In that mode SpamProbe scores the email and then
classifies it as either spam or good based on the score. It always
automatically adds the word counts for the email to the appropriate
database. This is essentially like running in score mode followed
immediately by either spam or good mode. It produces more database
I/O and a bigger database but ensures that every message has its terms
reflected in the database. Personally I use train mode. A sample
procmailrc file using the receive command looks like this:
MAILDIR=$HOME/IMAP
:0 c
saved
:0
SCORE=| /home/brian/bin/spamprobe receive
:0 wf
| formail -I "X-SpamProbe: $SCORE"
:0 a:
*^X-SpamProbe: SPAM
spamprobe
If you need to have a locale set when spamprobe is run by procmail you
may need to use a script inside of the procmailrc file rather than
running spamprobe directly. The script can set up any command line
options or environment variables before running spamprobe. The most
common use of this technique is to set the LANG environment variable.
MAKING CORRECTIONS
SpamProbe is not perfect. It is able to detect over 99% of the spams
that I receive but some still slip through. To correct these missed
emails I run SpamProbe periodically and have it scan a special mbox.
Since I use IMAP to retrieve my emails I can simply drop undetected
spams into this mbox from my mail client. If you use POP or some
other system then you will need to find a way get the undetected spams
into a mbox that spamprobe can see.
Periodically I run a script that scans three special mboxes to correct
errors in judgment:
#!/bin/sh
IMAPDIR=$HOME/IMAP
spamprobe remove $IMAPDIR/remove
spamprobe good $IMAPDIR/nonspam
spamprobe spam $IMAPDIR/spam
spamprobe train-spam $IMAPDIR/spamprobe
From this example you can see that I use three special mboxes to make
corrections. I copy emails that I don't want spamprobe to store into
the remove mbox. This is useful if you receive email from a friend or
colleague that looks like spam and you don't want it to dilute the
effectiveness of the terms it contains.
Undetected spams go into the spam mbox. SpamProbe will reclassify
those emails as spam and correct its database accordingly. Note that
doing this does not guarantee that the spam will always be scored as
spam in the future. Some spams are too bland to detect perfectly.
Fortunately those are very rare.
The nonspam mbox is for any false positives. These are always
possible and it is important to have a way to reclassify them when
they do occur.
If you are using receive mode rather than train mode then the above
script can be modified to remove the train-spam line. For example:
#!/bin/sh
IMAPDIR=$HOME/IMAP
spamprobe remove $IMAPDIR/remove
spamprobe good $IMAPDIR/nonspam
spamprobe spam $IMAPDIR/spam
Finally you'll need to build a starting database. Since SpamProbe
relies on word counts from past emails it requires a decent sized
database to be accurate. To build the database select some of your
mboxes containing past emails. Ideally you should have one mbox of
spams and one or more of non-spams. If you don't have any spams handy
then don't worry, SpamProbe will gradually become more accurate as you
receive more spams. Expect a fairly high false negative (i.e. missed
spams) rate as you first start using SpamProbe.
To import your starting messages use commands such as these. The
example assumes that you have non-spams stored in a file named mbox in
your home directory and some spams stored in a file named nasty-spams.
Replace these names with real ones.
spamprobe good ~/mbox
spamprobe spam ~/nasty-spams
COMMAND LINE USAGE
SpamProbe accepts a small set of commands and a growing set of options
on the command line in addition to zero or more file names of mboxes.
The general usage is:
spamprobe [options] <command> [filename...]
You can obtain a full list of all available command line options
by running the "help" command. For example:
spamprobe help
The recognized options are:
-a char
By default SpamProbe converts non-ascii characters (characters
with the most significant bit set to 1) into the letter 'z'. This
is useful for lumping all Asian characters into a single word for
easy recognition. The -a option allows you to change the
character to something else if you don't like the letter 'z' for
some reason.
-c
Tells spamprobe to create the database directory if it does not
already exist. Normally spamprobe exits with a usage error if
the database directory does not already exist.
-C number
Tells SpamProbe to assign a default, somewhat neutral, probability
to any term that does not have a weighted (good count doubled)
count of at least number in the database. This prevents terms
which have been seen only a few times from having an unreasonable
influence on the score of an email containing them.
The default value is 5. For example if number is 5 then in order
for a term to use its calculated probability it must have been
seen 3 times in good mails, or 2 times in good mails and once in
spam, or 5 times in spam, or some other combination adding up to
at least 5.
-d directory
By default SpamProbe stores its database in a directory named
.spamprobe under your home directory. The -d option allows you to
specify a different directory to use. This is necessary if your
home directory is NFS mounted for example.
The directory name can be prefixed with a special code to force
SpamProbe to use a particular type of data file format. The type
codes depend on how your copy of SpamProbe was compiled. Defined
types include:
Example Description
-d pbl:path Forces the use of PBL data file.
-d hash:path Forces the use of an mmapped hash file.
-d split:path Forces the use of a hash file and ISAM
file (may provide better precision than
plain hash in some cases).
The hash: option can also specify a desired file size in megabytes
before the path. For example -d hash:19:path would cause
SpamProbe to use a 19 MB hash file. The size must be in the range
of 1-100. The default hash file size is 16 MB. Because hash
files have a fixed size and capacity they should be cleaned
relatively often using the cleanup command (see below) to prevent
them from becoming full or being slowed by too many hash key
collisions.
Hash files provide better performance than either of the ISAM
options (PBL or Berkeley DB). However hash files do not store the
original terms. Only a 32 bit hash key is stored with each term.
This prevents a user from exploring the terms in the database
using the dump command to see what words are particularly spammy
or hammy.
-D directory
Tells SpamProbe to use the database in the specified directory
(must be different than the one specified with the -d option) as a
shared database from which to draw terms that are not defined in
the user's own database. This can be used to provide a baseline
database shared by all users on a system (in the -D directory) and
a private database unique to each user of the system
($HOME/.spamprobe or -d directory).
-g field_name
Tells SpamProbe what header to look for previous score and message
digest in. Default is X-SpamProbe. Field name is not case
sensitive. Used by all commands except receive.
-h
By default SpamProbe removes HTML markup from the text in emails
to help avoid false positives. The -h option allows you to
override this behavior and force SpamProbe to include words from
within HTML tags in its word counts. Note that SpamProbe always
counts any URLs in hrefs within tags whether -h is used or not.
Use of this option is discouraged. It can increase the rate of
spam detection slightly but unless the user receives a significant
amount of HTML emails it also tends to increase the number of
false positives.
-H option
By default SpamProbe only scans a meaningful subset of headers
from the email message when searching for words to score. The -H
option allows the user to specify additional headers to scan.
Legal values are "all", "nox", "none", or "normal". "all" scans
all headers, "nox" scans all headers except those starting with
X-, "none" does not scan headers, and "normal" scans the normal
set of headers.
In addition to those values you can also explicitly add a header
to the list of headers to process by adding the header name in
lower case preceded by a plus sign. Multiple headers can be
specified by using multiple -H options. For example, to include
only the From and Received headers in your train command you could
run spamprobe as follows:
spamprobe -Hnone -H+from -H+received train
To process the normal set of headers but also add the SpamAssassin
header X-SpamStatus you could run spamprobe as follows:
spamprobe -H+x-spam-status train
-l number
Changes the spam probability threshold for emails from the default
(0.7) to number. The number must be a between 0 and 1. Generally
the value should be above 0.5 to avoid a high false positive rate.
Lower numbers tend to produce more false positives while higher
numbers tend to reduce accuracy.
-m
Forces SpamProbe to use mbox format for reading emails in receive
mode. Normally SpamProbe assumes that the input to receive mode
contains a single message so it doesn't look for message breaks.
-M
Forces SpamProbe to treat the entire input as a single message.
This ignores From lines and Content-Length headers in the input.
-o option_name
Enables special options by name. Currently the only special
options are:
-o graham
Causes SpamProbe to emulate the filtering algorithm originally
outlined in A Plan For Spam.
-o honor-status-header
Causes SpamProbe to ignore messages if they have a Status:
header containing a capital D. Some mail servers use this
status to indicate a message that has been flagged for
deletion but has not yet been purged from the file.
DO NOT use this option with the receive or train command in
your procmailrc file! Doing so could allow spammers to bypass
the filter. This option is meant to be used with the
train-spam and train-good commands in scripts that
periodically update the database.
-o honor-xstatus-header
Causes SpamProbe to ignore messages if they have a X-Status:
header containing a capital D. Some mail servers use this
status to indicate a message that has been flagged for
deletion but has not yet been purged from the file.
DO NOT use this option with the receive or train command in
your procmailrc file! Doing so could allow spammers to bypass
the filter. This option is meant to be used with the
train-spam and train-good commands in scripts that
periodically update the database.
-o orig-score
Causes SpamProbe to use its original scoring algorithm that
produces excellent results but tends to generate scores of
either 0 or 1 for all messages.
-o suspicious-tags
Causes SpamProbe to scan the contents of "suspicious" tags for
tokens rather than simply throwing them out. Currently only
font tags are scanned but other tags may be added to this list
in later versions.
-o tokenized
Causes SpamProbe to read tokens one per line rather than
processing the input as mbox format. This allows users to
completely replace the standard spamprobe tokenizer if they
wish and instead use some external program as a tokenizer.
For example in your procmailrc file you could use:
SCORE=| tokenize.pl | /bin/spamprobe -o tokenized train
In this mode SpamProbe considers a blank line to indicate the
end of one message's tokens and the start of a new message's
tokens. SpamProbe computes a message digest based on the
lines of text containing the tokens.
The -o option can be used multiple times and all requested options
will be applied. Note that some options might conflict with each
other in which case the last option would take precedence.
-p number
Changes the maximum number of words per phrase. Default value is
two. Increasing the limit improves accuracy somewhat but
increases database size. Experiments indicate that increasing
beyond two is not worth the extra cost in space.
-P number
Causes spamprobe to perform a purge of all terms with junk count
less than or equal 2 after every number messages are processed.
Using this option when classifying a large collection of spam can
prevent the database from growing overly large at the cost of more
processing time and possible loss of precision.
-r number
Changes the number of times that a single word/phrase can occur
in the top words array used to calculate the score for each
message. Allowing repeats reduces the number of words overall
(since a single word occupies more than one slot) but allows words
which occur frequently in the message to have a higher weight.
Generally this is changed only for optimization purposes.
-R
Causes spamprobe to treat the input as a single message and to
base its exit code on whether or not that message was spam. The
exit code will be 0 if the message was spam or 1 if the message
was good.
-s number
SpamProbe maintains an in memory cache of the words it has seen in
previous messages to reduce disk I/O and improve performance. By
default the cache will contain the most recently accessed 2,500
terms. This number can be changed using the -s option. Using a
larger the cache size will cause SpamProbe to use more memory and,
potentially, to perform less database I/O.
A value of zero causes SpamProbe to use 100,000 as the limit which
effectively means that the cache will only be flushed at program
exit (unless you have really enormous mailbox files). The cache
doesn't affect receive, dump, or export but has a significant
impact on the others.
-T
Causes SpamProbe to write out the top terms associated with each
message in addition to its normal output. Works with find-good,
find-spam, and score.
-v
When it appears once on the command line this option tells
SpamProbe to write verbose information during processing. When it
appears twice on the command line this option tells SpamProbe to
write debugging information to stderr. This can be useful for
debugging or for seeing which terms SpamProbe used to score each
email.
-V
Prints version and copyright information and then exits.
-w number
Changes the number of most significant words/phrases used by
SpamProbe to calculate the score for each message. Generally this
is changed only for optimization purposes.
-x
Normally SpamProbe uses only a fixed number of top terms (as set
by the -w command line option) when scoring emails. The -x option
can be used to allow the array to be extended past the max size if
more terms are available with probabilities <= 0.1 or >= 0.9.
-X
An interesting variation on the scoring settings. Equivalent to
using "-w5 -r5 -x" so that generally only words with probabilites
<= 0.1 or >= 0.9 are used and word frequencies in the email count
heavily towards the score. Tests have shown that this setting
tends to be safer (fewer false positives) and have higher recall
(proper classification of spams previously scored as spam)
although its predictive power isn't quite as good as the default
settings. WARNING: This setting might work best with a fairly
large corpus, it has not been tested with a small corpus so it
might be very inaccurate with fewer than 1000 total messages.
-Y
Assume traditional Berkeley mailbox format, ignoring any
Content-Length: fields.
-7
Tells SpamProbe to ignore any characters with the most significant
bit set to 1 instead of mapping them to the letter 'z'.
-8
Tells SpamProbe to store all characters even if their most
significant bit is set to 1.
SpamProbe recognizes the following commands:
spamprobe help [command]
With no arguments spamprobe lists all of the valid commands.
If one or more commands are specified after the word help,
spamprobe will print a more verbose description of each command.
spamprobe create-db
If no database currently exists spamprobe will attempt to create
one and then exit. This can be used to bootstrap a new
installation. Strictly speaking this command is not necessary
since the train-spam, train-good, and auto-train commands will also
create a database if none already exists but some users like to
create a database as a separate installation step.
spamprobe create-config
Writes a new configuration file named spamprobe.hdl into the
database directory (normally $HOME/.spamprobe). Any existing
configuration file will be overwritten so be sure to make a copy
before invoking this command.
spamprobe receive [filename...]
Tells SpamProbe to read its standard input (or a file specified
after the receive command) and score it using the current
databases. Once the message has been scored the message is
classified as either spam or non-spam and its word counts are
written to the appropriate database. The message's score is
written to stdout along with a single word. For example:
SPAM 0.9999999 595f0150587edd7b395691964069d7af
or
GOOD 0.0200000 595f0150587edd7b395691964069d7af
The string of numbers and letters after the score is the message's
"digest", a 32 character number which uniquely identifies the
message. The digest is used by SpamProbe to recognize messages
that it has processed previously so that it can keep its word
counts consistent if the message is reclassified.
Using the -T option additionally lists the terms used to produce
the score along with their counts (number of times they were found
in the message).
spamprobe train [filename...]
Functionally identical to receive except that the database is only
modified if the message was "difficult" to classify. In practice
this can reduce the number of database updates to as little as 10%
of messages received.
spamprobe score [filename...]
Similar to receive except that the database is not modified in
any way.
spamprobe summarize [filename...]
Similar to score except that it prints a short summary and score
for each message. This can be useful when testing. Using the -T
option additionally lists the terms used to produce the score along
with their counts (number of times they were found in the message).
spamprobe find-spam [filename...]
Similar to score except that it prints a short summary and score
for each message that is determined to be spam. This can be useful
when testing. Using the -T option additionally lists the terms
used to produce the score along with their counts (number of times
they were found in the message).
spamprobe find-good [filename...]
Similar to score except that it prints a short summary and score
for each message that is determined to be good. This can be useful
when testing. Using the -T option additionally lists the terms
used to produce the score along with their counts (number of times
they were found in the message).
spamprobe auto-train {SPAM|GOOD filename...}...
Attempts to efficiently build a database from all of the named
files. You may specify one or more file of each type. Prior to
each set of file names you must include the word SPAM or GOOD to
indicate what type of mail is contained in the files which follow
on the command line.
The case of the SPAM and GOOD keywords is important. Any number of
file names can be specified between the keywords. The command line
format is very flexible. You can even use a find command in
backticks to process whole directory trees of files. For example:
spamprobe auto-train SPAM spams/* GOOD `find hams -type f`
SpamProbe pre-scans the files to determine how many emails of each
type exist and then trains on hams and spams in a random sequence
that balances the inflow of each type so that the train command can
work most effectively. For example if you had 400 hams and 400
spams, auto-train will generally process one spam, then one ham,
etc. If you had 4000 spams and 400 hams then auto-train will
generally process 10 spams, then one ham, etc.
Since this command will likely take a long time to run it is often
desireable to use it with the -v option to see progress information
as the messages are processed.
spamprobe -v auto-train SPAM spams/* GOOD hams/*
spamprobe good [filename...]
Scans each file (or stdin if no file is specified) and reclassifies
every email in the file as non-spam. The databases are updated
appropriately. Messages previously classified as good (recognized
using their MD5 digest or message ids) are ignored. Messages
previously classified as spam are reclassified as good.
spamprobe train-good [filename...]
Functionally identical to "good" command except that it only
updates the database for messages that are either incorrectly
classified (i.e. classified as spam) or are "difficult" to
classify. In practice this can reduce amount of database updates
to as little as 10% of messages.
spamprobe spam [filename...]
Scans each file (or stdin if no file is specified) and reclassifies
every email in the file as spam. The databases are updated
appropriately. Messages previously classified as spam (recognized
using their MD5 digest of message ids) are ignored. Messages
previously classified as good are reclassified as spam.
spamprobe train-spam [filename...]
Functionally identical to "spam" command except that it only
updates the database for messages that are either incorrectly
classified (i.e. classified as good) or are "difficult" to
classify. In practice this can reduce amount of database updates
to as little as 10% of messages.
spamprobe remove [filename...]
Scans each file (or stdin if no file is specified) and removes its
term counts from the database. Messages which are not in the
database (recognized using their MD5 digest) are ignored.
spamprobe cleanup [ junk_count [ max_age ] ]
Scans the database and removes all terms with junk_count or less
(default 2) which have not had their counts modified in at least
max_age days (default 7). You can specify multiple count/age pairs
on a single command line but must specify both a count and an age
for all but the last count. This should be run periodically to
keep the database from growing endlessly.
For my own email I use cron to run the cleanup command every day
and delete all terms with count of 2 or less that have not been
modified in the last two weeks. Here is the excerpt from my
crontab:
3 0 * * * /home/brian/bin/spamprobe cleanup 2 14
Alternatively you might want to use a much higher count (1000 in
this example) for terms that have not been seen in roughly six
months:
3 0 * * * /home/brian/bin/spamprobe cleanup 1000 180 2 14
Because of the way that PBL and BerkeleyDB work the database file
will not actually shrink, but newly added terms will be able to use
the space previously occupied by any removed terms so that the
file's growth should be significantly slower if this command is
used.
To actually shrink the database you can build a new one using the
BerkeleyDB utility programs db_dump and db_load (Berkeley DB only)
or the spamprobe import and export commands (either database
library). For example:
cd ~
mkdir new.spamprobe
spamprobe export | spamprobe -d new.spamprobe import
mv .spamprobe old.spamprobe
mv new.spamprobe .spamprobe
The -P option can also be used to limit the rate of growth of the
database when importing a large number of emails. For example if
you want to classify 1000 emails and want SP to purge rare terms
every 100 messages use a command such as:
spamprobe -P 100 good goodmailboxname
Using -P slows down the classification but can avoid the need to
use the db_dump trick. Using -P only makes sense when classifying
a large number of messages.
spamprobe purge [ junk_count ]
Similar to cleanup but forces the immediate deletion of all terms
with total count less than junk_count (default is 2) no matter how
long it has been since they were modified (i.e. even if they were
just added today). This could be handy immediately after
classifying a large mailbox of historical spam or good email to
make room for the next batch.
spamprobe purge-terms regex
Similar to purge except that it removes from the database all terms
which match the specified regular expression. Be careful with this
command because it could remove many more terms than you expect.
Use dump with the same regex before running this command to see
exactly what will be deleted.
spamprobe edit-term term good_count spam_count
Can be used to specifically set the good and spam counts of a term.
Whether this is truly useful is doubtful but it is provided for
completeness sake. For example it could be used to force a
particular word to be very spammy or very good:
spamprobe edit-term nigeria 0 1000000
spamprobe edit-term burton 10000000 0
spamprobe dump [ regex ]
Prints the contents of the word counts database one word per line
in human readable format with spam probability, good count, spam
count, flags, and word in columns separated by whitespace. PBL and
Berkeley DB sort terms alphabetically. The standard unix sort
command can be used to sort the terms as desired. For example to
list all words from "most good" to "least good" use this command:
spamprobe dump | sort -k 1nr -k 3nr
To list all words from "most spammy" to "least spammy" use this
command:
spamprobe dump | sort -k 1n -k 2nr
Optionally you can specify a regular expression. If specified
SpamProbe will only dump terms matching the regular expression.
For example:
spamprobe dump 'finance'
spamprobe dump '\bfinance\b'
spamprobe dump 'HSubject_.*finance'
spamprobe tokenize [ filename ]
Prints the tokens found in the file one word per line in human
readable format with spam probability, good count, spam count,
message count, and word in columns separated by whitespace. Terms
are listed in the order in which they were encountered in the
message. The standard unix sort command can be used to sort the
terms as desired. For example to list all words from "most good"
to "least good" use this command:
spamprobe tokenize filename | sort -k 1nr -k 3nr
To list all words from "most spammy" to "least spammy" use this
command:
spamprobe tokenize filename | sort -k 1n -k 2nr
spamprobe export
Similar to the dump command but prints the counts and words in a
comma separated format with the words surrounded by double quotes.
This can be more useful for importing into some databases.
spamprobe import
Reads the specified files which must contain export data written by
the export command. The terms and counts from this file are added
to the database. This can be used to convert a database from a
prior version.
spamprobe exec command
Obtains an exclusive lock on the database and then executes the
command using system(3). If multiple arguments are given after
"exec" they are combined to form the command to be executed. This
command can be used when you want to perform some operation on the
database without interference from incoming mail. For example, to
back up your .spamprobe directory using tar you could do something
like this:
cd
spamprobe exec tar cf spamprobe-data.tar.gz .spamprobe
If you simply want to hold the lock while interactively running
commands in a different xterm you could use "spamprobe exec read".
The linux read program simply reads a line of text from your
terminal so the lock would effectively be held until you pressed
the enter key. Another option would be to use a shell as the
command and type the commands into that shell:
spamprobe /bin/bash
ls
date
exit
Be careful not to run spamprobe in the shell though since the
spamprobe in the shell will wind up deadlocked waiting for the
spamprobe running the exec command to release its lock.
spamprobe exec-shared command
Same as exec except that a shared lock is used. This may be more
appropriate if you are backing up your database since operations
like score (but not train or receive) could still be performed on
the database while the backup was running.
SUPPORT AND WARRANTY
SpamProbe works well for me. However please keep in mind that there
is NO WARRANTY at all with this software. Read the QPL (LICENSE.txt)
for details. YOU ASSUME ALL RISK when using this software.
Be sure to visit the project page on sourceforge. There you can
submit bug reports or feature requests, read and post messages on the
forums, and download the latest version.
http://sourceforge.net/projects/spamprobe/
You can also join the spamprobe mailing list to discuss issues with
other SpamProbe users.
http://lists.sourceforge.net/lists/listinfo/spamprobe-users
Also feel free to contact me at bburton@users.sourceforge.net with any
suggestions for improvements that you don't want to post to the
forums.
Enjoy!
Brian Burton
Changes:
2006-01-30 Brian Burton <brian@burton-computer.com>
* Released as 1.4a
2006-01-28 Brian Burton <brian@burton-computer.com>
* src/includes/LRUCache.h (LRUCache): Reimplemented using STL list
class. Map uses Node ptr as key to avoid having two copies of
the key in memory. Changed iterators to use STL style syntax.
* src/parser/PhrasingTokenizer.cc (compactChars): Improved
efficiency.
2006-01-25 Brian Burton <brian@burton-computer.com>
* src/parser/MimeDecoder.cc (next_char64): Fixed potential array
bounds overflow bug in base64 decoding. Thanks to Chris Ross
for the bug report.
* src/spamprobe.cc: (and various other files) Added
min_phrase_chars option that causes parser to keep adding tokens
for phrases until they are at least min_phrase_chars long
instead of stopping at phrase word limit. This might be useful
for catching "v i a g r a" as a single term. So far though
experiments with using this option are not very promising.
* src/spamprobe/Command_auto_train.cc (execute): Added LOG
sub-command to log each processed message to stdout along with
whether or not it had been scored successfully prior to
training. This is useful for experiments to determine how fast
SP can learn.
* src/spamprobe/Command_receive.cc (createScoreCommand): Fixed
documentation bug and improved online help for score command.
Thanks to Chris Ross for the bug report.
2006-01-03 Brian Burton <brian@burton-computer.com>
* src/parser/MailMessage.cc (MailMessage): Removed redundant
bounds check.
* src/utility/MultiLineString.cc (line): Added bounds check.
* src/parser/HtmlTokenizer.cc (tokenize): Added TempPtr to ensure
reader and receiver are cleared between runs.
* src/utility/MultiLineSubString.cc (m_target): Fixed boundary
conditions if passed in indexes are outside bounds of target.
2005-12-28 Brian Burton <brian@burton-computer.com>
* Released as 1.4.
* src/includes/LRUPtrCache.h (LRUPtrCache): Fixed clear() bug that
caused -P command line option to crash on some architectures due
to an invalid delete.
* src/database/FrequencyDBImpl_bdb.cc (openDatabase): Fixed
relative path database opening bug in CDB mode. (Thanks to
Nicolas Duboc for report and suggested fix).
* src/spamprobe/AbstractCommand.cc (openDatabase): All non-read
only commands will now create the database directory if it's
missing when they open the database for the first time and
the -c option was specified on the command line.
* src/database/HashDataFile.cc (close): Replaced clear() with
erase().
* src/spamprobe/Command_help.cc (printCommandLineOptions): Added
comprehensive command line option help.
* src/input/IstreamCharReader.cc (forward): Now uses rdbuf
directly when reading characters and seeking to eliminate extra
overhead of istream class.
* src/spamprobe/spamprobe.cc (main): -V option now returns exit
code 0 instead of 1.
2005-12-25 Brian Burton <brian@burton-computer.com>
* src/database/FrequencyDBImpl_cache.cc (writeWord): Modified
database cache to keep modified/unmodified terms within cache
size limit.
(flush): number of records written to disk per transaction
limited to avoid problem with PBL using excessive amounts of
memory to write a large cache to disk if the database is large.
2005-12-23 Brian Burton <brian@burton-computer.com>
* src/database/DatabaseConfig.cc (createDatabaseImpl): Restored
use of database cache.
2005-12-21 Brian Burton <brian@burton-computer.com>
* src/includes/Ref.h (T>): Eliminated RefObject base class.
2005-12-18 Brian Burton <brian@burton-computer.com>
* src/utility/util.cc (to_7bits): Removed use of string::+=
* src/input/LineReader.cc (forward): Removed use of string::+=
* src/utility/MultiLineString.cc (parseText): Fixed bug that
caused each succeeding line in decoded text to contain all prior
lines as well. Thanks to Nico for catching that one.
(parseText): More efficient algorithm for breaking string into
multiple lines without use of string::+=
* src/input/SimpleMultiLineStringCharReader.cc (class
SimpleMultiLineStringCharReaderPosition): Refactored code into
position object.
(ensureCharReady): Cached calls to m_target->line() based on
profiler results showing they were consuming too much time
during parsing.
2005-12-17 Brian Burton <brian@burton-computer.com>
* src/parser/HtmlTokenizer.cc (processTagBody): Restored support
for -o suspicious-tags option.
* src/parser/HtmlTokenizer.cc (processTagBody): Added tag specific
prefixes when parsing HTML tags. Left in the old prefix (U_)
even though it collides with url terms for backward
compatibility with people who used -h option. The compatibility
code should be removed after a few months.
2005-12-16 Brian Burton <brian@burton-computer.com>
* src/parser/MaildirMailMessageReader.cc (readMessage): Fixed
skipping of hidden files and sorting of files in cur and new.
2005-12-15 Brian Burton <brian@burton-computer.com>
* Released as 1.3x3.
* Added support for maildir directories to all file based commands.
2005-12-13 Brian Burton <brian@burton-computer.com>
* src/spamprobe/AbstractMessageCommand.cc (processStream): Improved
auto-purge support to work for both token and mime streams and to
perform a final purge after processing all messages.
* src/spamprobe/Command_auto_train.cc (execute): Added support for
auto-purge (-P command line option).
2005-12-11 Brian Burton <brian@burton-computer.com>
* src/spamprobe/Command_create_config.cc (execute): Added
create-config command to write a new config file based on the
current configuration.
* src/spamprobe/Command_create_db.cc (execute): Added create-db
command to auto-create a database if none is present.
* src/spamprobe/spamprobe.cc: Moved code from spamprobe.cc into
separate strategy objects for each supported commands.
* Added help command to print a list of all available commands
and (optionally) also provide a verbose description of any
named command.
* Config file is not automatically generated if missing since that
caused some confusion for users who don't use config files.
2005-12-09 Brian Burton <brian@burton-computer.com>
* src/includes/Buffer.h (class Buffer): Added assertions and sanity
checks. Made reset() exception safe.
* src/spamprobe/SpamFilter.cc (getSortedTokens): Removed use of
qsort(). Now sorting with std::sort().
* Removed unnecessary uses of NewArray<T>. Now its only used by
Buffer<T>.
* Removed old RCPtr<T> in favor of new Ref<T>. This affected lots
of classes in all modules.
2005-12-02 Brian Burton <brian@burton-computer.com>
* Changes below were actually made over the last few weeks but I'm
catching up on previous changes that I hadn't added to
ChangeLog.
* Fixed include <ostream> that didn't work with older gcc versions.
* Added preliminary gif parser support using libungif. configure
attempts to auto-detect libungif id present and uses it to
extract terms from information about any gifs in the message. I
used gifs first since those seem to be the most common format
used in spams.
* Added -f command line option. -d option reloads config file.
* Moved spamprobe app code into its own directory. Added copyright
notices to hdl source.
* Restored deleted lock file code.
* Added DatabaseConfig.
* Added FilterConfig
* Now generates config file if none present.
* Spamprobe has a config file!
* Added HDL code with validation.
* Refactored source code into multiple directories and
non-installed libraries for better code structure and
organization.
* Removed broken (never worked right) BNR code.
* Removed obsolete data conversion utility left over from version
0.6 upgrade.
2005-06-23 Brian Burton <brian@burton-computer.com>
* SimpleTokenizer.cc (isLetterChar): Fixed broken -8 command line
option that was causing 8 bit characters to be treated as word
boundaries.
2005-06-22 Brian Burton <brian@burton-computer.com>
* Released version 1.2.
2005-03-29 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (cleanup_database): Added ability to specify
multiple counts and ages for the cleanup command. This allows
more efficient use of multiple criteria for cleaning the
database.
2005-03-28 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_hash.h (class FrequencyDBImpl_hash): Changed
default hash file size to 32 megs.
* FrequencyDB.cc: SP now defaults to using hash data file format
if neither PBL nor Berkeley DB are available.
(createDB): Now auto-detects database type based on files
in database directory if possible.
* FrequencyDBImpl_null.h (class FrequencyDBImpl_null): Added
"null" database instance to avoid null pointer issues when
command line arguments are invalid.
* spamprobe.cc (quick_close): removed code for closing the
database since it created a race condition that could corrupt
memory and crash out in ::delete.
(main): changed usage/version printing again to avoid crashes
when invalid command line used with -V option.
2005-03-26 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_hash.cc (initializeHeaderRecords): Added a
header record to hash data files to identify file format and
version.
2005-03-24 Brian Burton <brian@burton-computer.com>
* spamprobe.cc: Applied usage message/version reporting fix
supplied by Chris Ross.
* FrequencyDBImpl_split.cc (open): Removed addition of .hash
suffix to hash file name. The suffix is now added automatically
by the FrequencyDBImpl_hash class.
* FrequencyDBImpl_hash.cc: Lots of improvements such as hash
collision detection and mitigation (tries next array element).
Factored out hash file code into a new class (HashDataFile).
Added code to rehash the file whenever cleanup is run. Hash
data file size is now selectable in 1 MB increments instead of
the old use of powers of two. Actual number of elements in hash
table is now based on a prime number that yields as close to the
target file size as possible.
* FrequencyDB.cc: Changed hash: db prefix to use a pure hash file
instead of a split file. Added split: prefix for when that is
more desireable.
2004-11-19 Brian Burton <brian@burton-computer.com>
* spamprobe.cc: Added verbose mode as a less overwhelming
alternative to existing debug mode. Using -v once triggers
verbose mode. Twice triggers debug mode.
(auto_train): Added auto-train command to improve initial
training for new users.
2004-11-13 Brian Burton <brian@burton-computer.com>
* FrequencyDB.cc (class InterruptTest): Attempted to make shutdown
due to user interrupts cleaner by using a guard object in each
method that calls the database. If an interrupt is requested
during a database operation it will be noted and an exception
thrown after the call completes. Multiple interrupts will fall
back to the default signal handler and shut down the process
more forcefully.
2004-11-12 Brian Burton <brian@burton-computer.com>
* README.txt: Fixed -s command line option doc.
* spamprobe.1: Fixed -s command line option doc.
* spamprobe.cc (process_mime_stream): Added support for -Y option
to suppress content-length support in mailboxes.
* MailMessageReader.cc (readMessage): Added support for MIME's
Content-Length header as a way of bypassing embedded From_ lines
inside of a message. Only supported in outermost headers since
attachment bodies are already delimited using MIME boundaries.
* MultiLineString.cc (appendToLastLine): Added ability to append
to last line in string.
* IstreamCharReader.cc (createMark): Added ability to mark
and return when underlying stream is seekable.
* spamprobe.cc (process_mime_stream): Added support for MBX file
format. Added support for ignoring From_ line in mbox files.
2004-11-11 Brian Burton <brian@burton-computer.com>
* Replaced uses of string::clear() with string::erase().
2004-11-07 Brian Burton <brian@burton-computer.com>
* MimeDecoder.cc (decodeHeaderString): Fixed memory bug.
* Proximity phraser is history. It never performed well in
experiments anyway.
* All header terms are now prefixed instead of having some that
did not receive a prefix.
* Fully integrated new parser and removed code for old parser.
All headers are now run through the MIME decoder since the RFC
says the encoding can apply to more than just Subject.
2004-11-01 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_cache.cc; Cache size is now limited to a maximum
number of terms and is automatically flushed when the size is
exceeded. Uses LRUPtrCache instead of just a map so that the
most recently used terms can be kept in memory instead of being
periodically flushed.
2004-10-31 Brian Burton <brian@burton-computer.com>
* Added new email parsing implementation based on the experimental
C# implementation. This parser does less byte twiddling and
parses most emails in a single pass over the message. Many of
the old parsing related command line options are not yet enabled
but the standard processing of mbox files and scoring with basic
paramaters is working well.
2004-10-14 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (main): Added exec and exec-shared commands.
(import_words): modified import command to allow negative values
to be specified in the import file.
* Applied patches for configure.in and aclocal.m4 contributed by
Siggy Brentrup for debian compatibility.
2004-04-24 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_pbl.cc: Invokes new WordData methods to allow
storing data in big endian format.
* WordData.h: Added optional support for storing counts/flags
in big endian order for data portability.
2004-02-05 Brian Burton <brian@burton-computer.com>
* MimeLineReader.cc (readMBXFileHeader): UW IMAP MBX file format
is now auto detected from the first line of the mailbox file.
* spamprobe.cc (process_extended_options): Removed -o imap-mbx
option.
2004-02-04 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (process_extended_options): Added -o imap-mbx
option to process files as WU-IMAP MBX files rather than mbox
files.
* MimeLineReader.cc (readLine): Added support for WU-IMAP MBX file
format.
2004-02-02 Brian Burton <brian@burton-computer.com>
* Released as 0.9h.
2004-01-26 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (process_stream): Added -o tokenized option
to allow people to use an external tokenizer with spamprobe.
2004-01-22 Brian Burton <brian@burton-computer.com>
* SpamFilter.cc (scoreToken): Reduced sorting overhead by
pre-computing and integer sort value with sorting priorities
reflected in the value. This eliminates several calculations
inside of the sort routine.
2004-01-21 Brian Burton <brian@burton-computer.com>
* SpamFilter.cc (computeRatio): Capped ratios in calculations to
within MIN_PROB and MAX_PROB. Widened that range. This avoids
problems with div/0 and makes it easier to sort terms.
2004-01-20 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (dump_words): dump command can now optionally
accept a regular expression as an argument and will only dump
terms matching the regular expression.
(purge_terms): Added purge-terms command to purge from the
database all terms matching a regular expression.
2004-01-17 Brian Burton <brian@burton-computer.com>
* Released as 0.9g2.
* spamprobe.cc (main): Fixed bug in command line processing.
Thanks to Jem for bug report.
* Released as 0.9g.
2004-01-16 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (train_on_message): Code simplified. Eliminated
redundant recalculation of scores.
(train_on_message): Timestamps are now longer updated by
train-spam and train-good commands. They are still updated by
train command.
(main): Fixed assertion if -P option is specified in a read only
operation.
2004-01-14 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (main): Added -C command line option to allow users
to specify their own min word count.
* SpamFilter.cc (SpamFilter): Set default minimum word count back
to 5 (was 3).
* spamprobe.cc (process_extended_options): Removed "alt-score"
from -o options list because it distributes scores poorly. New
formula achieves the same end with better accuracy. Added
"orig-score" option to allow people to continue using the old
formula. Added "honor-xstatus-header" option for people whose
mail server uses X-Status: rather than Status: for the deleted
flag.
(main): Added -l command line option to allow people to set
their own spam threshold if they don't like the default value.
* SpamFilter.cc (scoreMessage): Added a new scoring formula based
on Paul's but taking the nth root of spam and good probabilities
to produce more evenly distributed scores. Lowered the spam
threshold to 0.6 to keep accuracy about the same as the original
formula. Highest score seen for a ham so far in tests is 0.44
so 0.6 seems safe. Made the new formula the default instead of
Paul's.
2004-01-12 Brian Burton <brian@burton-computer.com>
* Released as 0.9f
* spamprobe.cc (set_headers): Added -H+name command line option to
allow users to specifically add inidividual headers to the list
of headers to process.
(process_extended_options): Added -o option with graham and
honor-status-header options.
2004-01-09 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (edit_term): Removed validity check from edit term
command since it made it impossible to edit terms from headers.
(dump_message_words): Added "tokenize" command to allow a user
to see all of the terms in a message and their scores.
* What follows is a collection of changes not added here as they
were made:
* util.h (num_to_string3): Added function to produce a three digit
zero padded number.
* spamprobe.cc (train_on_message): Added option to have train mode
try to keep the spam/good counts balanced to minimize skewing
results towards whichever type we've seen the most.
* SpamFilter.cc (SpamFilter): Improved "extended top terms array"
logic to make the minimum distance from mean for the array
settable by caller of SpamFilter. Added ability to set a
minimum size for the top terms array.
* RegularExpression.cc (removeMatch): Added method for removing a
matched substring from the text.
(replaceMatch): Added method for replacing a matched substring
in the text.
* PhraseBuilder.h (class PhraseBuilder): Added ability to limit
the maximum length of a phrase so that the filter can use more
words per phrase without filling the database (i.e. min 2 and
max 8 words per phrase but limit phrases to max of 20
characters).
* MessageFactory.cc (addIPAddressTerm): Added a new logical term
for IP addresses found in a message.
(isSuspiciousTag): Added support for processing just
"suspicious" HTML tags (suggested by Paul Graham).
(processUrls): Added special prefix for terms found in URLs.
(addHeadersToMessage): Added support for processing arbitrary
headers.
* Message.cc (getAllTokensCount): Added AllTokensCount property
(total within document count of all terms).
* FrequencyDB.h (class FrequencyDB): Added MessageCount property.
2003-09-10 Brian Burton <brian@burton-computer.com>
* Released as 0.9e.
* spamprobe.cc (print_terms): Changed -T output to include overall
good/spam database counts of each term.
* SpamFilter.cc (token_qsort_criterion): Modified token sorting
algorithm to improve selection of top terms for scoring.
Changes appear to reduce the chances of false positives. The
new criteria are: higher distance from mean to 5 decimal places,
higher within document frequency div 3 (to make less selective),
less spammy score, higher count in database, and (final tie
break) alphabetical. The wdf div helps to make a small
difference in wdf to be less significant.
* MessageFactory.h (class MessageFactory): Added
useProximityPhraser().
* ProximityPhraseBuilder.h: Added "proximity" phrase builder that
stores distances between words instead of phrases themselves.
Not nearly as effective as phrases so far.
* AbstractPhraseBuilder.h: Added abstract super class for
PhraseBuilder to allow plugging in different kinds of phrasers.
2003-09-04 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_pbl.cc (sweepOutOldTerms): Changed to commit
based on number of records deleted instead of number of records
scanned.
(getWord): Changed to handle retrieval of current record
properly.
2003-09-03 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_pbl.h (class FrequencyDBImpl_pbl): Peter Graf
contributed a patch to switch over to using PBL's key files
instead of ISAM. This change cuts disk space usage by a factor
of 2 and seems to provide a comparable speed improvement as
well.
2003-09-01 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_pbl.cc (beginTransaction): Fixed some broken
assertions.
* spamprobe.cc (train_test): Added train-test message to
facilitate testing train mode. Reads a line at a time from
stdin. Each line contains a message type (spam/good) and a file
name. SP then reads the file and does a train-spam or
train-good on the message. Great for quickly building a
database from a lot of known emails using train mode.
* Released as 0.9d.
* Fixed configure to remove default -Wno-deprecated.
2003-08-30 Brian Burton <brian@burton-computer.com>
* LockFD.h (class LockFD): Changed SHARED to SHARED_LOCK to fix
compile problems on solaris 2.6. Thanks to Cornell Binder for
bug report.
* Released as 0.9c.
* README.txt: Updated for release 0.9c.
2003-08-29 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_split.cc (open): Modified to be compatible with
PBL in place of BDB for btree portion of database.
* FrequencyDBImpl_cache.cc (flush): Performs all writes to it's
impl db using a transaction for safety. Note that the cache
itself does not support transactions but only utilizes it's
impl's support for them (bug?).
* FrequencyDB.h (class FrequencyDB): Added beginTransaction() and
endTransaction() methods for impls that support transaction
semantics (currently only PBL). Also added createDB() static
method to allow other classes to create impl dbs without knowing
what type they are creating.
* FrequencyDBImpl.h (class FrequencyDBImpl): Added
beginTransaction() and endTransaction() empty default
implementations.
* FrequencyDBImpl_pbl.h (class FrequencyDBImpl_pbl): Added support
for Peter Graf's PBL (The Program Base Library) ISAM database as
an optional replacement for Berkeley DB. PBL offers transaction
semantics without all of the complicated background processing
of BDB but none of the locking. Since SP does its own locking
that should be fine. PBL files appear to be larger than BDB
files by a significant margin. PBL can be downloaded here:
http://mission.base.com/peter/source/
* FrequencyDBImpl_bdb.cc (writeWord): Any word with zero counts
can now be deleted on write. Previously __* terms were kept but
that's not really necessary and this will clear out redundant
empty digests.
* spamprobe.cc (quick_close): Fixed potential infinite loop when
processing signals.
* FrequencyDBImpl_bdb.cc: Improved error checking and reporting.
Made use of environment a compile time option controlled by
--enable-cdb passed to configure at build time.
(writeWord): Removed load/compare of existing record to speed
up writes to database except when in debug mode.
(flush): Added call to db->sync() during flush().
2003-08-23 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (process_test_cases): Added some more test cases.
Changed AUTO_PURGE_JUNK_COUNT to 2 instead of 4.
* SpamFilter.cc (token_qsort_criterion): When selecting top terms
now assigns terms to "bands" of roughly 0.005% rather than
sorting on raw probability. This helps to prevent almost
equally significant good terms from being overshadowed and
excluded by only slightly more significant spam terms and should
reduce number of false positives.
* PhraseBuilder.h (class PhraseBuilder): Dynamically resizes
buffer now rather than using a fixed size buffer. Supports min
as well as max number of words in phrases.
* MimeMessageReader.cc (unquoteText): Now converts _ to space
in quoted headers (thanks Junior for bug report!).
* MimeHeader.cc (getFieldName): Added accessor for field names.
* MessageFactory.cc (setMinPhraseLength): Phrases can now have a
minimum length as well as a maximum length.
(addHeadersToMessage): Improved header processing uses prefixes
for all headers, not just a subset of them. Better recognition
of ignored headers.
(getHeaderPrefix): Creates a prefix for any header with escaping
of non alphanumeric characters.
(addHeaderToMessage): terms from headers are only stored with
prefixes now instead of both prefixed and unprefixed.
2003-08-13 Brian Burton <brian@burton-computer.com>
* MimeMessageReader.cc (unquoteText): Added RFC 1522 support for _
as space in headers. Thanks jxz.
2003-08-07 Brian Burton <brian@burton-computer.com>
* Released as 0.9b.
* MessageFactory.cc (addHeadersToMessage): Modifed header
processing to decode RFC2047 encoded headers. Thanks to Junior
for the suggestion!
* MimeMessageReader.cc (decodeHeader): Added method for decoding
mime encoded headers.
* FrequencyDBImpl_bdb.cc (open): If berkeley db environment files
cannot be opened but the database is running in read only mode
we carry on without any environment. This allows shared
database directories to be kept purely read only for users.
* SpamFilter.cc (lock): locking now removes colon prefixes from
database filenames when creating filename for lock file. This
is done by nuking up to the last : so it will break windows
paths that include a drive letter.
2003-08-02 Brian Burton <brian@burton-computer.com>
* Released as 0.9a.
2003-08-01 Brian Burton <brian@burton-computer.com>
* Modified FrequencyDBImpl to accept file mode as an argument and
use that mode when creating database related files. This allows
shared and private dbs to have different modes.
2003-07-29 Brian Burton <brian@burton-computer.com>
* Added rebuilddb to contrib directory. This script from David
A. Lee automatically rebuilds your .spamprobe directory to
reclaim any space left unused by berkeley db.
2003-07-28 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_hash.cc (open): Removed obsolete reference to
MAP_FILE because it broke compilation on solaris 9 systems.
2003-07-27 Brian Burton <brian@burton-computer.com>
* Released as 0.9-dev-6.
2003-07-26 Brian Burton <brian@burton-computer.com>
* SpamFilter.cc (lock): Global lock file only used for commands
that write to the database. Using berkeley environment allows
reads to coexist safely with writes.
2003-07-25 Brian Burton <brian@burton-computer.com>
* FrequencyDB.cc (addWord): addWord() preserves flags if word
already in database or sets them to specified value if the word
is new.
* FrequencyDBImpl_cache.cc (close): no longer flushes
automatically. This allows SpamFilter to be closed
quickly if necessary. Caller must now specifically
flush() before closing.
* SpamFilter.cc (close): SpamFilter now can be closed in flush
mode or "abandon writes mode" so that cleanup code can avoid
writes if the user interrupted the program with ^C or kill.
* spamprobe.cc (close_on_exit): Added code to close database on
exit to ensure that berkeley db gets a chance to remove its
locks. Without this using ^C on one process could cause the
next SP process to hang when it tried to write because the
killed procs locks were still in the envronment (db_recover
could be used to clear them but that's a pain).
(import_words): import/export now include flags as well as
counts for each word so that timestamps can be preserved.
(train_on_message): Increased min message count for training
from 500 to 1500 to help ensure sufficient number of messages
for people using train from the beginning.
* SpamFilter.cc (lock): Added locking code to SpamFilter to ensure
locks are performed uniformly no matter what database is used.
Databases can still perform their own locking if needed. This
solved the weakness of berkeley db's concurrent data store
locking when performing read-update-write of terms (lack of
write locks while record being updated could cause counts to be
incorrect even though database was not corrupted).
* spamprobe.cc (main): Added -R option to return 0 if message was
spam and 1 otherwise. Based on patch from jxz@uol.com.br.
* FrequencyDBImpl_dbm.h: Removed locking code. Locks now
at spamprobe.cc level.
2003-07-24 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_bdb.cc (open): removed lock file code and
replaced it with use of a berkeley db environment and the
berkeley db concurrent data store to provide more concurrency
and better compatibility with other berkeley db routines.
* RegularExpression.cc (class RegularExpressionImpl): Fixed (yet
another) regexec() crash bug. Have to convert 8 bit chars to 7
bit before calling regexec() or it might crash on certain
sequences of 8 bit characters.
* Released as 0.9-dev-5 unstable package.
* spamprobe.cc (main): Temporarily disabled shared (read only)
locks in commands that used them as experiment to see if it
eliminates database corruption in berkeley db databases.
* MimeLineReader.cc: Using safe_char() to auto convert non-space
control chars to spaces.
2003-06-29 Brian Burton <brian@burton-computer.com>
* Added spamprobe-howto.html to contrib directory. Thanks to
Herman Oosthuysen.
* MessageFactory.cc (assignDigestToMessage): Added getMD5Digest()
call to top of function to fix an assertion thrown when messages
had digests in their headers.
* spamprobe.cc (main): Added setlocale() call (thanks to Junior
(don't know his name) for the suggestion) to fix tolower()
problems with accented characters in eight bit mode.
* FrequencyDBImpl_bdb.cc (writeWord): optimized writes to berkeley
db databases. Deletes records when their counts were going to
be written as zero to make purge 0 unnecessary.
(sweepOutOldTerms): Added code to remove MD5 records if they
have a count of zero. Removed code that wrote every record back
to the database (left over from mark and sweep days) for better
performance.
2003-05-20 Brian Burton <brian@galileo.burton-computer.com>
* spamprobe.cc (main): Cleaned up version printing (-V).
(main): summarize, find-good, and find-spam now print filename if
processing a file instead of stdin.
* FrequencyDBImpl_bdb.cc (open): Switched back to using a separate
lock file for berkeley db databases to avoid a possible
race condition.
2003-03-14 Brian Burton <brian@galileo.burton-computer.com>
* FrequencyDBImpl_cache.cc: Added feedback about whether a term is
from shared db to CacheEntry so that migration can be avoided if
counts don't change. This prevents terms from moving into the
private database if their time stamp changed but their counts
did not as might happen when running in training mode.
* FrequencyDBImpl_dual.cc (readWord): Added readWord()
implementation that gives a hint about whether or not the counts
came from the shared database.
2003-03-11 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_hash.cc (readWord): Fixed return value to allow
proper operation with shared database.
2003-03-09 Brian Burton <brian@burton-computer.com>
* Message.cc (addToken): Removed duplication when adding prefixed terms.
Previously term was added both with and without prefix. Now only
prefixed form is added.
* spamprobe.cc (main): Fixed -V option.
(train_on_message): Added current score test when training.
train-spam and train-good now use existing digest if any.
2003-03-08 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_hash.cc (setSize): Changed from using mod prime to
a bit mask for computing array indexes. Hash size can be specified
as number of bits in the range 33-63. Half that number of bits will
be used as a mask. The doubling allows file size to increase by
smaller increments than doublings. The most reasonable hash values
will be in the range 38 (4 MB) - 44 (32 MB). File sizes in this
range are roughly:
size megabytes terms
38 4 512k
39 6 768k
40 8 1024k
41 12 1536k (default)
42 16 2048k
43 24 3072k
44 32 4096k
* spamprobe.cc (process_stream): Train mode now updates timestamps
of terms in messages that don't need to be classified. This
prevents terms that are actually being used to score messages
from expiring.
* FrequencyDB.cc (touchMessage): Added new method to update timestamp
of terms in a message so that train commands can keep terms from
expiring.
2003-03-03 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (process_stream): Added train, train-good, and
train-spam commands for building database with a minimum number
of emails for better performance and less disk usage.
2003-02-28 Brian Burton <brian@burton-computer.com>
* MimeMessageReader.cc (readText): Fixed bug that caused rfc822
attachments to be treated as text rather than parsed into their
own mime parts. As a result base64 encoded attachments in
embedded messages wound up being tokenized rather than decoded
or ignored based on their mimetype.
* spamprobe.cc (main): Added support for combining multiple test
cases on the command line. Added counts command to print out
total message counts.
* SpamFilter.cc: Cleaned up code for computing score to share more
code. Fixed bug that ignored terms with wdf of 1. Added
support for wdf to alt1 scoring method. CHanged spam threshold of
alt1 method.
2003-02-26 Brian Burton <brian@burton-computer.com>
* MessageFactory.cc (removeHTMLFromText): Added test for each tag
to determine if it should add a space in its place. Previously
text like: j<br>u<br>n would be treated as "jun" instead of "j u
n". Prevents words from being combined if only space tags
separated them.
* configure.in: Added test for mmap.
* MessageFactory.cc (addWordToMessage): Fixed bug that did not
prefix word parts in prefixed headers.
* spamprobe.cc (main): Added Received header to list of headers
stored with a prefix. First Received header and subsequent ones
stored with different prefix to maybe detect falsified received
headers. First Received header should generally be more
trustworthy since it comes from your own mail server rather than
from the sending mail server or relay.
(main): Added From header to list of headers stored with prefix
since some spammers seem to use the same from line repeatedly - I
guess they think we trust their "brand".
* MessageFactory.cc (MessageFactory): Changed minimum word length
to 1.
* README.txt: Updated readme for -P option.
* spamprobe.1 (Content-Length): updated manpage for -P option.
* spamprobe.cc (main): Added -P command line option to automatically
purge terms with total count <= 4 after specified number of messages.
2003-02-07 Brian Burton <brian@burton-computer.com>
* Added new FrequencyDBImpl_hash class and made assorted changes
to FrequencyDB and other impl classes to support it. The hash
impl uses a fixed size array and Bob Jenkin's hash function to
provide an efficient though somewhat inaccurate database. Based
on the database structure in CRM114's mailfilter program. The
impl supports all the semantics of the other impls including
cleanup, dump, import, export, etc.
2003-02-06 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_bdb.cc (open): Modified berkeley db
implementation to lock the actual database file instead of
creating a separate lock file. This should work much more
smoothly with shared databases than the lock file did. Chose
not to use BDB's own locking environment because it seemed hard
to get right and prone to lock ups.
* LockFD.h (class LockFD): Added LockFD class to handle locking
an arbitrary file descriptor.
* LockFile.cc (lock): Changed LockFile to use a LockFD object
instead of calling fcntl() directly.
2003-01-30 Brian Burton <brian@burton-computer.com>
* Updated version of spamprobe.el from Dave Pearson's web site.
* Added README-mta-mda-mua.txt graciously contributed by Anto Veldre.
* MessageFactory.cc (removeHTMLFromText): replaces all whitespace
with space characters to avoid wierd crash in regex routines on
RedHat 8 systems.
2003-01-28 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (main): Added -M option to force a single message
per file (ignores content-length and From).
* Fixed manpage.
* Changed lock file mode to 0666 instead of 0600 so that shared
locks will work better. TODO: need to eliminate the need for
the lock file altogether.
2002-12-29 Brian Burton <brian@burton-computer.com>
* MessageFactory.cc (expandCharsInURL): Added decoding of %xx encoded
characters in URLs. Using this to prevent spammers from slipping
URLs through unchallenged by encoding them completely as hex.
2002-12-26 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (main): SpamProbe now stores words and phrases in
the to, cc, and subject headers both normally and with a special
prefix to improve accuracy since some words are spammier in the
subject than in the message body.
(main): Added -p option to limit number of words per phrase.
* Released version 0.8
2002-11-12 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (classify_message): spam and good commands now
count the words from messages multiple times if necessary to
ensure that they are recalled correctly. receive command does
not do this since it's decisions are not as reliable as manual
ones. This is intended to improve overall accuracy by
maximizing recall and making it harder for "spams of the future"
to slip through the cracks because of their low word counts.
2002-10-28 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (import_words): Fixed broken import command.
2002-10-27 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (process_stream): Added summarize command to print
find-good style output for every message whether good or spam.
2002-10-26 Brian Burton <brian@burton-computer.com>
* MimeMessageReader.cc (readNextHeader): Uses inexact content
length in case the mbox has incorrect content-length values.
* spamprobe.cc (process_stream): score and receive now print
message digest along with the score.
(main): all commands except receive look for digest in
X-SpamProbe header
* MessageFactory.cc (assignDigestToMessage): message digest now
taken from header if available.
2002-10-24 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (dump_words): flags now show all 8 digits in dump
(process_stream): receive mode supports -T option
2002-10-22 Brian Burton <brian@burton-computer.com>
* WordData.h (class WordData): Modified database to store a
16 bit time stamp (days since August 12, 2002) instead of
using sweep count for database cleanup.
2002-10-20 Brian Burton <brian@burton-computer.com>
* MessageFactory.cc (addTextToMessage): Removed the to_lower() to
avoid unecessary string copying. Made regex's case insensitive
so that they are not needed.
* spamprobe.cc (import_words): Uses regular expression to parse
import lines instead of hard coded logic.
* configure.in: Added test to verify existence of regex.h on
target system.
* MimeHeader.cc (isFromLine): Uses regular expression to detect
From lines instead of the hard coded scans.
* RegularExpression.h (class RegularExpression): Added
RegularExpression class as a front-end for POSIX regular
expression library.
* MessageFactory.cc: Uses regular expressions instead of hardcoded
logic to detect html tags in pages and find urls inside of tags.
* README.txt (including): Aded --enable-assert to configure script
so that assertions are off by default but can still be enabled
for debugging purposes.
2002-10-16 Brian Burton <brian@burton-computer.com>
* spamprobe.1: Changed version to just 0.7 so I don't have to keep
it up to date constantly.
* contrib/spamprobe.el: Updated to latest version of spamprobe.el
Thanks Dave!
* spamprobe.cc (main): Added --enable-8bit option to configure
script.
2002-10-15 Brian Burton <brian@burton-computer.com>
* configure.in (have_database): Moved berkeleydb tests into a
common spot. Added -ldb3 to the list of libraries to check.
* Switched to autoconf generated Makefile instead of the manual
one. The original makefile is now named Makefile.orig.
* Moved md5 files out of thirdparty and into the top level
directory.
2002-10-14 Brian Burton <brian@burton-computer.com>
* countscores.rb (goods): Changed to accomodate change to score
output.
* spamprobe.cc (process_stream): score command prints in same
format as receive command to simplify using it in procmailrc.
(find_message): Prints subject of message to make output more
human understandable.
* SpamFilter.cc (normalScoreMessage): Fixed NAN bug if inner and
outer both approx. 0. Returns 0.5 in that case to be safe.
* spamprobe.cc (main): Added command name validation and support
for shared locks for read only commands. Moved database lock
acquisition into the FrequencyDBImpl classes.
2002-10-11 Brian Burton <brian@burton-computer.com>
* FrequencyDBImpl_dual.h (class FrequencyDBImpl_dual): Added new
database impl class that uses a shared read only database and a
private read-write one. Also added -D option to program to
allow user to specify the shared db dir and made numerous
changes to other classes to put this new option into effect.
* spamprobe.cc (main): Modified to process multiple mboxes much
faster by opening and closing the database only once instead of
once per file.
* Released SpamProbe-0.7d
* spamprobe.cc (main): Added purge and edit-term commands.
* FrequencyDBImpl_bdb.cc (sweepOutJunk): Added purge mode.
2002-10-08 Brian Burton <brian@burton-computer.com>
* FrequencyDB.h (class FrequencyDB, class FrequencyDBImpl*): added
sweepOutJunk method for use by cleanup function.
* spamprobe.cc (cleanup_database): Added cleanup command to do a
mark and sweep database cleanup.
* WordData.h (class WordData): Promoted WordData to its own
class so that the cleanup function could be implemented.
2002-10-06 Brian Burton <brian@burton-computer.com>
* MimeMessageReader.cc (getMD5Digest): Replaced sprintf call with
hex_digit() util.cc function call.
* spamprobe.cc (import_words): import and export now use the
encode_string and decode_string functions from util.cc to
properly handle non-printable characters.
(main): Added -X option to rely almost exclusively on terms with
distance from mean >= 0.4 and allow word repeats of 5
(equivalent to -w 5 -r 5 -x)
2002-10-02 Brian Burton <brian@burton-computer.com>
* SpamFilter.cc (computeRatio): Fixed bug that returned word score
of 0.5 for messages which had 0 in one count. Only happened if
corresponding message count was also zero.
* FrequencyDB.cc: Removed uses of message id as database key.
* spamprobe.cc (dump_words): spamprobe dump now prints word
probabilities in addition to counts.
(main): Added -x command line option to allow top terms array to
extend past size limit if there are more significant terms than
can fit.
2002-09-20 Brian Burton <brian@burton-computer.com>
* SpamFilter.cc (scoreMessage): Relaxed the maximum top terms
array size limit to allow more terms to be used if their
distance from the mean is at least 0.4. This way emails with
many good and spam words get a more accurate evaluation since
the good terms don't squeeze all of the spammy words out. Seems
to yield a slight improvement on new, difficult spams without
increasing false positives.
2002-09-19 Brian Burton <brian@burton-computer.com>
* NewPtr.h (class NewPtr): Added NewPtr class to use in place of
auto_ptr. I'd rather follow the standard but some older
versions of g++ came with a broken auto_ptr.
* FrequencyDBImpl_bdb.cc (open): Added #if condition to handle the
gratuitous api change made by SleepyCat to the open() function.
2002-09-17 Brian Burton <brian@burton-computer.com>
* Released 0.7b with better mbox support, domain name break down,
and md5 digests for message identification.
* spamprobe.cc (import_words): Changed constructor arguments as
suggested by Xavier Nodet to work around problem with MSVC.
(set_headers): Added -H none command line option to ignore
headers when scoring a message.
* FrequencyDB.cc and lots of other files: Added MD5 digest as
unique identifier for emails instead of using message-id. For
now message-id is still used if digest not found but eventually
will remove it since digest is better identifier anyway.
2002-09-16 Brian Burton <brian@burton-computer.com>
* SpamFilter.cc (scoreMessage): Added code to put top tokens into
the Message object while scoring.
* Message.h (class Message): Added code to store and retrieve top
tokens.
* spamprobe.cc (main): Added -T command line option to print
top terms and their score and message count.
* Tokenizer.cc (is_special_char): Removed ' from special chars
since it seemed to hurt accuracy to include it.
2002-09-15 Brian Burton <brian@burton-computer.com>
* MimeMessageReader.cc (readText): Content type is now returned
with each text block so that it can be added to the token list.
2002-09-13 Brian Burton <brian@burton-computer.com>
* MimeHeader.cc (isFromLine): Improved mbox reading logic by
incorporating the From line format specification as defined in
the qmail mbox man page. Tried to be a little flexible for
flawed variations but still strict enough to not think a
sentence starting with From is a new message.
* MessageFactory.cc (addWordPartsToMessage): Now breaks tokens
containing non-alnums into pieces adding each sub-word plus each
suffix. This breaks host names down into their host and domain
names. This seems to improve accurracy.
* MimeHeader.cc (read): Added extra argument to control whether or
not to allow the header to begin with a From_ line. This fixes
a bug causing SP to miss some emails in mboxes if the preceeding
email was multipart and did not have a terminator.
* Released 0.7a with receive mode bug fix, solaris ctype functions
bug fix, and better tokenizer.
2002-09-12 Brian Burton <brian@burton-computer.com>
* Tokenizer.h (class Tokenizer): Changed tokenizing of text to
involve less copying.
* util.h: Added ctype front-end functions to work around problems
on solaris with non-ascii chars.
* MimeLineReader.cc (readLine): Rewrote loop to make it handle
lines terminated by only CR as well as CR or CRLF.
2002-09-11 Brian Burton <brian@burton-computer.com>
* MessageFactory.cc (addStringToMessage): Fixed bug that dropped 8
bit characters when m_replaceNonAsciiChars was false.
* MimeLineReader.cc (readLine): Converts null bytes into spaces.
* spamprobe.cc (import_words): Added import command to import
terms previously saved using export command.
2002-09-10 Brian Burton <brian@burton-computer.com>
* util.cc (is_all_digits): Added is_all_digits.
* MessageFactory.cc (addWordToMessage): Fixed all digits token
removal so that IP addresses are added as tokens.
* MimeMessageReader.cc (readToBoundary): When reading messages
from mboxes now honor content-length fields in the headers
unless -Y option was specified.
* FrequencyDBImpl_cache.h (class FrequencyDBImpl_cache): Added
is_dirty flag to cache entries so that values that haven't
changed don't get written to the database.
* spamprobe.cc (main): Added -S command line option to allow
messages per cache flush to be controlled from command line.
Some other code cleanup as well.
* FrequencyDBImpl_cache.h (class FrequencyDBImpl_cache): Added a
caching proxy frequency db impl class that uses an STL map to
cache term counts to reduce disk i/o at the expense of more cpu
time and memory usage.
* FrequencyDBImpl.h (class FrequencyDBImpl): Added an abstract
base class for frequency db impls so that I could have a caching
proxy.
* SpamFilter.cc (token_qsort_criterion): Fixed incorrect sort
order that put spammy words ahead of good words in the tie
breaker. Also imposed a limit on the term count when sorting
since counts above a certain number become basically identical.
* FrequencyDB.h (class FrequencyDB): Modified to use an
implementation class for all database access. This will make it
easier to plug in new ones later.
* FrequencyDBImpl_bdb.h (class FrequencyDBImpl): Added berkeley db
based implementation class to isolate the rest of the code from
the choice of database. This version uses btree files instead
of hash for better performance, smaller file sizes, and sorted
output during traversals.
* FrequencyDBImpl_dbm.h (class FrequencyDBImpl): Added dbm based
implementation class to isolate the rest of the code from the
choice of database.
* MessageFactory.cc (addWordToMessage): Fixed bug that allowed all
digit tokens to slip in.
2002-09-07 Brian Burton <brian@burton-computer.com>
* contrib/README-maildrop.txt: Added Matthias Andree's maildrop
howto to the contrib directory.
* MimeLineReader.cc (readLine): Fixed to properly handle null
bytes in lines. Not that those are valid but bugged mailer
sometimes embed them.
2002-09-06 Brian Burton <brian@burton-computer.com>
* spamprobe.cc: added two new commands: dump and export.
* SpamFilter.cc: Fixed a memory leak in scoreToken(). Converted
to use only a single FrequencyDB now. Added an accessor to
allow clients to get access to the db. Changed comparisons to
zero to allow for inexact floating point differences.
* FrequencyDB.cc: FrequencyDB modified to store both spam and good
word counts for each word in a single dbm file. Added a pair of
traveral functions for the export command.
* util.h: Moved iostream inclusion into util.h. Also added cctype
include there at Matthias Andree's suggestion for better gcc 3
compatibility.
2002-09-05 Brian Burton <brian@burton-computer.com>
* FrequencyDB.h: Added hooks for switching to berkeley db in ndbm
compatibility mode. GDBM does not scale well for large
databases. Will continue to use GDBM until 0.7 but will switch
over at that release.
* util.h: Added using namespace std to avoid problems on modern
C++ compilers. Thanks to Matthias Andree for bug report.
* README.txt: Put in fix to procmail recipe. Thanks to Steven
Grimm for bug report.
* FrequencyDB.cc (removeMessage): Will not attempt to remove a
message which has no message id.
(addMessage): Will not attempt to add a message which has no
message id.
* MessageFactory.cc (initMessage): Fixed bug 604808 which caused
messages with no message id to not have their bodies read.
Thanks to Steven Grimm for bug report.
* MimeMessageReader.cc (readText): Fixed potential bug which could
incorrectly skip part of message body for non-multipart
messages.
* MessageFactory.cc (addWordToMessage): Allows leading $ in
tokens. Ignores tokens consisting entirely of digits.
2002-09-03 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (set_headers): Removed obsolete test cases.
(process_stream): Added find-spam and find-good commands.
2002-09-02 Brian Burton <brian@burton-computer.com>
* Changed the use to string::find() == 0 to use a new inline
function that used strncmp() for greater efficiency.
* SpamFilter.cc (SpamFilter): Changed default scoring params to
use top 27 words and max of 2 repeats. Found this to be a good
option based on test runs with sample corpus.
* spamprobe.cc (set_headers): Added -H command line option to
control which headers are parsed to find tokens.
2002-08-30 Brian Burton <brian@burton-computer.com>
* spamprobe.cc (main): Added -h command line option to retain html
tags when generating tokens.
* MessageFactory.cc (expandEntitiesInHtml): When not removing html
we still expand any entities in the html.
* SpamFilter.cc (token_qsort_criterion): Modified token sort
criteria to favor good words over spammy ones if their distance
from mean and counts are equal.
* spamprobe.cc (process_test_case): Removed tune_1 test case.
* MessageFactory.cc (MessageFactory): Changed default settings for
better spam detection.
* SpamFilter.cc (SpamFilter): Changed default settings for better
spam detection.
* MessageFactory.cc (initMessage): Scoring additional headers.
Scoring subject header twice for extra emphasis.
* MessageFactory.h: Made the scoring parameters setting member
variables.
* SpamFilter.h: Made the scoring parameters setting member variables.
* MimeMessageReader.cc (readText): skips junk at end of message in
mime multipart messages. Previously became confused by the
extra junk and generated spurious scores for some messages which
threw off accuracy.
* Misc: Added scripts for testing accuracy of results.
