Re: [Dspam-user] Increase Spam Hit Rate

[Stevan B. <st...@ba...>]

On 20.04.2012 07:32, Steve Fatula wrote:
> [...]

> If you give me the SPAM corpus, I can just run dspam_train on it (and 
> I'd even add my 80). But it will be pretty unbalanced since I have few 
> HAM messages since I only keep a month (maybe a few thousand 
> messages). I am not sure that matters much? In the end, won't the 
> detection still work, maybe biased towards SPAM at first, but, surely, 
> it woudln't take too long to stop false positives?
>
Lets say you want to make that merged global group. Then this is what 
you should do:

1) Create a new DSPAM user. If you can create a flat user (no 
loc...@do...d) because a flat user name will be easier to 
recognize on your setup where you usually have full blown up email 
addresses as user name. Lets say that new created user is called 
"SpamHitRate".

2) Change preferences for that user to:
dspam_admin change preference "SpamHitRate" "dailyQuarantineSummary" "off"
dspam_admin change preference "SpamHitRate" "enableBNR" "on"
dspam_admin change preference "SpamHitRate" "enableWhitelist" "off"
dspam_admin change preference "SpamHitRate" "fallbackDomain" "off"
dspam_admin change preference "SpamHitRate" "ignoreGroups" "on"
dspam_admin change preference "SpamHitRate" "ignoreRBLLookups" "on"
dspam_admin change preference "SpamHitRate" "makeCorpus" "off"
dspam_admin change preference "SpamHitRate" "optIn" "on"
dspam_admin change preference "SpamHitRate" "optOut" "off"
dspam_admin change preference "SpamHitRate" "optOutClamAV" "on"
dspam_admin change preference "SpamHitRate" "processorBias" "off"
dspam_admin change preference "SpamHitRate" "showFactors" "off"
dspam_admin change preference "SpamHitRate" "signatureLocation" "headers"
dspam_admin change preference "SpamHitRate" "spamAction" "deliver"
dspam_admin change preference "SpamHitRate" "spamSubject" ""
dspam_admin change preference "SpamHitRate" "statisticalSedation" "0"
dspam_admin change preference "SpamHitRate" "storeFragments" "off"
dspam_admin change preference "SpamHitRate" "tagNonspam" "off"
dspam_admin change preference "SpamHitRate" "tagSpam" "off"
dspam_admin change preference "SpamHitRate" "trainingMode" "TOE"
dspam_admin change preference "SpamHitRate" "trainPristine" "off"
dspam_admin change preference "SpamHitRate" "whitelistThreshold" "9999999"

Basically you want that user to not use ClamAV, nor any groups, nor any 
RBL, nor do you want whitelisting or any other mambo jambo. Usually you 
would not turn off that many helper mechanism on a normal user but this 
is not a normal user. You want that user to be as hard as possible. You 
don't care about false positive or false negative on that user. In fact 
this is exactly what you want. You want that user to generate as much 
false positive / negative as needed. Because the more FP/FN you have the 
more you can make DSPAM to learn. And this is what you are going to do 
mainly with that user. You are going to use dspam_train with Spam/Ham corpi.

3) Now go on and train with dspam_train: dspam_train SpamHitRate 
[spam_corpus maildir or mbox] [nonspam_corpus maildir or mbox]

4) After you are finished with dspam_train you should go on and run 
dspam_clean: dspam_clean -s0 -p0 -u0,0,0,0 SpamHitRate

5) Now you enable the merged global group by editing the DSPAM group 
file and there you add:
SpamHitRate:merged:*

6) You are using MySQL right? Now it is time to delete all users tokens 
except for SpamHitRate. To do that you just execute this (assuming the 
uid of SpamHitRate is 1000):

delete from dspam_signature_data where uid!=1000;
delete from dspam_stats where uid!=1000;
delete from dspam_token_data where uid!=1000;

analyze table dspam_signature_data;
analyze table dspam_stats;
analyze table dspam_token_data;

optimize table dspam_signature_data;
optimize table dspam_stats;
optimize table dspam_token_data;

After you have done that all old tokens and signatures and statistics 
for each user should be removed. This will lead to problems if user are 
going to try to retrain stuff that they got in the last days (since the 
signature data is purged). I don't think this will be a big issue on 
your setup since your users are using the dovecot anti-spam plugin and 
all DSPAM stuff is masked/hidden for them.

7) Change your dspam.conf to run in TOE instead of TEFT. Don't forget to 
check the preferences of each user if they don't have set by accident 
"trainingMode" to anything other than "TOE". Actually you could delete 
"trainingMode" if the user has that preference (it will fall back to 
that what you have set in dspam.conf, which should be in your case TOE).

8) Restart the DSPAM daemon.


> Thanks in advance!



-- 
Kind Regards from Switzerland,

Stevan Bajić