[Dspam-devel] [ dspam-Bug Tracker-3142744 ] Undo whitelisting suggestion

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Bug Tracker item #3142744, was opened at 2010-12-23 12:31
Message generated for change (Comment added) made by sbajic
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3142744&group_id=250683

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Jens Finkhaeuser (unwesen)
Assigned to: Nobody/Anonymous (nobody)
Summary: Undo whitelisting suggestion

Initial Comment:
As I mentioned in a different issue (that I just found while looking for this), it seems very hard to train dspam to *not* whitelist some sender. This all seems to boil down to this code here:

    if (CTX->flags & DSF_WHITELIST) {                                                    
      if (ds_term->key == whitelist_token              &&.                               
          ds_term->s.spam_hits <= (ds_term->s.innocent_hits / 15) &&.                    
          ds_term->s.innocent_hits > CTX->wh_threshold &&.                               
          CTX->classification == DSR_NONE)                                               
      {                                                                                  
        do_whitelist = 1;                                                                
      }                                                                                  
    }

Ca. line 930 in libdspam.c.

The whitelist_token appears to be calculated from the sender address (or from: line); so I understand the logic that if a sender is found, and it's got 15x as many innocent hits as spam hits, then whitelist the message (leaving out a few details here).

I think that logic works well enough for deciding that a sender can be presumed innocent, but it doesn't work very well for suggesting that the sender might in fact not be a good candidate for whitelisting. That logic seems to be in there because the whitelist_token's spam probability is hardcoded to 0.5 (in _ds_calc_stats). Wouldn't it make much more sense to calculate its probability properly, and use wh_threshold as a probability threshold, i.e. if the spam probability is below 0.3 or whatever, then whitelist it?

That way you can use the same probability calculation as for other terms and therefore train dspam, but still treat the whitelist token as special in that if it is trained to be ok, then the rest of the tokens get disregarded because the message is whitelisted.

I've attached a patch that compiles, but is otherwise untested - mostly because I have no idea of what ramifications the change might have outside the code I touched. Also, it changes the meaning and format of the wh_token config variable, which is most likely *not* what you want. But it'll convey what I mean better than writing more text :)

----------------------------------------------------------------------

>Comment By: Stevan Bajic (sbajic)
Date: 2011-05-14 12:00

Message:
> The whitelist_token appears to be calculated from the
> sender address (or from: line);
>
Correct.

> so I understand the logic that if a sender is found,
> and it's got 15x as many innocent hits as spam hits,
> then whitelist the message (leaving out a few details
> here).
>
Here you are wrong. If you use whitelisting feature in DSPAM then you can
specify the whitelist threshold either in dspam.conf or with global/user
preferences. So the computation is:
1) if whitelisting is enabled then
1.1) if the token is a whitelist token AND the whitelist token has minimum
15 more innocent hits than spam hits AND the whitelist token innocent hit
count is bigger than the whitelist threshold AND the message is not yet
classified THEN do whitelist the sender

Your patch would completely mess up the whitelisting. The reason why your
patch will not work in production is because every one uses different
tokenizers and they all lead to different results. So assume you got this
from line:
From: Test User <te...@ex...>

The whitelisting code would now not process that line with the tokenizer
but make a hard coded whitelist token:
From*Test User <te...@ex...>

And it would assign a neutral score for it (aka 0.5)

Now assume you change the whitelist threshold to be a probabilistic value
of 0.3 and assume you just got one innocent hit and no spam hit for
"From*Test User <te...@ex...>" then the computation will quickly
produce either 0.0 or 1.0. Just from one single token. And assume a user
has completely messed up his token data and has a lot of spam tokens but
almost no innocent tokens or the other way around. That as well would
completely mess up the whitelist computation. Do you understand what I
mean?

Beside that it's hard to explain to some one that he now needs to enter
0.2 for the whitelist threshold. I already see the mailing list flooded
with questions what a good whitelist threshold is. Where the current
implementation allows to say: if you add 100 then you need at least 100
time to get a innocent message (or you need to train them as innocent) from
"Test User <te...@ex...>" to have "Test User <te...@ex...>"
whitelisted AND you need minimum 15 times more innocent hits than spam hits
for "Test User <te...@ex...>" to be whitelisted.

While looking at the code I think that one enhancement that could be done
to the whitelisting code would be to normalize the innocent hit counter by
subtracting the spam hits and then compare against the whitelist threshold.
Something like this:
if (CTX->flags & DSF_WHITELIST) {
  if (ds_term->key == whitelist_token &&
      ds_term->s.spam_hits <= (ds_term->s.innocent_hits / 15) &&
      (ds_term->s.innocent_hits - ds_term->s.spam_hits) >
CTX->wh_threshold &&
      CTX->classification == DSR_NONE)
  {
    do_whitelist = 1;
  }
}

> I think that logic works well enough for deciding that
> a sender can be presumed innocent, but it doesn't work
> very well for suggesting that the sender might in fact
> not be a good candidate for whitelisting.
>
So you got a from line that has 15 times more innocent hits than spam hits
and that has a innocent hit bigger than your specified whitelist thrashold
that you DO NOT consider worth to whitelist? Have you thought about
increasing the whitelist threshold or about blocklisting that
sender/domain?

IMHO a user and/or domain dependent whitelisting threshold would be a good
way to work around such issues. Currently DSPAM does not have that (only
has one single whitelist threshold for every sender. Senders are way to
easy to forge and using the whole From header line is a IMHO a good
trade-off).

----------------------------------------------------------------------

Comment By: Jens Finkhaeuser (unwesen)
Date: 2011-05-14 11:11

Message:
Did anyone have a chance to look at this?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=3142744&group_id=250683