Improving the pyzor digester

[Oli S. <py...@li...>]

Hi

I noticed that pyzor recently received a feature to locally whitelist
digests - thanks a lot for this! This got me thinking of how the root
cause of these 'digest collisions' could be fixed (see
https://github.com/SpamExperts/pyzor/issues/3 ). 

I came up with a few ideas - but maybe I'm completely wrong about this,
so im hoping we could discuss it on the mailing list before I submit
any actual code.

#1 : Don't feed "invalid" data to the normalizer

The normalizer assumes to be working with text (it strips urls, emails,
unique identifiers, ...) but pyzor also feeds *undecoded* non-text
attachments. (https://github.com/SpamExperts/pyzor/blob/master/pyzor/digest.py#L173-L175) 

This causes messages with little or no body text and a base64 encoded
attachment to end up with a single line of base64 padding data used for
creating the digest. 
Ideas:
 - ignore non-text parts completely ( probably requires code to detect
text parts which don't actually have a text/* content-type header)
 - use a hash of all non-text parts to update the digest and don't feed
this hash to the normalizer

#2 : If a message body only contains a links/short text lines/very long
words (identifiers), the normalizer strips away everything and creates
a digest of the empty string which causes digest collisions. 

Let's say we have a body like this: 
"buy: http://shop1.example.com" 
In the current implementation, pyzor strips away both the url and the
text (because the text is shorter than 8 characters) and the message
gets the same digest as one with an empty body.

Idea: Don't *remove* dynamic parts like emails/urls/identifiers but
replace them with static markers like '[EMAIL ADDRESS]',
'[URLPATTERN]' , '[LONG STRING]' in order to keep the text structure
intact. So, in the example above, we'd actually hash the string
"buy:[URLPATTERN]"

#3 : Add more uniqueness to empty  body messages

Even with #2 implemented, there are situations where the digester
simply doesn't get any  data. For example, we see a lot of people
sending themselves reminder mails with no body ("Subject: 'buy milk'")

Idea: if there is no usable body data, use the Subject header to create
the digest

What do you think?

Best regards
Oli

-- 
message transmitted on 100% recycled electrons