Re: [Crm114-general] New Public release - 20041209.BlameFidelisEvenMore

[Bill Y. <ws...@me...>]

   From: Fidelis Assis <fi...@po...>

   Jonathan wrote:
   > On Fri, 10 Dec 2004 09:20:35 -0500 Bill wrote: 
   [...]
   > Personally I intend to
   > switch to OSB(F) with thick thresholds, but I'm not sure I'd recommend
   > it to someone who just wanted to spend less time on spam.

   I've made some tests and the following tables might help others in that
   decision:

   OSBF <microgroom> - 94.321 buckets
   --------------------------------------------
     Threshold  | Avg. errors  |  Avg. trainings
   --------------------------------------------
       10       |       1.0    |     16.9
	5       |       1.5    |     10.1
	1       |       2.6    |      4.6
	0 (TOE) |       4.4    |      4.4
   --------------------------------------------


   OSB <unique microgroom> - 282.001 buckets
   --------------------------------------------
     Threshold  | Avg. errors  | Avg. trainings
   --------------------------------------------
       10       |       2.1    |     37.9
	5       |       2.4    |     15.6
	1       |       3.8    |      5.9
	0 (TOE) |       4.8    |      4.8
   --------------------------------------------

Note that the difference at TOE is 0.4 messages; statistically
not very significant.  


   Errors and trainings in the tables above are relative to the last 500
   messages.

   Here is another table, with the average total trainings per shuffle:

   ----------------------------------------------------------
	      |  Avg. total trainings per shuffle (4147 msgs)
   Threshold |        OSBF        OSB
   ----------------------------------------------------------
	0               92          80
	1              107         102
	5              173         229
       10              263         473
   ----------------------------------------------------------

   All these tests were run on the 6 10xshuffles of SpamAssassin corpus
   I've been using.

   OSBF performed better in all aspects:
   - better accuracy for all thresholds, including TOE;
   - less trainings to keep final accuracy;
   - faster;
   - less space for the databases;
   - converges faster - less total trainings.

   The negative aspects are the fact that it's not as well tested as OSB
   and a special "voodoo" ingredient I introduced in the last confidence
   factor formula, which I myself don't quite understand why it works (or
   have been working) so well. So, unless one is afraid of some "voodoo", 
   OSBF seems to be a better option :).

The 'tested' aspect isn't the issue; it's based on the same core as
OSB, which is the same core as Markovian; your changes are actually 
pretty minimal (less than 30 lines, I think; pretty darn good).

The problem is the cognitive load of a third category for thickness-based
training... and how to interface that in Really Stupid Mail Readers.

That and the voodoo... :)  But as far as Joe User is concerned, it's all
voodoo anyway and a little more voodoo is no difference.  

     -Bill Yerazunis