[popfile-commit] engine/Classifier Bayes.pm,1.187,1.188

[<jgr...@us...>]

Update of /cvsroot/popfile/engine/Classifier
In directory sc8-pr-cvs1:/tmp/cvs-serv27342/Classifier

Modified Files:
	Bayes.pm 
Log Message:
English.msg:

Fixed a message that had no trailing </font> and would produce
incorrect HTML.


HTML.pm:

Further work on the performance of the loading and organizing
of the history from disk.  Start up is now much faster when
messages have previously been cached to disk.  Removal of old
files is delayed until a TICKD event.

Removed various lines of dead/legacy code revealed by the 
test suite.

Changed the various checkboxes that are used for history deletion
from all being called remove to having individual names, this
makes them easier to test.

Removed the code for invalidating the history cache, this is 
no longer needed because the progressive history update does
away with the need to go looking on disk for changes except
for at startup.


Bayes.pm:

Make QuickMagnets work again.
Change cutoff for unsure messages to 0.4.  
Add some close statements to the matrix loader which were missing.


TestModule.tst:

Initialize the logger to prevent a Perl warning.


TestConfiguration.tst:

Fix tests that break with ActivePerl because sort is returning
values in a different order.


TestProxy.pm:

Started working with ActivePerl for the test suite and 
discovered that it considers 0 to be a valid port number so
use the port number -1 for bad port number tests.


TestHTTP.pm:

Started working with ActivePerl for the test suite and 
discovered that it considers 0 to be a valid port number so
use the port number -1 for bad port number tests.


TestHTML.tst/TestHTML.script:

Tests for QuickMagnets, reclassification, message deletion,
reading of ICO, GIF, PNG, LOG, HTML and CSS files, tests for
the disk caching of history data, archiving of history files.

Additional safety code in the test runner to spot bad 
scripts or return codes from HTTP requests.  New code to 
follow redirects on form submission.

TestHTML.tst now covers 97% of HTML.pm's code.  Remaining 
tests needed are for the display of magnetized mails.


TestMailParse009.wrd/TestMailParse023.wrd:

Update words list since we now correctly add the domain name
from within an img src tag.









Index: Bayes.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Classifier/Bayes.pm,v
retrieving revision 1.187
retrieving revision 1.188
diff -C2 -d -r1.187 -r1.188
*** Bayes.pm	5 Sep 2003 21:18:51 -0000	1.187
--- Bayes.pm	8 Sep 2003 18:03:57 -0000	1.188
***************
*** 505,511 ****
--- 505,513 ----
              if ( $1 != $self->{corpus_version__} )  {
                  print STDERR "Incompatible corpus version in $bucket\n";
+                 close WORDS;
                  return 0;
              }
          } else {
+             close WORDS;
              return 0;
          }
***************
*** 519,528 ****
                  my $value = $2;
                  if ( $value > 0 )  {
- 
-                     # Here we do a simple "add one" smoothing on the data being
-                     # loaded
- 
-                     $value += 1;
- 
                      $self->{total__}{$bucket}        += $value;
                      $self->{unique__}{$bucket}       += 1;
--- 521,524 ----
***************
*** 742,746 ****
      my $certainty = ($c1-$c0 + 1) / 2;
  
!     if ( $certainty < 0.9 ) {
          $class = 'unsure';
      }
--- 738,742 ----
      my $certainty = ($c1-$c0 + 1) / 2;
  
!     if ( $certainty < 0.4 ) {
          $class = 'unsure';
      }
***************
*** 764,769 ****
              foreach my $type ( keys %types ) {
  
!                 if (defined $qm{$type})
!                 {
                      $i += 1;
  
--- 760,764 ----
              foreach my $type ( keys %types ) {
  
!                 if (defined $qm{$type}) {
                      $i += 1;
  
***************
*** 771,776 ****
                      $self->{scores__} .= "<select name=\"text$i\" id=\"\">\n";
  
!                     foreach my $magnet ( 0 .. $#{$qm{$type}} ) {
!                         $self->{scores__} .= "<option>" . Classifier::MailParse::splitline(@{$qm{$type}}[$magnet], 0) . "</option>\n";
                      }
                      $self->{scores__} .= "</select>\n";
--- 766,771 ----
                      $self->{scores__} .= "<select name=\"text$i\" id=\"\">\n";
  
!                     foreach my $magnet ( @{$qm{$type}} ) {
!                         $self->{scores__} .= "<option>$magnet</option>\n";
                      }
                      $self->{scores__} .= "</select>\n";