[popfile-commit] engine/tests TestLogger.tst,NONE,1.1 TestMailParse.tst,1.7,1.8

[<jgr...@us...>]

Update of /cvsroot/popfile/engine/tests
In directory sc8-pr-cvs1:/tmp/cvs-serv2831/tests

Modified Files:
	TestMailParse.tst 
Added Files:
	TestLogger.tst 
Log Message:
PERFORMANCE CHANGES

Bayes.pm:     Added new add_messages_to_bucket API to add multiple messages to a 
              bucket at the same time with a single read/write of the appropriate 
              corpus table for speed.

              New write_line__ method to write a line to a MSG file and optionally
              to the parse_line API of MailParse.pm.  Now we write a file to disk
              and parse it without reloading the MSG file from disk for speed.

              The MSG gets a temporary name until the CLS file is written to prevent
              the history from reloading in the middle of a download ending up with
              a message with a class file error

              classify_file becomes classify and can classify either from a file
              or from the preparsed information in the parser

              classify_and_modify returns the name of the file where the message
              was stored in addition to the classification.

HTML.pm:      Use add_messages_to_bucket API to reclassification for speed.

              Use the new classify method in Bayes.pm to classify a file after it
              has been digested by the parser for colorization and get the word
              scores.  This means we only load the MSG file once (used to be 
              twice) and hence double the speed of viewing a colorized message.

              New method load_disk_cache__ and save_disk_cache__ are used to 
              keep a copy of the history cache on disk between sessions so that
              session start up is as fast as possible.  There will be no need
              to parse messages for header information on start up if the
              history_cache file is present.

              Removed the boundary feature because it is incompatible with the
              concept of a "download" since we now send new history file messages
              async. through the MQ.

              Load the history cache progessively as files are written.  The proxies
              send the message NEWFL and the method new_history_file__ adds the 
              file to the history.  This is done so that when the user hits the 
              History tab button after a mail download the history cache is
              already loaded and there should be no delay in displaying the 
              history page.

MailParse.pm: Renamed parse_stream to parse_file since that's a better name
              New start_parse, stop_parse and parse_line APIs so that a file can 
              be parsed line by line.

MQ.pm:        Defined a new message type NEWFL which is used to indicate that
              a file has been added to the history cache.  NEWFL's message
              is the name of the file (the MSG file) that was added.

POP3.pm:      Send the NEWFL message through the pipe to the parent so that 
              the history is aware of new messages.

SMTP.pm:
NNTP.pm:      Send CLASS and NEWFL messages through the pipe to the parent.

insert.pl:    Updated to use new parse_file API

bayes.pl:     Updated to use new classify not classify_file API.

TEST SUITE CHANGES
 
tests.pl:     New test_assert_regexp function for doing fuzzy matching of
              test results.

              Returns 0 if all tests run successfully, and 1 if there are 
              any errors

TestLogger.tst: New file for testing POPFile::Logger functionality.

Makefile:     The test target has a variable TESTARGS can be set with the 
              specific module (or modules using glob patterns) to run.

              For example: gmake test TESTARGS='TestLogger'

              There's a new coverage target to run the test suite and output
              code coverage information for the modules used.

TestCoverage.pm: New module that provides line coverage information for 
              the test suite.  Executed as a Perl debugger using the -d
              switch and outputs code coverage information for all
              POPFile files tested.




--- NEW FILE: TestLogger.tst ---
# ---------------------------------------------------------------------------------------------
#
# Tests for Logger.pm
#
# Copyright (c) 2003 John Graham-Cumming
#
# ---------------------------------------------------------------------------------------------


use POPFile::Configuration;
use POPFile::MQ;
use POPFile::Logger;

my $c = new POPFile::Configuration;
my $mq = new POPFile::MQ;
my $l = new POPFile::Logger;

$c->configuration( $c );
$c->mq( $mq );
$c->logger( $l );

$l->configuration( $c );
$l->mq( $mq );
$l->logger( $l );

$l->initialize();

$mq->configuration( $c );
$mq->mq( $mq );
$mq->logger( $l );

# Test basic setup, name of the module, debug file location
# logging option
test_assert_equal( $l->name(), 'logger' );
test_assert_equal( $l->config_('logdir'), './' );
test_assert_equal( $l->global_config_('debug'), 1 );

# Change the log locatioin
$l->config_( 'logdir', 'tests/' );
test_assert_equal( $l->config_('logdir'), 'tests/' );
$l->calculate_today__();
test_assert_regexp( $l->debug_filename(), 'tests/' );

# Test that the last ten functionality works
my @last_ten = $l->last_ten();

test_assert_equal( $#last_ten, 0 );
test_assert_equal( $last_ten[0], 'log empty' );

$l->debug( 'test1' );

@last_ten = $l->last_ten();

test_assert_equal( $#last_ten, 0 );
test_assert_regexp( $last_ten[0], 'test1' );

$l->debug( 'test2' );

@last_ten = $l->last_ten();

test_assert_equal( $#last_ten, 1 );
test_assert_regexp( $last_ten[0], 'test1' );
test_assert_regexp( $last_ten[1], 'test2' );

# Check the time function is working to generate times to the nearest day
test_assert_equal( $l->{today__},  int( time / 86400 ) * 86400 );

# Read the debug file and check that it contains what we expect
open DEBUG, '<' . $l->debug_filename();
my $line = <DEBUG>;
test_assert_regexp( $line, 'test1' );
$line = <DEBUG>;
test_assert_regexp( $line, 'test2' );
close DEBUG;

# Check that we get a tick when a day passes
$l->{today__} -= 86400;
$l->service();

test_assert( defined( $mq->{queue__}{TICKD}[0][0] ), "checking TICKD message" );
test_assert( defined( $mq->{queue__}{TICKD}[0][1] ), "checking TICKD message" );

# Move the date ahead three days and check that the debug
# file gets deleted, this relies on the GNU date program
my $file = $l->debug_filename();
`date --set='2 days'`; 
$l->service();
my $exists = ( -e $file );
test_assert( $exists, "checking that debug file was deleted" );
`date --set='1 day'`; 
$l->service();
$exists = ( -e $file );
test_assert( !$exists, "checking that debug file was deleted" );
`date --set='3 days ago'`;


Index: TestMailParse.tst
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestMailParse.tst,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** TestMailParse.tst	26 Jun 2003 14:09:00 -0000	1.7
--- TestMailParse.tst	9 Jul 2003 18:18:25 -0000	1.8
***************
*** 117,121 ****
      # wrd file
  
!     $cl->parse_stream( $parse_test );
  
      open WORDS, "<$words";
--- 117,121 ----
      # wrd file
  
!     $cl->parse_file( $parse_test );
  
      open WORDS, "<$words";