Menu
▾
▴
popfile-commit — CVS commit list messages
You can subscribe to this list here.
| 2003 |
Jan
|
Feb
(160) |
Mar
(119) |
Apr
(111) |
May
(118) |
Jun
(101) |
Jul
(304) |
Aug
(113) |
Sep
(140) |
Oct
(137) |
Nov
(87) |
Dec
(122) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2004 |
Jan
(78) |
Feb
(125) |
Mar
(131) |
Apr
(59) |
May
(121) |
Jun
(166) |
Jul
(150) |
Aug
(137) |
Sep
(73) |
Oct
(58) |
Nov
(27) |
Dec
(60) |
| 2005 |
Jan
(131) |
Feb
(84) |
Mar
(36) |
Apr
(8) |
May
(28) |
Jun
(20) |
Jul
(10) |
Aug
(72) |
Sep
(76) |
Oct
(34) |
Nov
(3) |
Dec
(29) |
| 2006 |
Jan
(13) |
Feb
(92) |
Mar
(7) |
Apr
(1) |
May
(1) |
Jun
(2) |
Jul
(4) |
Aug
(17) |
Sep
(5) |
Oct
(2) |
Nov
(8) |
Dec
(12) |
| 2007 |
Jan
(28) |
Feb
(15) |
Mar
|
Apr
|
May
(8) |
Jun
(4) |
Jul
(5) |
Aug
(8) |
Sep
(20) |
Oct
(38) |
Nov
(65) |
Dec
(92) |
| 2008 |
Jan
(21) |
Feb
(56) |
Mar
(27) |
Apr
(174) |
May
(25) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: <jgr...@us...> - 2003-09-09 05:14:49
|
Update of /cvsroot/popfile/engine/Classifier
In directory sc8-pr-cvs1:/tmp/cvs-serv14716/Classifier
Modified Files:
Bayes.pm
Log Message:
HTML.pm:
Fix minor bugs pointed out by the test suite.
TestHTML.tst/TestHTML.script:
Tests for magnet classified mails, showing them in the
main history list, filter for magnet classified and
non-magnet classified mails, looking at a message in
the single message view and checking the magnet use
highlighting.
With this commit we have 100% line coverage for the
HTML.pm module (and with the previous commits today
100% coverage for all shipping POPFile modules).
Index: Bayes.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Classifier/Bayes.pm,v
retrieving revision 1.188
retrieving revision 1.189
diff -C2 -d -r1.188 -r1.189
*** Bayes.pm 8 Sep 2003 18:03:57 -0000 1.188
--- Bayes.pm 9 Sep 2003 00:28:19 -0000 1.189
***************
*** 132,149 ****
# No default unclassified weight is the number of times more sure POPFile
! # must be of the top class vs the second class, default is 100 times more
! $self->config_( 'unclassified_weight', 100 );
# The corpus is kept in the 'corpus' subfolder of POPFile
$self->config_( 'corpus', 'corpus' );
# The characters that appear before and after a subject modification
$self->config_( 'subject_mod_left', '[' );
$self->config_( 'subject_mod_right', ']' );
# Get the hostname for use in the X-POPFile-Link header
$self->{hostname__} = hostname;
# Allow the user to override the hostname
$self->config_( 'hostname', $self->{hostname__} );
--- 132,154 ----
# No default unclassified weight is the number of times more sure POPFile
! # must be of the top class vs the second class, default is 10 times more
!
! $self->config_( 'unclassified_weight', 10 );
# The corpus is kept in the 'corpus' subfolder of POPFile
+
$self->config_( 'corpus', 'corpus' );
# The characters that appear before and after a subject modification
+
$self->config_( 'subject_mod_left', '[' );
$self->config_( 'subject_mod_right', ']' );
# Get the hostname for use in the X-POPFile-Link header
+
$self->{hostname__} = hostname;
# Allow the user to override the hostname
+
$self->config_( 'hostname', $self->{hostname__} );
***************
*** 946,950 ****
if ( open CLASS, '<' . $self->global_config_( 'msgdir' ) . $filename ) {
$bucket = <CLASS>;
! if ( $bucket =~ /([^ ]+) MAGNET (.+)/ ) {
$bucket = $1;
$magnet = $2;
--- 951,955 ----
if ( open CLASS, '<' . $self->global_config_( 'msgdir' ) . $filename ) {
$bucket = <CLASS>;
! if ( $bucket =~ /([^ ]+) MAGNET ([^\r\n]+)/ ) {
$bucket = $1;
$magnet = $2;
|
|
From: <jgr...@us...> - 2003-09-09 05:09:02
|
Update of /cvsroot/popfile/engine/tests
In directory sc8-pr-cvs1:/tmp/cvs-serv27342/tests
Modified Files:
TestBayes.tst TestConfiguration.tst TestHTML.script
TestHTML.tst TestHTTP.tst TestLogger.tst TestMailParse009.wrd
TestMailParse023.wrd TestModule.tst TestProxy.tst
Log Message:
English.msg:
Fixed a message that had no trailing </font> and would produce
incorrect HTML.
HTML.pm:
Further work on the performance of the loading and organizing
of the history from disk. Start up is now much faster when
messages have previously been cached to disk. Removal of old
files is delayed until a TICKD event.
Removed various lines of dead/legacy code revealed by the
test suite.
Changed the various checkboxes that are used for history deletion
from all being called remove to having individual names, this
makes them easier to test.
Removed the code for invalidating the history cache, this is
no longer needed because the progressive history update does
away with the need to go looking on disk for changes except
for at startup.
Bayes.pm:
Make QuickMagnets work again.
Change cutoff for unsure messages to 0.4.
Add some close statements to the matrix loader which were missing.
TestModule.tst:
Initialize the logger to prevent a Perl warning.
TestConfiguration.tst:
Fix tests that break with ActivePerl because sort is returning
values in a different order.
TestProxy.pm:
Started working with ActivePerl for the test suite and
discovered that it considers 0 to be a valid port number so
use the port number -1 for bad port number tests.
TestHTTP.pm:
Started working with ActivePerl for the test suite and
discovered that it considers 0 to be a valid port number so
use the port number -1 for bad port number tests.
TestHTML.tst/TestHTML.script:
Tests for QuickMagnets, reclassification, message deletion,
reading of ICO, GIF, PNG, LOG, HTML and CSS files, tests for
the disk caching of history data, archiving of history files.
Additional safety code in the test runner to spot bad
scripts or return codes from HTTP requests. New code to
follow redirects on form submission.
TestHTML.tst now covers 97% of HTML.pm's code. Remaining
tests needed are for the display of magnetized mails.
TestMailParse009.wrd/TestMailParse023.wrd:
Update words list since we now correctly add the domain name
from within an img src tag.
Index: TestBayes.tst
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestBayes.tst,v
retrieving revision 1.25
retrieving revision 1.26
diff -C2 -d -r1.25 -r1.26
*** TestBayes.tst 24 Aug 2003 04:04:52 -0000 1.25
--- TestBayes.tst 8 Sep 2003 18:03:57 -0000 1.26
***************
*** 66,76 ****
# Test the unclassifier_probability parameter
! test_assert_equal( $b->{unclassified__}, log(0.5) );
! $b->config_( 'unclassified_probability', 0.42 );
test_assert( $b->start() );
! test_assert_equal( $b->{unclassified__}, log(0.42) );
! $b->config_( 'unclassified_probability', 0.5 );
test_assert( $b->start() );
! test_assert_equal( $b->{unclassified__}, log(0.5) );
# test the API functions
--- 66,76 ----
# Test the unclassifier_probability parameter
! test_assert_equal( $b->{unclassified__}, log(100) );
! $b->config_( 'unclassified_weight', 10 );
test_assert( $b->start() );
! test_assert_equal( $b->{unclassified__}, log(10) );
! $b->config_( 'unclassified_weight', 5 );
test_assert( $b->start() );
! test_assert_equal( $b->{unclassified__}, log(5) );
# test the API functions
***************
*** 518,542 ****
# get_stopword_list
! my @stopwords = $b->get_stopword_list();
test_assert_equal( $#stopwords, 1 );
! test_assert_equal( $stopwords[0], 'notthis' );
! test_assert_equal( $stopwords[1], 'andnotthat' );
# add_stopword
test_assert( $b->add_stopword( 'northat' ) );
! @stopwords = $b->get_stopword_list();
test_assert_equal( $#stopwords, 2 );
! test_assert_equal( $stopwords[0], 'northat' );
! test_assert_equal( $stopwords[1], 'notthis' );
! test_assert_equal( $stopwords[2], 'andnotthat' );
# remove_stopword
test_assert( $b->remove_stopword( 'northat' ) );
! @stopwords = $b->get_stopword_list();
test_assert_equal( $#stopwords, 1 );
! test_assert_equal( $stopwords[0], 'notthis' );
! test_assert_equal( $stopwords[1], 'andnotthat' );
# Test history class file reading and writing
--- 518,542 ----
# get_stopword_list
! my @stopwords = sort $b->get_stopword_list();
test_assert_equal( $#stopwords, 1 );
! test_assert_equal( $stopwords[0], 'andnotthat' );
! test_assert_equal( $stopwords[1], 'notthis' );
# add_stopword
test_assert( $b->add_stopword( 'northat' ) );
! @stopwords = sort $b->get_stopword_list();
test_assert_equal( $#stopwords, 2 );
! test_assert_equal( $stopwords[0], 'andnotthat' );
! test_assert_equal( $stopwords[1], 'northat' );
! test_assert_equal( $stopwords[2], 'notthis' );
# remove_stopword
test_assert( $b->remove_stopword( 'northat' ) );
! @stopwords = sort $b->get_stopword_list();
test_assert_equal( $#stopwords, 1 );
! test_assert_equal( $stopwords[0], 'andnotthat' );
! test_assert_equal( $stopwords[1], 'notthis' );
# Test history class file reading and writing
***************
*** 583,587 ****
print FILE "From: test\@test.com\n";
print FILE "Subject: Your attention please\n\n";
! print FILE "This is the body\n.\n";
close FILE;
--- 583,587 ----
print FILE "From: test\@test.com\n";
print FILE "Subject: Your attention please\n\n";
! print FILE "This is the body www.supersizewebhosting.com www.gamelink.com\n.\n";
close FILE;
Index: TestConfiguration.tst
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestConfiguration.tst,v
retrieving revision 1.7
retrieving revision 1.8
diff -C2 -d -r1.7 -r1.8
*** TestConfiguration.tst 31 Jul 2003 16:32:22 -0000 1.7
--- TestConfiguration.tst 8 Sep 2003 18:03:57 -0000 1.8
***************
*** 126,129 ****
--- 126,131 ----
open FILE, "<popfile.cfg";
my $line = <FILE>;
+ test_assert_regexp( $line, 'config_piddir ../tests/' );
+ $line = <FILE>;
test_assert_regexp( $line, 'GLOBAL_debug 0' );
$line = <FILE>;
***************
*** 139,144 ****
$line = <FILE>;
test_assert_regexp( $line, 'GLOBAL_xtc 1' );
- $line = <FILE>;
- test_assert_regexp( $line, 'config_piddir ../tests/' );
$line = <FILE>;
test_assert_regexp( $line, 'logger_logdir ./' );
--- 141,144 ----
Index: TestHTML.script
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestHTML.script,v
retrieving revision 1.15
retrieving revision 1.16
diff -C2 -d -r1.15 -r1.16
*** TestHTML.script 28 Aug 2003 15:27:20 -0000 1.15
--- TestHTML.script 8 Sep 2003 18:03:57 -0000 1.16
***************
*** 37,40 ****
--- 37,41 ----
# INPUTIS a b Assert that form item a has value b
# SETINPUT a b Sets the form item a to value b
+ # SETINPUTN a b c Sets the bth input named a to c
# SUBMIT a Finds the form with input a and submits it
# SETSUBMIT a b Just like doing SETINPUT a b followed by SUBMIT a
***************
*** 253,257 ****
GET /history?session=$sk&setsort=
! # TODO Check reclassification
# Check click through from bucket name
--- 254,270 ----
GET /history?session=$sk&setsort=
! # Check reclassification
!
! GET /history
! MATCH Testing Refresh
! SETINPUT 0 spam
! CLICK change
! MATCH Changed to <font color="blue">spam</font>
! MATCH Reclassified as <font color="blue">spam</font>
!
! # Check undo
!
! CLICK undo_0
! MATCH Changed to <font color="green">personal</font>
# Check click through from bucket name
***************
*** 271,277 ****
MATCH <font color=\"green\">personal</font>
MATCH Scores
! MATCH 0.769209
! MATCH 0.230763
! MATCH 2.829186e-05
MATCH <font color="red">here</font>
MATCH 0.0017
--- 284,290 ----
MATCH <font color=\"green\">personal</font>
MATCH Scores
! MATCH 0. [28 zeroes] 265393
! MATCH 0. [29 zeroes] 796179
! MATCH 0. [33 zeroes] 976128
MATCH <font color="red">here</font>
MATCH 0.0017
***************
*** 289,295 ****
MATCH <font color=\"green\">personal</font>
MATCH Scores
! MATCH 0.769209
! MATCH 0.230763
! MATCH 2.829186e-05
MATCH <font color="red">here</font>
MATCH 0.0017
--- 302,308 ----
MATCH <font color=\"green\">personal</font>
MATCH Scores
! MATCH 0. [28 zeroes] 265393
! MATCH 0. [29 zeroes] 796179
! MATCH 0. [33 zeroes] 976128
MATCH <font color="red">here</font>
MATCH 0.0017
***************
*** 303,318 ****
MATCH Recent Messages
! # TODO quick magnets
! # TODO reclassify from single message view
! # TODO Check delete one message
MATCH Testing Refresh
GET /history
! #SETINPUT remove on
! #CLICK deletemessage
! #NOTMATCH Testing Refresh
! # TODO Check delete some messages
# Check delete page of messages
--- 316,369 ----
MATCH Recent Messages
! # Quick Magnets
! GET /jump_to_message?view=popfile10000=1.msg
! MATCH Single Message View
! MATCH <option>Testing</option>
! MATCH <option>Refresh</option>
! MATCH <option>John</option>
! SETINPUT text2 John
! SETINPUT bucket2 other
! CLICK create
! MATCH Current Magnets
! MATCH <input type="text" name="text1" value="John" size="50" />
! SETINPUT remove1 on
! CLICK delete
!
! # Reclassify from single message view
+ GET /view?view=popfile10000=1.msg&start_message=0&session=$sk
+ MATCH Single Message View
MATCH Testing Refresh
+ SETINPUT 0 spam
+ CLICK change
+ MATCH Changed to <font color="blue">spam</font>
+ MATCH Reclassified as <font color="blue">spam</font>
+
+ # Undo on single message view
+
+ GET /view?view=popfile10000=1.msg&start_message=0&session=$sk
+ MATCH Single Message View
+ CLICK undo_0
+ MATCH Changed to <font color="green">personal</font>
+
+ # Check delete one message
+
GET /history
! MATCH Testing Refresh
! SETINPUT remove_1 on
! CLICK deletemessage
! MATCH Recent Messages
! NOTMATCH Testing Refresh
! # Check delete some messages
!
! MATCH test for various HTML parts
! MATCH Enlarge your package Doctor ApprovedNHOB
! SETINPUT remove_4 on
! SETINPUT remove_5 on
! CLICK deletemessage
! NOTMATCH test for various HTML parts
! NOTMATCH Enlarge your package Doctor ApprovedNHOB
# Check delete page of messages
***************
*** 607,616 ****
INPUTIS ui_port 8081
- # TODO Check XMLRPC port (write when doing TestXMLRPC)
- # TODO Check SMTP port (write when doing TestSMTP)
- # TODO Check NNTP port (write when doing TestNNTP)
- # TODO Check NNTP separator (write when doing TestNNTP)
- # TODO Check Windows icon
-
# Security Page
--- 658,661 ----
***************
*** 693,701 ****
MATCH Updated password to
- # TODO Check SMTP stealth (write when doing TestSMTP)
- # TODO Check SMTP chain options (write when doing TestSMTP)
- # TODO Check NNTP stealth (write when doing TestNNTP)
- # TODO Check XMLRPC stealth (write when doing TestXMLRPC)
-
# Advanced Page
--- 738,741 ----
***************
*** 884,887 ****
--- 924,940 ----
MATCH <td width="1%"> </td><td align="right">0</td><td width="1%"> </td>
+ # Test removal of history files that are out of date (old and new style)
+
+ CODE
+ open TEMP, ">messages/popfile1_1.msg";
+ print TEMP, "1\n";
+ close TEMP;
+ open TEMP, ">messages/popfile1=1.msg";
+ print TEMP, "1\n";
+ close TEMP;
+ ENDCODE
+
+ SENDMSG TICKD dummy
+
# Check the login message
***************
*** 895,899 ****
MATCH POPFile Session Expired
! # TODO Get ICO, GIF, CSS, PNG, LOG and HTML file
# Get a page that does not exist
--- 948,995 ----
MATCH POPFile Session Expired
! # Get ICO, GIF, CSS, PNG, LOG and HTML file
!
! CODE
! open TEMP, ">temp.ico";
! print TEMP "10 GOTO 10\n";
! close TEMP;
! ENDCODE
!
! GET /temp.ico
! MATCH 10 GOTO 10
!
! CODE
! rename 'temp.ico', 'temp.gif';
! ENDCODE
!
! GET /temp.gif
! MATCH 10 GOTO 10
!
! CODE
! rename 'temp.gif', 'temp.png';
! ENDCODE
!
! GET /temp.png
! MATCH 10 GOTO 10
!
! CODE
! rename 'temp.png', 'popfile1.log';
! ENDCODE
!
! GET /popfile1.log?session=$sk
! MATCH 10 GOTO 10
!
! CODE
! mkdir 'manual';
! open TEMP, ">manual/temp.html";
! print TEMP "10 GOTO 10\n";
! close TEMP;
! ENDCODE
!
! GET /manual/temp.html
! MATCH 10 GOTO 10
!
! GET /skins/default.css
! MATCH .menuSelected
# Get a page that does not exist
***************
*** 918,921 ****
--- 1014,1026 ----
SETSUBMIT password secret
GET /advanced?session=123456
+ MATCH Enter password:
+ SETSUBMIT password wrong
+ MATCH Enter password:
+ MATCH Incorrect password
+ GET /?session=123456
+ MATCH Enter password:
+ SETSUBMIT password secret
+ MATCH Recent Messages
+ GET /advanced?session=123456&dummy=foo&dummy=bar
MATCH Enter password:
SETSUBMIT password secret
Index: TestHTML.tst
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestHTML.tst,v
retrieving revision 1.18
retrieving revision 1.19
diff -C2 -d -r1.18 -r1.19
*** TestHTML.tst 24 Aug 2003 01:20:34 -0000 1.18
--- TestHTML.tst 8 Sep 2003 18:03:58 -0000 1.19
***************
*** 64,71 ****
sub find_form
{
! my ( $name ) = @_;
foreach my $form (@forms) {
! my $input = $form->find_input( $name );
if ( defined( $input ) ) {
--- 64,71 ----
sub find_form
{
! my ( $name, $nth ) = @_;
foreach my $form (@forms) {
! my $input = $form->find_input( $name, undef, $nth );
if ( defined( $input ) ) {
***************
*** 100,105 ****
sub form_input
{
! my ( $name, $value ) = @_;
! my ( $form, $input ) = find_form( $name );
if ( defined( $form ) ) {
--- 100,105 ----
sub form_input
{
! my ( $name, $value, $nth ) = @_;
! my ( $form, $input ) = find_form( $name, $nth );
if ( defined( $form ) ) {
***************
*** 217,220 ****
--- 217,259 ----
if ( $command =~ /^__QUIT/ ) {
$h->stop();
+
+ # Test the history disk caching function
+
+ open TEMP, ">messages/popfile1=1.msg";
+ print TEMP "From: John\n\nBody\n";
+ close TEMP;
+
+ $h->start();
+ $h->stop();
+
+ $h->start();
+ $h->stop();
+
+ unlink( 'messages/popfile1=1.msg' );
+
+ $h->start();
+ $h->stop();
+
+ open TEMP, ">messages/popfile1=1.msg";
+ print TEMP "From: John\n\nBody\n";
+ close TEMP;
+ open TEMP, ">messages/popfile1=1.cls";
+ print TEMP "spam\n";
+ close TEMP;
+ open TEMP, ">messages/popfile1_1.msg";
+ print TEMP "From: John\n\nBody\n";
+ close TEMP;
+
+ $h->config_( 'archive', 1 );
+ $h->config_( 'archive_dir', 'archive' );
+ $h->config_( 'archive_classes', 1 );
+
+ `date --set='3 days'`;
+ $h->remove_mail_files();
+ `date --set='3 days ago'`;
+ test_assert( !( -e 'messages/popfile1=1.msg' ) );
+ test_assert( !( -e 'messages/popfile1_1.msg' ) );
+ test_assert( -e 'archive/spam/0/popfile1=1.msg' );
+
print $uwriter "OK\n";
last;
***************
*** 313,319 ****
if ( defined( $request ) ) {
my $response = $ua->request( $request );
! $content = $response->content;
@forms = HTML::Form->parse( $content, "http://127.0.0.1:$port" );
! }
next;
}
--- 352,365 ----
if ( defined( $request ) ) {
my $response = $ua->request( $request );
! if ( $response->code == 302 ) {
! $content = get(url("http://127.0.0.1:$port" . $response->headers->header('Location')));
! } else {
! test_assert_equal( $response->code, 200, "From script line $line_number" );
! $content = $response->content;
! }
@forms = HTML::Form->parse( $content, "http://127.0.0.1:$port" );
! } else {
! test_assert( 0, "Failed to create request form at script line $line_number" );
! }
next;
}
***************
*** 380,383 ****
--- 426,436 ----
}
+ if ( $line =~ /^SETINPUTN +([^ ]+) +(\d+) ?(.+)?$/ ) {
+ my ( $name, $nth, $value ) = ( $1, $2, $3 );
+ $value = '' if ( !defined( $value ) );
+ form_input( $name, $value, $nth );
+ next;
+ }
+
if ( $line =~ /^(SETINPUT|SETSUBMIT) +([^ ]+) ?(.+)?$/ ) {
my ( $name, $value ) = ( $2, $3 );
***************
*** 450,453 ****
--- 503,510 ----
next;
}
+
+ if ( $line =~ /[^ \t\r\n]/ ) {
+ test_assert( 0, "Don't understand line $line_number" );
+ }
}
Index: TestHTTP.tst
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestHTTP.tst,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** TestHTTP.tst 31 Jul 2003 16:32:22 -0000 1.2
--- TestHTTP.tst 8 Sep 2003 18:03:58 -0000 1.3
***************
*** 208,212 ****
test_assert( $line =~ /^$eol$/ );
$line = <FILE>;
! test_assert_equal( $line, "somechars\n" );
$line = <FILE>;
test_assert( !defined( $line ) );
--- 208,212 ----
test_assert( $line =~ /^$eol$/ );
$line = <FILE>;
! test_assert_equal( $line, "somechars$eol" );
$line = <FILE>;
test_assert( !defined( $line ) );
***************
*** 223,227 ****
$h2->initialize();
$h2->name( 'simple' );
! $h2->config_( 'port', 0 );
open (STDERR, ">stdout.tmp");
--- 223,227 ----
$h2->initialize();
$h2->name( 'simple' );
! $h2->config_( 'port', -1 );
open (STDERR, ">stdout.tmp");
Index: TestLogger.tst
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestLogger.tst,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** TestLogger.tst 31 Jul 2003 16:32:22 -0000 1.4
--- TestLogger.tst 8 Sep 2003 18:03:58 -0000 1.5
***************
*** 23,27 ****
# ---------------------------------------------------------------------------------------------
-
use POPFile::Configuration;
use POPFile::MQ;
--- 23,26 ----
Index: TestMailParse009.wrd
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestMailParse009.wrd,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** TestMailParse009.wrd 26 Jun 2003 14:19:41 -0000 1.6
--- TestMailParse009.wrd 8 Sep 2003 18:03:58 -0000 1.7
***************
*** 28,32 ****
header:Date 1
yherkhsdkjfhsdkjfhksjdfhkjsdhfiuweyrkhsdkfjhs 1
! subject:thr 1
perry 1
header:Return-Path 1
--- 28,36 ----
header:Date 1
yherkhsdkjfhsdkjfhksjdfhkjsdhfiuweyrkhsdkfjhs 1
! subject:thrôugh 1
! subject:unçõlleçted 1
! subject:eárn 1
! subject:fántástìç 1
! subject:mõnéy 1
perry 1
header:Return-Path 1
***************
*** 38,42 ****
html:fontcolorwhite 2
header:X-Priority 1
- subject:ted 1
wekrhksjdfhweyhfk 1
luke 1
--- 42,45 ----
***************
*** 50,54 ****
html:fontcolorfuchsia 1
kjsdhfkjsdhfkshdfiwueyrwhsfkjsdhfkh 1
- subject:lle 1
header:X-Mailer 1
header:Received 1
--- 53,56 ----
***************
*** 61,63 ****
subject:judgments 1
header:From 1
! subject:ugh 1
--- 63,65 ----
subject:judgments 1
header:From 1
! 202.108.221.16 4
Index: TestMailParse023.wrd
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestMailParse023.wrd,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** TestMailParse023.wrd 26 Jul 2003 22:53:24 -0000 1.1
--- TestMailParse023.wrd 8 Sep 2003 18:03:58 -0000 1.2
***************
*** 3,7 ****
000000 3
0078b3 1
! 27meg.com 11
access 1
action 1
--- 3,7 ----
000000 3
0078b3 1
! 27meg.com 19
access 1
action 1
***************
*** 75,79 ****
transference 1
utu 1
! www.27meg.com 11
young 2
younger 1
--- 75,79 ----
transference 1
utu 1
! www.27meg.com 19
young 2
younger 1
Index: TestModule.tst
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestModule.tst,v
retrieving revision 1.4
retrieving revision 1.5
diff -C2 -d -r1.4 -r1.5
*** TestModule.tst 31 Jul 2003 16:32:22 -0000 1.4
--- TestModule.tst 8 Sep 2003 18:03:58 -0000 1.5
***************
*** 89,92 ****
--- 89,93 ----
$m->logger( $l );
$l->configuration( $c );
+ $l->initialize();
$l->calculate_today__();
$m->global_config_( 'debug', 1 );
Index: TestProxy.tst
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestProxy.tst,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** TestProxy.tst 24 Aug 2003 03:30:10 -0000 1.12
--- TestProxy.tst 8 Sep 2003 18:03:58 -0000 1.13
***************
*** 528,532 ****
$sp2->initialize();
! $sp2->config_( 'port', 0 );
open (STDERR, ">stdout.tmp");
--- 528,532 ----
$sp2->initialize();
! $sp2->config_( 'port', -1 );
open (STDERR, ">stdout.tmp");
|
|
From: <jgr...@us...> - 2003-09-09 05:05:54
|
Update of /cvsroot/popfile/engine/UI
In directory sc8-pr-cvs1:/tmp/cvs-serv27342/UI
Modified Files:
HTML.pm
Log Message:
English.msg:
Fixed a message that had no trailing </font> and would produce
incorrect HTML.
HTML.pm:
Further work on the performance of the loading and organizing
of the history from disk. Start up is now much faster when
messages have previously been cached to disk. Removal of old
files is delayed until a TICKD event.
Removed various lines of dead/legacy code revealed by the
test suite.
Changed the various checkboxes that are used for history deletion
from all being called remove to having individual names, this
makes them easier to test.
Removed the code for invalidating the history cache, this is
no longer needed because the progressive history update does
away with the need to go looking on disk for changes except
for at startup.
Bayes.pm:
Make QuickMagnets work again.
Change cutoff for unsure messages to 0.4.
Add some close statements to the matrix loader which were missing.
TestModule.tst:
Initialize the logger to prevent a Perl warning.
TestConfiguration.tst:
Fix tests that break with ActivePerl because sort is returning
values in a different order.
TestProxy.pm:
Started working with ActivePerl for the test suite and
discovered that it considers 0 to be a valid port number so
use the port number -1 for bad port number tests.
TestHTTP.pm:
Started working with ActivePerl for the test suite and
discovered that it considers 0 to be a valid port number so
use the port number -1 for bad port number tests.
TestHTML.tst/TestHTML.script:
Tests for QuickMagnets, reclassification, message deletion,
reading of ICO, GIF, PNG, LOG, HTML and CSS files, tests for
the disk caching of history data, archiving of history files.
Additional safety code in the test runner to spot bad
scripts or return codes from HTTP requests. New code to
follow redirects on form submission.
TestHTML.tst now covers 97% of HTML.pm's code. Remaining
tests needed are for the display of magnetized mails.
TestMailParse009.wrd/TestMailParse023.wrd:
Update words list since we now correctly add the domain name
from within an img src tag.
Index: HTML.pm
===================================================================
RCS file: /cvsroot/popfile/engine/UI/HTML.pm,v
retrieving revision 1.198
retrieving revision 1.199
diff -C2 -d -r1.198 -r1.199
*** HTML.pm 5 Sep 2003 21:11:58 -0000 1.198
--- HTML.pm 8 Sep 2003 18:03:57 -0000 1.199
***************
*** 91,97 ****
# in the current filter, sort or search set.
#
- # history_invalid is set to cause the history cache to be reloaded by a call to
- # load_history_cache__, and is set by a call to invalidate_history_cache
- #
# If new items have been added to the history the set need_resort__ to 1 to ensure
# that the next time a history page is being displayed the appropriate sort, search
--- 91,94 ----
***************
*** 100,104 ****
$self->{history__} = {};
$self->{history_keys__} = ();
- $self->{history_invalid__} = 0;
$self->{need_resort__} = 0;
--- 97,100 ----
***************
*** 263,272 ****
# or History_NoSubject in while loading the cache
- $self->invalidate_history_cache();
$self->load_disk_cache__();
$self->load_history_cache__();
- $self->remove_mail_files();
-
return $self->SUPER::start();
}
--- 259,265 ----
***************
*** 284,288 ****
$self->copy_pre_cache__();
-
$self->save_disk_cache__();
}
--- 277,280 ----
***************
*** 413,418 ****
$self->parse_form_( $1 );
}
- } else {
- $url = '/';
}
} else {
--- 405,408 ----
***************
*** 466,470 ****
my $file = $self->{form_}{view};
! $self->copy_pre_cache__() if ($self->{need_resort__});
foreach my $akey ( keys %{ $self->{history__} } ) {
--- 456,460 ----
my $file = $self->{form_}{view};
! $self->copy_pre_cache__();
foreach my $akey ( keys %{ $self->{history__} } ) {
***************
*** 481,490 ****
$self->{form_}{setsearch} = 1;
! # Force a history_reload if we did not find this file in the history cache
! # but we do find it on disk using perl's -e file test operator (returns
! # true if the file exists).
!
! $self->invalidate_history_cache() if ( !$found );
! if ( -e ( $self->global_config_( 'msgdir' ) . $file ) ) {
$self->http_redirect_( $client, "/view?session=$self->{session_key__}&view=$self->{form_}{view}" );
} else {
--- 471,475 ----
$self->{form_}{setsearch} = 1;
! if ( $found ) {
$self->http_redirect_( $client, "/view?session=$self->{session_key__}&view=$self->{form_}{view}" );
} else {
***************
*** 2268,2273 ****
$body .= sprintf( $self->{language__}{Bucket_DoesNotAppear}, $word );
}
- } else {
- $body .= "<div class=\"error01\">$self->{language__}{Bucket_Error4}</div>";
}
--- 2253,2256 ----
***************
*** 2309,2327 ****
# ---------------------------------------------------------------------------------------------
#
- # invalidate_history_cache
- #
- # Called to notify the module that new files have been added to the history cache on disk
- # and the cache needs to be reloaded.
- #
- # ---------------------------------------------------------------------------------------------
- sub invalidate_history_cache
- {
- my ( $self ) = @_;
-
- $self->{history_invalid__} = 1;
- }
-
- # ---------------------------------------------------------------------------------------------
- #
# sort_filter_history
#
--- 2292,2295 ----
***************
*** 2583,2587 ****
}
- $self->{history_invalid__} = 0;
$self->{need_resort__} = 0;
$self->sort_filter_history( '', '', '' );
--- 2551,2554 ----
***************
*** 2686,2690 ****
}
! $self->{history__}{$file}{index} = $index;
}
--- 2653,2657 ----
}
! $self->{$cache}{$file}{index} = $index;
}
***************
*** 3102,3111 ****
# we are in control of deleting messages
! if ( defined($self->{form_}{remove_array}) ) {
! for my $i ( 0 .. $#{$self->{form_}{remove_array}} ) {
! $self->history_delete_file( $self->{history_keys__}[$self->{form_}{remove_array}[$i] - 1], 0);
! }
! } elsif ( defined($self->{form_}{remove}) ) {
! $self->history_delete_file( $self->{history_keys__}[$self->{form_}{remove} - 1], 0);
}
}
--- 3069,3076 ----
# we are in control of deleting messages
! for my $i ( keys %{$self->{form_}} ) {
! if ( $i =~ /^remove_(\d+)$/ ) {
! $self->history_delete_file( $self->{history_keys__}[$1 - 1], 0);
! }
}
}
***************
*** 3136,3140 ****
# applied. The watch word here is to avoid doing work
- $self->load_history_cache__() if ( $self->{history_invalid__} == 1 );
$self->sort_filter_history( $self->{form_}{filter}, # PROFILE BLOCK START
$self->{form_}{search},
--- 3101,3104 ----
***************
*** 3159,3163 ****
if ( !$self->history_cache_empty() ) {
- my $highlight_message = '';
my $start_message = 0;
--- 3123,3126 ----
***************
*** 3248,3258 ****
$body .= "<tr";
! if ( ( ( defined($self->{form_}{file}) && ( $self->{form_}{file} eq $mail_file ) ) ) || # PROFILE BLOCK START
! ( $highlight_message eq $mail_file ) ) { # PROFILE BLOCK STOP
! $body .= " class=\"rowHighlighted\"";
! } else {
! $body .= " class=\"";
! $body .= $stripe?"rowEven\"":"rowOdd\"";
! }
$stripe = 1 - $stripe;
--- 3211,3216 ----
$body .= "<tr";
! $body .= " class=\"";
! $body .= $stripe?"rowEven\"":"rowOdd\"";
$stripe = 1 - $stripe;
***************
*** 3270,3278 ****
$body .= " <input type=\"submit\" class=\"undoButton\" name=\"undo_$i\" value=\"$self->{language__}{Undo}\">\n";
} else {
! if ( !defined($self->{classifier__}->get_bucket_color($bucket))) {
! $body .= "$bucket</td>\n<td>";
! } else {
! $body .= "<a href=\"buckets?session=$self->{session_key__}&showbucket=$bucket\"><font color=\"" . $self->{classifier__}->get_bucket_color($bucket) . "\">$bucket</font></a></td>\n<td>";
! }
if ( $self->{history__}{$mail_file}{magnet} eq '' ) {
--- 3228,3232 ----
$body .= " <input type=\"submit\" class=\"undoButton\" name=\"undo_$i\" value=\"$self->{language__}{Undo}\">\n";
} else {
! $body .= "<a href=\"buckets?session=$self->{session_key__}&showbucket=$bucket\"><font color=\"" . $self->{classifier__}->get_bucket_color($bucket) . "\">$bucket</font></a></td>\n<td>";
if ( $self->{history__}{$mail_file}{magnet} eq '' ) {
***************
*** 3294,3298 ****
$body .= "</td>\n<td>\n";
$body .= "<label class=\"removeLabel\" for=\"remove_" . ( $i+1 ) . "\">$self->{language__}{Remove}</label>\n";
! $body .= "<input type=\"checkbox\" id=\"remove_" . ( $i+1 ) . "\" class=\"checkbox\" name=\"remove\" value=\"" . ( $i+1 ) . "\" />\n";
$body .= "</td>\n</tr>\n";
--- 3248,3252 ----
$body .= "</td>\n<td>\n";
$body .= "<label class=\"removeLabel\" for=\"remove_" . ( $i+1 ) . "\">$self->{language__}{Remove}</label>\n";
! $body .= "<input type=\"checkbox\" id=\"remove_" . ( $i+1 ) . "\" class=\"checkbox\" name=\"remove_" . ($i+1) . "\"/>\n";
$body .= "</td>\n</tr>\n";
***************
*** 3346,3359 ****
my ( $self, $client ) = @_;
- $self->load_history_cache__() if ( $self->{history_invalid__} == 1 );
-
- if ( $self->{need_resort__} == 1 ) {
-
- $self->copy_pre_cache__();
- $self->sort_filter_history( (defined($self->{form_}{filter})?$self->{form_}{filter}:''),
- (defined($self->{form_}{search})?$self->{form_}{search}:''),
- (defined($self->{form_}{sort})?$self->{form_}{sort}:'') );
- }
-
my $mail_file = $self->{form_}{view};
my $start_message = $self->{form_}{start_message} || 0;
--- 3300,3303 ----
***************
*** 3368,3400 ****
my $index = -1;
! foreach my $i ( $start_message .. $start_message + $page_size - 1) {
if ( $self->{history_keys__}[$i] eq $mail_file ) {
! $index = $i;
last;
}
}
- # If we fail to find the index of the message we are looking for then
- # do a full search of the history for it
-
- if ( $index == -1 ) {
- foreach my $i ( 0 .. $self->history_size()-1 ) {
- if ( $self->{history_keys__}[$i] eq $mail_file ) {
- use integer;
- $index = $i;
- $start_message = ($i / $page_size ) * $page_size;
- $self->{form_}{start_message} = $start_message;
- last;
- }
- }
- }
-
- # If we still can't find the message then return to the history page
-
- if ( $index == -1 ) {
- $self->log_("view page cache miss");
- return $self->http_redirect_( $client, '/history' );
- }
-
my $body = "<table width=\"100%\" summary=\"\">\n<tr>\n<td align=\"left\">\n";
--- 3312,3326 ----
my $index = -1;
!
! foreach my $i ( 0 .. $self->history_size()-1 ) {
if ( $self->{history_keys__}[$i] eq $mail_file ) {
! use integer;
! $index = $i;
! $start_message = ($i / $page_size ) * $page_size;
! $self->{form_}{start_message} = $start_message;
last;
}
}
my $body = "<table width=\"100%\" summary=\"\">\n<tr>\n<td align=\"left\">\n";
***************
*** 3696,3699 ****
--- 3622,3626 ----
# ---------------------------------------------------------------------------------------------
+ #
# copy_pre_cache__
#
***************
*** 3706,3714 ****
# Copy the history pre-cache over AFTER any possibly index-based remove operations are complete
! foreach my $file ( keys( %{$self->{history_pre_cache__}} ) ) {
! $self->{history__}{$file} = \%{$self->{history_pre_cache__}{$file}};
delete $self->{history_pre_cache__}{$file};
}
$self->{history_pre_cache__} = {};
}
--- 3633,3649 ----
# Copy the history pre-cache over AFTER any possibly index-based remove operations are complete
!
! my $index = $self->history_size() + 1;
! my $added = 0;
! foreach my $file (sort compare_mf keys %{$self->{history_pre_cache__}} ) {
! $self->{history__}{$file} = $self->{history_pre_cache__}{$file};
! $self->{history__}{$file}{index} = $index;
! $index += 1;
! $added = 1;
delete $self->{history_pre_cache__}{$file};
}
+
$self->{history_pre_cache__} = {};
+ $self->sort_filter_history( '', '', '' ) if ( $added );
}
***************
*** 3728,3732 ****
while ( my $mail_file = readdir MESSAGES ) {
! if ( $mail_file =~ /popfile\d+=\d+\.msg$/ ) {
my ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,$blksize,$blocks) = stat( $self->global_config_( 'msgdir' ) . $mail_file);
--- 3663,3667 ----
while ( my $mail_file = readdir MESSAGES ) {
! if ( $mail_file =~ /popfile(\d+)=\d+\.msg$/ ) {
my ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,$blksize,$blocks) = stat( $self->global_config_( 'msgdir' ) . $mail_file);
***************
*** 3784,3788 ****
my ($reclassified, $bucket, $usedtobe, $magnet) = $self->{classifier__}->history_read_class( $mail_file );
! if ( ( $bucket ne 'unclassified' ) && ( $bucket ne 'unknown class' ) ) {
$path .= "\/" . $bucket;
mkdir( $path );
--- 3719,3723 ----
my ($reclassified, $bucket, $usedtobe, $magnet) = $self->{classifier__}->history_read_class( $mail_file );
! if ( ( $bucket ne 'unclassified' ) && ( $bucket ne 'unknown class' ) && ( $bucket ne 'unsure' ) ) {
$path .= "\/" . $bucket;
mkdir( $path );
***************
*** 3884,3889 ****
}
! return $formstring if ($count > 0);
! return '';
}
--- 3819,3823 ----
}
! return ($count>0)?$formstring:'';
}
|
|
From: <jgr...@us...> - 2003-09-09 05:05:54
|
Update of /cvsroot/popfile/engine/tests/languages In directory sc8-pr-cvs1:/tmp/cvs-serv27342/tests/languages Modified Files: English.msg Log Message: English.msg: Fixed a message that had no trailing </font> and would produce incorrect HTML. HTML.pm: Further work on the performance of the loading and organizing of the history from disk. Start up is now much faster when messages have previously been cached to disk. Removal of old files is delayed until a TICKD event. Removed various lines of dead/legacy code revealed by the test suite. Changed the various checkboxes that are used for history deletion from all being called remove to having individual names, this makes them easier to test. Removed the code for invalidating the history cache, this is no longer needed because the progressive history update does away with the need to go looking on disk for changes except for at startup. Bayes.pm: Make QuickMagnets work again. Change cutoff for unsure messages to 0.4. Add some close statements to the matrix loader which were missing. TestModule.tst: Initialize the logger to prevent a Perl warning. TestConfiguration.tst: Fix tests that break with ActivePerl because sort is returning values in a different order. TestProxy.pm: Started working with ActivePerl for the test suite and discovered that it considers 0 to be a valid port number so use the port number -1 for bad port number tests. TestHTTP.pm: Started working with ActivePerl for the test suite and discovered that it considers 0 to be a valid port number so use the port number -1 for bad port number tests. TestHTML.tst/TestHTML.script: Tests for QuickMagnets, reclassification, message deletion, reading of ICO, GIF, PNG, LOG, HTML and CSS files, tests for the disk caching of history data, archiving of history files. Additional safety code in the test runner to spot bad scripts or return codes from HTTP requests. New code to follow redirects on form submission. TestHTML.tst now covers 97% of HTML.pm's code. Remaining tests needed are for the display of magnetized mails. TestMailParse009.wrd/TestMailParse023.wrd: Update words list since we now correctly add the domain name from within an img src tag. Index: English.msg =================================================================== RCS file: /cvsroot/popfile/engine/tests/languages/English.msg,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -d -r1.1 -r1.2 *** English.msg 20 Aug 2003 03:09:04 -0000 1.1 --- English.msg 8 Sep 2003 18:03:58 -0000 1.2 *************** *** 161,165 **** History_MagnetUsed Magnet used History_MagnetBecause <b>Magnet Used</b><p>Classified to <font color="%s">%s</font> because of magnet %s </p> ! History_ChangedTo Changed to <font color="%s">%s History_Already Reclassified as <font color="%s">%s</font> History_RemoveAll Remove All --- 161,165 ---- History_MagnetUsed Magnet used History_MagnetBecause <b>Magnet Used</b><p>Classified to <font color="%s">%s</font> because of magnet %s </p> ! History_ChangedTo Changed to <font color="%s">%s</font> History_Already Reclassified as <font color="%s">%s</font> History_RemoveAll Remove All |
|
From: <jgr...@us...> - 2003-09-09 02:39:27
|
Update of /cvsroot/popfile/engine/UI
In directory sc8-pr-cvs1:/tmp/cvs-serv14716/UI
Modified Files:
HTML.pm
Log Message:
HTML.pm:
Fix minor bugs pointed out by the test suite.
TestHTML.tst/TestHTML.script:
Tests for magnet classified mails, showing them in the
main history list, filter for magnet classified and
non-magnet classified mails, looking at a message in
the single message view and checking the magnet use
highlighting.
With this commit we have 100% line coverage for the
HTML.pm module (and with the previous commits today
100% coverage for all shipping POPFile modules).
Index: HTML.pm
===================================================================
RCS file: /cvsroot/popfile/engine/UI/HTML.pm,v
retrieving revision 1.199
retrieving revision 1.200
diff -C2 -d -r1.199 -r1.200
*** HTML.pm 8 Sep 2003 18:03:57 -0000 1.199
--- HTML.pm 9 Sep 2003 00:28:20 -0000 1.200
***************
*** 2413,2420 ****
if ( $first =~ /___HISTORY__ __ VERSION__ 1/ ) {
while ( my $line = <CACHE> ) {
! if ( !( $line =~ /__HISTORY__ __BOUNDARY__/ ) ) {
! $self->log_( "Problem in history_cache file, expecting boundary got $line" );
! last;
! }
$line = <CACHE>;
--- 2413,2417 ----
if ( $first =~ /___HISTORY__ __ VERSION__ 1/ ) {
while ( my $line = <CACHE> ) {
! last if ( !( $line =~ /__HISTORY__ __BOUNDARY__/ ) );
$line = <CACHE>;
***************
*** 2630,2634 ****
$short_subject =~ s/>/>/g;
! # If the index is known, stick it straight into the history else# go into
# the precache for merging into history when the history is viewed next
--- 2627,2631 ----
$short_subject =~ s/>/>/g;
! # If the index is known, stick it straight into the history else go into
# the precache for merging into history when the history is viewed next
***************
*** 2638,2642 ****
}
-
$self->{$cache}{$file}{bucket} = $bucket;
$self->{$cache}{$file}{reclassified} = $reclassified;
--- 2635,2638 ----
***************
*** 3041,3048 ****
if ( $filter eq '__filter__magnet' ) {
$filtered .= $self->{language__}{History_Magnet};
- } elsif ( $filter eq '__filter__no__magnet' ) {
- $filtered .= $self->{language__}{History_NoMagnet};
} else {
! $filtered = sprintf( $self->{language__}{History_Filter}, $self->{classifier__}->get_bucket_color($self->{form_}{filter}), $self->{form_}{filter} ) if ( $self->{form_}{filter} ne '' );
}
}
--- 3037,3046 ----
if ( $filter eq '__filter__magnet' ) {
$filtered .= $self->{language__}{History_Magnet};
} else {
! if ( $filter eq '__filter__no__magnet' ) {
! $filtered .= $self->{language__}{History_NoMagnet};
! } else {
! $filtered = sprintf( $self->{language__}{History_Filter}, $self->{classifier__}->get_bucket_color($self->{form_}{filter}), $self->{form_}{filter} ) if ( $self->{form_}{filter} ne '' );
! }
}
}
***************
*** 3242,3246 ****
$body .= "</select>\n";
} else {
! $body .= " ($self->{language__}{History_MagnetUsed}: " . Classifier::MailParse::splitline( $self->{history__}{$mail_file}{magnet}, 0 ) . ")";
}
}
--- 3240,3244 ----
$body .= "</select>\n";
} else {
! $body .= " ($self->{language__}{History_MagnetUsed}: " . $self->{history__}{$mail_file}{magnet} . ")";
}
}
***************
*** 3388,3392 ****
$body .= "</select>\n<input type=\"submit\" class=\"reclassifyButton\" name=\"change\" value=\"$self->{language__}{Reclassify}\" />";
} else {
! $body .= " ($self->{language__}{History_MagnetUsed}: " . Classifier::MailParse::splitline( $self->{history__}{$mail_file}{magnet} , 0) . ")";
}
}
--- 3386,3390 ----
$body .= "</select>\n<input type=\"submit\" class=\"reclassifyButton\" name=\"change\" value=\"$self->{language__}{Reclassify}\" />";
} else {
! $body .= " ($self->{language__}{History_MagnetUsed}: " . $self->{history__}{$mail_file}{magnet} . ")";
}
}
***************
*** 3461,3468 ****
$self->{classifier__}->scores('');
} else {
! $body .= sprintf( $self->{language__}{History_MagnetBecause},
$color, $bucket,
Classifier::MailParse::splitline($self->{history__}{$mail_file}{magnet},0)
! );
}
--- 3459,3466 ----
$self->{classifier__}->scores('');
} else {
! $body .= sprintf( $self->{language__}{History_MagnetBecause}, # PROFILE BLOCK START
$color, $bucket,
Classifier::MailParse::splitline($self->{history__}{$mail_file}{magnet},0)
! ); # PROFILE BLOCK STOP
}
|
|
From: <jgr...@us...> - 2003-09-09 01:35:22
|
Update of /cvsroot/popfile/engine/languages In directory sc8-pr-cvs1:/tmp/cvs-serv27342/languages Modified Files: English.msg Log Message: English.msg: Fixed a message that had no trailing </font> and would produce incorrect HTML. HTML.pm: Further work on the performance of the loading and organizing of the history from disk. Start up is now much faster when messages have previously been cached to disk. Removal of old files is delayed until a TICKD event. Removed various lines of dead/legacy code revealed by the test suite. Changed the various checkboxes that are used for history deletion from all being called remove to having individual names, this makes them easier to test. Removed the code for invalidating the history cache, this is no longer needed because the progressive history update does away with the need to go looking on disk for changes except for at startup. Bayes.pm: Make QuickMagnets work again. Change cutoff for unsure messages to 0.4. Add some close statements to the matrix loader which were missing. TestModule.tst: Initialize the logger to prevent a Perl warning. TestConfiguration.tst: Fix tests that break with ActivePerl because sort is returning values in a different order. TestProxy.pm: Started working with ActivePerl for the test suite and discovered that it considers 0 to be a valid port number so use the port number -1 for bad port number tests. TestHTTP.pm: Started working with ActivePerl for the test suite and discovered that it considers 0 to be a valid port number so use the port number -1 for bad port number tests. TestHTML.tst/TestHTML.script: Tests for QuickMagnets, reclassification, message deletion, reading of ICO, GIF, PNG, LOG, HTML and CSS files, tests for the disk caching of history data, archiving of history files. Additional safety code in the test runner to spot bad scripts or return codes from HTTP requests. New code to follow redirects on form submission. TestHTML.tst now covers 97% of HTML.pm's code. Remaining tests needed are for the display of magnetized mails. TestMailParse009.wrd/TestMailParse023.wrd: Update words list since we now correctly add the domain name from within an img src tag. Index: English.msg =================================================================== RCS file: /cvsroot/popfile/engine/languages/English.msg,v retrieving revision 1.40 retrieving revision 1.41 diff -C2 -d -r1.40 -r1.41 *** English.msg 20 Aug 2003 01:47:41 -0000 1.40 --- English.msg 8 Sep 2003 18:03:57 -0000 1.41 *************** *** 161,165 **** History_MagnetUsed Magnet used History_MagnetBecause <b>Magnet Used</b><p>Classified to <font color="%s">%s</font> because of magnet %s </p> ! History_ChangedTo Changed to <font color="%s">%s History_Already Reclassified as <font color="%s">%s</font> History_RemoveAll Remove All --- 161,165 ---- History_MagnetUsed Magnet used History_MagnetBecause <b>Magnet Used</b><p>Classified to <font color="%s">%s</font> because of magnet %s </p> ! History_ChangedTo Changed to <font color="%s">%s</font> History_Already Reclassified as <font color="%s">%s</font> History_RemoveAll Remove All |
|
From: <jgr...@us...> - 2003-09-09 01:35:11
|
Update of /cvsroot/popfile/engine/Classifier
In directory sc8-pr-cvs1:/tmp/cvs-serv27342/Classifier
Modified Files:
Bayes.pm
Log Message:
English.msg:
Fixed a message that had no trailing </font> and would produce
incorrect HTML.
HTML.pm:
Further work on the performance of the loading and organizing
of the history from disk. Start up is now much faster when
messages have previously been cached to disk. Removal of old
files is delayed until a TICKD event.
Removed various lines of dead/legacy code revealed by the
test suite.
Changed the various checkboxes that are used for history deletion
from all being called remove to having individual names, this
makes them easier to test.
Removed the code for invalidating the history cache, this is
no longer needed because the progressive history update does
away with the need to go looking on disk for changes except
for at startup.
Bayes.pm:
Make QuickMagnets work again.
Change cutoff for unsure messages to 0.4.
Add some close statements to the matrix loader which were missing.
TestModule.tst:
Initialize the logger to prevent a Perl warning.
TestConfiguration.tst:
Fix tests that break with ActivePerl because sort is returning
values in a different order.
TestProxy.pm:
Started working with ActivePerl for the test suite and
discovered that it considers 0 to be a valid port number so
use the port number -1 for bad port number tests.
TestHTTP.pm:
Started working with ActivePerl for the test suite and
discovered that it considers 0 to be a valid port number so
use the port number -1 for bad port number tests.
TestHTML.tst/TestHTML.script:
Tests for QuickMagnets, reclassification, message deletion,
reading of ICO, GIF, PNG, LOG, HTML and CSS files, tests for
the disk caching of history data, archiving of history files.
Additional safety code in the test runner to spot bad
scripts or return codes from HTTP requests. New code to
follow redirects on form submission.
TestHTML.tst now covers 97% of HTML.pm's code. Remaining
tests needed are for the display of magnetized mails.
TestMailParse009.wrd/TestMailParse023.wrd:
Update words list since we now correctly add the domain name
from within an img src tag.
Index: Bayes.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Classifier/Bayes.pm,v
retrieving revision 1.187
retrieving revision 1.188
diff -C2 -d -r1.187 -r1.188
*** Bayes.pm 5 Sep 2003 21:18:51 -0000 1.187
--- Bayes.pm 8 Sep 2003 18:03:57 -0000 1.188
***************
*** 505,511 ****
--- 505,513 ----
if ( $1 != $self->{corpus_version__} ) {
print STDERR "Incompatible corpus version in $bucket\n";
+ close WORDS;
return 0;
}
} else {
+ close WORDS;
return 0;
}
***************
*** 519,528 ****
my $value = $2;
if ( $value > 0 ) {
-
- # Here we do a simple "add one" smoothing on the data being
- # loaded
-
- $value += 1;
-
$self->{total__}{$bucket} += $value;
$self->{unique__}{$bucket} += 1;
--- 521,524 ----
***************
*** 742,746 ****
my $certainty = ($c1-$c0 + 1) / 2;
! if ( $certainty < 0.9 ) {
$class = 'unsure';
}
--- 738,742 ----
my $certainty = ($c1-$c0 + 1) / 2;
! if ( $certainty < 0.4 ) {
$class = 'unsure';
}
***************
*** 764,769 ****
foreach my $type ( keys %types ) {
! if (defined $qm{$type})
! {
$i += 1;
--- 760,764 ----
foreach my $type ( keys %types ) {
! if (defined $qm{$type}) {
$i += 1;
***************
*** 771,776 ****
$self->{scores__} .= "<select name=\"text$i\" id=\"\">\n";
! foreach my $magnet ( 0 .. $#{$qm{$type}} ) {
! $self->{scores__} .= "<option>" . Classifier::MailParse::splitline(@{$qm{$type}}[$magnet], 0) . "</option>\n";
}
$self->{scores__} .= "</select>\n";
--- 766,771 ----
$self->{scores__} .= "<select name=\"text$i\" id=\"\">\n";
! foreach my $magnet ( @{$qm{$type}} ) {
! $self->{scores__} .= "<option>$magnet</option>\n";
}
$self->{scores__} .= "</select>\n";
|
|
From: <jgr...@us...> - 2003-09-09 01:28:09
|
Update of /cvsroot/popfile/engine/tests
In directory sc8-pr-cvs1:/tmp/cvs-serv14716/tests
Modified Files:
TestBayes.tst TestHTML.script
Log Message:
HTML.pm:
Fix minor bugs pointed out by the test suite.
TestHTML.tst/TestHTML.script:
Tests for magnet classified mails, showing them in the
main history list, filter for magnet classified and
non-magnet classified mails, looking at a message in
the single message view and checking the magnet use
highlighting.
With this commit we have 100% line coverage for the
HTML.pm module (and with the previous commits today
100% coverage for all shipping POPFile modules).
Index: TestBayes.tst
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestBayes.tst,v
retrieving revision 1.26
retrieving revision 1.27
diff -C2 -d -r1.26 -r1.27
*** TestBayes.tst 8 Sep 2003 18:03:57 -0000 1.26
--- TestBayes.tst 9 Sep 2003 00:28:20 -0000 1.27
***************
*** 66,73 ****
# Test the unclassifier_probability parameter
- test_assert_equal( $b->{unclassified__}, log(100) );
- $b->config_( 'unclassified_weight', 10 );
- test_assert( $b->start() );
test_assert_equal( $b->{unclassified__}, log(10) );
$b->config_( 'unclassified_weight', 5 );
test_assert( $b->start() );
--- 66,73 ----
# Test the unclassifier_probability parameter
test_assert_equal( $b->{unclassified__}, log(10) );
+ $b->config_( 'unclassified_weight', 9 );
+ test_assert( $b->start() );
+ test_assert_equal( $b->{unclassified__}, log(9) );
$b->config_( 'unclassified_weight', 5 );
test_assert( $b->start() );
Index: TestHTML.script
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestHTML.script,v
retrieving revision 1.16
retrieving revision 1.17
diff -C2 -d -r1.16 -r1.17
*** TestHTML.script 8 Sep 2003 18:03:57 -0000 1.16
--- TestHTML.script 9 Sep 2003 00:28:20 -0000 1.17
***************
*** 187,190 ****
--- 187,222 ----
MATCH popfile10000=1
+ # Check magnet classified mails
+
+ NOTMATCH Magnetic Attraction for Ferrous Females
+ MATCH (<a class="history" href="/history?session=$sk&setfilter=">Refresh</a>)
+ CODE
+ open NEWMSG, ">messages/popfile10002=1.msg";
+ print NEWMSG <<EOM;
+ From: foo-magnet\@magnetmania.com
+ Subject: Magnetic Attraction for Ferrous Females
+
+ Align your pole to magnetic north
+ EOM
+ close NEWMSG;
+ open NEWCLS, ">messages/popfile10002=1.cls";
+ print NEWCLS "personal MAGNET from: foo\n";
+ close NEWCLS;
+ ENDCODE
+ SENDMSG NEWFL popfile10002=1.msg
+ GET /history?session=$sk&setfilter=
+ MATCH foo-magnet\@magnetmania.com
+ MATCH Magnetic Attraction for Ferrous Females
+ MATCH personal
+ MATCH popfile10002=1
+ MATCH Magnet used: from: foo
+
+ GET /view?view=popfile10002=1.msg&start_message=0&session=$sk
+ MATCH Single Message View
+ MATCH foo-magnet\@magnetmania.com
+ MATCH Magnetic Attraction for Ferrous Females
+ MATCH Magnet used: from: foo
+ MATCH <font color="green">foo</font>
+
# Check search
***************
*** 241,244 ****
--- 273,286 ----
MATCH (just showing bucket <font color="red">other</font>)
MATCH No messages.
+ SETINPUT filter __filter__magnet
+ CLICK setfilter
+ MATCH (just showing magnet classified messages)
+ MATCH Magnetic Attraction for Ferrous Females
+ NOTMATCH Testing Refresh
+ SETINPUT filter __filter__no__magnet
+ CLICK setfilter
+ MATCH (just showing non-magnet classified messages)
+ NOTMATCH Magnetic Attraction for Ferrous Females
+ MATCH Testing Refresh
CLICK reset_filter_search
***************
*** 258,262 ****
GET /history
MATCH Testing Refresh
! SETINPUT 0 spam
CLICK change
MATCH Changed to <font color="blue">spam</font>
--- 300,304 ----
GET /history
MATCH Testing Refresh
! SETINPUT 1 spam
CLICK change
MATCH Changed to <font color="blue">spam</font>
***************
*** 265,269 ****
# Check undo
! CLICK undo_0
MATCH Changed to <font color="green">personal</font>
--- 307,311 ----
# Check undo
! CLICK undo_1
MATCH Changed to <font color="green">personal</font>
***************
*** 336,340 ****
MATCH Single Message View
MATCH Testing Refresh
! SETINPUT 0 spam
CLICK change
MATCH Changed to <font color="blue">spam</font>
--- 378,382 ----
MATCH Single Message View
MATCH Testing Refresh
! SETINPUT 1 spam
CLICK change
MATCH Changed to <font color="blue">spam</font>
***************
*** 345,349 ****
GET /view?view=popfile10000=1.msg&start_message=0&session=$sk
MATCH Single Message View
! CLICK undo_0
MATCH Changed to <font color="green">personal</font>
--- 387,391 ----
GET /view?view=popfile10000=1.msg&start_message=0&session=$sk
MATCH Single Message View
! CLICK undo_1
MATCH Changed to <font color="green">personal</font>
***************
*** 352,356 ****
GET /history
MATCH Testing Refresh
! SETINPUT remove_1 on
CLICK deletemessage
MATCH Recent Messages
--- 394,398 ----
GET /history
MATCH Testing Refresh
! SETINPUT remove_2 on
CLICK deletemessage
MATCH Recent Messages
***************
*** 361,366 ****
MATCH test for various HTML parts
MATCH Enlarge your package Doctor ApprovedNHOB
- SETINPUT remove_4 on
SETINPUT remove_5 on
CLICK deletemessage
NOTMATCH test for various HTML parts
--- 403,408 ----
MATCH test for various HTML parts
MATCH Enlarge your package Doctor ApprovedNHOB
SETINPUT remove_5 on
+ SETINPUT remove_6 on
CLICK deletemessage
NOTMATCH test for various HTML parts
|
|
From: <xue...@us...> - 2003-09-07 11:31:12
|
Update of /cvsroot/popfile/windows
In directory sc8-pr-cvs1:/tmp/cvs-serv12284
Modified Files:
CBP.nsh
Log Message:
All macro definitions moved outside functions (to simplify coverage tests).
Index: CBP.nsh
===================================================================
RCS file: /cvsroot/popfile/windows/CBP.nsh,v
retrieving revision 1.21
retrieving revision 1.22
diff -C2 -d -r1.21 -r1.22
*** CBP.nsh 10 Aug 2003 19:35:35 -0000 1.21
--- CBP.nsh 7 Sep 2003 11:31:07 -0000 1.22
***************
*** 685,688 ****
--- 685,739 ----
#==============================================================================================
+ ;--------------------------------------------------------------------------------------
+ ; Macros used by 'CBP_CreateINIfile' (outside the function to simplify coverage tests)
+ ;--------------------------------------------------------------------------------------
+
+ ; Basic macro used to create the INI file
+
+ !macro CBP_WRITE_INI SECTION KEY VALUE
+ WriteINIStr "$PLUGINSDIR\${CBP_C_INIFILE}" "${SECTION}" "${KEY}" "${VALUE}"
+ !macroend
+
+ ; Macro used to define a standard control for the custom page
+ ; (used for ComboBoxes, the GroupBox and the info Labels)
+
+ !macro CBP_DEFINE_CONTROL FIELD TYPE TEXT LEFT RIGHT TOP BOTTOM
+ !insertmacro CBP_WRITE_INI "${FIELD}" "Type" "${TYPE}"
+
+ StrCmp "${TYPE}" "ComboBox" 0 +3
+ ; ComboBox control
+ !insertmacro CBP_WRITE_INI "${FIELD}" "ListItems" "${TEXT}"
+ goto +2
+
+ ; GroupBox or Label control
+ !insertmacro CBP_WRITE_INI "${FIELD}" "Text" "${TEXT}"
+
+ ; Remainder is common to "ComboBox", "GroupBox" and "Label" controls
+ !insertmacro CBP_WRITE_INI "${FIELD}" "Left" "${LEFT}"
+ !insertmacro CBP_WRITE_INI "${FIELD}" "Right" "${RIGHT}"
+ !insertmacro CBP_WRITE_INI "${FIELD}" "Top" "${TOP}"
+ !insertmacro CBP_WRITE_INI "${FIELD}" "Bottom" "${BOTTOM}"
+ !macroend
+
+ ; Macro used to define a label which holds one of the 8 bucket names
+
+ !macro CBP_DEFINE_BN_TEXT FIELD TEXT ROW
+ !insertmacro CBP_DEFINE_CONTROL "${FIELD}" \
+ "Label" \
+ "${TEXT}" \
+ "${CBP_BN_NAME_LEFT}" "${CBP_BN_NAME_RIGHT}" \
+ "${CBP_BN_${ROW}_TOP}" "${CBP_BN_${ROW}_BOTTOM}"
+ !macroend
+
+ ; Macro used to define a checkbox for marking a bucket name for deletion
+
+ !macro CBP_DEFINE_BN_REMOVE FIELD ROW
+ !insertmacro CBP_DEFINE_CONTROL "${FIELD}" \
+ "CheckBox" \
+ "$(PFI_LANG_CBP_IO_REMOVE)" \
+ "${CBP_BN_REMOVE_LEFT}" "${CBP_BN_REMOVE_RIGHT}" \
+ "${CBP_BN_${ROW}_TOP}" "${CBP_BN_${ROW}_BOTTOM}"
+ !macroend
+
Function CBP_CreateINIfile
***************
*** 760,810 ****
!define CBP_BN_ROW_8_BOTTOM 118
- ; Basic macro used to create the INI file
-
- !macro CBP_WRITE_INI SECTION KEY VALUE
- WriteINIStr "$PLUGINSDIR\${CBP_C_INIFILE}" "${SECTION}" "${KEY}" "${VALUE}"
- !macroend
-
- ; Macro used to define a standard control for the custom page
- ; (used for ComboBoxes, the GroupBox and the info Labels)
-
- !macro CBP_DEFINE_CONTROL FIELD TYPE TEXT LEFT RIGHT TOP BOTTOM
- !insertmacro CBP_WRITE_INI "${FIELD}" "Type" "${TYPE}"
-
- StrCmp "${TYPE}" "ComboBox" 0 +3
- ; ComboBox control
- !insertmacro CBP_WRITE_INI "${FIELD}" "ListItems" "${TEXT}"
- goto +2
-
- ; GroupBox or Label control
- !insertmacro CBP_WRITE_INI "${FIELD}" "Text" "${TEXT}"
-
- ; Remainder is common to "ComboBox", "GroupBox" and "Label" controls
- !insertmacro CBP_WRITE_INI "${FIELD}" "Left" "${LEFT}"
- !insertmacro CBP_WRITE_INI "${FIELD}" "Right" "${RIGHT}"
- !insertmacro CBP_WRITE_INI "${FIELD}" "Top" "${TOP}"
- !insertmacro CBP_WRITE_INI "${FIELD}" "Bottom" "${BOTTOM}"
- !macroend
-
- ; Macro used to define a label which holds one of the 8 bucket names
-
- !macro CBP_DEFINE_BN_TEXT FIELD TEXT ROW
- !insertmacro CBP_DEFINE_CONTROL "${FIELD}" \
- "Label" \
- "${TEXT}" \
- "${CBP_BN_NAME_LEFT}" "${CBP_BN_NAME_RIGHT}" \
- "${CBP_BN_${ROW}_TOP}" "${CBP_BN_${ROW}_BOTTOM}"
- !macroend
-
- ; Macro used to define a checkbox for marking a bucket name for deletion
-
- !macro CBP_DEFINE_BN_REMOVE FIELD ROW
- !insertmacro CBP_DEFINE_CONTROL "${FIELD}" \
- "CheckBox" \
- "$(PFI_LANG_CBP_IO_REMOVE)" \
- "${CBP_BN_REMOVE_LEFT}" "${CBP_BN_REMOVE_RIGHT}" \
- "${CBP_BN_${ROW}_TOP}" "${CBP_BN_${ROW}_BOTTOM}"
- !macroend
-
#----------------------------------------------------------------------------------------------
# Now create the INI file for the "Create Buckets" custom page
--- 811,814 ----
***************
*** 1006,1010 ****
#==============================================================================================
! Function CBP_CreateBucketsPage
; The CBP_CreateBucketsPage function creates a custom page which uses the CBP_HandleUserInput
--- 1010,1016 ----
#==============================================================================================
! ;------------------------------------------------------------------------------------------
! ; Macros used by 'CBP_CreateBucketsPage' (outside the function to simplify coverage tests)
! ;------------------------------------------------------------------------------------------
; The CBP_CreateBucketsPage function creates a custom page which uses the CBP_HandleUserInput
***************
*** 1032,1035 ****
--- 1038,1043 ----
!undef CBP_L_TEMP
!macroend
+
+ Function CBP_CreateBucketsPage
!insertmacro CBP_HUI_SharedDefs
***************
*** 1191,1195 ****
Function CBP_HandleUserInput
! !insertmacro CBP_HUI_SharedDefs ; this macro is defined in CBP_CreateBucketsPage
; Check the user input... starting with the "Remove" check boxes as deletion has higher
--- 1199,1203 ----
Function CBP_HandleUserInput
! !insertmacro CBP_HUI_SharedDefs ; defined just before the 'CBP_CreateBucketsPage' function
; Check the user input... starting with the "Remove" check boxes as deletion has higher
***************
*** 1340,1344 ****
Return
! !insertmacro CBP_HUI_SharedUnDefs ; this macro is defined in CBP_CreateBucketsPage
FunctionEnd
--- 1348,1352 ----
Return
! !insertmacro CBP_HUI_SharedUnDefs ; defined just before the 'CBP_CreateBucketsPage' function
FunctionEnd
|
|
From: <jgr...@us...> - 2003-09-05 21:18:56
|
Update of /cvsroot/popfile/engine/Classifier
In directory sc8-pr-cvs1:/tmp/cvs-serv5019/Classifier
Modified Files:
Bayes.pm
Log Message:
Default values for the dummy unsure bucket
Index: Bayes.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Classifier/Bayes.pm,v
retrieving revision 1.186
retrieving revision 1.187
diff -C2 -d -r1.186 -r1.187
*** Bayes.pm 4 Sep 2003 16:30:41 -0000 1.186
--- Bayes.pm 5 Sep 2003 21:18:51 -0000 1.187
***************
*** 407,419 ****
$self->{colors__}{unclassified} = 'black';
# SLM for unclassified "bucket" will always match the global setting
$self->{parameters__}{unclassified}{subject} = $self->global_config_('subject');
# Quarantine for unclassified will be off:
$self->{parameters__}{unclassified}{quarantine} = 0;
!
}
--- 407,421 ----
$self->{colors__}{unclassified} = 'black';
+ $self->{colors__}{unsure} = 'black';
# SLM for unclassified "bucket" will always match the global setting
$self->{parameters__}{unclassified}{subject} = $self->global_config_('subject');
+ $self->{parameters__}{unsure}{subject} = $self->global_config_('subject');
# Quarantine for unclassified will be off:
$self->{parameters__}{unclassified}{quarantine} = 0;
! $self->{parameters__}{unsure}{quarantine} = 0;
}
***************
*** 608,612 ****
my $term = $sum;
! for my $i (1..$free/2) {
$term *= $m / $i;
$sum += $term;
--- 610,614 ----
my $term = $sum;
! for my $i (1..($free/2-1)) {
$term *= $m / $i;
$sum += $term;
***************
*** 697,701 ****
--- 699,706 ----
# P(word|bucket) ^ word count and multiply to the score
+ my $word_count = 0;
+
foreach my $word (keys %{$self->{parser__}->{words__}}) {
+ $word_count += 2;
my $wmax = -10000;
***************
*** 719,723 ****
foreach my $bucket (@buckets) {
! $chi{$bucket} = chi2( $score{$bucket}, 2 * $matchcount{$bucket}, -int($score{$ranking[0]}/log(10)) * log(10) );
}
--- 724,728 ----
foreach my $bucket (@buckets) {
! $chi{$bucket} = chi2( $score{$bucket}, $word_count, -int($score{$ranking[0]}/log(10)) * log(10) );
}
***************
*** 785,789 ****
}
! $self->{scores__} .= "<hr><b>$language{Scores}</b><p>\n<b>Verdict: <font color=\"$self->{colors__}{$class}\">$class ($certainty)</font></b><p>\n<table class=\"top20Words\">\n<tr>\n<th scope=\"col\">$language{Bucket}</th>\n<th> </th>\n";
$self->{scores__} .= "<th scope=\"col\">$language{Count} </th><th scope=\"col\">$language{Probability}</th></tr>\n";
--- 790,794 ----
}
! $self->{scores__} .= "<hr><b>$language{Scores}</b><p>\n<b>Verdict: <font color=\"$self->{colors__}{$class}\">$class ($certainty $chi{$ranking[0]} $chi{$ranking[1]})</font></b><p>\n<table class=\"top20Words\">\n<tr>\n<th scope=\"col\">$language{Bucket}</th>\n<th> </th>\n";
$self->{scores__} .= "<th scope=\"col\">$language{Count} </th><th scope=\"col\">$language{Probability}</th></tr>\n";
***************
*** 1160,1164 ****
# Add the Subject line modification or the original line back again
! if ( $classification ne 'unclassified' ) {
if ( $self->global_config_( 'subject' ) ) {
# Don't add the classification unless it is not present
--- 1165,1169 ----
# Add the Subject line modification or the original line back again
! if ( ( $classification ne 'unclassified' ) && ( $classification ne 'unsure' ) ) {
if ( $self->global_config_( 'subject' ) ) {
# Don't add the classification unless it is not present
***************
*** 1198,1202 ****
# information from POPFile and wrapping the original message in a MIME encoding
! if ( $classification ne 'unclassified' ) {
if ( $self->{parameters__}{$classification}{quarantine} == 1 ) {
print $client "From: " . $self->{parser__}->get_header( 'from' ) . "$eol";
--- 1203,1207 ----
# information from POPFile and wrapping the original message in a MIME encoding
! if ( ( $classification ne 'unclassified' ) && ( $classification ne 'unsure' ) ) {
if ( $self->{parameters__}{$classification}{quarantine} == 1 ) {
print $client "From: " . $self->{parser__}->get_header( 'from' ) . "$eol";
***************
*** 1236,1240 ****
my $before_dot = '';
! if ( $classification ne 'unclassified' ) {
if ( ( $self->{parameters__}{$classification}{quarantine} == 1 ) && $echo ) {
$before_dot = "$eol--$nopath_temp_file--$eol";
--- 1241,1245 ----
my $before_dot = '';
! if ( ( $classification ne 'unclassified' ) && ( $classification ne 'unsure' ) ) {
if ( ( $self->{parameters__}{$classification}{quarantine} == 1 ) && $echo ) {
$before_dot = "$eol--$nopath_temp_file--$eol";
|
|
From: <jgr...@us...> - 2003-09-05 21:12:05
|
Update of /cvsroot/popfile/engine/UI
In directory sc8-pr-cvs1:/tmp/cvs-serv3906/UI
Modified Files:
HTML.pm
Log Message:
Speed optimization of history reload after successful shutdown and message deletion. Both use readdir instead of glob
Index: HTML.pm
===================================================================
RCS file: /cvsroot/popfile/engine/UI/HTML.pm,v
retrieving revision 1.197
retrieving revision 1.198
diff -C2 -d -r1.197 -r1.198
*** HTML.pm 24 Aug 2003 01:20:33 -0000 1.197
--- HTML.pm 5 Sep 2003 21:11:58 -0000 1.198
***************
*** 266,270 ****
$self->load_disk_cache__();
$self->load_history_cache__();
- $self->sort_filter_history( '', '', '' );
$self->remove_mail_files();
--- 266,269 ----
***************
*** 2544,2570 ****
# for non-culling and new entries that need to be added to the end
! my @history_files = sort compare_mf glob( $self->global_config_( 'msgdir' ) . "popfile*=*.msg" );
! foreach my $i ( 0 .. $#history_files ) {
! # Strip any directory portion of the name in the current file so that we
! # just get the base name of the file that we are dealing with
! $history_files[$i] =~ /(popfile(\d+)=\d+\.msg)$/;
! $history_files[$i] = $1;
! if ( $2 > $max ) {
! $max = $2;
}
! # If this file already exists in the history cache then just mark it not
! # to be culled and move on.
! if ( defined( $self->{history__}{$history_files[$i]} ) ) {
! $self->{history__}{$history_files[$i]}{cull} = 0;
! $self->{history__}{$history_files[$i]}{index} = $i;
! } else {
! $self->new_history_file__( $history_files[$i], $i );
! }
}
--- 2543,2570 ----
# for non-culling and new entries that need to be added to the end
! opendir MESSAGES, $self->global_config_( 'msgdir' );
! my @history_files;
! while ( my $entry = readdir MESSAGES ) {
! if ( $entry =~ /(popfile(\d+)=\d+\.msg)$/ ) {
! $entry = $1;
! if ( $2 > $max ) {
! $max = $2;
! }
! if ( defined( $self->{history__}{$entry} ) ) {
! $self->{history__}{$entry}{cull} = 0;
! } else {
! push @history_files, ($entry);
! }
}
+ }
! closedir MESSAGES;
! foreach my $i ( 0 .. $#history_files ) {
! $self->new_history_file__( $history_files[$i] );
}
***************
*** 2572,2578 ****
# comment at the start of this function for more detail
! foreach my $key (keys %{$self->{history__}}) {
if ( $self->{history__}{$key}{cull} == 1 ) {
delete $self->{history__}{$key};
}
}
--- 2572,2583 ----
# comment at the start of this function for more detail
! my $index = 0;
!
! foreach my $key (sort compare_mf keys %{$self->{history__}}) {
if ( $self->{history__}{$key}{cull} == 1 ) {
delete $self->{history__}{$key};
+ } else {
+ $self->{history__}{$key}{index} = $index;
+ $index += 1;
}
}
***************
*** 3720,3736 ****
my ( $self ) = @_;
! my @mail_files = glob( $self->global_config_( 'msgdir' ) . "popfile*=*.msg" );
! foreach my $mail_file (@mail_files) {
! my ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,$blksize,$blocks) = stat($mail_file);
! if ( $ctime < (time - $self->config_( 'history_days' ) * $seconds_per_day) ) {
! $self->history_delete_file( $mail_file, $self->config_( 'archive' ) );
}
}
# Clean up old style msg/cls files
! @mail_files = glob( $self->global_config_( 'msgdir' ) . "popfile*_*.???" );
foreach my $mail_file (@mail_files) {
--- 3725,3745 ----
my ( $self ) = @_;
! opendir MESSAGES, $self->global_config_( 'msgdir' );
! while ( my $mail_file = readdir MESSAGES ) {
! if ( $mail_file =~ /popfile\d+=\d+\.msg$/ ) {
! my ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,$blksize,$blocks) = stat( $self->global_config_( 'msgdir' ) . $mail_file);
! if ( $ctime < (time - $self->config_( 'history_days' ) * $seconds_per_day) ) {
! $self->history_delete_file( $mail_file, $self->config_( 'archive' ) );
! }
}
}
+ closedir MESSAGES;
+
# Clean up old style msg/cls files
! my @mail_files = glob( $self->global_config_( 'msgdir' ) . "popfile*_*.???" );
foreach my $mail_file (@mail_files) {
|
|
From: <jgr...@us...> - 2003-09-05 20:38:46
|
Update of /cvsroot/popfile/engine/Classifier
In directory sc8-pr-cvs1:/tmp/cvs-serv28979/Classifier
Modified Files:
MailParse.pm
Log Message:
For some reason we weren't adding the domain names of img src tags as a word, now we do
Index: MailParse.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Classifier/MailParse.pm,v
retrieving revision 1.158
retrieving revision 1.159
diff -C2 -d -r1.158 -r1.159
*** MailParse.pm 24 Aug 2003 04:04:52 -0000 1.158
--- MailParse.pm 5 Sep 2003 20:38:10 -0000 1.159
***************
*** 582,586 ****
} else {
! my $host = add_url( $self, $value, $encoded, $quote, $end_quote, '', 1 );
# If the host name is not blank (i.e. there was a hostname in the url
--- 582,586 ----
} else {
! my $host = add_url( $self, $value, $encoded, $quote, $end_quote, '' );
# If the host name is not blank (i.e. there was a hostname in the url
***************
*** 896,900 ****
# http://www.0dns.org has a good reference of ccTLD's and their sub-tld's if desired
! if ( $hostform eq "name" ) {
while ( $host =~ s/^([^\.])+\.(.*\.(.*))$/$2/ ) {
update_word( $self, $2, $encoded, '[\.]', '[<]', $prefix) if ( !defined( $noadd ) );
--- 896,900 ----
# http://www.0dns.org has a good reference of ccTLD's and their sub-tld's if desired
! if ( $hostform eq 'name' ) {
while ( $host =~ s/^([^\.])+\.(.*\.(.*))$/$2/ ) {
update_word( $self, $2, $encoded, '[\.]', '[<]', $prefix) if ( !defined( $noadd ) );
|
|
From: <jgr...@us...> - 2003-09-04 16:31:26
|
Update of /cvsroot/popfile/engine/Classifier
In directory sc8-pr-cvs1:/tmp/cvs-serv7999/Classifier
Modified Files:
Bayes.pm
Log Message:
Use the same combining method as SpamBayes to take the two chi2 calculations and come up with a certainty value
Index: Bayes.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Classifier/Bayes.pm,v
retrieving revision 1.185
retrieving revision 1.186
diff -C2 -d -r1.185 -r1.186
*** Bayes.pm 4 Sep 2003 13:31:06 -0000 1.185
--- Bayes.pm 4 Sep 2003 16:30:41 -0000 1.186
***************
*** 613,617 ****
}
! return $sum;
}
--- 613,617 ----
}
! return ($sum<1)?$sum:1;
}
***************
*** 719,723 ****
foreach my $bucket (@buckets) {
! $chi{$bucket} = chi2( $score{$bucket}, 2 * $matchcount{$ranking[0]}, -int($score{$ranking[0]}/log(10)) * log(10) );
}
--- 719,723 ----
foreach my $bucket (@buckets) {
! $chi{$bucket} = chi2( $score{$bucket}, 2 * $matchcount{$bucket}, -int($score{$ranking[0]}/log(10)) * log(10) );
}
***************
*** 732,736 ****
# we are unsure
! if ( ( $chi{$ranking[0]} * 0.5 ) < $chi{$ranking[1]} ) {
$class = 'unsure';
}
--- 732,741 ----
# we are unsure
! my $c0 = 1.0 - $chi{$ranking[0]};
! my $c1 = 1.0 - $chi{$ranking[1]};
!
! my $certainty = ($c1-$c0 + 1) / 2;
!
! if ( $certainty < 0.9 ) {
$class = 'unsure';
}
***************
*** 780,784 ****
}
! $self->{scores__} .= "<hr><b>$language{Scores}</b><p>\n<b>Verdict: <font color=\"$self->{colors__}{$class}\">$class</font></b><p>\n<table class=\"top20Words\">\n<tr>\n<th scope=\"col\">$language{Bucket}</th>\n<th> </th>\n";
$self->{scores__} .= "<th scope=\"col\">$language{Count} </th><th scope=\"col\">$language{Probability}</th></tr>\n";
--- 785,789 ----
}
! $self->{scores__} .= "<hr><b>$language{Scores}</b><p>\n<b>Verdict: <font color=\"$self->{colors__}{$class}\">$class ($certainty)</font></b><p>\n<table class=\"top20Words\">\n<tr>\n<th scope=\"col\">$language{Bucket}</th>\n<th> </th>\n";
$self->{scores__} .= "<th scope=\"col\">$language{Count} </th><th scope=\"col\">$language{Probability}</th></tr>\n";
|
|
From: <jgr...@us...> - 2003-09-04 13:31:13
|
Update of /cvsroot/popfile/engine/Classifier
In directory sc8-pr-cvs1:/tmp/cvs-serv4573/Classifier
Modified Files:
Bayes.pm
Log Message:
Small change to chi2 calculation to use degrees of freedom from the top bucket and not normalize to 1
Index: Bayes.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Classifier/Bayes.pm,v
retrieving revision 1.184
retrieving revision 1.185
diff -C2 -d -r1.184 -r1.185
*** Bayes.pm 3 Sep 2003 21:53:11 -0000 1.184
--- Bayes.pm 4 Sep 2003 13:31:06 -0000 1.185
***************
*** 613,617 ****
}
! return ($sum < 1)?$sum:1;
}
--- 613,617 ----
}
! return $sum;
}
***************
*** 719,723 ****
foreach my $bucket (@buckets) {
! $chi{$bucket} = chi2( $score{$bucket}, 2 * $matchcount{$bucket}, -int($score{$ranking[0]}/log(10)) * log(10) );
}
--- 719,723 ----
foreach my $bucket (@buckets) {
! $chi{$bucket} = chi2( $score{$bucket}, 2 * $matchcount{$ranking[0]}, -int($score{$ranking[0]}/log(10)) * log(10) );
}
|
|
From: <jgr...@us...> - 2003-09-03 21:53:15
|
Update of /cvsroot/popfile/engine/Classifier
In directory sc8-pr-cvs1:/tmp/cvs-serv12610/Classifier
Modified Files:
Bayes.pm
Log Message:
Use chi2 test taken similar to SpamBayes to implement an unsure feature, when POPFile is close on the top two buckets it will mark an email as unsure instead of picking the top classification
Index: Bayes.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Classifier/Bayes.pm,v
retrieving revision 1.183
retrieving revision 1.184
diff -C2 -d -r1.183 -r1.184
*** Bayes.pm 3 Sep 2003 20:47:47 -0000 1.183
--- Bayes.pm 3 Sep 2003 21:53:11 -0000 1.184
***************
*** 589,592 ****
--- 589,621 ----
# ---------------------------------------------------------------------------------------------
#
+ # chi2
+ #
+ # $val The value on which we do the chi2 test
+ # $free Number of degrees of freedom
+ # $modifier log() of a power of 10 to make values come in range
+ #
+ # Performs a chi-squared calculation on the passed in log(probability), liberally inspired
+ # by code in SpamBayes and work by Gary Robinson
+ #
+ # ---------------------------------------------------------------------------------------------
+
+ sub chi2
+ {
+ my ( $val, $free, $modifier ) = @_;
+
+ my $m = $val + $modifier;
+ my $sum = exp(-$m);
+ my $term = $sum;
+
+ for my $i (1..$free/2) {
+ $term *= $m / $i;
+ $sum += $term;
+ }
+
+ return ($sum < 1)?$sum:1;
+ }
+
+ # ---------------------------------------------------------------------------------------------
+ #
# classify
#
***************
*** 658,661 ****
--- 687,691 ----
my %score;
my %matchcount;
+ my %chi;
for my $bucket (@buckets) {
***************
*** 688,691 ****
--- 718,739 ----
my @ranking = sort {$score{$b} <=> $score{$a}} keys %score;
+ foreach my $bucket (@buckets) {
+ $chi{$bucket} = chi2( $score{$bucket}, 2 * $matchcount{$bucket}, -int($score{$ranking[0]}/log(10)) * log(10) );
+ }
+
+ # If no bucket has a probability better than 0.5, call the message "unclassified".
+ my $class = 'unclassified';
+
+ if ( $score{$ranking[0]} > ( $score{$ranking[1]} + $self->{unclassified__} ) ) {
+ $class = $ranking[0];
+ }
+
+ # Now take a look at the top two chi tests, if they are close to each other then
+ # we are unsure
+
+ if ( ( $chi{$ranking[0]} * 0.5 ) < $chi{$ranking[1]} ) {
+ $class = 'unsure';
+ }
+
if ($self->{wordscores__} && defined($ui) ) {
my %qm = %{$self->{parser__}->quickmagnets()};
***************
*** 732,736 ****
}
! $self->{scores__} .= "<hr><b>$language{Scores}</b><p>\n<table class=\"top20Words\">\n<tr>\n<th scope=\"col\">$language{Bucket}</th>\n<th> </th>\n";
$self->{scores__} .= "<th scope=\"col\">$language{Count} </th><th scope=\"col\">$language{Probability}</th></tr>\n";
--- 780,784 ----
}
! $self->{scores__} .= "<hr><b>$language{Scores}</b><p>\n<b>Verdict: <font color=\"$self->{colors__}{$class}\">$class</font></b><p>\n<table class=\"top20Words\">\n<tr>\n<th scope=\"col\">$language{Bucket}</th>\n<th> </th>\n";
$self->{scores__} .= "<th scope=\"col\">$language{Count} </th><th scope=\"col\">$language{Probability}</th></tr>\n";
***************
*** 804,814 ****
$self->{scores__} .= "</table></p>";
- }
-
- # If no bucket has a probability better than 0.5, call the message "unclassified".
- my $class = 'unclassified';
-
- if ( $score{$ranking[0]} > ( $score{$ranking[1]} + $self->{unclassified__} ) ) {
- $class = $ranking[0];
}
--- 852,855 ----
|
|
From: <jgr...@us...> - 2003-09-03 20:47:51
|
Update of /cvsroot/popfile/engine/Classifier
In directory sc8-pr-cvs1:/tmp/cvs-serv30888/Classifier
Modified Files:
Bayes.pm
Log Message:
Implemented add one smoothing of the loaded corpus, change the unclassified_probability to unclassified_weight, it is now used to compare the difference between the top chosen bucket and the second, the top must be unclassified_weight times greater than the second to avoid an unclassified
Index: Bayes.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Classifier/Bayes.pm,v
retrieving revision 1.182
retrieving revision 1.183
diff -C2 -d -r1.182 -r1.183
*** Bayes.pm 3 Sep 2003 19:37:06 -0000 1.182
--- Bayes.pm 3 Sep 2003 20:47:47 -0000 1.183
***************
*** 102,107 ****
$self->{magnet_count__} = 0;
! # The unclassified cutoff probability
! $self->{unclassified__} = 0.5;
# Used to tell the caller whether a magnet was used in the last
--- 102,108 ----
$self->{magnet_count__} = 0;
! # The unclassified cutoff this value means that the top probabilily must be n times greater than the
! # second probability, default is 100 times more likely
! $self->{unclassified__} = log(100);
# Used to tell the caller whether a magnet was used in the last
***************
*** 130,135 ****
my ( $self ) = @_;
! # No default unclassified probability
! $self->config_( 'unclassified_probability', 0 );
# The corpus is kept in the 'corpus' subfolder of POPFile
--- 131,137 ----
my ( $self ) = @_;
! # No default unclassified weight is the number of times more sure POPFile
! # must be of the top class vs the second class, default is 100 times more
! $self->config_( 'unclassified_weight', 100 );
# The corpus is kept in the 'corpus' subfolder of POPFile
***************
*** 165,174 ****
my ( $self ) = @_;
! if ( $self->config_( 'unclassified_probability' ) != 0 ) {
! $self->{unclassified__} = $self->config_( 'unclassified_probability' );
! }
!
! $self->{unclassified__} = log($self->{unclassified__});
!
$self->load_word_matrix_();
--- 167,171 ----
my ( $self ) = @_;
! $self->{unclassified__} = log( $self->config_( 'unclassified_weight' ) );
$self->load_word_matrix_();
***************
*** 520,523 ****
--- 517,526 ----
my $value = $2;
if ( $value > 0 ) {
+
+ # Here we do a simple "add one" smoothing on the data being
+ # loaded
+
+ $value += 1;
+
$self->{total__}{$bucket} += $value;
$self->{unique__}{$bucket} += 1;
***************
*** 806,812 ****
my $class = 'unclassified';
! # if ( ( $total != 0 ) && ( $score{$ranking[0]} > $self->{unclassified__} + log($total) ) ) {
$class = $ranking[0];
! # }
return $class;
--- 809,815 ----
my $class = 'unclassified';
! if ( $score{$ranking[0]} > ( $score{$ranking[1]} + $self->{unclassified__} ) ) {
$class = $ranking[0];
! }
return $class;
|
|
From: <jgr...@us...> - 2003-09-03 19:37:41
|
Update of /cvsroot/popfile/engine/Classifier
In directory sc8-pr-cvs1:/tmp/cvs-serv16348/Classifier
Modified Files:
Bayes.pm
Log Message:
Change the way we display the probability numbers for single message view so that there is no normalization going on
Index: Bayes.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Classifier/Bayes.pm,v
retrieving revision 1.181
retrieving revision 1.182
diff -C2 -d -r1.181 -r1.182
*** Bayes.pm 20 Aug 2003 06:12:15 -0000 1.181
--- Bayes.pm 3 Sep 2003 19:37:06 -0000 1.182
***************
*** 664,683 ****
# P(word|bucket) ^ word count and multiply to the score
- my $logbuck = 1;
- $logbuck = log( $#buckets + 1 ) if ( $#buckets > 0 );
-
- # Ideally, the "raw score" in the score display would reflect the sum of the
- # scores for the individual words, as shown by the lookup GUI. Actually
- # doing this requires a fair amount of computation to compute the sum of the
- # probabilities. If we assume that only the most probable choice is significant
- # (that is, that the max probability and the sum of the probabilities are the
- # same), we do much less computation, and still end up with results that are
- # "close enough for jazz". Note that this makes *no* difference for
- # classification - it only matters for the debug (bayes.pl) display.
-
- my $correction = -$logbuck;
-
- # Switching from using *= to += and using the log of every probability instead
-
foreach my $word (keys %{$self->{parser__}->{words__}}) {
my $wmax = -10000;
--- 664,667 ----
***************
*** 695,704 ****
$score{$bucket} += ( $probability * $self->{parser__}{words__}{$word} );
}
-
- if ($wmax > $self->{not_likely__}) {
- $correction += ($wmax - $logbuck) * $self->{parser__}{words__}{$word};
- } else {
- $correction += $wmax * $self->{parser__}{words__}{$word};
- }
}
--- 679,682 ----
***************
*** 707,729 ****
my @ranking = sort {$score{$b} <=> $score{$a}} keys %score;
- my %raw_score;
- my $base_score = $score{$ranking[0]};
- my $total = 0;
-
- $self->log_( "Base score is $base_score for $ranking[0]" );
-
- # Compute the total of all the scores to generate the normalized scores and probability
- # estimate. $total is always 1 after the first loop iteration, so any additional term
- # less than 2 ** -54 is insignificant, and need not be computed.
-
- my $ln2 = log(2);
-
- foreach my $b (@ranking) {
- $raw_score{$b} = $score{$b};
- $score{$b} -= $base_score;
-
- $total += exp($score{$b}) if ($score{$b} > ( -54 * $ln2 ) );
- }
-
if ($self->{wordscores__} && defined($ui) ) {
my %qm = %{$self->{parser__}->quickmagnets()};
--- 685,688 ----
***************
*** 774,787 ****
foreach my $b (@ranking) {
- my $prob = exp($score{$b})/$total;
- my $probstr;
! if ($prob >= 0.1 || $prob == 0.0) {
! $probstr = sprintf("%12.6f", $prob);
! } else {
! $probstr = sprintf("%17.6e", $prob);
! }
! $self->{scores__} .= "<tr>\n<td><font color=\"$self->{colors__}{$b}\"><b>$b</b></font></td>\n<td> </td>\n<td align=\"right\">$matchcount{$b} </td>\n<td>$probstr</td>\n</tr>\n";
}
--- 733,747 ----
foreach my $b (@ranking) {
! # Take a score value (which is log of the probability) and write it out as 0.000000 lots 00000001234, to do this we
! # calculate the number of 0 between the . and the first significant digit and output the number of zeroes and
! # then the significant digits
! my $zero_count = -int($score{$b}/log(10));
! my $significant = sprintf( "%.6f", exp($score{$b} + $zero_count * log(10)) );
! $significant =~ s/^0\.//;
! my $probstr = sprintf( "0. [%d zeroes] %s", $zero_count, $significant );
!
! $self->{scores__} .= "<tr>\n<td><font color=\"$self->{colors__}{$b}\"><b>$b</b></font></td>\n<td> </td>\n<td align=\"right\">$matchcount{$b} </td>\n<td>$probstr</td>\n</tr>\n";
}
***************
*** 846,852 ****
my $class = 'unclassified';
! if ( ( $total != 0 ) && ( $score{$ranking[0]} > $self->{unclassified__} + log($total) ) ) {
$class = $ranking[0];
! }
return $class;
--- 806,812 ----
my $class = 'unclassified';
! # if ( ( $total != 0 ) && ( $score{$ranking[0]} > $self->{unclassified__} + log($total) ) ) {
$class = $ranking[0];
! # }
return $class;
|
|
From: <ssc...@us...> - 2003-08-31 22:25:58
|
Update of /cvsroot/popfile/engine/UI
In directory sc8-pr-cvs1:/tmp/cvs-serv7489
Modified Files:
XMLRPC.pm
Log Message:
basic XMLRPC tests, 48% XMRPC coverage
failure of daemon start and UI plugin tests required
Index: XMLRPC.pm
===================================================================
RCS file: /cvsroot/popfile/engine/UI/XMLRPC.pm,v
retrieving revision 1.5
retrieving revision 1.6
diff -C2 -d -r1.5 -r1.6
*** XMLRPC.pm 31 Jul 2003 16:32:22 -0000 1.5
--- XMLRPC.pm 31 Aug 2003 22:25:55 -0000 1.6
***************
*** 81,91 ****
# item that needs a UI component
! $self->register_configuration_item_( 'configuration',
'xmlrpc_port',
! $self );
! $self->register_configuration_item_( 'security',
'xmlrpc_local',
! $self );
return 1;
--- 81,91 ----
# item that needs a UI component
! $self->register_configuration_item_( 'configuration', # PROFILE BLOCK START
'xmlrpc_port',
! $self ); # PROFILE BLOCK STOP
! $self->register_configuration_item_( 'security', # PROFILE BLOCK START
'xmlrpc_local',
! $self ); # PROFILE BLOCK STOP
return 1;
***************
*** 106,115 ****
# Classifier::Bayes object
! $self->{server__} = XMLRPC::Transport::HTTP::Daemon->new(
Proto => 'tcp',
$self->config_( 'local' ) == 1 ? (LocalAddr => 'localhost') : (),
LocalPort => $self->config_( 'port' ),
Listen => SOMAXCONN,
! Reuse => 1 );
if ( !defined( $self->{server__} ) ) {
--- 106,115 ----
# Classifier::Bayes object
! $self->{server__} = XMLRPC::Transport::HTTP::Daemon->new( # PROFILE BLOCK START
Proto => 'tcp',
$self->config_( 'local' ) == 1 ? (LocalAddr => 'localhost') : (),
LocalPort => $self->config_( 'port' ),
Listen => SOMAXCONN,
! Reuse => 1 ); # PROFILE BLOCK STOP
if ( !defined( $self->{server__} ) ) {
***************
*** 178,183 ****
my ( $remote_port, $remote_host ) = sockaddr_in( $client->peername() );
! if ( ( $self->config_( 'local' ) == 0 ) ||
! ( $remote_host eq inet_aton( "127.0.0.1" ) ) ) {
my $request = $client->get_request();
--- 178,183 ----
my ( $remote_port, $remote_host ) = sockaddr_in( $client->peername() );
! if ( ( $self->config_( 'local' ) == 0 ) || # PROFILE BLOCK START
! ( $remote_host eq inet_aton( "127.0.0.1" ) ) ) { # PROFILE BLOCK STOP
my $request = $client->get_request();
|
|
From: <ssc...@us...> - 2003-08-31 22:25:43
|
Update of /cvsroot/popfile/engine/tests
In directory sc8-pr-cvs1:/tmp/cvs-serv7434
Added Files:
TestXMLRPC.tst
Log Message:
basic XMLRPC tests, 48% XMRPC coverage
failure of daemon start and UI plugin tests required
--- NEW FILE: TestXMLRPC.tst ---
# ---------------------------------------------------------------------------------------------
#
# Tests for XMLRPC.pm
#
# Copyright (c) 2003 John Graham-Cumming
#
# This file is part of POPFile
#
# POPFile is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# POPFile is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with POPFile; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
# Modified by Sam Schinke (ssc...@us...)
#
# ---------------------------------------------------------------------------------------------
use Classifier::MailParse;
use Classifier::Bayes;
use POPFile::Configuration;
use POPFile::MQ;
use POPFile::Logger;
use UI::XMLRPC;
# Load the test corpus
my $c = new POPFile::Configuration;
my $mq = new POPFile::MQ;
my $l = new POPFile::Logger;
my $b = new Classifier::Bayes;
my $x = new UI::XMLRPC;
$c->configuration( $c );
$c->mq( $mq );
$c->logger( $l );
$l->configuration( $c );
$l->mq( $mq );
$l->logger( $l );
$l->initialize();
$mq->configuration( $c );
$mq->mq( $mq );
$mq->logger( $l );
$b->configuration( $c );
$b->mq( $mq );
$b->logger( $l );
$x->configuration( $c );
$x->mq( $mq );
$x->logger( $l );
$x->{classifier__} = $b;
$b->initialize();
test_assert( $b->start() );
$x->initialize();
# TODO: make this work
# similar code fails in all tests
# Test XMLRPC startup
# $x->config_("port", "aaa");
#open STDERR, ">stdout.tmp";
#test_assert(!$x->start());
#close STDERR;
#open TEMP, "<stdout.tmp";
#my $line = <TEMP>;
#$line = <TEMP>;
#$line = <TEMP>;
#$line = <TEMP>;
#test_assert_regexp($line,"Couldn't start the XMLRPC HTTP interface because POPFile could not bind to the");
#close TEMP;
#unlink "stdout.tmp";
my $xport = 12000 + int(rand(2000));
$x->config_("port", $xport);
my $pid = fork();
if ($pid == 0) {
# CHILD THAT WILL RUN THE XMLRPC SERVER
$x->start();
while ( $x->service() && $b->alive()) {
select(undef,undef,undef, 0.1);
}
$x->stop();
$b->stop();
exit(0);
} else {
# PARENT -- test the XMLRPC server
select(undef,undef,undef,1);
use XMLRPC::Lite;
my $set_bucket_color = XMLRPC::Lite
-> proxy("http://127.0.0.1:" . $xport . "/RPC2")
-> call('Classifier/Bayes.set_bucket_color','foobarbaz', 'somecolour')
-> result;
select(undef,undef,undef,.2);
my $bucket_color = XMLRPC::Lite
-> proxy("http://127.0.0.1:" . $xport . "/RPC2")
-> call('Classifier/Bayes.get_bucket_color','foobarbaz')
-> result;
test_assert_equal($bucket_color,"somecolour");
select(undef,undef,undef,.2);
my $alive = XMLRPC::Lite
-> proxy("http://127.0.0.1:" . $xport . "/RPC2")
-> call('Classifier/Bayes.alive')
-> result;
test_assert($alive);
select(undef,undef,undef,.2);
my $alive = XMLRPC::Lite
-> proxy("http://127.0.0.1:" . $xport . "/RPC2")
-> call('Classifier/Bayes.alive',0)
-> result;
test_assert(!$alive);
select(undef,undef,undef,1);
}
|
|
From: <jgr...@us...> - 2003-08-28 17:54:58
|
Update of /cvsroot/popfile/engine/tests
In directory sc8-pr-cvs1:/tmp/cvs-serv16226/tests
Modified Files:
TestPOP3.tst
Log Message:
Bring Bayes tests up to 100% coverage by adding testing of the X-POPFile-TimeoutPrevention header
Index: TestPOP3.tst
===================================================================
RCS file: /cvsroot/popfile/engine/tests/TestPOP3.tst,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** TestPOP3.tst 23 Aug 2003 19:01:39 -0000 1.12
--- TestPOP3.tst 28 Aug 2003 17:54:55 -0000 1.13
***************
*** 81,84 ****
--- 81,85 ----
my ( $client ) = @_;
my @messages = sort glob 'TestMailParse*.msg';
+ my $goslow = 0;
print $client "+OK Ready$eol";
***************
*** 91,96 ****
if ( $command =~ /^USER (.*)/i ) {
! if ( $1 =~ /(gooduser)/ ) {
print $client "+OK Welcome $1$eol";
} else {
print $client "-ERR Unknown user $1$eol";
--- 92,98 ----
if ( $command =~ /^USER (.*)/i ) {
! if ( $1 =~ /(gooduser|goslow)/ ) {
print $client "+OK Welcome $1$eol";
+ $goslow = ( $1 =~ /goslow/ );
} else {
print $client "-ERR Unknown user $1$eol";
***************
*** 179,182 ****
--- 181,187 ----
while ( <FILE> ) {
print $client $_;
+ if ( $goslow ) {
+ select( undef, undef, undef, 3 );
+ }
}
close FILE;
***************
*** 902,905 ****
--- 907,960 ----
close FILE;
+ $result = <$client>;
+ test_assert_equal( $result, ".$eol" );
+
+ print $client "QUIT$eol";
+ $result = <$client>;
+ test_assert_equal( $result, "+OK Bye$eol" );
+
+ close $client;
+
+ # Check insertion of the X-POPFile-Timeout headers
+
+ my $client = IO::Socket::INET->new(
+ Proto => "tcp",
+ PeerAddr => 'localhost',
+ PeerPort => $port );
+
+ test_assert( defined( $client ) );
+ test_assert( $client->connected );
+
+ my $result = <$client>;
+ test_assert_equal( $result, "+OK POP3 POPFile (test suite) server ready$eol" );
+
+ print $client "USER 127.0.0.1:8110:goslow$eol";
+ $result = <$client>;
+ test_assert_equal( $result, "+OK Welcome goslow$eol" );
+
+ print $client "RETR 1$eol";
+ $result = <$client>;
+ test_assert_equal( $result, "+OK " . ( -s $messages[0] ) . "$eol" );
+ my $cam = $messages[0];
+ $cam =~ s/msg$/cam/;
+
+ test_assert( open FILE, "<$cam" );
+ binmode FILE;
+ $line = <$client>;
+ test_assert_equal( $line, "X-POPFile-TimeoutPrevention: 0$eol" );
+ $line = <$client>;
+ test_assert_equal( $line, "X-POPFile-TimeoutPrevention: 1$eol" );
+ $line = <$client>;
+ test_assert_equal( $line, "X-POPFile-TimeoutPrevention: 2$eol" );
+ while ( <FILE> ) {
+ my $line = $_;
+ $result = <$client>;
+ $result =~ s/popfile3=1/popfile0=0/;
+ test_assert_equal( $result, $line );
+ }
+ close FILE;
+
+ $result = <$client>;
+ test_assert_equal( $result, "$eol" );
$result = <$client>;
test_assert_equal( $result, ".$eol" );
|
|
From: <jgr...@us...> - 2003-08-28 15:27:59
|
Update of /cvsroot/popfile/engine/Proxy
In directory sc8-pr-cvs1:/tmp/cvs-serv21023/Proxy
Modified Files:
Proxy.pm
Log Message:
Use PROFILE PLATFORM START to enable profiler to ignore platform code in Proxy and bring tests to 100%
Index: Proxy.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Proxy/Proxy.pm,v
retrieving revision 1.32
retrieving revision 1.33
diff -C2 -d -r1.32 -r1.33
*** Proxy.pm 21 Aug 2003 20:53:59 -0000 1.32
--- Proxy.pm 28 Aug 2003 15:27:56 -0000 1.33
***************
*** 211,218 ****
if ( $^O eq "MSWin32" ) {
- # PLATFORM SPECIFIC CODE
# bypasses bug in -s $pipe under ActivePerl
! my $message;
if ( &{ $self->{pipeready_} }($handle) ) {
--- 211,217 ----
if ( $^O eq "MSWin32" ) {
# bypasses bug in -s $pipe under ActivePerl
! my $message; # PROFILE PLATFORM START MSWin32
if ( &{ $self->{pipeready_} }($handle) ) {
***************
*** 231,236 ****
$message = $1 if ($self->{pipe_cache__} =~ s/(.*?\n)//);
! return $message;
!
} else {
--- 230,234 ----
$message = $1 if ($self->{pipe_cache__} =~ s/(.*?\n)//);
! return $message; # PROFILE PLATFORM STOP
} else {
|
|
From: <jgr...@us...> - 2003-08-28 15:27:23
|
Update of /cvsroot/popfile/engine/tests In directory sc8-pr-cvs1:/tmp/cvs-serv20924/tests Modified Files: TestHTML.script Log Message: Bring TestPOP3 up to 100% coverage by handling the error message that occurs when the secure port is out of range Index: TestHTML.script =================================================================== RCS file: /cvsroot/popfile/engine/tests/TestHTML.script,v retrieving revision 1.14 retrieving revision 1.15 diff -C2 -d -r1.14 -r1.15 *** TestHTML.script 24 Aug 2003 01:20:34 -0000 1.14 --- TestHTML.script 28 Aug 2003 15:27:20 -0000 1.15 *************** *** 654,657 **** --- 654,661 ---- CONFIGIS pop3_secure_port 111 MATCH Updated POP3 SPA/AUTH port to 111; this change will not take affect until you restart POPFile + SETSUBMIT sport 0 + INPUTIS sport 111 + CONFIGIS pop3_secure_port 111 + MATCH The port must be a number between 1 and 65535 # Check autoupdate |
|
From: <jgr...@us...> - 2003-08-28 15:27:20
|
Update of /cvsroot/popfile/engine/Devel
In directory sc8-pr-cvs1:/tmp/cvs-serv20847/Devel
Modified Files:
TestCoverage.pm
Log Message:
Add new PROFILE PLATFORM START directive to mark blocks of code as platform specific so that the profiler can ignore them
Index: TestCoverage.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Devel/TestCoverage.pm,v
retrieving revision 1.12
retrieving revision 1.13
diff -C2 -d -r1.12 -r1.13
*** TestCoverage.pm 31 Jul 2003 16:32:21 -0000 1.12
--- TestCoverage.pm 28 Aug 2003 15:26:44 -0000 1.13
***************
*** 59,68 ****
{
# Print out information for each file
! for my $file (keys %count)
{
if ( ( $file =~ /^[^\/]/ ) && ( $file ne '../tests.pl' ) && !( $file =~ /^..\/\/Test\// ) ) {
my $current_line = 0;
my $block_executed = 0;
!
open SOURCE_FILE, "<$file";
my $clean = $file;
--- 59,69 ----
{
# Print out information for each file
!
! for my $file (keys %count)
{
if ( ( $file =~ /^[^\/]/ ) && ( $file ne '../tests.pl' ) && !( $file =~ /^..\/\/Test\// ) ) {
my $current_line = 0;
my $block_executed = 0;
!
open SOURCE_FILE, "<$file";
my $clean = $file;
***************
*** 75,86 ****
# %count hash for each file: total_lines, total_executable_lines
# and total_executed
while (<SOURCE_FILE>)
{
# Keep count of the total number of lines in this file
$current_line += 1;
!
# We do not count lines that are blank or exclusively
# comments or just have braces on them or
# just an else or just a subroutine definition
if ( ( ( /^\s*\#/ == 0 ) &&
( /^\s*$/ == 0 ) &&
--- 76,90 ----
# %count hash for each file: total_lines, total_executable_lines
# and total_executed
+
while (<SOURCE_FILE>)
{
# Keep count of the total number of lines in this file
+
$current_line += 1;
!
# We do not count lines that are blank or exclusively
# comments or just have braces on them or
# just an else or just a subroutine definition
+
if ( ( ( /^\s*\#/ == 0 ) &&
( /^\s*$/ == 0 ) &&
***************
*** 89,96 ****
( /^\s*package / == 0 ) ) || ( $block_executed ) ) {
$count{$file}{total_executable_lines} += 1;
!
# If this line was executed then keep count of
# that fact
!
if ( ( $count{$file}{$current_line} > 0 ) || ( $block_executed ) ) {
print LINE_DATA "1\n";
--- 93,104 ----
( /^\s*package / == 0 ) ) || ( $block_executed ) ) {
$count{$file}{total_executable_lines} += 1;
!
# If this line was executed then keep count of
# that fact
!
! if ( /\# PROFILE PLATFORM START ([^\r\n]+)/ ) {
! $block_executed = ( $1 ne $^O );
! }
!
if ( ( $count{$file}{$current_line} > 0 ) || ( $block_executed ) ) {
print LINE_DATA "1\n";
***************
*** 107,110 ****
--- 115,122 ----
$block_executed = 0;
}
+
+ if ( /\# PROFILE PLATFORM STOP/ ) {
+ $block_executed = 0;
+ }
} else {
print LINE_DATA "0\n";
***************
*** 114,118 ****
}
}
!
close SOURCE_FILE;
close LINE_DATA;
--- 126,130 ----
}
}
!
close SOURCE_FILE;
close LINE_DATA;
***************
*** 121,125 ****
if ( $block_executed ) {
! print "WARNING: unterminated PROFILE BLOCK detected; did you forget a PROFILE BLOCK STOP in $file?\n";
}
}
--- 133,137 ----
if ( $block_executed ) {
! print "WARNING: unterminated PROFILE BLOCK/PLATFORM detected; did you forget a PROFILE BLOCK/PLATFORM STOP in $file?\n";
}
}
|
|
From: <xue...@us...> - 2003-08-27 17:23:06
|
Update of /cvsroot/popfile/windows In directory sc8-pr-cvs1:/tmp/cvs-serv26275 Modified Files: ioA.ini Log Message: Preset the "Run POPFile in the background at startup" flag [see Bug #795555] Index: ioA.ini =================================================================== RCS file: /cvsroot/popfile/windows/ioA.ini,v retrieving revision 1.15 retrieving revision 1.16 diff -C2 -d -r1.15 -r1.16 *** ioA.ini 31 Jul 2003 16:32:23 -0000 1.15 --- ioA.ini 27 Aug 2003 17:22:49 -0000 1.16 *************** *** 68,71 **** --- 68,72 ---- Top=80 Bottom=96 + State=1 [Field 6] |
|
From: <jgr...@us...> - 2003-08-24 04:58:35
|
Update of /cvsroot/popfile/engine/Classifier
In directory sc8-pr-cvs1:/tmp/cvs-serv31384/Classifier
Modified Files:
WordMangle.pm
Log Message:
Fix minor problems in test suite
Index: WordMangle.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Classifier/WordMangle.pm,v
retrieving revision 1.28
retrieving revision 1.29
diff -C2 -d -r1.28 -r1.29
*** WordMangle.pm 23 Aug 2003 00:48:24 -0000 1.28
--- WordMangle.pm 24 Aug 2003 03:30:08 -0000 1.29
***************
*** 148,157 ****
my ( $self, $stopword ) = @_;
- $stopword = $self->mangle( $stopword, 0, 1 );
-
if ( $stopword =~ /[^[:lower:]\-_\.\@0-9]/i ) {
return 0;
}
if ( $stopword ne '' ) {
$self->{stop__}{$stopword} = 1;
--- 148,157 ----
my ( $self, $stopword ) = @_;
if ( $stopword =~ /[^[:lower:]\-_\.\@0-9]/i ) {
return 0;
}
+ $stopword = $self->mangle( $stopword, 0, 1 );
+
if ( $stopword ne '' ) {
$self->{stop__}{$stopword} = 1;
***************
*** 168,176 ****
my ( $self, $stopword ) = @_;
- $stopword = $self->mangle( $stopword, 0, 1 );
-
if ( $stopword =~ /[^[:lower:]\-_\.\@0-9]/i ) {
return 0;
}
if ( $stopword ne '' ) {
--- 168,176 ----
my ( $self, $stopword ) = @_;
if ( $stopword =~ /[^[:lower:]\-_\.\@0-9]/i ) {
return 0;
}
+
+ $stopword = $self->mangle( $stopword, 0, 1 );
if ( $stopword ne '' ) {
|
11 messages has been excluded from this view by a project administrator.
