From: naoki i. <am...@us...> - 2008-04-25 16:27:16
|
Update of /cvsroot/popfile/engine/Classifier In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv20326/Classifier Modified Files: Bayes.pm MailParse.pm Log Message: Change Log 1. UI and POP3 proxy are now able to be connected via SSL 2. New Proxy module 'Proxy::POP3S' (POP3 over SSL) 3. New global options: GLOBAL_cert_file, GLOBAL_key_file, GLOBAL_ca_file 4. New html options: html_https_enabled, html_https_port 5. Secure cookies are used when accessed to the HTTPS server Proxy/POP3S.pm Proxy/POP3.pm Proxy/Proxy.pm UI/HTML.pm UI/HTTP.pm POPFile/Configuration.pm Classifier/Bayes.pm skins/default/pop3s-configuration-panel.thtml skins/default/pop3s-security-panel.thtml tests/Configuration.tst The new Proxy::POP3S module supports connections via SSL. It has following options: pop3s_enabled Enable(=1)/Disable(=0;default) the module pop3s_force_fork Enable(=1)/Disable(=0) forking pop3s_local Allow(=1)/Disallow(=0) connections from remote pop3s_port POP3S proxy port(default:995) pop3s_socks_port SOCKS proxy port pop3s_socks_server SOCKS proxy server pop3s_welcome_string POP3S proxy welcome string The other POP3 options (e.g. pop3_separator) are same as the Proxy::POP3 module. Here's the new options of the UI::HTML module: html_https_enabled Enable(=1)/Disable(=0;default) the HTTPS server html_https_port HTTPS server port(default:8443) The new global options: GLOBAL_cert_file Location of the certification file of the server GLOBAL_key_file Location of the key file of the server GLOBAL_ca_file Location of the CA file NOTE: I've implemented POP3S and the HTTPS in the different way. I've made a new module Proxy::POP3S for POP3S, but I don't make the module for HTTPS. This is because I think making the new module for HTTPS ( UI:HTTPS ? ) is very hard. BUG: Concurrent POP3S connections cause an error in some environment: SSL3 alert write:fatal:bad record mac 7664:error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac:s3_pkt.c:424: 6. AUTH PLAIN support Proxy/POP3.pm 7. Supress the verbose status messages on the administration tab 8. Add some status messages (administration tab) UI/HTML.pm UI/XMLRPC.pm Proxy/Proxy.pm Proxy/POP3.pm Proxy/NNTP.pm Proxy/SMTP.pm languages/English.msg languages/Nihongo.msg tests/TestPOP3.tst tests/TestHTML.script 9. The message files are no longer cached by the web browser UI/HTTP.pm 10. if html_allow_javascript == 0, don't disable the radio buttons skins/default/administration-page.thtml skins/default/pop3-security-panel.thtml skins/default/nntp-security-local.thtml skins/default/smtp-security-local.thtml skins/default/xmlrpc-local.thtml 11. Minor updates of the skins skins/smtp-chain-server.thtml skins/smtp-chain-server-port.thtml (merged to the above file) skins/pop3-chain-panel.thtml 12. Source code cleanup UI/HTML.pm UI/HTTP.pm UI/XMLRPC.pm Proxy/POP3.pm Proxy/Proxy.pm Proxy/NNTP.pm Proxy/SMTP.pm Classifier/Bayes.pm Classifier/MailParse.pm POPFile/Configuration.pm POPFile/Database.pm POPFile/History.pm POPFile/Loader.pm POPFile/Logger.pm POPFile/Module.pm POPFile/MQ.pm POPFile/Mutex.pm Add some 'PROFILE BLOCK START' and 'PROFILE BLOCK STOP's. These are used by Devel::TestCoverage to get the correct coverage. 13. Add some tests tests/TestBayes.tst Current state of the test suite: TestBayesScript PASS TestBayes PASS TestConfiguration PASS * TestHistory PASS TestHTML PASS * TestHTTP PASS TestIMAP PASS TestInsertScript PASS * TestLogger PASS TestMailParse PASS TestModule PASS TestMQ PASS TestMutex PASS TestPipeScript PASS TestPOP3 PASS TestProxy PASS TestWordMangle PASS TestXMLRPC PASS * : TODO : needs to add tests for multi user support Index: Bayes.pm =================================================================== RCS file: /cvsroot/popfile/engine/Classifier/Bayes.pm,v retrieving revision 1.368 retrieving revision 1.369 diff -C2 -d -r1.368 -r1.369 *** Bayes.pm 18 Apr 2008 12:41:49 -0000 1.368 --- Bayes.pm 25 Apr 2008 16:26:04 -0000 1.369 *************** *** 335,345 **** # Windows and using the fork. ! if ( ( $nihongo_parser eq 'kakasi' ) && ( $^O eq 'MSWin32' ) && ! ( ( ( $self->module_config_( 'pop3', 'enabled' ) ) && ! ( $self->module_config_( 'pop3', 'force_fork' ) ) ) || ! ( ( $self->module_config_( 'nntp', 'enabled' ) ) && ! ( $self->module_config_( 'nntp', 'force_fork' ) ) ) || ! ( ( $self->module_config_( 'smtp', 'enabled' ) ) && ! ( $self->module_config_( 'smtp', 'force_fork' ) ) ) ) ) { $self->{parser__}->{need_kakasi_mutex__} = 1; --- 335,347 ---- # Windows and using the fork. ! if ( ( $nihongo_parser eq 'kakasi' ) && ( $^O eq 'MSWin32' ) && # PROFILE BLOCK START ! ( ( ( $self->module_config_( 'pop3', 'enabled' ) ) && ! ( $self->module_config_( 'pop3', 'force_fork' ) ) ) || ! ( ( $self->module_config_( 'pop3s', 'enabled' ) ) && ! ( $self->module_config_( 'pop3s', 'force_fork' ) ) ) || ! ( ( $self->module_config_( 'nntp', 'enabled' ) ) && ! ( $self->module_config_( 'nntp', 'force_fork' ) ) ) || ! ( ( $self->module_config_( 'smtp', 'enabled' ) ) && ! ( $self->module_config_( 'smtp', 'force_fork' ) ) ) ) ) { # PROFILE BLOCK STOP $self->{parser__}->{need_kakasi_mutex__} = 1; *************** *** 460,490 **** if ( $bucket ne $newbucket ) { ! my $count = $self->get_bucket_parameter( ! $session, $newbucket, 'count' ); my $newcount = $count + $c; $newcount = 0 if ( $newcount < 0 ); ! $self->set_bucket_parameter( ! $session, $newbucket, 'count', $newcount ); ! $count = $self->get_bucket_parameter( ! $session, $bucket, 'count' ); $newcount = $count - $c; $newcount = 0 if ( $newcount < 0 ); ! $self->set_bucket_parameter( ! $session, $bucket, 'count', $newcount ); ! my $fncount = $self->get_bucket_parameter( ! $session, $newbucket, 'fncount' ); my $newfncount = $fncount + $c; $newfncount = 0 if ( $newfncount < 0 ); ! $self->set_bucket_parameter( ! $session, $newbucket, 'fncount', $newfncount ); ! my $fpcount = $self->get_bucket_parameter( ! $session, $bucket, 'fpcount' ); my $newfpcount = $fpcount + $c; $newfpcount = 0 if ( $newfpcount < 0 ); ! $self->set_bucket_parameter( ! $session, $bucket, 'fpcount', $newfpcount ); } } --- 462,492 ---- if ( $bucket ne $newbucket ) { ! my $count = $self->get_bucket_parameter( # PROFILE BLOCK START ! $session, $newbucket, 'count' ); # PROFILE BLOCK STOP my $newcount = $count + $c; $newcount = 0 if ( $newcount < 0 ); ! $self->set_bucket_parameter( # PROFILE BLOCK START ! $session, $newbucket, 'count', $newcount ); # PROFILE BLOCK STOP ! $count = $self->get_bucket_parameter( # PROFILE BLOCK START ! $session, $bucket, 'count' ); # PROFILE BLOCK STOP $newcount = $count - $c; $newcount = 0 if ( $newcount < 0 ); ! $self->set_bucket_parameter( # PROFILE BLOCK START ! $session, $bucket, 'count', $newcount ); # PROFILE BLOCK STOP ! my $fncount = $self->get_bucket_parameter( # PROFILE BLOCK START ! $session, $newbucket, 'fncount' ); # PROFILE BLOCK STOP my $newfncount = $fncount + $c; $newfncount = 0 if ( $newfncount < 0 ); ! $self->set_bucket_parameter( # PROFILE BLOCK START ! $session, $newbucket, 'fncount', $newfncount ); # PROFILE BLOCK STOP ! my $fpcount = $self->get_bucket_parameter( # PROFILE BLOCK START ! $session, $bucket, 'fpcount' ); # PROFILE BLOCK STOP my $newfpcount = $fpcount + $c; $newfpcount = 0 if ( $newfpcount < 0 ); ! $self->set_bucket_parameter( # PROFILE BLOCK START ! $session, $bucket, 'fpcount', $newfpcount ); # PROFILE BLOCK STOP } } *************** *** 529,534 **** if ( $prob > $max ) { $max = $prob; ! $color = $self->get_bucket_parameter( $session, $bucket, ! 'color' ); } } --- 531,536 ---- if ( $prob > $max ) { $max = $prob; ! $color = $self->get_bucket_parameter( $session, $bucket, # PROFILE BLOCK START ! 'color' ); # PROFILE BLOCK STOP } } *************** *** 579,584 **** # much slower than this single log with a division in it ! return log( $value / ! $self->get_bucket_word_count( $session, $bucket ) ); } else { return 0; --- 581,586 ---- # much slower than this single log with a division in it ! return log( $value / # PROFILE BLOCK START ! $self->get_bucket_word_count( $session, $bucket ) ); # PROFILE BLOCK STOP } else { return 0; *************** *** 611,616 **** my ( $self, $session, $bucket, $word, $value ) = @_; ! if ( $self->db_put_word_count__( $session, $bucket, ! $word, $value ) == 1 ) { # If we set the word count to zero then clean it up by deleting the --- 613,618 ---- my ( $self, $session, $bucket, $word, $value ) = @_; ! if ( $self->db_put_word_count__( $session, $bucket, # PROFILE BLOCK START ! $word, $value ) == 1 ) { # PROFILE BLOCK STOP # If we set the word count to zero then clean it up by deleting the *************** *** 674,679 **** if ( $total != 0 ) { ! $self->{bucket_start__}{$userid}{$bucket} = log( $total / ! $wc ); } else { $self->{bucket_start__}{$userid}{$bucket} = 0; --- 676,681 ---- if ( $total != 0 ) { ! $self->{bucket_start__}{$userid}{$bucket} = log( $total / # PROFILE BLOCK START ! $wc ); # PROFILE BLOCK STOP } else { $self->{bucket_start__}{$userid}{$bucket} = 0; *************** *** 781,789 **** and matrix.bucketid = ?;' ); # PROFILE BLOCK STOP ! $self->{db_get_user_list__} = $self->db_()->prepare( ! 'select id, name from users order by name;' ); ! $self->{db_get_user_from_account__} = $self->db_()->prepare( ! 'select userid from accounts where account = ?' ); # Get the mapping from parameter names to ids into a local hash --- 783,791 ---- and matrix.bucketid = ?;' ); # PROFILE BLOCK STOP ! $self->{db_get_user_list__} = $self->db_()->prepare( # PROFILE BLOCK START ! 'select id, name from users order by name;' ); # PROFILE BLOCK STOP ! $self->{db_get_user_from_account__} = $self->db_()->prepare( # PROFILE BLOCK START ! 'select userid from accounts where account = ?' ); # PROFILE BLOCK STOP # Get the mapping from parameter names to ids into a local hash *************** *** 947,957 **** $word = $self->db_()->quote($word); ! my $result = $self->db_()->selectrow_arrayref( ! "select words.id from words where words.word = $word limit 1;"); if ( !defined( $result ) ) { $self->db_()->do( "insert into words ( word ) values ( $word );" ); ! $result = $self->db_()->selectrow_arrayref( ! "select words.id from words where words.word = $word limit 1;"); } --- 949,959 ---- $word = $self->db_()->quote($word); ! my $result = $self->db_()->selectrow_arrayref( # PROFILE BLOCK START ! "select words.id from words where words.word = $word limit 1;" ); # PROFILE BLOCK STOP if ( !defined( $result ) ) { $self->db_()->do( "insert into words ( word ) values ( $word );" ); ! $result = $self->db_()->selectrow_arrayref( # PROFILE BLOCK START ! "select words.id from words where words.word = $word limit 1;" ); # PROFILE BLOCK STOP } *************** *** 1238,1243 **** my $value = $self->configuration_()->{deprecated_parameters__}{$parameter}; ! if ( defined($module) && defined($config) && defined($value) && ! defined($self->user_module_config_( 1, $module, $config )) ) { # Upgrade parameters to admin's --- 1240,1245 ---- my $value = $self->configuration_()->{deprecated_parameters__}{$parameter}; ! if ( defined($module) && defined($config) && defined($value) && # PROFILE BLOCK START ! defined($self->user_module_config_( 1, $module, $config )) ) { # PROFILE BLOCK STOP # Upgrade parameters to admin's *************** *** 1568,1577 **** $self->log_( 1, "Generating random octet using $module" ); ! my $random = $self->random_()->generate_random_string( $module, 128, $self->global_config_( 'crypt_strength' ), $self->global_config_( 'crypt_device' ) ! ); my $now = time; return sha256_hex( "$$" . "$random$now" ); --- 1570,1579 ---- $self->log_( 1, "Generating random octet using $module" ); ! my $random = $self->random_()->generate_random_string( # PROFILE BLOCK START $module, 128, $self->global_config_( 'crypt_strength' ), $self->global_config_( 'crypt_device' ) ! ); # PROFILE BLOCK STOP my $now = time; return sha256_hex( "$$" . "$random$now" ); *************** *** 1873,1877 **** # accounts. ! if ( ( $module ne 'pop3' ) && ( $module ne 'insert' ) ) { return $self->get_single_user_session_key(); } --- 1875,1879 ---- # accounts. ! if ( ( $module !~ /^pop3s?|insert$/ ) ) { return $self->get_single_user_session_key(); } *************** *** 1879,1883 **** my $user; ! if ( $module eq 'pop3' ) { my ( $server, $username ) = split( /:/, $token ); --- 1881,1885 ---- my $user; ! if ( $module =~ /^pop3s?$/ ) { my ( $server, $username ) = split( /:/, $token ); *************** *** 1902,1907 **** # figure out which user is being talked about ! my $result = $self->{db_get_user_from_account__}->execute( ! "$module:$token" ); if ( !defined( $result ) ) { $self->log_( 1, "Unknown account $module:$token" ); --- 1904,1909 ---- # figure out which user is being talked about ! my $result = $self->{db_get_user_from_account__}->execute( # PROFILE BLOCK START ! "$module:$token" ); # PROFILE BLOCK STOP if ( !defined( $result ) ) { $self->log_( 1, "Unknown account $module:$token" ); *************** *** 2847,2850 **** --- 2849,2860 ---- print $client $msg_head_before; print $client $msg_head_after; + + # Workaround for Win32 compatibility + + while ( length( $msg_body ) > 16383 ) { + my $msg_body_tmp = substr( $msg_body, 0, 16383 ); + print $client $msg_body_tmp; + $msg_body = substr( $msg_body, 16383 ); + } print $client $msg_body; } *************** *** 2892,2897 **** # append any data we got to the actual temp file ! if ( ( (-s "$msg_file.flush") > 0 ) && ! ( open FLUSH, "<$msg_file.flush" ) ) { binmode FLUSH; if ( open TEMP, ">>$msg_file" ) { --- 2902,2907 ---- # append any data we got to the actual temp file ! if ( ( (-s "$msg_file.flush") > 0 ) && # PROFILE BLOCK START ! ( open FLUSH, "<$msg_file.flush" ) ) { # PROFILE BLOCK STOP binmode FLUSH; if ( open TEMP, ">>$msg_file" ) { *************** *** 2962,2973 **** while ( my ( $slot, $newbucket ) = each %messages ) { $self->log_(2, "Message $slot will be reclassified to $newbucket" ); ! push @{$work{$newbucket}}, ! $self->history_()->get_slot_file( $slot ); my @fields = $self->history_()->get_slot_fields( $slot, $session ); my $bucket = $fields[8]; ! $self->classifier_()->reclassified( ! $session, $bucket, $newbucket, 0 ); ! $self->history_()->change_slot_classification( ! $slot, $newbucket, $session, 0); } --- 2972,2983 ---- while ( my ( $slot, $newbucket ) = each %messages ) { $self->log_(2, "Message $slot will be reclassified to $newbucket" ); ! push @{$work{$newbucket}}, # PROFILE BLOCK START ! $self->history_()->get_slot_file( $slot ); # PROFILE BLOCK STOP my @fields = $self->history_()->get_slot_fields( $slot, $session ); my $bucket = $fields[8]; ! $self->classifier_()->reclassified( # PROFILE BLOCK START ! $session, $bucket, $newbucket, 0 ); # PROFILE BLOCK STOP ! $self->history_()->change_slot_classification( # PROFILE BLOCK START ! $slot, $newbucket, $session, 0); # PROFILE BLOCK STOP } *************** *** 3510,3516 **** # If there is a non-default value for this parameter then return it. ! $self->{db_get_bucket_parameter__}->execute( $self->{db_bucketid__}{$userid}{$bucket}{id}, ! $self->{db_parameterid__}{$parameter} ); my $result = $self->{db_get_bucket_parameter__}->fetchrow_arrayref; --- 3520,3526 ---- # If there is a non-default value for this parameter then return it. ! $self->{db_get_bucket_parameter__}->execute( # PROFILE BLOCK START $self->{db_bucketid__}{$userid}{$bucket}{id}, ! $self->{db_parameterid__}{$parameter} ); # PROFILE BLOCK STOP my $result = $self->{db_get_bucket_parameter__}->fetchrow_arrayref; *************** *** 3575,3580 **** my $quoted_user = $self->db_()->quote( $new_user ); my $quoted_hash = $self->db_()->quote( $password_hash ); ! $self->db_()->do( "insert into users ( name, password ) ! values ( $quoted_user, $quoted_hash );" ); my $id = $self->get_user_id( $session, $new_user ); --- 3585,3591 ---- my $quoted_user = $self->db_()->quote( $new_user ); my $quoted_hash = $self->db_()->quote( $password_hash ); ! $self->db_()->do( # PROFILE BLOCK START ! "insert into users ( name, password ) ! values ( $quoted_user, $quoted_hash );" ); # PROFILE BLOCK STOP my $id = $self->get_user_id( $session, $new_user ); *************** *** 3600,3605 **** # Clone user's parameters ! my $h = $self->db_()->prepare( ! "select utid, val from user_params where userid = ?;" ); $h->execute( $clid ); --- 3611,3616 ---- # Clone user's parameters ! my $h = $self->db_()->prepare( # PROFILE BLOCK START ! "select utid, val from user_params where userid = ?;" ); # PROFILE BLOCK STOP $h->execute( $clid ); *************** *** 3610,3616 **** $h->finish; ! $h = $self->db_()->prepare( "insert into user_params ( userid, utid, val ) ! values ( ?, ?, ? );" ); foreach my $utid (keys %add) { $h->execute( $id, $utid, $add{$utid} ); --- 3621,3627 ---- $h->finish; ! $h = $self->db_()->prepare( # PROFILE BLOCK START "insert into user_params ( userid, utid, val ) ! values ( ?, ?, ? );" ); # PROFILE BLOCK STOP foreach my $utid (keys %add) { $h->execute( $id, $utid, $add{$utid} ); *************** *** 3620,3625 **** # Clone buckets (optional) ! $h = $self->db_()->prepare( ! "select name, pseudo from buckets where userid = ?;" ); $h->execute( $clid ); my %buckets; --- 3631,3636 ---- # Clone buckets (optional) ! $h = $self->db_()->prepare( # PROFILE BLOCK START ! "select name, pseudo from buckets where userid = ?;" ); # PROFILE BLOCK STOP $h->execute( $clid ); my %buckets; *************** *** 3629,3635 **** $h->finish; ! $h = $self->db_()->prepare( "insert into buckets ( userid, name, pseudo ) ! values ( ?, ?, ? );" ); foreach my $name (keys %buckets) { $h->execute( $id, $name, $buckets{$name} ); --- 3640,3646 ---- $h->finish; ! $h = $self->db_()->prepare( # PROFILE BLOCK START "insert into buckets ( userid, name, pseudo ) ! values ( ?, ?, ? );" ); # PROFILE BLOCK STOP foreach my $name (keys %buckets) { $h->execute( $id, $name, $buckets{$name} ); *************** *** 3639,3645 **** # Fetch new bucket ids and cloned bucket ids ! $h = $self->db_()->prepare( "select bucket1.id, bucket2.id from buckets as bucket1, buckets as bucket2 ! where bucket1.userid = $id and bucket1.name = bucket2.name and bucket2.userid = ?;" ); $h->execute( $clid ); --- 3650,3658 ---- # Fetch new bucket ids and cloned bucket ids ! $h = $self->db_()->prepare( # PROFILE BLOCK START "select bucket1.id, bucket2.id from buckets as bucket1, buckets as bucket2 ! where bucket1.userid = $id and ! bucket1.name = bucket2.name and ! bucket2.userid = ?;" ); # PROFILE BLOCK STOP $h->execute( $clid ); *************** *** 3652,3658 **** # Clone bucket parameters ! $h = $self->db_()->prepare( "select bucketid, btid, val from buckets, bucket_params ! where userid = ? and buckets.id = bucket_params.bucketid;" ); $h->execute( $clid ); --- 3665,3672 ---- # Clone bucket parameters ! $h = $self->db_()->prepare( # PROFILE BLOCK START "select bucketid, btid, val from buckets, bucket_params ! where userid = ? and ! buckets.id = bucket_params.bucketid;" ); # PROFILE BLOCK STOP $h->execute( $clid ); *************** *** 3663,3669 **** $h->finish; ! $h = $self->db_()->prepare( "insert into bucket_params ( bucketid, btid, val) ! values ( ?, ?, ? );" ); foreach my $bucketid ( keys %bucket_params ) { foreach my $btid ( keys %{$bucket_params{$bucketid}} ) { --- 3677,3683 ---- $h->finish; ! $h = $self->db_()->prepare( # PROFILE BLOCK START "insert into bucket_params ( bucketid, btid, val) ! values ( ?, ?, ? );" ); # PROFILE BLOCK STOP foreach my $bucketid ( keys %bucket_params ) { foreach my $btid ( keys %{$bucket_params{$bucketid}} ) { *************** *** 3677,3683 **** if ( $copy_magnets ) { ! $h = $self->db_()->prepare( "select magnets.bucketid, magnets.mtid, magnets.val from magnets, buckets ! where magnets.bucketid = buckets.id and buckets.userid = ?;" ); $h->execute( $clid ); --- 3691,3698 ---- if ( $copy_magnets ) { ! $h = $self->db_()->prepare( # PROFILE BLOCK START "select magnets.bucketid, magnets.mtid, magnets.val from magnets, buckets ! where magnets.bucketid = buckets.id and ! buckets.userid = ?;" ); # PROFILE BLOCK STOP $h->execute( $clid ); *************** *** 3688,3694 **** $h->finish; ! $h = $self->db_()->prepare( "insert into magnets ( bucketid, mtid, val ) ! values ( ?, ?, ? );" ); foreach my $bucketid ( keys %magnets ) { foreach my $mtid ( keys %{$magnets{$bucketid}} ) { --- 3703,3709 ---- $h->finish; ! $h = $self->db_()->prepare( # PROFILE BLOCK START "insert into magnets ( bucketid, mtid, val ) ! values ( ?, ?, ? );" ); # PROFILE BLOCK STOP foreach my $bucketid ( keys %magnets ) { foreach my $mtid ( keys %{$magnets{$bucketid}} ) { *************** *** 3703,3709 **** if ( $copy_corpus ) { ! $h = $self->db_()->prepare( "select matrix.bucketid, matrix.wordid, matrix.times from matrix, buckets ! where matrix.bucketid = buckets.id and buckets.userid = ?;" ); $h->execute( $clid ); --- 3718,3725 ---- if ( $copy_corpus ) { ! $h = $self->db_()->prepare( # PROFILE BLOCK START "select matrix.bucketid, matrix.wordid, matrix.times from matrix, buckets ! where matrix.bucketid = buckets.id and ! buckets.userid = ?;" ); # PROFILE BLOCK STOP $h->execute( $clid ); *************** *** 3714,3720 **** $h->finish; ! $h = $self->db_()->prepare( "insert into matrix ( bucketid, wordid, times ) ! values ( ?, ?, ? );" ); foreach my $bucketid ( keys %matrix ) { foreach my $wordid ( keys %{$matrix{$bucketid}} ) { --- 3730,3736 ---- $h->finish; ! $h = $self->db_()->prepare( # PROFILE BLOCK START "insert into matrix ( bucketid, wordid, times ) ! values ( ?, ?, ? );" ); # PROFILE BLOCK STOP foreach my $bucketid ( keys %matrix ) { foreach my $wordid ( keys %{$matrix{$bucketid}} ) { *************** *** 3736,3746 **** # default settings ! $self->db_()->do( "insert into buckets ( name, pseudo, userid ) ! values ( 'unclassified', 1, $id );" ); # Copy the global language setting to the user's language setting ! $self->user_module_config_( $id, 'html', 'language', ! $self->global_config_( 'language' ) ); } --- 3752,3763 ---- # default settings ! $self->db_()->do( # PROFILE BLOCK START ! "insert into buckets ( name, pseudo, userid ) ! values ( 'unclassified', 1, $id );" ); # PROFILE BLOCK STOP # Copy the global language setting to the user's language setting ! $self->user_module_config_( $id, 'html', 'language', # PROFILE BLOCK START ! $self->global_config_( 'language' ) ); # PROFILE BLOCK STOP } *************** *** 4201,4206 **** if ( exists( $self->{cached_user_parameters__}{$user}{$parameter} ) ) { $self->{cache_user_parameters__}{$user}{$parameter}{lastused} = time; ! return ($self->{cached_user_parameters__}{$user}{$parameter}{value}, ! $self->{cached_user_parameters__}{$user}{$parameter}{default}); } --- 4218,4223 ---- if ( exists( $self->{cached_user_parameters__}{$user}{$parameter} ) ) { $self->{cache_user_parameters__}{$user}{$parameter}{lastused} = time; ! return ($self->{cached_user_parameters__}{$user}{$parameter}{value}, # PROFILE BLOCK START ! $self->{cached_user_parameters__}{$user}{$parameter}{default}); # PROFILE BLOCK STOP } *************** *** 4215,4220 **** my $default = 0; if ( !defined( $result ) ) { ! $self->{db_get_user_parameter_default__}->execute( # PROFILE BLOCK START ! $self->{db_user_parameterid__}{$parameter} ); # PROFILE BLOCK STOP $result = $self->{db_get_user_parameter_default__}->fetchrow_arrayref; $default = 1; --- 4232,4237 ---- my $default = 0; if ( !defined( $result ) ) { ! $self->{db_get_user_parameter_default__}->execute( # PROFILE BLOCK START ! $self->{db_user_parameterid__}{$parameter} ); # PROFILE BLOCK STOP $result = $self->{db_get_user_parameter_default__}->fetchrow_arrayref; $default = 1; *************** *** 4222,4231 **** if ( defined( $result ) ) { ! $self->{cached_user_parameters__}{$user}{$parameter}{value} = ! $result->[0]; ! $self->{cached_user_parameters__}{$user}{$parameter}{default} = ! $default; ! $self->{cached_user_parameters__}{$user}{$parameter}{lastused} = ! time; return ( $result->[0], $default ); } else { --- 4239,4248 ---- if ( defined( $result ) ) { ! $self->{cached_user_parameters__}{$user}{$parameter}{value} = # PROFILE BLOCK START ! $result->[0]; # PROFILE BLOCK STOP ! $self->{cached_user_parameters__}{$user}{$parameter}{default} = # PROFILE BLOCK START ! $default; # PROFILE BLOCK STOP ! $self->{cached_user_parameters__}{$user}{$parameter}{lastused} = # PROFILE BLOCK START ! time; # PROFILE BLOCK STOP return ( $result->[0], $default ); } else { *************** *** 4317,4323 **** # Prevent user 1 from stopping being an admin ! if ( ( $user == 1 ) && ( $parameter eq 'GLOBAL_can_admin' ) && ! ( $value != 1 ) ) { return 0; } --- 4334,4340 ---- # Prevent user 1 from stopping being an admin ! if ( ( $user == 1 ) && # PROFILE BLOCK START ( $parameter eq 'GLOBAL_can_admin' ) && ! ( $value != 1 ) ) { # PROFILE BLOCK STOP return 0; } *************** *** 4407,4412 **** $self->{parser__}->{bayes__} = bless $self; ! my $result = $self->{parser__}->parse_file( $file, ! $self->global_config_( 'message_cutoff' ) ); $self->{parser__}->{color__} = ''; --- 4424,4430 ---- $self->{parser__}->{bayes__} = bless $self; ! my $result = $self->{parser__}->parse_file( # PROFILE BLOCK START ! $file, ! $self->global_config_( 'message_cutoff' ) ); # PROFILE BLOCK STOP $self->{parser__}->{color__} = ''; *************** *** 4439,4444 **** $bucket = $self->db_()->quote( $bucket ); ! $self->db_()->do( # PROFILE BLOCK START ! "insert into buckets ( name, pseudo, userid ) values ( $bucket, 0, $userid );" ); # PROFILE BLOCK STOP $self->db_update_cache__( $session ); --- 4457,4463 ---- $bucket = $self->db_()->quote( $bucket ); ! $self->db_()->do( # PROFILE BLOCK START ! "insert into buckets ( name, pseudo, userid ) ! values ( $bucket, 0, $userid );" ); # PROFILE BLOCK STOP $self->db_update_cache__( $session ); *************** *** 4659,4663 **** my $bucketid = $self->{db_bucketid__}{$userid}{$bucket}{id}; ! my $h = $self->db_()->prepare( "select magnet_types.mtype from magnet_types, magnets, buckets where magnet_types.id = magnets.mtid and --- 4678,4682 ---- my $bucketid = $self->{db_bucketid__}{$userid}{$bucket}{id}; ! my $h = $self->db_()->prepare( # PROFILE BLOCK START "select magnet_types.mtype from magnet_types, magnets, buckets where magnet_types.id = magnets.mtid and *************** *** 4665,4669 **** buckets.id = ? group by magnet_types.mtype ! order by magnet_types.mtype;" ); $h->execute( $bucketid ); --- 4684,4688 ---- buckets.id = ? group by magnet_types.mtype ! order by magnet_types.mtype;" ); # PROFILE BLOCK STOP $h->execute( $bucketid ); *************** *** 4742,4751 **** my $bucketid = $self->{db_bucketid__}{$userid}{$bucket}{id}; ! my $h = $self->db_()->prepare( "select magnets.val from magnets, magnet_types where magnets.bucketid = $bucketid and magnets.id != 0 and magnet_types.id = magnets.mtid and ! magnet_types.mtype = ? order by magnets.val;" ); $h->execute( $type ); --- 4761,4770 ---- my $bucketid = $self->{db_bucketid__}{$userid}{$bucket}{id}; ! my $h = $self->db_()->prepare( # PROFILE BLOCK START "select magnets.val from magnets, magnet_types where magnets.bucketid = $bucketid and magnets.id != 0 and magnet_types.id = magnets.mtid and ! magnet_types.mtype = ? order by magnets.val;" ); # PROFILE BLOCK STOP $h->execute( $type ); *************** *** 4778,4784 **** my $bucketid = $self->{db_bucketid__}{$userid}{$bucket}{id}; ! my $result = $self->db_()->selectrow_arrayref( "select magnet_types.id from magnet_types ! where magnet_types.mtype = '$type';" ); my $mtid = $result->[0]; --- 4797,4803 ---- my $bucketid = $self->{db_bucketid__}{$userid}{$bucket}{id}; ! my $result = $self->db_()->selectrow_arrayref( # PROFILE BLOCK START "select magnet_types.id from magnet_types ! where magnet_types.mtype = '$type';" ); # PROFILE BLOCK STOP my $mtid = $result->[0]; *************** *** 4786,4791 **** $text = $self->db_()->quote( $text ); ! $self->db_()->do( "insert into magnets ( bucketid, mtid, val ) ! values ( $bucketid, $mtid, $text );" ); } --- 4805,4811 ---- $text = $self->db_()->quote( $text ); ! $self->db_()->do( # PROFILE BLOCK START ! "insert into magnets ( bucketid, mtid, val ) ! values ( $bucketid, $mtid, $text );" ); # PROFILE BLOCK STOP } *************** *** 4808,4814 **** my %result; ! my $h = $self->db_()->prepare( "select magnet_types.mtype, magnet_types.header ! from magnet_types order by mtype;" ); $h->execute; --- 4828,4834 ---- my %result; ! my $h = $self->db_()->prepare( # PROFILE BLOCK START "select magnet_types.mtype, magnet_types.header ! from magnet_types order by mtype;" ); # PROFILE BLOCK STOP $h->execute; *************** *** 4841,4855 **** my $bucketid = $self->{db_bucketid__}{$userid}{$bucket}{id}; ! my $result = $self->db_()->selectrow_arrayref( "select magnet_types.id from magnet_types ! where magnet_types.mtype = '$type';" ); my $mtid = $result->[0]; $text = $self->db_()->quote( $text ); ! $self->db_()->do( "delete from magnets ! where magnets.bucketid = $bucketid and ! magnets.mtid = $mtid and ! magnets.val = $text;" ); } --- 4861,4876 ---- my $bucketid = $self->{db_bucketid__}{$userid}{$bucket}{id}; ! my $result = $self->db_()->selectrow_arrayref( # PROFILE BLOCK START "select magnet_types.id from magnet_types ! where magnet_types.mtype = '$type';" ); # PROFILE BLOCK STOP my $mtid = $result->[0]; $text = $self->db_()->quote( $text ); ! $self->db_()->do( # PROFILE BLOCK START ! "delete from magnets ! where magnets.bucketid = $bucketid and ! magnets.mtid = $mtid and ! magnets.val = $text;" ); # PROFILE BLOCK STOP } *************** *** 4871,4877 **** return undef if ( !defined( $userid ) ); ! my $m = $self->db_()->prepare( "select magnet_types.header, magnets.val from magnet_types, magnets ! where magnet_types.id = magnets.mtid and magnets.id = ?;" ); $m->execute( $magnetid ); --- 4892,4898 ---- return undef if ( !defined( $userid ) ); ! my $m = $self->db_()->prepare( # PROFILE BLOCK START "select magnet_types.header, magnets.val from magnet_types, magnets ! where magnet_types.id = magnets.mtid and magnets.id = ?;" ); # PROFILE BLOCK STOP $m->execute( $magnetid ); *************** *** 4924,4932 **** return undef if ( !defined( $userid ) ); ! my $result = $self->db_()->selectrow_arrayref( "select count(*) from magnets, buckets where buckets.userid = $userid and magnets.id != 0 and ! magnets.bucketid = buckets.id;" ); if ( defined( $result ) ) { --- 4945,4953 ---- return undef if ( !defined( $userid ) ); ! my $result = $self->db_()->selectrow_arrayref( # PROFILE BLOCK START "select count(*) from magnets, buckets where buckets.userid = $userid and magnets.id != 0 and ! magnets.bucketid = buckets.id;" ); # PROFILE BLOCK STOP if ( defined( $result ) ) { *************** *** 4962,4966 **** # Pass language parameter to add_stopword() ! return $self->{parser__}->{mangle__}->add_stopword( $stopword, $self->global_config_( 'language' ) ); } --- 4983,4988 ---- # Pass language parameter to add_stopword() ! return $self->{parser__}->{mangle__}->add_stopword( # PROFILE BLOCK START ! $stopword, $self->global_config_( 'language' ) ); # PROFILE BLOCK STOP } *************** *** 4978,4982 **** # Pass language parameter to remove_stopword() ! return $self->{parser__}->{mangle__}->remove_stopword( $stopword, $self->global_config_( 'language' ) ); } --- 5000,5005 ---- # Pass language parameter to remove_stopword() ! return $self->{parser__}->{mangle__}->remove_stopword( # PROFILE BLOCK START ! $stopword, $self->global_config_( 'language' ) ); # PROFILE BLOCK STOP } Index: MailParse.pm =================================================================== RCS file: /cvsroot/popfile/engine/Classifier/MailParse.pm,v retrieving revision 1.231 retrieving revision 1.232 diff -C2 -d -r1.231 -r1.232 *** MailParse.pm 5 Apr 2008 16:50:45 -0000 1.231 --- MailParse.pm 25 Apr 2008 16:26:04 -0000 1.232 *************** *** 44,50 **** # These are used for Japanese support ! my %encoding_candidates = ( 'Nihongo' => [ 'cp932', 'euc-jp', '7bit-jis' ] ! ); my $ascii = '[\x00-\x7F]'; # ASCII chars --- 44,50 ---- # These are used for Japanese support ! my %encoding_candidates = ( # PROFILE BLOCK START 'Nihongo' => [ 'cp932', 'euc-jp', '7bit-jis' ] ! ); # PROFILE BLOCK STOP my $ascii = '[\x00-\x7F]'; # ASCII chars *************** *** 79,83 **** my $euc_jp_kanji = '[\xB0-\xF4][\xA1-\xFE](?:[\xB0-\xF4][\xA1-\xFE]|\xA1\xB9)?'; # One or two Kanji characters ! my $euc_jp_word = '(' . $euc_jp_alphanum . '|' . $euc_jp_hiragana . '|' . --- 79,83 ---- my $euc_jp_kanji = '[\xB0-\xF4][\xA1-\xFE](?:[\xB0-\xF4][\xA1-\xFE]|\xA1\xB9)?'; # One or two Kanji characters ! my $euc_jp_word = '(' . # PROFILE BLOCK START $euc_jp_alphanum . '|' . $euc_jp_hiragana . '|' . *************** *** 87,91 **** $euc_jp_symbol . '|' . $ascii . '+|' . ! $three_bytes_euc_jp . ')'; # HTML entity mapping to character codes, this maps things like & --- 87,91 ---- $euc_jp_symbol . '|' . $ascii . '+|' . ! $three_bytes_euc_jp . ')'; # PROFILE BLOCK STOP # HTML entity mapping to character codes, this maps things like & *************** *** 284,293 **** my @buckets = $self->{bayes__}->get_buckets( $self->{color__} ); ! return $self->{bayes__}->get_bucket_color( $self->{color__}, $self->{bayes__}->get_top_bucket__( $self->{color_userid__}, $id, $self->{color_matrix__}, ! \@buckets ) ); } else { return 'black'; --- 284,293 ---- my @buckets = $self->{bayes__}->get_buckets( $self->{color__} ); ! return $self->{bayes__}->get_bucket_color( $self->{color__}, # PROFILE BLOCK START $self->{bayes__}->get_top_bucket__( $self->{color_userid__}, $id, $self->{color_matrix__}, ! \@buckets ) ); # PROFILE BLOCK STOP } else { return 'black'; *************** *** 351,356 **** if ( $self->{htmlfontcolor__} ne '' && $self->{htmlbackcolor__} ne '' ) { ! $self->{htmlcolordistance__} = $self->compute_rgb_distance( ! $self->{htmlfontcolor__}, $self->{htmlbackcolor__} ); } } --- 351,356 ---- if ( $self->{htmlfontcolor__} ne '' && $self->{htmlbackcolor__} ne '' ) { ! $self->{htmlcolordistance__} = $self->compute_rgb_distance( # PROFILE BLOCK START ! $self->{htmlfontcolor__}, $self->{htmlbackcolor__} ); # PROFILE BLOCK STOP } } *************** *** 613,617 **** if ( $self->{lang__} eq 'Korean' || $self->{lang__} eq 'Nihongo' ) { ! $to = ' '; } else { $to = chr($to); --- 613,617 ---- if ( $self->{lang__} eq 'Korean' || $self->{lang__} eq 'Nihongo' ) { ! $to = ' '; } else { $to = chr($to); *************** *** 627,633 **** # Don't decode odd (nonprintable) characters or < >'s. ! if ( ( ( $2 < 255 ) && ( $2 > 63 ) ) || ( $2 == 61 ) || ! ( ( $2 < 60 ) && ( $2 > 31 ) ) ) { my $from = $1; my $to = chr($2); --- 627,633 ---- # Don't decode odd (nonprintable) characters or < >'s. ! if ( ( ( $2 < 255 ) && ( $2 > 63 ) ) || # PROFILE BLOCK START ( $2 == 61 ) || ! ( ( $2 < 60 ) && ( $2 > 31 ) ) ) { # PROFILE BLOCK STOP my $from = $1; my $to = chr($2); *************** *** 637,642 **** $self->{ut__} =~ s/$from/$to/g; print "$from -> $to\n" if $self->{debug__}; ! $self->update_pseudoword( 'html', ! 'numericentity', $encoded, $from ); } } --- 637,642 ---- $self->{ut__} =~ s/$from/$to/g; print "$from -> $to\n" if $self->{debug__}; ! $self->update_pseudoword( 'html', # PROFILE BLOCK START ! 'numericentity', $encoded, $from ); # PROFILE BLOCK STOP } } *************** *** 908,913 **** if ($value =~ /^mailto:/i) { if ( $tag =~ /^a$/ && $value =~ /^mailto:([[:alpha:]0-9\-_\.]+?@([[:alpha:]0-9\-_\.]+?))([>\&\?\:\/\" \t]|$)/i ) { ! update_word( $self, $1, $encoded, 'mailto:', ($3?'[\\\>\&\?\:\/]':$end_quote), '' ); ! add_url( $self, $2, $encoded, '@', ($3?'[\\\&\?\:\/]':$end_quote), '' ); } } else { --- 908,913 ---- if ($value =~ /^mailto:/i) { if ( $tag =~ /^a$/ && $value =~ /^mailto:([[:alpha:]0-9\-_\.]+?@([[:alpha:]0-9\-_\.]+?))([>\&\?\:\/\" \t]|$)/i ) { ! update_word( $self, $1, $encoded, 'mailto:', ($3?'[\\\>\&\?\:\/]':$end_quote), '' ); ! add_url( $self, $2, $encoded, '@', ($3?'[\\\&\?\:\/]':$end_quote), '' ); } } else { *************** *** 926,930 **** add_line($self, $value, $encoded, ''); next; ! } # Tags with working background attributes --- 926,930 ---- add_line($self, $value, $encoded, ''); next; ! } # Tags with working background attributes *************** *** 1501,1506 **** $size_read += length($_); $self->parse_line( $_ ); ! if ( ( $max_size > 0 ) && ! ( $size_read > $max_size ) ) { last; } --- 1501,1506 ---- $size_read += length($_); $self->parse_line( $_ ); ! if ( ( $max_size > 0 ) && # PROFILE BLOCK START ! ( $size_read > $max_size ) ) { # PROFILE BLOCK STOP last; } *************** *** 1783,1788 **** # If we are in a mime document then spot the boundaries ! if ( ( $self->{mime__} ne '' ) && ( $line =~ ! /^\-\-($self->{mime__})(\-\-)?/ ) ) { # approach each mime part with fresh eyes --- 1783,1788 ---- # If we are in a mime document then spot the boundaries ! if ( ( $self->{mime__} ne '' ) && # PROFILE BLOCK START ! ( $line =~ /^\-\-($self->{mime__})(\-\-)?/ ) ) { # PROFILE BLOCK STOP # approach each mime part with fresh eyes *************** *** 1948,1953 **** ($pre, $atom, $charset, $encoding, $value) = ($1, $2, $3, $4, $5); ! $output .= $pre unless ($last_is_encoded && defined($atom) # Per RFC 2047 section 6.2 ! && $pre =~ /^[\t ]+$/); if (defined($atom)) { --- 1948,1953 ---- ($pre, $atom, $charset, $encoding, $value) = ($1, $2, $3, $4, $5); ! $output .= $pre unless ($last_is_encoded && defined($atom) # PROFILE BLOCK START ! && $pre =~ /^[\t ]+$/); # PROFILE BLOCK STOP( Per RFC 2047 section 6.2 ) if (defined($atom)) { *************** *** 2044,2048 **** if ( $argument =~ /=\?([^\r\n\t ]{1,40})\?(Q|B)/i ) { ! update_word( $self, $1, 0, '', '', 'charset' ); } --- 2044,2048 ---- if ( $argument =~ /=\?([^\r\n\t ]{1,40})\?(Q|B)/i ) { ! update_word( $self, $1, 0, '', '', 'charset' ); } *************** *** 2720,2725 **** # Insert white spaces after words. ! $self->{nihongo_parser__}{obj_mecab} ! = MeCab::Tagger->new('-F %M\s -U %M\s -E \n'); } --- 2720,2725 ---- # Insert white spaces after words. ! $self->{nihongo_parser__}{obj_mecab} # PROFILE BLOCK START ! = MeCab::Tagger->new('-F %M\s -U %M\s -E \n'); # PROFILE BLOCK STOP } |