Thread: [popfile-commit] engine/Classifier Bayes.pm,1.362,1.363

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Update of /cvsroot/popfile/engine/Classifier
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv12124/Classifier

Modified Files:
	Bayes.pm 
Log Message:
Change Log

1. The parameters are now automatically upgraded

  Classifier/Bayes.pm

  Parameters in the configration file (popfile.cfg) are automatically 
  upgraded to the POPFile database as the admin's parameters exept for
  the password of the UI (html_password).

2. A new global option 'language' (GLOBAL_language)

  POPFile/Configuration.pm
  Classifier/Bayes.pm
  tests/TestBayes.tst
  tests/TestMailParse.tst

  This setting is used by POPFile to decide whether it runs some 
  language specific codes. e.g. if GLOBAL_language is 'Nihongo', 
  POPFile will handle Japanese texts correctly.

  And it is used as the default language setting for the users.

  TODO:

  In the single user mode, POPFile's 'global' language setting should
  be equaled to the admin's setting.

3. Localized UI strings are now cached per user (session)

  UI/HTML.pm
  tests/TestHTML.tst

  All of the users shared the language cache till now.
  If a user changed the language setting, the cache will be updated,
  and affect the other users' UI.

  The user's cache will be cleared when the session is rleased.

4. The magnet tab always show the localized magnet types

  UI/HTML.pm
  skins/default/magnet-page.thtml

5. Cookie will be deleted when closing web browsers

  UI/HTML.pm

  I think there's no need to store the cookie for two weeks.

6. Removed almost all of the hardcoded user id 1

  UI/HTML.pm
  Classifier/Bayes.pm
  POPFile/History.pm

7. Login page for the single user mode

  UI/HTML.pm
  skins/default/password-page.thtml
  languages/English.msg
  languages/Nihongo.msg

  If admin has a password, login page will be shown even in the single
  user mode 

8. commit_history is called before the session key is released

  Classifier/Bayes.pm
  POPFile/History.pm

  Avoid the 'Invalid session key' error occuring when POPFile is updating
  the history.

9. Messages are now ordered by their types.

  POPFile/MQ.pm

  Messages are handled in following order:

    1 CREAT
    2 LOGIN
    3 UIREG
    4 COMIT
    5 TICKD
    6 RELSE

  In some case, COMIT had been processed before the session key is
  created by CREAT. This caused the 'Invalid session key' errors.

10. TestPOP3 is now passed

  tests/TestPOP3.tst
  tests/TestMailParse029.msg
  tests/TestMailParse029.cam
  tests/TestMailParse029.wrd

11. Fixed some tests to output their result correctly

  tests/TestBayesScript.tst
  tests/TestConfiguration.tst
  tests/TestHTTP.tst
  tests/TestInsertScript.tst
  tests/TestProxy.tst

Current state of the test suite:

    TestBayesScript                 PASS
    TestBayes                       PASS
    TestConfiguration               PASS *
    TestHistory                     fail *
    TestHTML                        PASS *
    TestHTTP                        PASS
    TestIMAP               not tested yet
    TestInsertScript                PASS *
    TestLogger                      PASS
    TestMailParse                   PASS
    TestModule                      PASS
    TestMQ                          PASS
    TestMutex                       PASS
    TestPipeScript                  PASS
    TestPOP3                        PASS
    TestProxy                       PASS
    TestWordMangle                  PASS
    TestXMLRPC                      PASS (but child process does not terminate)

    * :  needs to add tests for multi user support

TODO

1. multi user mode tests are needed

  tests/TestConfiguration.tst
  tests/TestHistory.tst
  tests/TestHTML.tst
  tests/TestInsertScript.tst

Index: Bayes.pm
===================================================================
RCS file: /cvsroot/popfile/engine/Classifier/Bayes.pm,v
retrieving revision 1.362
retrieving revision 1.363
diff -C2 -d -r1.362 -r1.363
*** Bayes.pm	6 Apr 2008 08:27:07 -0000	1.362
--- Bayes.pm	9 Apr 2008 17:20:47 -0000	1.363
***************
*** 262,265 ****
--- 262,270 ----

      if ( $type eq 'RELSE' ) {
+         # Before releasing the session key we have to make sure that all of
+         # the histories are committed
+ 
+         $self->history_()->commit_history() if ( defined($self->history_()) );
+ 
          $self->release_session_key_private__( $message[0] );
      }
***************
*** 297,304 ****
      # characters(EUC-JP).

!     # TODO : hardcoded 1

!     if ( defined( $self->user_module_config_( 1, 'html', 'language' ) ) &&
!        ( $self->user_module_config_( 1, 'html', 'language' ) =~ /^Nihongo|Korean$/ )) {
          use POSIX qw( locale_h );
          setlocale( LC_COLLATE, 'C' );
--- 302,308 ----
      # characters(EUC-JP).

!     my $language = $self->global_config_( 'language' );

!     if ( defined( $language ) && ( $language =~ /^Nihongo|Korean$/ ) ) {
          use POSIX qw( locale_h );
          setlocale( LC_COLLATE, 'C' );
***************
*** 307,317 ****
      # Pass in the current interface language for language specific parsing

!     # TODO : hardcoded 1
! 
!     $self->{parser__}->{lang__}  = $self->user_module_config_( 1, 'html', 'language' ) || '';
!     $self->{unclassified__} = log( $self->user_config_( 1, 'unclassified_weight' ) );

      $self->upgrade_predatabase_data__();

      # Since Text::Kakasi is not thread-safe, we use it under the
      # control of a Mutex to avoid a crash if we are running on
--- 311,320 ----
      # Pass in the current interface language for language specific parsing

!     $self->{parser__}->{lang__}  = $language || '';

      $self->upgrade_predatabase_data__();

+     $self->upgrade_v1_data__();
+ 
      # Since Text::Kakasi is not thread-safe, we use it under the
      # control of a Mutex to avoid a crash if we are running on
***************
*** 1194,1197 ****
--- 1197,1239 ----
  #----------------------------------------------------------------------------
  #
+ # upgrade_v1_data__
+ #
+ # If the deprecated parameters found, upgrades them to the SQL database.
+ #
+ #----------------------------------------------------------------------------
+ sub upgrade_v1_data__
+ {
+     my ( $self ) = @_;
+ 
+     # Copy deprecated parameters to database
+ 
+     if ( defined($self->configuration_()->{deprecated_parameters__}) ) {
+         $self->log_( 1, "Upgrade the deprecated parameters in popfile.cfg to the admin's parameters\n" );
+ 
+         foreach my $parameter ( keys %{$self->configuration_()->{deprecated_parameters__}} ) {
+             $parameter =~ m/^([^_]+)_(.*)$/;
+             my $module = $1;
+             my $config = $2;
+             my $value  = $self->configuration_()->{deprecated_parameters__}{$parameter};
+ 
+             if ( defined($module) && defined($config) && defined($value) && 
+                     defined($self->user_module_config_( 1, $module, $config )) ) {
+ 
+                 # Upgrade parameters to admin's
+ 
+                 $self->user_module_config_( 1, $module, $config, $value );
+ 
+                 # Upgrade language parameter to global
+ 
+                 if ( ( $module eq 'html' ) && ( $config eq 'language' ) ) {
+                     $self->global_config_( 'language', $value );
+                 }
+             }
+         }
+     }
+ }
+ 
+ #----------------------------------------------------------------------------
+ #
  # magnet_match_helper__
  #
***************
*** 1214,1221 ****
      $match = lc($match);

-     # In Japanese and Korean mode, disable locale.  Sorting Japanese
-     # and Korean with "use locale" is memory and time consuming, and
-     # may cause perl crash.
- 
      my @magnets;

--- 1256,1259 ----
***************
*** 1905,1909 ****
      return undef if ( !defined( $userid ) );

!     $self->{unclassified__} = log( $self->user_config_( 1, 'unclassified_weight' ) );

      $self->{magnet_used__}   = 0;
--- 1943,1947 ----
      return undef if ( !defined( $userid ) );

!     $self->{unclassified__} = log( $self->user_config_( $userid, 'unclassified_weight' ) );

      $self->{magnet_used__}   = 0;
***************
*** 2469,2473 ****
      if ( $class eq '' ) {
          $self->{parser__}->start_parse();
!         ( $slot, $msg_file ) = $self->history_()->reserve_slot( $session, $userid );
      } else {
          $msg_file = $self->history_()->get_slot_file( $slot );
--- 2507,2511 ----
      if ( $class eq '' ) {
          $self->{parser__}->start_parse();
!         ( $slot, $msg_file ) = $self->history_()->reserve_slot( $session );
      } else {
          $msg_file = $self->history_()->get_slot_file( $slot );
***************
*** 3216,3223 ****
      # perl crash.

!     if ( $self->user_module_config_( 1, 'html', 'language' ) eq 'Nihongo' ) {
          return grep {$_ ne $prev && ($prev = $_, 1)} sort map {substr_euc__($_,0,1)} @{$result};
      } else {
!         if  ( $self->user_module_config_( 1, 'html', 'language' ) eq 'Korean' ) {
              return grep {$_ ne $prev && ($prev = $_, 1)} sort map {$_ =~ /([\x20-\x80]|$eksc)/} @{$result};
          } else {
--- 3254,3261 ----
      # perl crash.

!     if ( $self->global_config_( 'language' ) eq 'Nihongo' ) {
          return grep {$_ ne $prev && ($prev = $_, 1)} sort map {substr_euc__($_,0,1)} @{$result};
      } else {
!         if  ( $self->global_config_( 'language' ) eq 'Korean' ) {
              return grep {$_ ne $prev && ($prev = $_, 1)} sort map {$_ =~ /([\x20-\x80]|$eksc)/} @{$result};
          } else {
***************
*** 4647,4653 ****
      # Pass language parameter to add_stopword()

!     # TODO : hardcoded 1
! 
!     return $self->{parser__}->{mangle__}->add_stopword( $stopword, $self->user_module_config_( 1, 'html', 'language' ) );
  }

--- 4685,4689 ----
      # Pass language parameter to add_stopword()

!     return $self->{parser__}->{mangle__}->add_stopword( $stopword, $self->global_config_( 'language' ) );
  }

***************
*** 4661,4667 ****
      # Pass language parameter to remove_stopword()

!     # TODO : hardcoded 1
! 
!     return $self->{parser__}->{mangle__}->remove_stopword( $stopword, $self->user_module_config_( 1, 'html', 'language' ) );
  }

--- 4697,4701 ----
      # Pass language parameter to remove_stopword()

!     return $self->{parser__}->{mangle__}->remove_stopword( $stopword, $self->global_config_( 'language' ) );
  }