Menu

v0.18.0 Windows and Cross Platform Release

2003-02-06
2003-02-07
  • John Graham-Cumming

    INTRODUCTION

    It has been close to 2 months since the last POPFile release and boy have I
    and others been busy.  POPFile v0.18.0 is a *major* change from v0.17.9 and
    you'd do well to read the section entitled ESSENTIAL READING before you
    install it. This version is backwards compatible with previous versions (even
    autofixing some errors that the old versions created) but there are some
    differences.

    This is also the first release in which two additional developers (additional
    to me that is) really contributed a lot of code to POPFile.  They are Stanley
    Krute and Sam Schinke.  Stan and Sam worked on many of the changes to POPFile
    (which you can read all about in the CVS logs) including massive changes to
    the mail parser, whole new skinning system, HTML 4.01 compliance, tons of
    testing, lots of user interface changes.  Guys, *you* made a big difference to
    POPFile, and I want to say a big "Thank you" in public.

    This version of POPFile supports unlimited numbers of connections, multiple
    natural languages, has a much improved UI, has many improvements to help catch
    spam, has a new Quarantine feature that lets POPFile automatically hide all
    the mail in a specific bucket (each mail is replaced with a new mail stating
    that POPFile quarantined a message with an option to see the original), and
    incorporates features and bug fixes submitted by users.

    ESSENTIAL READING

    1. BACK UP YOUR OLD INSTALLATION: POPFile makes this really easy, just copy
       the entire POPFile directory somewhere.  You can then safely install
       POPFile v0.18.0 on top of your current installation; I just think a back up
       is a sensible precaution.

    2. YOUR HISTORY WILL CLEAR: I have changed the format of the History files
       used in this version which means that the old History files will not be
       read by POPFile and will in fact be deleted to save disk space.  If you
       need to do any reclassifications prior to installing v0.18.0 do them!
       POPFile does NOT clear buckets, statistics or anything else on install of
       this version, just the History files.  To refresh the History view click
       the History tab at any time; POPFile will check disk for new messages.

    3. IF YOU HAD BROKEN MAGNETS: I have added automatic update of broken magnets.
       If you had a magnet like [foo] which POPFile mistakenly changed to \[foo\]
       it should get magically fixed and start working.

    4. ACCURACY MIGHT DROP FOR A SHORT WHILE: because of some changes made in the
       mail parser it is possible that you might see accuracy drop initially and
       you may find yourself reclassifying a few messages that used to work.  This
       is unfortunate but necessary to make POPFile even more accurate than before
       and v0.18.0 incorporates changes that make POPFile's classification
       accuracy better; however old corpuses might need a little retraining.

    DOWNLOADING

    You can obtain the latest release of POPFile by visiting

        http://sourceforge.net/project/showfiles.php?group_id=63137

    UPGRADING

    Just install POPFile on top of the currently installed version.  But did you
    read the ESSENTIAL READING above first?

    FAQ

    zonk3r has spent a great deal of time on a POPFile FAQ.  Please check it out
    as it covers many questions that you might have:

        http://sourceforge.net/docman/display_doc.php?docid=14421&group_id=63137

    THE GORY DETAILS

    1. Total rearchitecture of the POPFile code into a collection of loadable
        modules to allow for future expansion of POPFile with different proxy
        services and different UIs.  The major modules are now:

        - popfile.pl: controlling process that loads the modules that implement
          POPFile's main functionality;

        - Proxy::POP3 the original POP3 proxying code from POPFile separated into
          its own module.  In future we should expect to be able to have other
          Proxy:: modules such as Proxy::IMAP.

        - UI::HTML contains the current HTML user interface.  I hope that we'll
          add other UI modules such as UI::XMLRPC or UI::SOAP in future.

        - Classifier::Bayes is the module that does the Naive Bayes
          classification. (Note this module also includes a method called
          classify_and_modify that performs the classification and modification of
          email messages that may be of use to other module writers)

        POPFile now supports two process types: the top level process round robins
        between the main modules listed above providing concurrent access to the
        UI and POP3 proxy.  The POP3 proxy itself (see below) forks to provide
        multiple simultaneous connections. (That last part was originally
        requested by sschinke)

    2. Introduction of a scheme for localization of the user interface using
        message catalogs stored in the languages/ subfolder. Currently shipping
        with languages: English, French, German, Danish, Dutch, Brazilian, and
        (for fun and for testing) H4X0R. Translators can take the English.msg file
        and translate the strings within to create their own language for the user
        interface. POPFile detects available languages on start up and they are
        available from the Configuration page. (Originally requested by
        pfaelzerchen)

    3. Rework of much of the user interface to make pages shorter and clearer.
        This is mostly limited to look and feel and no items have changed tab in
        the UI.

    4. Total rewrite of the history caching code for clarity with a change from
        multiple arrays to a single hash containing all the cached information.

    5. Automatic update checking now defaults to off and there's a large
        explanation of what it does in the user interface to allay the fears of
        many people concerning the possibility that programs like POPFile are
        spying on them. (Requested by quite a few people)

    6. When creating the user interface password the password is now hidden when
        typing it. (Requested by kraelen, pfaelzerchen)

    7. The Proxy::POP3 now forks after a new connection is accepted from a valid
        client and calls a special child() method that handles the connection
        between a client and mail server via POPFile for the life of the client
        connection.  This is the beginning of a version of POPFile that supports a
        very large number of clients. (Requested by many people)

    8. A new command line parameter -test_language can be set to 1 for UI
        translators.  The UI will show the language identifiers used for each
        element rather than the strings associated with the identifier.

    9. Rework of the History page to take into account ideas and suggestions from
        many people: Search now operates on From and Subject, From and Subject
        have popup windows to show the full text of each, you can sort by ID,
        From, Subject or Classification, you can filter by magnet classified mail,
        navigation is at the top and the bottom, state is remembered so that you
        can sort/filter and remove pages and reclassify and stay in the same
        place, many small improvements.

    10. Major work on the MailParse.pm for dealing with improved mail parsing and
        the handling of spammers tricks (like S P A C E D out words etc.).  Switch
        to using the Base64 and QuotedPrint modules from CPAN for improved mail
        cracking. (Work done mainly by sschinke)

    11. Automated test suite now part of POPFile. (Suggested by aziegler)

    12. Rework of all the HTML and CSS so that everything can be skinned and so
        that the complete UI is HTML 4.01 compliant. (Work done mainly by
        stanley_krute; there was a request from nobody for this)

    13. You can now filter the History page by magnet filtered mails to look at
        them or delete them. (Requested by jarmstro, tcdk, msc_buff, fitzsimons,
        dyoungmcwicom)

    14. The UI password no longer appears in URLs in plain text (requested by
        gribnif)

    15. You can do multiple classifications at a time by selecting multiple
        messages and changing the drop down and hitting any Reclassify button.
        (Requested by dave823; implemented by sschinke)

    16. You can now sort the History page by message ID, From, Subject and
        Classification (bucket). (Requested by jerometremblay, tomvoss, biobear)

    17. When you remove a page the History no longer goes back to the start but
        stays at the same place in the History if it can. (Requested by ajweiner)

    18. After you reclassify a message the History page stays at the same position
        (at the reclassified message). (Requested by harleythebrave, transit)

    19. The installer now picks up old settings from a current POPFile install to
        fill in the options on the final dialog. (Requested by harleythebrave)

    20. Added two new options -logdir and -msgdir to specify the location of the
        log directory (default to ./) and the messages (where the history is
        stored; defaults to messages/). (Requested by nobody)

    21. Search in History now works on From as well as Subject. (Requested by
        transit)

    22. The navigation from page to page in the History is now at the top and at
        the bottom of the page. (Requested by transit, nobody, nobody, nobody)

    23. There are new [Remove] buttons on each message in the History to let you
        remove messages one by one. (Requested by nobody)

    24. The X-POPFile-Link: repositions the History page at the message selected
        when you click the link. (Requested by zonk3r)

    25. There is a new filter for unclassified messages. (Requested by nobody)

    26. New -toptoo command line option enables classification on the POP3 TOP
        command and is only recommended for people on fast connections. (Requested
        by jberkes)

    27. When viewing a colorized message there is a blank line between the header
        and the body of the message. (Requested by whiteg)

    28. Message IDs (and History filenames) are now globally unique per POPFile
        install. (Requested by helphand)

    29. You can now control the inclusion of the X-Text-Classification and X-
        POPFile-Link headers. (Requested by nobody)

    30. Refreshing a page does not repeat the action any more. (Reported by
        nobody)

    31. Fixed a bug where magnets with certain characters in them (e.g. [ or <)
        did not work and could not be deleted. (Reported by nobody, nobody,
        nobody, alanmuk, boneill, bana-sea-)

    32. Fixed a bug where you could make POPFile exit with a divide by zero error
        if you started it with no corpus and then tried to Lookup a word's
        probability. (Reported by sunshine, gribnif)

    33. Fixed a bug where the Shutdown link on the POPFile Control Center did not
        work if the UI password had been set. (Reported by actoews, nobody)

    34. Fixed an embarassing bug where any POP server that contained the word
        'stat' would not work with POPFile. (Reported by nobody, markabcxyz)

    35. Fixed a bug where To magnets did not work correctly in the presence of
        other To style email headers. (Reported by dyoungmcwicom)

    36. Fixed a bug where a specific email was causing the History page to lock up
        when you tried to view the colorized version. (Reported by ajweiner)

    37. Fixed a bug where having pages in POPFile automatically refresh every ten
        minutes could cause POPFile to repeat an action (such as Reclassification)
        every 10 minutes. (Reported by mayfoev)

    38. Fixed a bug where parts of a &gt; attribute could appear in the From or
        Subject lines on the History (Reported and fixed by sschinke)

    39. Fixed a bug where some CLS and MSG files were not getting deleted from the
        History when they were no longer needed. (Reported by helphand)

    TRANSLATIONS

    Thanks to the following for helping to make POPFile a global piece of
    software:

        Brazilian       Adriano RG Danish          Nicki Brchner Nielsen Dutch
        Jilles Oldenbeuving French          Antony Boucher German
        Matthias Deege Norweigian      Jo Inge Fjellstad Russian         Alexander
        Saltanov

    Please translate the UI and the manual into your language.

    DONATIONS

    Thank you to everyone who as clicked the Feed Me! button and donated their
    hard earned cash to me in support of POPFile.  Thank you also to the people
    who have contributed patches, feature requests and bug reports.

    Keep the ideas and bug reports coming.

    John.

     
    • Wally Hartshorn

      Wally Hartshorn - 2003-02-07

      Looking at the 0.18 cross-platform file on SourceForge, the size of the zip file is listed as being 2,455,815 bytes. Is that correct? That's far, far larger than the earlier releases, and is even larger than the Windows-only release (which, at about 2MB, is far larger than the 0.179 release).

      Not a complaint really, just want to make sure that there hasn't been a booboo.

       
      • John Graham-Cumming

        It is correct and is caused by all the languages we are now supporting.  In future POPFile is likely to come out in English, French, etc. versions to minimize size.

        John.

         
    • Brian Bloom

      Brian Bloom - 2003-02-07

      From the "release notes":
      23. There are new [Remove] buttons on each message in the History to let you remove messages one by one. (Requested by nobody)

      I haven't downloaded 0.18 to look at it yet, but if we are now using checkboxes to allow [re]classification of multiple messages, can/do we offer the same thing for "remove"?  That is, can I select say 9 messages using the checkboxes and remove them?  That would also eliminate the need to have a "remove" button for each message, a feature that I wonder if it will be used particularly often?

      Again, if this is already supported and I'm just smoking crack, feel free to point it out to me.  I'm at work now and won't be able to actually snag 0.18 until tonight or tomorrow...

       
      • Sam Schinke

        Sam Schinke - 2003-02-07

        Brian,

        Good idea.

        John,

        I can add a "<remove message>" option to the drop-boxes, which would acheive this. Yea, Nay?

        Regards,
        Sam

         
    • Sam Schinke

      Sam Schinke - 2003-02-07

      John,

      Would you consider doing a quick 0.18.0 "minimal" cross-platform release that only includes english.msg, as well as a seperate "languages" zipfile?

      A 2+mb download may discourage some potential users. This "english + languages file" distribution is fairly common among multi-language software I have seen.

      Regards,
      Sam

       
    • Nobody/Anonymous

      >10. Major work on the MailParse.pm for dealing with improved mail parsing and
      the handling of spammers tricks (like S P A C E D out words etc.). Switch
      to using the Base64 and QuotedPrint modules from CPAN for improved mail
      cracking. (Work done mainly by sschinke)

      This should improve filtering accuracy right?

      >11. Automated test suite now part of POPFile. (Suggested by aziegler)

      What's this?

       
      • John Graham-Cumming

        Yes and part of POPFile is an automated test suite that will let the developers test all of POPFile's functionality automatically each time they release a version to make sure nothing got broken.

        It's still in its infancy, but this is the next priority to make POPFile a *real* piece of software.

        John.

         
    • Nobody/Anonymous

      >Yes and part of POPFile is an automated test suite that will let the developers test all of POPFile's functionality automatically each time they release a version to make sure nothing got broken.

      Oh.. Not important for normal idiot users like me than. I thought it was a way to craft a fake email and see how it is processed by popfile.

      >It's still in its infancy, but this is the next priority to make POPFile a *real* piece of software.

      Not sure what you mean by *real* but perhaps one of the beginner skins should have popup help reminding users that they should classify only mail that is wrong (after the training period anyway). Not sure what to do during the training period..reclassify everything?

       
      • Hollywood at monkeysvsrobots.com

        i think you'll find some of the answers to your questions in the faq.

        the automated test suite is to test the effectiveness of the classifier.

        i think what is meant by *real* is to make popfile be seen as a rock solid product.  something people feel comfortable installing and using.  something you would install for your mother.  something that supermegaco, inc. would feel comfy installing for their employees.  thats the way i take it imho.  :)

         

Log in to post a comment.