[Popfile-announce] POPFile v0.18.0 Released

[John Graham-C. <jg...@jg...>]

INTRODUCTION

It has been close to 2 months since the last POPFile release and boy have=
 I
and others been busy.  POPFile v0.18.0 is a *major* change from v0.17.9 a=
nd
you'd do well to read the section entitled ESSENTIAL READING before you
install it. This version is backwards compatible with previous versions
(even
autofixing some errors that the old versions created) but there are some
differences.

This is also the first release in which two additional developers
(additional
to me that is) really contributed a lot of code to POPFile.  They are
Stanley
Krute and Sam Schinke.  Stan and Sam worked on many of the changes to
POPFile
(which you can read all about in the CVS logs) including massive changes =
to
the mail parser, whole new skinning system, HTML 4.01 compliance, tons of
testing, lots of user interface changes.  Guys, *you* made a big differen=
ce
to
POPFile, and I want to say a big "Thank you" in public.

This version of POPFile supports unlimited numbers of connections, multip=
le
natural languages, has a much improved UI, has many improvements to help
catch
spam, has a new Quarantine feature that lets POPFile automatically hide a=
ll
the mail in a specific bucket (each mail is replaced with a new mail stat=
ing
that POPFile quarantined a message with an option to see the original), a=
nd
incorporates features and bug fixes submitted by users.


ESSENTIAL READING

1. BACK UP YOUR OLD INSTALLATION: POPFile makes this really easy, just co=
py
   the entire POPFile directory somewhere.  You can then safely install
   POPFile v0.18.0 on top of your current installation; I just think a ba=
ck
up
   is a sensible precaution.

2. YOUR HISTORY WILL CLEAR: I have changed the format of the History file=
s
   used in this version which means that the old History files will not b=
e
   read by POPFile and will in fact be deleted to save disk space.  If yo=
u
   need to do any reclassifications prior to installing v0.18.0 do them!
   POPFile does NOT clear buckets, statistics or anything else on install=
 of
   this version, just the History files.  To refresh the History view cli=
ck
   the History tab at any time; POPFile will check disk for new messages.

3. IF YOU HAD BROKEN MAGNETS: I have added automatic update of broken
magnets.
   If you had a magnet like [foo] which POPFile mistakenly changed to
\[foo\]
   it should get magically fixed and start working.

4. ACCURACY MIGHT DROP FOR A SHORT WHILE: because of some changes made in
the
   mail parser it is possible that you might see accuracy drop initially =
and
   you may find yourself reclassifying a few messages that used to work.
This
   is unfortunate but necessary to make POPFile even more accurate than
before
   and v0.18.0 incorporates changes that make POPFile's classification
   accuracy better; however old corpuses might need a little retraining.


DOWNLOADING

You can obtain the latest release of POPFile by visiting

    http://sourceforge.net/project/showfiles.php?group_id=3D63137


UPGRADING

Just install POPFile on top of the currently installed version.  But did =
you
read the ESSENTIAL READING above first?


FAQ

zonk3r has spent a great deal of time on a POPFile FAQ.  Please check it =
out
as it covers many questions that you might have:

    http://sourceforge.net/docman/display_doc.php?docid=3D14421&group_id=3D=
63137


THE GORY DETAILS

 1. Total rearchitecture of the POPFile code into a collection of loadabl=
e
    modules to allow for future expansion of POPFile with different proxy
    services and different UIs.  The major modules are now:

    - popfile.pl: controlling process that loads the modules that impleme=
nt
      POPFile's main functionality;

    - Proxy::POP3 the original POP3 proxying code from POPFile separated
into
      its own module.  In future we should expect to be able to have othe=
r
      Proxy:: modules such as Proxy::IMAP.

    - UI::HTML contains the current HTML user interface.  I hope that we'=
ll
      add other UI modules such as UI::XMLRPC or UI::SOAP in future.

    - Classifier::Bayes is the module that does the Naive Bayes
      classification. (Note this module also includes a method called
      classify_and_modify that performs the classification and modificati=
on
of
      email messages that may be of use to other module writers)

    POPFile now supports two process types: the top level process round
robins
    between the main modules listed above providing concurrent access to =
the
    UI and POP3 proxy.  The POP3 proxy itself (see below) forks to provid=
e
    multiple simultaneous connections. (That last part was originally
    requested by sschinke)

 2. Introduction of a scheme for localization of the user interface using
    message catalogs stored in the languages/ subfolder. Currently shippi=
ng
    with languages: English, French, German, Danish, Dutch, Brazilian, an=
d
    (for fun and for testing) H4X0R. Translators can take the English.msg
file
    and translate the strings within to create their own language for the
user
    interface. POPFile detects available languages on start up and they a=
re
    available from the Configuration page. (Originally requested by
    pfaelzerchen)

 3. Rework of much of the user interface to make pages shorter and cleare=
r.
    This is mostly limited to look and feel and no items have changed tab=
 in
    the UI.

 4. Total rewrite of the history caching code for clarity with a change f=
rom
    multiple arrays to a single hash containing all the cached informatio=
n.

 5. Automatic update checking now defaults to off and there's a large
    explanation of what it does in the user interface to allay the fears =
of
    many people concerning the possibility that programs like POPFile are
    spying on them. (Requested by quite a few people)

 6. When creating the user interface password the password is now hidden
when
    typing it. (Requested by kraelen, pfaelzerchen)

 7. The Proxy::POP3 now forks after a new connection is accepted from a
valid
    client and calls a special child() method that handles the connection
    between a client and mail server via POPFile for the life of the clie=
nt
    connection.  This is the beginning of a version of POPFile that suppo=
rts
a
    very large number of clients. (Requested by many people)

 8. A new command line parameter -test_language can be set to 1 for UI
    translators.  The UI will show the language identifiers used for each
    element rather than the strings associated with the identifier.

 9. Rework of the History page to take into account ideas and suggestions
from
    many people: Search now operates on From and Subject, From and Subjec=
t
    have popup windows to show the full text of each, you can sort by ID,
    From, Subject or Classification, you can filter by magnet classified
mail,
    navigation is at the top and the bottom, state is remembered so that =
you
    can sort/filter and remove pages and reclassify and stay in the same
    place, many small improvements.

10. Major work on the MailParse.pm for dealing with improved mail parsing
and
    the handling of spammers tricks (like S P A C E D out words etc.).
Switch
    to using the Base64 and QuotedPrint modules from CPAN for improved ma=
il
    cracking. (Work done mainly by sschinke)

11. Automated test suite now part of POPFile. (Suggested by aziegler)

12. Rework of all the HTML and CSS so that everything can be skinned and =
so
    that the complete UI is HTML 4.01 compliant. (Work done mainly by
    stanley_krute; there was a request from nobody for this)

13. You can now filter the History page by magnet filtered mails to look =
at
    them or delete them. (Requested by jarmstro, tcdk, msc_buff, fitzsimo=
ns,
    dyoungmcwicom)

14. The UI password no longer appears in URLs in plain text (requested by
    gribnif)

15. You can do multiple classifications at a time by selecting multiple
    messages and changing the drop down and hitting any Reclassify button.
    (Requested by dave823; implemented by sschinke)

16. You can now sort the History page by message ID, From, Subject and
    Classification (bucket). (Requested by jerometremblay, tomvoss, biobe=
ar)

17. When you remove a page the History no longer goes back to the start b=
ut
    stays at the same place in the History if it can. (Requested by
ajweiner)

18. After you reclassify a message the History page stays at the same
position
    (at the reclassified message). (Requested by harleythebrave, transit)

19. The installer now picks up old settings from a current POPFile instal=
l
to
    fill in the options on the final dialog. (Requested by harleythebrave=
)

20. Added two new options -logdir and -msgdir to specify the location of =
the
    log directory (default to ./) and the messages (where the history is
    stored; defaults to messages/). (Requested by nobody)

21. Search in History now works on From as well as Subject. (Requested by
    transit)

22. The navigation from page to page in the History is now at the top and=
 at
    the bottom of the page. (Requested by transit, nobody, nobody, nobody=
)

23. There are new [Remove] buttons on each message in the History to let =
you
    remove messages one by one. (Requested by nobody)

24. The X-POPFile-Link: repositions the History page at the message selec=
ted
    when you click the link. (Requested by zonk3r)

25. There is a new filter for unclassified messages. (Requested by nobody=
)

26. New -toptoo command line option enables classification on the POP3 TO=
P
    command and is only recommended for people on fast connections.
(Requested
    by jberkes)

27. When viewing a colorized message there is a blank line between the
header
    and the body of the message. (Requested by whiteg)

28. Message IDs (and History filenames) are now globally unique per POPFi=
le
    install. (Requested by helphand)

29. You can now control the inclusion of the X-Text-Classification and X-
    POPFile-Link headers. (Requested by nobody)

30. Refreshing a page does not repeat the action any more. (Reported by
    nobody)

31. Fixed a bug where magnets with certain characters in them (e.g. [ or =
<)
    did not work and could not be deleted. (Reported by nobody, nobody,
    nobody, alanmuk, boneill, bana-sea-)

32. Fixed a bug where you could make POPFile exit with a divide by zero
error
    if you started it with no corpus and then tried to Lookup a word's
    probability. (Reported by sunshine, gribnif)

33. Fixed a bug where the Shutdown link on the POPFile Control Center did
not
    work if the UI password had been set. (Reported by actoews, nobody)

34. Fixed an embarassing bug where any POP server that contained the word
    'stat' would not work with POPFile. (Reported by nobody, markabcxyz)

35. Fixed a bug where To magnets did not work correctly in the presence o=
f
    other To style email headers. (Reported by dyoungmcwicom)

36. Fixed a bug where a specific email was causing the History page to lo=
ck
up
    when you tried to view the colorized version. (Reported by ajweiner)

37. Fixed a bug where having pages in POPFile automatically refresh every
ten
    minutes could cause POPFile to repeat an action (such as
Reclassification)
    every 10 minutes. (Reported by mayfoev)

38. Fixed a bug where parts of a &gt; attribute could appear in the From =
or
    Subject lines on the History (Reported and fixed by sschinke)

39. Fixed a bug where some CLS and MSG files were not getting deleted fro=
m
the
    History when they were no longer needed. (Reported by helphand)


TRANSLATIONS

Thanks to the following for helping to make POPFile a global piece of
software:

    Brazilian       Adriano RG Danish          Nicki Br=F8chner Nielsen D=
utch
    Jilles Oldenbeuving French          Antony Boucher German
    Matthias Deege Norweigian      Jo Inge Fjellstad Russian
Alexander
    Saltanov

Please translate the UI and the manual into your language.


DONATIONS

Thank you to everyone who as clicked the Feed Me! button and donated thei=
r
hard earned cash to me in support of POPFile.  Thank you also to the peop=
le
who have contributed patches, feature requests and bug reports.

Keep the ideas and bug reports coming.

John.