This version consists of a major update to v0.21.0 with many improvements
and bug fixes:
1. History in database
We've finally done away with the old method of keeping the message history
in a single directory full of files (which was frankly flaky and hard to
maintain). Messages are now placed in subdirectories with no more than 256
per directory. History is now moved it into the database along with all
the other information that POPFile uses.
This change brings two benefits: firstly it's a lot faster than previous
versions because we can rely on the SQL database to do searching and sorting
for us; secondly, it brings extra flexibility and we now have additional
columns available in the history (we now have two time values (the date/time
the message was received by POPFile and the date/time in the message header),
the From, To, Cc and Subject headers and message size).
2. Switch UI to HTML templates
Back in the mists of POPFile time I grafted an HTML interface onto my little
Perl script and POPFile was born. Unfortunately that HTML interface had
grown into thousands of lines of ugly Perl and HTML interwined. This release
completely separates out the HTML from the Perl.
This has a number of advantages: firstly, POPFile's code is easier to maintain
because it's simpler; secondly, skinning POPFile can now take advantage of the
full power of HTML and not just a limited set of classes and lastly it's made
it easy to expand the UI, which brings us to...
3. UI improvements
The History page is redesigned to be cleaner and support multiple, configurable
columns (From, To, Cc, Arrived Date, Message Date, Subject, etc.). Filtering
options have been expanded. The page now shows the total number of messages
and the line break showing when different download sessions occurred.
Many people will notice that the 'unmagnetized' filter has been removed; this
is because we've added an 'Invert' option that allows inversion of any search
or filter. Now it's possible to filter on 'not spam', for example. To filter
on unmagnetized messages select magnetized and Invert.
Buckets page has been simplified to make it clearer. The confusing accuracy
graph has been removed and unclassified messages are now correctly counted
in total messages and total errors. The old color picker is replaced with
a simple drop down.
The Advanced page now shows which parameters have been changed from the
default. If a parameter has a non-default value then its name is bolded.
Many of the skins had minor improvements. And we have two new skins:
The first, 'osx' gives you a bit of the Mac feel.
As part of the UI improvement a new skin called 'oceanblue' has been added
that demonstrates how the UI for POPFile can be changed dramatically with
the new skinning scheme using only a few files.
4. Anti-spam improvements
The mail parsing engine has been updated to deal with the latest spammer
trickery: notably the Flex Hex trick which you can read about in The
Spammers' Compendium: http://www.jgc.org/tsc/
5. Support for SMTP and NNTP
For a long time we've had experimental support for two new protocols:
SMTP (often used for incoming email) and NNTP (most commonly used for
Usenet-style news). They are no longer experimental and are now distributed
with POPFile. The SMTP and NNTP modules are disabled by default
but can be enabled on the Advanced page.
The SMTP module is designed to act as a front-end for a real SMTP server
accepting messages and proxying them into a single SMTP server after
performing classification. The NNTP module acts as a proxy between your
News client and News server performing classification on articles.
6. Experimental support for IMAP
The new experimental IMAP module is not included in the default distribution.
You can find the documentation and a download link here:
The IMAP module is not a proxy, but acts as a service which will regularily
check an IMAP server for new messages, classifies them and then moves them into
IMAP folders that correspond to your buckets. Reclassification can be done
outside the POPFile user interface simply by moving messages between folders
on the IMAP server.
IMAP module users should watch out about upsetting their ISP if they have a
very large number of buckets. The IMAP module will keep a connection open for
each folder you monitor. This has caused a complaint from at least one ISP to
a user with an extrememly large number of buckets.
7. XML-RPC interface
For some time we've had an experimental external interface to POPFile through
XML-RPC. For the first time we are shipping this XML-RPC component, though
it is disabled by default. If it is enabled an external program written in
many different programming languages can communicate with POPFile to use its
services.
The POP3 proxy has new support for the APOP authentication method. To use
APOP authentication you change the POP3 server string in your email client
from 'host:username' to 'host:username:apop'.
SSL support is added for all proxies. POPFile can now establish secure
connections to servers. To use change the POP3 server string in your email
client from 'host:username' to 'host:username:ssl'. POPFile does not
accept SSL connections, but will establish connections to secure servers.
SOCKS proxying is added to all proxies. To configure the host and port
to use go to the Configuration tab and set it for the appropriate proxy.
Outbound proxy connections will go via the SOCKS server specified.
The Windows installer DOES NOT include the components necessary for SSL
support. If you need SSL support on Windows YOU MUST download the Perl
SSL components as follows:
IO::Socket::SSL
Net::SSLeay
OpenSSL
9. Transparent POP3 Proxying
v0.22.0 includes support for transparent proxying of POP3 connections where
the destination POP3 server is configured in POPFile and the mail client
talks to POPFile without using the special host:user syntax for the hostname.
This is most useful in a small office environment with a dedicated POP3
server where POPFile can act as a front end for the real POP3 server and
by installing it on the same machine as the POP3 server there's no need to
change mail client settings. To use this mode set the host and port of the
real POP3 server on POPFile's Security tab in the SPA/AUTH settings.
ESSENTIAL READING IF YOU ARE UPGRADING TO v0.22.0
1. BACK UP YOUR OLD INSTALLATION: POPFile makes this really easy, just copy
the entire POPFile directory somewhere. You can then safely install
POPFile v0.22.0 on top of your current installation; I just think a back up
is a sensible precaution.
On Windows you may want to check out Brian's Backup and Restore Utilities:
2. ON WINDOWS POPFILE IS AN EXE. Windows users will now be able to see
POPFile running in the Task Manager with an executable called popfileXX.exe
where the XX is one of f, if, b, ib depending on configuration. POPFile
is started by running popfile.exe which chooses the appropriate popfileXX.exe
This might cause your firewall to ask about giving popfileXX.exe permissions,
in addition if you had allowed Perl permissions in your firewall they are
NO LONGER needed.
3. IGNORE "PRIMARY KEY" ERRORS. When the history is being upgraded you may
see some "PRIMARY KEY" errors which can safely be ignored. Note that the
date on entries in the upgraded history will be incorrect: it will appear
that all the messages arrived at the moment of upgrade. Subsequent messages
will be correctly time and date stamped.
KNOWN WINDOWS ISSUES
1. THE TRAYICON ON WINDOWS CAN CAUSE POPFILE TO FREEZE. With certain setups
if you attempt to view the trayicon's tooltip or hover your cursor over it
while checking or downloading mail POPFile may freeze. The exact cause is
still unknown but it may be related to some video drivers. If this problem
affects you turn off the trayicon. A solution is being looked into.
2. ON WINDOWS I WANT TO CHECK MULTIPLE EMAIL ACCOUNTS SIMULTANEOUSLY.
Because the time taken to start a new process on Windows is long under Perl
there is an optimization for Windows that is present by default: when a new
connection is made between your email program and POPFile, POPFile handles it
in the 'parent' process. This means that the connect happens fast and mail
starts downloading very quickly, but is means that you can only downloaded
messages from one server at a time (up to 6 other connections will be queued
up and dealt with in the order they arrive) and the UI is unavaiable while
downloading email.
You can turn this behavior off (and get simultaneous UI/email access and as
many email connections as you like) by going to the Configuration panel in
the UI and making sure that "Allow concurrent POP3 connections:" is set to
Yes, or by specifying --set pop3_force_fork=1 on the command line.
I AM USING THE CROSS PLATFORM VERSION
POPFile requires a number of Perl modules that are available from CPAN. New
in v0.22.0 are the need for the following:
I AM USING THE CROSS PLATFORM VERSION AND WANT TO USE SQLITE 3
Since there is currently a great deal of change in SQLite and the Perl
modules that interface with SQLite please read the following page for
a full explanation:
Just install POPFile on top of the currently installed version. But did you
read the ESSENTIAL READING above first.
DONATIONS
Thank you to everyone who has clicked the Donate! button and donated their
hard earned cash to me in support of POPFile. Thank you also to the people
who have contributed patches, feature requests, bug reports and translations.
This release sees an enormous improvement in the Windows installer which
is the exclusive playground of Brian Smith. Brian has worked tirelessly to
make the Windows installer robust, clear and easy. Nice work, Brian!
A new POPFile developer is Manni Heumann who has made a number of contributions
to the release and wiki, the most significant of which is the implementation of
the experimental IMAP module.
Another new developer, Joseph Connors, has been very active in the HTML
skinning of POPFile and has made major contributions to the POPFile
documentation on the Wiki.
Amatubu is always working to translate and expand the Wiki into Japanese and
Yasuo Honda is spent a great deal of time reviewing the Japanese Wiki
pages.
CONCLUSION
Keep the ideas and bug reports coming. If you are interested in knowing
more about what's planned for future POPFile versions (or just learning
about POPFile's history) visit the POPFile Roadmap:
Do I need to worry about those "PRIMARY KEY" msgs during the database upgrade from 0.21? Got quit a few but otherwise 0.22 appears to have upgraded and is working fine :)
Great work!
Fedorov.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
3. IGNORE "PRIMARY KEY" ERRORS. When the history is being upgraded you may
see some "PRIMARY KEY" errors which can safely be ignored. Note that the
date on entries in the upgraded history will be incorrect: it will appear
that all the messages arrived at the moment of upgrade. Subsequent messages
will be correctly time and date stamped.
So don't worry about it.
John.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The accuracy probably will change a little because we are now including 'unclassified' messages correctly in the error statistics (i.e. when you reclassify and unclassified message it will be counted as an error) and hence your number of errors count probably went up and accuracy went down. You can actually test this hypothesis by going to the buckets page and do the following:
1. Make a note of the "Messages Classified"
2. Make a note of the "Classification Errors"
3. Make a note of the value in the "False Positives" column for unclassified
4. Subtract #3 from #2
5. Now work out the accuracy using the new count of classification errors, my guess is it'll be 99.75%.
John.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi, guys.
I can't find the 'show classification in subject' button in the configuration page. This is disappointing since it's a very important feature to disable (or enable).
But I'm sure I'm missing something here. Any help?
Thanks
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Welcome to POPFile v0.22.0
This version consists of a major update to v0.21.0 with many improvements
and bug fixes:
1. History in database
We've finally done away with the old method of keeping the message history
in a single directory full of files (which was frankly flaky and hard to
maintain). Messages are now placed in subdirectories with no more than 256
per directory. History is now moved it into the database along with all
the other information that POPFile uses.
This change brings two benefits: firstly it's a lot faster than previous
versions because we can rely on the SQL database to do searching and sorting
for us; secondly, it brings extra flexibility and we now have additional
columns available in the history (we now have two time values (the date/time
the message was received by POPFile and the date/time in the message header),
the From, To, Cc and Subject headers and message size).
2. Switch UI to HTML templates
Back in the mists of POPFile time I grafted an HTML interface onto my little
Perl script and POPFile was born. Unfortunately that HTML interface had
grown into thousands of lines of ugly Perl and HTML interwined. This release
completely separates out the HTML from the Perl.
More details are here:
http://popfile.sf.net/cgi-bin/wiki.pl?Template
This has a number of advantages: firstly, POPFile's code is easier to maintain
because it's simpler; secondly, skinning POPFile can now take advantage of the
full power of HTML and not just a limited set of classes and lastly it's made
it easy to expand the UI, which brings us to...
3. UI improvements
The History page is redesigned to be cleaner and support multiple, configurable
columns (From, To, Cc, Arrived Date, Message Date, Subject, etc.). Filtering
options have been expanded. The page now shows the total number of messages
and the line break showing when different download sessions occurred.
Many people will notice that the 'unmagnetized' filter has been removed; this
is because we've added an 'Invert' option that allows inversion of any search
or filter. Now it's possible to filter on 'not spam', for example. To filter
on unmagnetized messages select magnetized and Invert.
Buckets page has been simplified to make it clearer. The confusing accuracy
graph has been removed and unclassified messages are now correctly counted
in total messages and total errors. The old color picker is replaced with
a simple drop down.
The Advanced page now shows which parameters have been changed from the
default. If a parameter has a non-default value then its name is bolded.
Many of the skins had minor improvements. And we have two new skins:
The first, 'osx' gives you a bit of the Mac feel.
As part of the UI improvement a new skin called 'oceanblue' has been added
that demonstrates how the UI for POPFile can be changed dramatically with
the new skinning scheme using only a few files.
4. Anti-spam improvements
The mail parsing engine has been updated to deal with the latest spammer
trickery: notably the Flex Hex trick which you can read about in The
Spammers' Compendium: http://www.jgc.org/tsc/
5. Support for SMTP and NNTP
For a long time we've had experimental support for two new protocols:
SMTP (often used for incoming email) and NNTP (most commonly used for
Usenet-style news). They are no longer experimental and are now distributed
with POPFile. The SMTP and NNTP modules are disabled by default
but can be enabled on the Advanced page.
The SMTP module is designed to act as a front-end for a real SMTP server
accepting messages and proxying them into a single SMTP server after
performing classification. The NNTP module acts as a proxy between your
News client and News server performing classification on articles.
6. Experimental support for IMAP
The new experimental IMAP module is not included in the default distribution.
You can find the documentation and a download link here:
http://popfile.sf.net/cgi-bin/wiki.pl?ExperimentalModules/Imap
The IMAP module is not a proxy, but acts as a service which will regularily
check an IMAP server for new messages, classifies them and then moves them into
IMAP folders that correspond to your buckets. Reclassification can be done
outside the POPFile user interface simply by moving messages between folders
on the IMAP server.
IMAP module users should watch out about upsetting their ISP if they have a
very large number of buckets. The IMAP module will keep a connection open for
each folder you monitor. This has caused a complaint from at least one ISP to
a user with an extrememly large number of buckets.
7. XML-RPC interface
For some time we've had an experimental external interface to POPFile through
XML-RPC. For the first time we are shipping this XML-RPC component, though
it is disabled by default. If it is enabled an external program written in
many different programming languages can communicate with POPFile to use its
services.
Full details of the API can be found here:
http://popfile.sf.net/cgi-bin/wiki.pl?ExperimentalModules/XmlRpc
8. Support for APOP, SSL and SOCKS
The POP3 proxy has new support for the APOP authentication method. To use
APOP authentication you change the POP3 server string in your email client
from 'host:username' to 'host:username:apop'.
SSL support is added for all proxies. POPFile can now establish secure
connections to servers. To use change the POP3 server string in your email
client from 'host:username' to 'host:username:ssl'. POPFile does not
accept SSL connections, but will establish connections to secure servers.
SOCKS proxying is added to all proxies. To configure the host and port
to use go to the Configuration tab and set it for the appropriate proxy.
Outbound proxy connections will go via the SOCKS server specified.
The Windows installer DOES NOT include the components necessary for SSL
support. If you need SSL support on Windows YOU MUST download the Perl
SSL components as follows:
IO::Socket::SSL
Net::SSLeay
OpenSSL
9. Transparent POP3 Proxying
v0.22.0 includes support for transparent proxying of POP3 connections where
the destination POP3 server is configured in POPFile and the mail client
talks to POPFile without using the special host:user syntax for the hostname.
This is most useful in a small office environment with a dedicated POP3
server where POPFile can act as a front end for the real POP3 server and
by installing it on the same machine as the POP3 server there's no need to
change mail client settings. To use this mode set the host and port of the
real POP3 server on POPFile's Security tab in the SPA/AUTH settings.
ESSENTIAL READING IF YOU ARE UPGRADING TO v0.22.0
1. BACK UP YOUR OLD INSTALLATION: POPFile makes this really easy, just copy
the entire POPFile directory somewhere. You can then safely install
POPFile v0.22.0 on top of your current installation; I just think a back up
is a sensible precaution.
On Windows you may want to check out Brian's Backup and Restore Utilities:
http://www.sugelan.co.uk/popfile/databru.html
2. ON WINDOWS POPFILE IS AN EXE. Windows users will now be able to see
POPFile running in the Task Manager with an executable called popfileXX.exe
where the XX is one of f, if, b, ib depending on configuration. POPFile
is started by running popfile.exe which chooses the appropriate popfileXX.exe
This might cause your firewall to ask about giving popfileXX.exe permissions,
in addition if you had allowed Perl permissions in your firewall they are
NO LONGER needed.
3. IGNORE "PRIMARY KEY" ERRORS. When the history is being upgraded you may
see some "PRIMARY KEY" errors which can safely be ignored. Note that the
date on entries in the upgraded history will be incorrect: it will appear
that all the messages arrived at the moment of upgrade. Subsequent messages
will be correctly time and date stamped.
KNOWN WINDOWS ISSUES
1. THE TRAYICON ON WINDOWS CAN CAUSE POPFILE TO FREEZE. With certain setups
if you attempt to view the trayicon's tooltip or hover your cursor over it
while checking or downloading mail POPFile may freeze. The exact cause is
still unknown but it may be related to some video drivers. If this problem
affects you turn off the trayicon. A solution is being looked into.
2. ON WINDOWS I WANT TO CHECK MULTIPLE EMAIL ACCOUNTS SIMULTANEOUSLY.
Because the time taken to start a new process on Windows is long under Perl
there is an optimization for Windows that is present by default: when a new
connection is made between your email program and POPFile, POPFile handles it
in the 'parent' process. This means that the connect happens fast and mail
starts downloading very quickly, but is means that you can only downloaded
messages from one server at a time (up to 6 other connections will be queued
up and dealt with in the order they arrive) and the UI is unavaiable while
downloading email.
You can turn this behavior off (and get simultaneous UI/email access and as
many email connections as you like) by going to the Configuration panel in
the UI and making sure that "Allow concurrent POP3 connections:" is set to
Yes, or by specifying --set pop3_force_fork=1 on the command line.
I AM USING THE CROSS PLATFORM VERSION
POPFile requires a number of Perl modules that are available from CPAN. New
in v0.22.0 are the need for the following:
Date::Parse
HTML::Template
HTML::Tagset
DBD::SQLite
DBI
Date::TimeDate
Please refer to the detailed installation instructions on the POPFile wiki
http://popfile.sf.net/cgi-bin/wiki.pl?HowTos/CrossPlatformInstall
I AM USING THE CROSS PLATFORM VERSION AND WANT TO USE SQLITE 3
Since there is currently a great deal of change in SQLite and the Perl
modules that interface with SQLite please read the following page for
a full explanation:
http://popfile.sf.net/cgi-bin/wiki.pl?SQLite3
(Thanks helphand for the detailed explanation of the issues.)
DOWNLOADING
You can obtain the latest releases of POPFile by visiting
http://sf.net/project/showfiles.php?group_id=63137
UPGRADING
Just install POPFile on top of the currently installed version. But did you
read the ESSENTIAL READING above first.
DONATIONS
Thank you to everyone who has clicked the Donate! button and donated their
hard earned cash to me in support of POPFile. Thank you also to the people
who have contributed patches, feature requests, bug reports and translations.
http://sf.net/forum/forum.php?forum_id=213876
PRAISE
This release sees an enormous improvement in the Windows installer which
is the exclusive playground of Brian Smith. Brian has worked tirelessly to
make the Windows installer robust, clear and easy. Nice work, Brian!
A new POPFile developer is Manni Heumann who has made a number of contributions
to the release and wiki, the most significant of which is the implementation of
the experimental IMAP module.
Another new developer, Joseph Connors, has been very active in the HTML
skinning of POPFile and has made major contributions to the POPFile
documentation on the Wiki.
Amatubu is always working to translate and expand the Wiki into Japanese and
Yasuo Honda is spent a great deal of time reviewing the Japanese Wiki
pages.
CONCLUSION
Keep the ideas and bug reports coming. If you are interested in knowing
more about what's planned for future POPFile versions (or just learning
about POPFile's history) visit the POPFile Roadmap:
http://sf.net/docman/display_doc.php?docid=17906&group_id=63137
John.
Do I need to worry about those "PRIMARY KEY" msgs during the database upgrade from 0.21? Got quit a few but otherwise 0.22 appears to have upgraded and is working fine :)
Great work!
Fedorov.
From the release notes:
3. IGNORE "PRIMARY KEY" ERRORS. When the history is being upgraded you may
see some "PRIMARY KEY" errors which can safely be ignored. Note that the
date on entries in the upgraded history will be incorrect: it will appear
that all the messages arrived at the moment of upgrade. Subsequent messages
will be correctly time and date stamped.
So don't worry about it.
John.
The 0.22.0 Windows installer does not include the IMAP module or the extra files required for the optional SSL support.
A couple of small "add on" installers are available to add IMAP and SSL support to an installation created by the 0.22.0 Windows installer:
IMAP Updater (185 KB zip file) which downloads and installs the IMAP module:
http://www.sugelan.co.uk/popfile/download/cvsbuild/updateimap.zip
SSL Setup (84 KB zip file) which downloads and installs extra Perl components and the SSL libraries:
http://www.sugelan.co.uk/popfile/download/cvsbuild/addssl.zip
Brian
Great - must have read Release notes too fast ;)
My accuracy went from 99.75% down to 99.71% after the upgrade??? Not complaining, still bloody amazing accuracy!!! :)
Fedorov.
The accuracy probably will change a little because we are now including 'unclassified' messages correctly in the error statistics (i.e. when you reclassify and unclassified message it will be counted as an error) and hence your number of errors count probably went up and accuracy went down. You can actually test this hypothesis by going to the buckets page and do the following:
1. Make a note of the "Messages Classified"
2. Make a note of the "Classification Errors"
3. Make a note of the value in the "False Positives" column for unclassified
4. Subtract #3 from #2
5. Now work out the accuracy using the new count of classification errors, my guess is it'll be 99.75%.
John.
Yup, nice thanks :)
Damn quiet in this thread, that either means:
a) 0.22.0 is running fantastic for everyone
b) Nobody else has installed it yet ;)
Fedorov.
Hi, guys.
I can't find the 'show classification in subject' button in the configuration page. This is disappointing since it's a very important feature to disable (or enable).
But I'm sure I'm missing something here. Any help?
Thanks
See the Buckets page.
--
Jim
Hi, Jim. Got it. It was a bad (constructive criticism;-) ) translation (I was using the IT language file) issue. Thanks
Installed on WinXP Pro over 0.21, running OK. I reset the statistics after the install and accuracy has dropped from 99.89% to 99.34% so far.
Bill
Works fine. Thanks, John et al.
Brian (the other one!)