[maildb-devel] Server-side filtering

[Jeff S. <jsq...@os...>]

We've talked about maildb built-in filtering before.  Indeed, that's
one of the main strengths of maildb -- that you can/should have
millions of rules that will attach all kinds of categories to messages
(e.g., as opposed to common thinking/usage today where most users file
a message away in a *single* target folder; with maildb you should add
*lots* of categories to each message -- this actually *increases* the
possibility of you seeing important mails, as opposed to only filing a
message away in a single [potentially obscure] folder).

So we [eventually] need to support server-side filtering somehow.

Up until now, we've only concentrated on the storage of messages -- we
need to get this thing working before we tackle the complex issues of
built-in server-side filtering.  I think that's been a good decision.

But since this is a major feature/capability of maildb, it would be
good to support it *somehow* -- even in our initial versions.

The thought occurred to me today: what about procmail?

Procmail is a slick server-side user filtering agent that is typically
invoked directly by the MTA (e.g., via .forward).

Obviously, procmail can write to the conventional mbox and mh formats,
but it won't know how to write to the maildb data store.  But perhaps
there's a quick-n-dirty way to make procmail work with maild: instead
of having procmail write the actual output message to a mailbox file,
have it simply add a header line telling maildb what to do when the
message eventually gets written to the database.  Perhaps, something
like:

-----
:0 fc
* ^TO_...@li...
| formail -A "X-Maildb-Category: maildb/devel"

:0 fc
* ^FROM_.*@squyres.com
| formail -A "X-Maildb-Category: received/squyres/family"
-----

Then when the message finally gets written to the db, maildb will see
any X-Maildb-Category line(s) and attach the appropriate category
name(s) to the message in the database.  This is actually more
efficient, because procmail won't write out the message N times --
it'll only add N header lines and then write out the message *once* to
the backing store.

To make it work, there will need to be a final, all-encompassing
procmail rule that actually writes the resulting message (including
any added X-Maildb-Category header lines) into maildb by invoking some
custom executable:

-----
:0 f
| /usr/local/bin/maildb.insert
-----

...or something along those lines.

Does this sound too hack-ish?  Any other thoughts/ideas?

-- 
{+} Jeff Squyres
{+} jsq...@os...
{+} Research Associate, Open Systems Lab, Indiana University
{+} http://www.osl.iu.edu/