CRM114 Discriminator - Controllable Rege

on 4/28/05 9:08 PM, Fidelis Assis <fidelis@...> wrote:

> I'm interested in your scripts. Could you send me? Do they work for any
> MUA, say, Outlook 2002? I've a co-worker that uses it and we couldn't
> find a way to get the complete message source, only the header.

Hmm.  Well you understand I'm using server-side scripts, right?  It sounds
like you are wanting to run the filter under your local machine's MUA, which
is not something I can help you with, especially if your local machine is
not *nix, which is apparently so if it is running Outlook.  You could
probably beat DOS-mode into running some modification of these scripts if
you are running a mail server on your MUA machine, but it may not be pretty.

The scripts should be able to be modified to work on any *nix server, as
long as your LDA is sufficiently similar.  But they were designed to be
inserted into the qmail-vpopmail delivery pipeline.  There scripts have some
dependencies currently on the qmail/vpopmail methodology, which has to do
mostly with how the Delivered-To and Return-Path information are kept in
environment variables and added to the message at the moment of actual
delivery to the Maildir, which happens *after* the mail filtering step.

The filtering does not do routing to different folders, but depends on the
X-CRM114 headers having been added, allowing the MUA to filter as
spam/nonspam based on the status header.  Currently there is no provision
for shades of spam because the current method depends on mailfilter.crm's
added headers being utilized by MUA rule-based filtering, and the numeric
information would be too much for MUA rules to handle.

I'll put them up on the web when I get the chance and post a link here with
some explanation.  But if this does not sound like what you need, then I may
do this in less of a hurry.

-Kurt

on 4/28/05 9:38 PM, Kurt Bigler <kkb@...> wrote:

> on 4/28/05 9:08 PM, Fidelis Assis <fidelis@...> wrote:
> 
>> I'm interested in your scripts. Could you send me? Do they work for any
>> MUA, say, Outlook 2002? I've a co-worker that uses it and we couldn't
>> find a way to get the complete message source, only the header.
> 
> Hmm.  Well you understand I'm using server-side scripts, right?  It sounds
> like you are wanting to run the filter under your local machine's MUA, which
> is not something I can help you with, especially if your local machine is
> not *nix, which is apparently so if it is running Outlook.  You could
> probably beat DOS-mode into running some modification of these scripts if
> you are running a mail server on your MUA machine, but it may not be pretty.
> 
> The scripts should be able to be modified to work on any *nix server, as
> long as your LDA is sufficiently similar.  But they were designed to be
> inserted into the qmail-vpopmail delivery pipeline.  There scripts have some
> dependencies currently on the qmail/vpopmail methodology, which has to do
> mostly with how the Delivered-To and Return-Path information are kept in
> environment variables and added to the message at the moment of actual
> delivery to the Maildir, which happens *after* the mail filtering step.
> 
> The filtering does not do routing to different folders, but depends on the
> X-CRM114 headers having been added, allowing the MUA to filter as
> spam/nonspam based on the status header.  Currently there is no provision
> for shades of spam because the current method depends on mailfilter.crm's
> added headers being utilized by MUA rule-based filtering, and the numeric
> information would be too much for MUA rules to handle.
> 
> I'll put them up on the web when I get the chance and post a link here with
> some explanation.  But if this does not sound like what you need, then I may
> do this in less of a hurry.

Ok, I put up the beginnings of a recipe here:

    http://breathhost.net/crm114-vpopmail-recipe/

People here should feel free to comment by replying here rather than
privately to the address indicated on the website.  I assume that it will be
a while before this proves to create any undo burden on this list.

As I post this, I have tried to cover all ths basics, but no doubt there is
missing information, and probably even typos.  It may take a while before
this achieves a really usable form true to the word "recipe".

Also please note that some of the content may be useful in non-vpopmail
contexts.  The really distinguishing features of this recipe are the support
for a POP-only single-account approach that permits training nonspam by
resend-to-self and training spam by resend-to-special-spam-address.  The
connection to vpopmail is superficial.

Thanks to everyone for their help in this project.

-Kurt

Kurt:

Is the scrip ready enough to get referenced from the CRM114 home 
page?

	-Bill Yerazunis

Kurt Bigler wrote:
> on 4/28/05 9:08 PM, Fidelis Assis <fidelis@...> wrote:
> 
> 
>>I'm interested in your scripts. Could you send me? Do they work for any
>>MUA, say, Outlook 2002? I've a co-worker that uses it and we couldn't
>>find a way to get the complete message source, only the header.
> 
> 
> Hmm.  Well you understand I'm using server-side scripts, right?  It sounds
> like you are wanting to run the filter under your local machine's MUA, which
> is not something I can help you with, especially if your local machine is
> not *nix, which is apparently so if it is running Outlook.

I understood you're using server-side scripts. What I misunderstood was that I thought you had, somehow, managed to make any MUA to have the resend facility. I saw later on your page that you require the email client to be capable of a message resend, what is similar to requiring it to be able to get the message source. So we seem to still have a problem here, because not all MUA can do that and not all users can or want to change their email client.

Perhaps the only generic solution (for POP3, single mailbox) is to keep a copy of the original message on the server for a reasonable period, and add an unique ID so that the user can send it back when training.
 
> I'll put them up on the web when I get the chance and post a link here with
> some explanation.  But if this does not sound like what you need, then I may
> do this in less of a hurry.

Thanks, I'll take a look. It doesn't seem to fit exactly what I'm looking for, but we can always steal the good ideas :)

-- 
Fidelis Assis

on 4/29/05 8:21 AM, wsy <wsy@...> wrote:

> 
> Kurt:
> 
> Is the scrip ready enough to get referenced from the CRM114 home
> page?

As far as the scripts go, read the bugs and caveats on the page I
referenced, and see what you think.  As far as the presentation goes, I
think it is too incomplete.  That may change a little over the next couple
of weeks, if I have the time.

-Kurt

on 4/29/05 8:28 AM, Fidelis Assis <fidelis@...> wrote:

> Kurt Bigler wrote:
>> on 4/28/05 9:08 PM, Fidelis Assis <fidelis@...> wrote:
>> 
>> 
>>> I'm interested in your scripts. Could you send me? Do they work for any
>>> MUA, say, Outlook 2002? I've a co-worker that uses it and we couldn't
>>> find a way to get the complete message source, only the header.
>> 
>> 
>> Hmm.  Well you understand I'm using server-side scripts, right?  It sounds
>> like you are wanting to run the filter under your local machine's MUA, which
>> is not something I can help you with, especially if your local machine is
>> not *nix, which is apparently so if it is running Outlook.
> 
> I understood you're using server-side scripts. What I misunderstood was that I
> thought you had, somehow, managed to make any MUA to have the resend facility.
> I saw later on your page that you require the email client to be capable of a
> message resend, what is similar to requiring it to be able to get the message
> source.

Ah, I see.  Both of the MUA's I have here have that feature.  So I wasn't
really aware of how big of a problem this was.  The only MUA I know of that
doesn't have it is my webmail.

> So we seem to still have a problem here, because not all MUA can do
> that and not all users can or want to change their email client.
> 
> Perhaps the only generic solution (for POP3, single mailbox) is to keep a copy
> of the original message on the server for a reasonable period, and add an
> unique ID so that the user can send it back when training.

If the user has to type the id then that is already too much work I'd say.
I'd prefer to let users without a resend simply use forward and have less
accurate training as a result.  My scripts don't currently recognize
forward-to-self, but they could probably be modified for that purpose.

Ideally filtering/training based on forwards might be structured a little
differently.  Perhaps the global+local classifier approach could be used and
the global classifier looks at entire messages whereas the local classifier
can be configured to ignore headers, and specifically to ignore them in such
a way that a forward-to-self would result in exactly the same text being
trained as was originally classified.

-Kurt

On Thu, Apr 28, 2005 at 10:31:30PM +0200, Milan Zamazal wrote:
> I received the following bug report on BlameTheIRS from a Debian user
> and I can reproduce the bug:
> 
>           This is a regression; I used to be able to run cssdiff on
>    files I had read access to. Even if the write access is required,
>    cssdiff should fail with a diagnostic, thather than segfaulting, when

easy... in cssdiff.c (@ ~53):

    h1 = (FEATUREBUCKET_TYPE *) crm_mmap_file (0, argv[1], hfsize);

+   if (h1 == NULL)
+       {
+         fprintf (stderr, "\n Could not open file %s\n",
+           argv[1]);
+         exit (EXIT_FAILURE);
+       };
    if (h1 == MAP_FAILED)
       {
          fprintf (stderr, "\n MMAP failed on file %s\n",
             argv[1]);
          exit (EXIT_FAILURE);
       };

and same stuff a few lines below for h2.

We should also add a MODE for crm_mmap_file(), as we don't always need
to fopen() the mmap'd file O_RDWR (like in cssdiff), so fopen() should 
succeed even on RO .css and then crm_mmap_file() would not return NULL.


-- 
 paolo
 
 GPG/PGP id:0x21426690 kfp:EDFB 0103 A8D8 4180 8AB5  D59E 9771 0F28 2142 6690
 "Indeed, it does come with warranty: it *will* fail, sometimes, somehow..."
                                                           - software vendor

From: Jeffrey Rice <jeffrice@...>


   This is perplexing.  For some reason, CRM bombs out when I try to call my 
   filter from Procmail under BlameTheIRS, but not under Bigler.  Procmail 
   gives me a program failure (-11).  Strangely, the filter runs fine if I 
   invoke if from the command line.

-11?  Negative numbers usually indicate a pipe problem; CRM114 itself 
uses the C exit() routine which wraps negative numbers to positive.

e.g. 

     crm '-{ exit /-11/ }' ; echo $?    

yields 245.

(and the default build only yields exit codes of 0, 1, 2, and 666, which
wraps to 154) 

So, look higher to find the problem... maybe memory allocation space?

The fact that it *does* run fine from command line is a clue.


   Could this be something to do with the user directory or something?  I have 
   been calling the script with the "-u /home/jeffrice" flag.  Something 
   definately changed between these versions, because it is running fine now 
   under Bigler without any other changes.

Hurm...

I didn't change anything that should have affected the -u stuff; at 
least I don't think so.

Question: this is under *NIX, right?  (because *huge* amounts of stuff
changed under the W32 subbuild)

	-Bill Yerazunis

Jeffrey Rice wrote:

<> This is perplexing.  For some reason, CRM bombs out when I try to call=
 my=20
<> filter from Procmail under BlameTheIRS, but not under Bigler.  Procmai=
l=20
<> gives me a program failure (-11).  Strangely, the filter runs fine if =
I=20
<> invoke if from the command line.

wsy wrote:

<> -11?  Negative numbers usually indicate a pipe problem; CRM114 itself=20
<> uses the C exit() routine which wraps negative numbers to positive.

Hah. Welcome to the wonderful world of procmail ...

From the procmail(1) man page:

  Program failure (nnn) of "x"
                                Program that was started  by  proc=AD
				mail returned nnn instead of EX=AD
				IT_SUCCESS (=3D0); if nnn is nega=AD
				tive, then this is the signal the
				program died on.

As Paulo said, "SEGV"

Jeffrey, has anything else changed on your system except the version of
CRM114?  No changes to the procmail scripts?

Reto
--=20
R A Lichtensteiger		rali@...

		Newsflash: You're not cute when I'm angry

From: Jeffrey Rice <jeffrice@...>

   This is perplexing.  For some reason, CRM bombs out when I try to call my 
   filter from Procmail under BlameTheIRS, but not under Bigler.  Procmail 
   gives me a program failure (-11).  Strangely, the filter runs fine if I 
   invoke if from the command line.

   Could this be something to do with the user directory or something?  I have 
   been calling the script with the "-u /home/jeffrice" flag.  Something 
   definately changed between these versions, because it is running fine now 
   under Bigler without any other changes.

Has any clue arisen on this?  

Did the problem go away, or is it still a problem?

(and if so, do you want to try to debug it?)

     -Bill Yerazunis

From: Kurt Bigler <kkb@...>

   > The question of "biggest message allowed in CRM11"4 has been long debated.
   > 
   > The best answer I have seen is that the *right* thing to do is
   > to have the MDA (such as Procmail) give CRM114 only the first
   > N bytes (where N goes from 4 kilobytes up to 1 megabyte depending
   > on how the sysadmin feels about it), and then the return code
   > is used by the MDA to dispatch the full mail.

   Ah, ok.  When I read about :decision_length: I thought this subsumed that
   consideration and I didn't realize I'd be revisiting it at another level.  I
   also *knew* that for the moment I didn't want to involve any other
   additional software.  I wanted mailfilter.crm to produce the message being
   delivered directly.

   I can revisit this design but for now I'm going to try to get my virtual
   server admin to increase my limits so I can use an acceptable -w value.  I
   want to keep my 50MB message limit working!

   Do you have information about approximately how the -w limit relates to the
   net memory requirement of crm114, for example by a factor of 20 or
   something?  If I knew it would help me clearly ask my vps provider for
   exactly what I need.

Yes, it's about a factor of 5 (once in :_dw:, once in the ISOLATED vars,
and then there are three other statically allocated buffers used
in various statements, all of that size as well.

So, to a first order, it's 5 * the W value, plus a constant on the order
of 2 megabytes.

   >> So is the usual practice to set the -w value to be larger than
   >> the largest supported message size?
   > 
   > Yes.  Or to 

   Hmm.  Can you finish that sentence?

Sorry... "Or to use it with procmail, and filter only on the first
part of the message."

   >> Also the fact that the variable :broken_program_message: was
   >> flagged seems a rather strange symptom.
   > 
   > No, that's correct.  The trap *captures* the error into the variable
   > named :broken_program_message: .  It actually worked correctly.

   I'm glad you understand it.  Seems both strange and unhelpful to me.  The
   :broken_program_message: wasn't too big, was it?  Maybe I just need to read
   up on fault and trap.  But it looks to me like the error message could be in
   some way clearer.

Yeah... I should recode the TRAP at the bottom of mailfilter.crm; that's
where the message actually came from.

      -Bill Yerazunis

Jason Barnett wrote:
[...] 
> Secondly: as my subject line indicates I'd like to use CRM as an
> extension to Mozilla Thunderbird.

Great! I'm very interested in an extension like that, specially if it makes training CRM as simple as clicking a button: "It's SPAM" or "It's not SPAM". I have been trying to find something that I could use or requiring simple adaptations but no luck. If you want someone to test it, let me know when you have something :)

-- 
Fidelis Assis

Fidelis Assis wrote:
> Jason Barnett wrote:
> [...]
> 
>> Secondly: as my subject line indicates I'd like to use CRM as an
>> extension to Mozilla Thunderbird.
> 
> 
> Great! I'm very interested in an extension like that, specially if it 
> makes training CRM as simple as clicking a button: "It's SPAM" or "It's 
> not SPAM". I have been trying to find something that I could use or 
> requiring simple adaptations but no luck. If you want someone to test 
> it, let me know when you have something :)
> 

Since I'm literally starting from scratch then it might be best if you 
give me a little input on the UI.  What I'm thinking is I'd like to 
override the default SPAM filtering to make things as transparent as 
possible.

First of all we'd train errors with the SPAM button in each mail column. 
  Either that or the user can train with the "Junk" or "Not Junk" 
options in the "Mark" context menu.

Also we'll override the default "Tools -> Junk Mail Controls" so that 
they'll hook directly into CRM.

If we go that far then we'd also need to override "Tools -> Message 
Filters".  Most likely I'd set the message filters to go into CRM's 
priolist, but if I get ambitious then I might try to add layers - so 
that the end user can make full use of priolist / whitelist / blacklist.

I'm thinking of going this route because it keeps the interface 
(virtually) identical to the end user.  The main downside with this 
approach is that the user won't be able to use the default filter, but 
I'm guessing that people that switch to CRM will do it because the 
default Bayesian filter just isn't good enough.

(in my best Johnny 5 voice) "Neeeeed innnnput!"

Jason Barnett wrote:
> Fidelis Assis wrote:
> 
>> Jason Barnett wrote:
>> [...]
[...]
> 
> I'm thinking of going this route because it keeps the interface 
> (virtually) identical to the end user.  The main downside with this 
> approach is that the user won't be able to use the default filter, but 
> I'm guessing that people that switch to CRM will do it because the 
> default Bayesian filter just isn't good enough.
> 
> (in my best Johnny 5 voice) "Neeeeed innnnput!"

I think that replacing the TB filter with CRM would be a very good project but I had something much simpler in mind to start with, Just to automate the usual CRM training process for server-side filtering:

- Start a new message;
- Remove the signature, if any;
- Insert the proper CRM command in the first line;
- Append the a copy of the message source;
- Send it to myself.

I would be more than happy with this and guess it could be done with a macro editor, but the one I found is for older TB versions:

  http://extensionroom.mozdev.org/more-info/macroeditor

Perhaps that macroeditor could be a starting point but I don't know how/where to start.

-- 
Fidelis Assis

Fidelis Assis wrote:
...
 >
 > I think that replacing the TB filter with CRM would be a very good
 > project but I had something much simpler in mind to start with, Just to
 > automate the usual CRM training process for server-side filtering:
 >

*If* server-side filtering was an option for me then I wouldn't have to 
bother with a TB extension in the first place.

- Install CRM to the server
- Create message filters in Thunderbird that react to the X-CRM-Whatever 
headers to send mail to the Junk folder, the Inbox, or whatever

Then to train the errors:
- Create a folder on the server to train SPAM
- Create a second folder to train NONSPAM
- For CRM errors I move the message to the appropriate folder
- Run a cron job on the server to LEARN through each message in these 
directories as SPAM / NONSPAM

Then you can "Right Click -> Move" the message and *poof* CRM learns 
from its errors.  Unfortunately this isn't an option for me though 
because I don't control my companies' mail server.  Bummer.

 > - Start a new message;
 > - Remove the signature, if any;
 > - Insert the proper CRM command in the first line;
 > - Append the a copy of the message source;
 > - Send it to myself.
 >
 > I would be more than happy with this and guess it could be done with a
 > macro editor, but the one I found is for older TB versions:
 >
 >  http://extensionroom.mozdev.org/more-info/macroeditor

That sure is a cool extension, but yeah that's a bummer that it doesn't 
support the newer versions of Mozilla.  I'll bet it was a lot of work to 
maintain it.

In any case: from everything that I've read (Bill's paper, some stuff 
from Paul Graham and probably a few others) it seems like having a 
client-based solution is "A Good Thing (tm)" because:

- I receive different kinds of email than other people (I especially get 
a lot for PHP, Telesuite, etc.)
- I am more / less tolerant of certain kinds of SPAM

Basically, when filtering is done at the client level it allows for more 
customized filters which (hopefully) cut out everything that is SPAM for 
me.  Of course I'm probably preaching to the choir here so I'll quit 
blabbing now.

Jason Barnett wrote:
[...]
> Then to train the errors:
> - Create a folder on the server to train SPAM
> - Create a second folder to train NONSPAM
> - For CRM errors I move the message to the appropriate folder
> - Run a cron job on the server to LEARN through each message in these 
> directories as SPAM / NONSPAM
> 
> Then you can "Right Click -> Move" the message and *poof* CRM learns 
> from its errors.  Unfortunately this isn't an option for me though 
> because I don't control my companies' mail server.  Bummer.

That's fine if you have folders on the server. Most of my servers have POP3 only, so I need a more generic solution for server-side filtering.

> In any case: from everything that I've read (Bill's paper, some stuff 
> from Paul Graham and probably a few others) it seems like having a 
> client-based solution is "A Good Thing (tm)" because:
> 
> - I receive different kinds of email than other people (I especially get 
> a lot for PHP, Telesuite, etc.)
> - I am more / less tolerant of certain kinds of SPAM
> 
> Basically, when filtering is done at the client level it allows for more 
> customized filters which (hopefully) cut out everything that is SPAM for 
> me.  Of course I'm probably preaching to the choir here so I'll quit 
> blabbing now.

I agree with you, that'd be a very good project and would reach a much broader audience. It could eventually become the default filter in TB :)

-- 
Fidelis Assis

on 4/26/05 11:06 AM, Fidelis Assis <fidelis@...> wrote:

> I think that replacing the TB filter with CRM would be a very good project but
> I had something much simpler in mind to start with, Just to automate the usual
> CRM training process for server-side filtering:
> 
> - Start a new message;
> - Remove the signature, if any;
> - Insert the proper CRM command in the first line;
> - Append the a copy of the message source;
> - Send it to myself.

I basically do all that on the server side by having the alternative of
resend-to-myself versus resend-to-special-spam-address.  The spam address
handler hard-codes the train spam function via the command-line options, so
I don't need the command embedded in the message.  The send-to-myself is
handled by the server detecting resend-to-self at any filter-enabled address
and thus distinguishing a classify from a train-nonspam (again via
command-line options).

I had similar goals to you, but chose a different approach because I want to
be able to train normal users to do this in whatever MUA they happen to use.
Thus the embedded-command approach was not acceptable.  Macros would have
made it ok, but most MUAs don't support macros.

I used maybe 2 or 3 little additional crm filters beyond mailfilter.crm to
achieve this, plus a couple more shell scripts, and managed an integration
with vpopmail that requires no special LDA.  There is the disadvantage that
messages exceeding the crm limit get silently truncated to length 0 and
delivered containing only a Return-Path and a Delivered-To line (nothing
else).

But I could share the scripts here if anyone is interested.  They work well
except for the mentioned limitation.  I would have waited a little longer
before sharing them since they are a work in progress, but if you or someone
needs them now to save some work, then just say so.

-Kurt