fetchmail-devel

[fetchmail-devel] [Bug #15103] Fetchmail Fails with ASSP (http://assp.sourceforge.net/)

[<ad...@be...>]

Bug #15103, was updated on 2009-Jan-23 07:36
Here is a current snapshot of the bug.

Project: Community Fetchmail
Category: None
Status: Open
Resolution: None
Bug Group: None
Priority: 5
Submitted by: darthasp
Assigned to : none
Summary: Fetchmail Fails with ASSP (http://assp.sourceforge.net/)

Details: Hi There,

I am trying to download messages from a catchall mailbox and forward them on to a mail server that sits been an ASSP installation.

The problem I am having is that ASSP only looks at the first 2000 bytes of the message in order to determine whether it is SPAM or not.

If the message that Fetchmail is sending is larger than this size and ASSP considers it to be SPAM - ASSP closes the connection with a 550: response before Fetchmail has finished copying the mail.

Fetchmail then errors with Query status=6 (IOERR)

It does not delete the message or mark it as seen.

next time fetchmail logs in try's to send the message again with the same result.

I can increase the amount of bytes that ASSP looks at and while this solves the issue temporary it is not a permanent solution.  ASSP was not designed to look that the whole message.
When I increase the amount of bytes ASSP looks at the system resources go through the roof.

Is it possible that Fetchmail could somehow recognise the 550 response from ASSP half way through message sending.

I have enclosed log file for you to look at

Hope you can help 

Regards

Phil 

fetchmail: SMTP> MAIL FROM:<externaluser> SIZE=861742
fetchmail: SMTP< 250 2.1.0 externaluser....Sender OK
fetchmail: SMTP> RCPT TO:<internaluser>
fetchmail: SMTP< 250 2.1.5 internaluser
fetchmail: SMTP> DATA
fetchmail: SMTP< 354 Start mail input; end with <CRLF>.<CRLF>
fetchmail: error writing message text
fetchmail: POP3> QUIT
fetchmail: POP3< O07etM6Coi+kFL/DyxZouBPCFDIAEiwZWHF6pjPuZt2vAR+/F0pCL8Re3Qva/m6kfYL7EsD+Vd52
fetchmail: MDA error while fetching from shrelay@do...@ma...
fetchmail: 6.3.9 querying server1 (protocol POP3) at Fri Jan 23 07:26:46 2009: poll completed
fetchmail: Query status=6 (IOERR)
fetchmail: normal termination, status 6
fetchmail: 1 message for shrelay@domain at mail.domain (861742 octets).
fetchmail: reading message shrelay@do...@ma...:1 of 1 (861742 octets) (log message incomplete)fetchmail: error writing message text
fetchmail: MDA error while fetching from shrelay@do...@ma...
fetchmail: Query status=6 (IOERR)

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=15103&group_id=1824

Re: [fetchmail-devel] [Bug #15103] Fetchmail Fails with ASSP (http://assp.sourceforge.net/)

[Frederic M. <fre...@wo...>]

ad...@be... a écrit :
> Bug #15103, was updated on 2009-Jan-23 07:36
> Here is a current snapshot of the bug.
>
> Project: Community Fetchmail
> Category: None
> Status: Open
> Resolution: None
> Bug Group: None
> Priority: 5
> Submitted by: darthasp
> Assigned to : none
> Summary: Fetchmail Fails with ASSP (http://assp.sourceforge.net/)
>
> Details: Hi There,
>
> I am trying to download messages from a catchall mailbox and forward them on to a mail server that sits been an ASSP installation.
>
> The problem I am having is that ASSP only looks at the first 2000 bytes of the message in order to determine whether it is SPAM or not.
>
> If the message that Fetchmail is sending is larger than this size and ASSP considers it to be SPAM - ASSP closes the connection with a 550: response before Fetchmail has finished copying the mail.
>
> Fetchmail then errors with Query status=6 (IOERR)
>
> It does not delete the message or mark it as seen.
>
> next time fetchmail logs in try's to send the message again with the same result.
>
> I can increase the amount of bytes that ASSP looks at and while this solves the issue temporary it is not a permanent solution.  ASSP was not designed to look that the whole message.
> When I increase the amount of bytes ASSP looks at the system resources go through the roof.
>
> Is it possible that Fetchmail could somehow recognise the 550 response from ASSP half way through message sending.
>
> I have enclosed log file for you to look at
>
> Hope you can help 
>
> Regards
>
> Phil 
>
> fetchmail: SMTP> MAIL FROM:<externaluser> SIZE=861742
> fetchmail: SMTP< 250 2.1.0 externaluser....Sender OK
> fetchmail: SMTP> RCPT TO:<internaluser>
> fetchmail: SMTP< 250 2.1.5 internaluser
> fetchmail: SMTP> DATA
> fetchmail: SMTP< 354 Start mail input; end with <CRLF>.<CRLF>
> fetchmail: error writing message text
> fetchmail: POP3> QUIT
> fetchmail: POP3< O07etM6Coi+kFL/DyxZouBPCFDIAEiwZWHF6pjPuZt2vAR+/F0pCL8Re3Qva/m6kfYL7EsD+Vd52
> fetchmail: MDA error while fetching from shrelay@do...@ma...
> fetchmail: 6.3.9 querying server1 (protocol POP3) at Fri Jan 23 07:26:46 2009: poll completed
> fetchmail: Query status=6 (IOERR)
> fetchmail: normal termination, status 6
> fetchmail: 1 message for shrelay@domain at mail.domain (861742 octets).
> fetchmail: reading message shrelay@do...@ma...:1 of 1 (861742 octets) (log message incomplete)fetchmail: error writing message text
> fetchmail: MDA error while fetching from shrelay@do...@ma...
> fetchmail: Query status=6 (IOERR)
>   

I solved this issue by running ASSP in test mode (check option "Bayesian 
Test Mode") and prefixing the subject with something like [SPAM] (in 
option "Prepend Spam Subject"). It means ASSP won't stop mails seen as 
bayesian spam but will tag the subject. The mails can then be filtered 
out automatically by the final recipient.

Since you are downloading your mails with fetchmail, you can't simply 
reject them because it is too late to do that. The purpose in rejecting 
them is to leave the bad mails on the hand of the one that accepted them 
in the first place. But in your case, you already have accepted them and 
you have to deal with them...

If your server is also accepting external mails through direct 
connections via port 25 and you want ASSP to reject those mails, then 
you have to open some kind of backdoor on your mail server to let 
fetchmail inject the mails it is reading. Be sure to properly firewall 
that port to prevent anybody but fetchmail to inject mails.

By the way, I don't advise to let ASSP delete or reject blindly bayesian 
spams because, as you noticed, it doesn't take the whole mail into 
account and, in general, it does a poor job at filtering spams. It tends 
to have a high rate of false positives and you will loose good e-mails 
if you don't keep an eye on what it does.

Frederic

Re: [fetchmail-devel] [Bug #15103] Fetchmail Fails with ASSP (http://assp.sourceforge.net/)

[Matthias A. <mat...@gm...>]

Frederic Marchal schrieb:

[in response to BerliOS Bug #15103]

> I solved this issue by running ASSP in test mode (check option "Bayesian 
> Test Mode") and prefixing the subject with something like [SPAM] (in 
> option "Prepend Spam Subject"). It means ASSP won't stop mails seen as 
> bayesian spam but will tag the subject. The mails can then be filtered 
> out automatically by the final recipient.

This, and my being bogofilter's co-maintainer (bogofilter also does
Bayesian filtering), prompted me to have a glimpse at ASSP, and I must say
that the promises it makes are way too bold. ASSP's self-advertising as the
best tool aside, the assertion that Bayes filter were to intelligently
decide is just crap. Bayes filters are making statistical decisions based
on past training. "Maintenance free" doesn't work for them.

> Since you are downloading your mails with fetchmail, you can't simply 
> reject them because it is too late to do that. The purpose in rejecting 
> them is to leave the bad mails on the hand of the one that accepted them 
> in the first place. But in your case, you already have accepted them and 
> you have to deal with them...

True enough. I had, before reading your message, already closed the bug
report as invalid as ASSP violates the SMTP protocol. You've just handed me
another good reason not to change fetchmail, because the whole setup is
just twisted.

Beyond that, the assertion of the original were that ASSP were inefficient
(the wording was like 'resource use goes through the roof' or similar) if
it were to look at the whole message -- and that's a particular drawback or
implementation issue of ASSP that fetchmail certainly isn't meant to fix,
and I'm definitely not meaning to.

> If your server is also accepting external mails through direct 
> connections via port 25 and you want ASSP to reject those mails, then 
> you have to open some kind of backdoor on your mail server to let 
> fetchmail inject the mails it is reading. Be sure to properly firewall 
> that port to prevent anybody but fetchmail to inject mails.

Well, while a firewall certainly isn't a bad idea, it's not required
either. Most SMTP servers can be made to listen on particular addresses,
which for most of them is equivalent to binding them to particular
interfaces. Binding the SMTP server to interfaces that are only locally
accessible, such as the loopback interface or its address 127.0.0.1, is
sufficient to prevent access from the outside. It's doable with Postfix and
Exim. I don't know about Sendmail or Courier. It's also possible with qmail
running off tcpsvd or tcpserver, but I'd recommend against using qmail for
the delayed-bounce behaviour and dozens of other reasons beyond this
discussion.

> By the way, I don't advise to let ASSP delete or reject blindly bayesian 
> spams because, as you noticed, it doesn't take the whole mail into 
> account and, in general, it does a poor job at filtering spams. It tends 
> to have a high rate of false positives and you will loose good e-mails 
> if you don't keep an eye on what it does.

But then it seems to me that if you're using fetchmail or getmail, that
integrating bogofilter, spamprobe or qsf in the mail system is more
efficient than ASSP if it's prone to high resource use.

At least bogofilter and SpamProbe are much faster than SpamAssassin's bayes
mode. :-)

HTH

-- 
Matthias Andree

Re: [fetchmail-devel] [Bug #15103] Fetchmail Fails with ASSP (http://assp.sourceforge.net/)

[Frederic M. <fre...@wo...>]

Matthias Andree a écrit :
> Frederic Marchal schrieb:
>
> [in response to BerliOS Bug #15103]
>
>   
>> I solved this issue by running ASSP in test mode (check option "Bayesian 
>> Test Mode") and prefixing the subject with something like [SPAM] (in 
>> option "Prepend Spam Subject"). It means ASSP won't stop mails seen as 
>> bayesian spam but will tag the subject. The mails can then be filtered 
>> out automatically by the final recipient.
>>     
>
> This, and my being bogofilter's co-maintainer (bogofilter also does
> Bayesian filtering), prompted me to have a glimpse at ASSP, and I must say
> that the promises it makes are way too bold. ASSP's self-advertising as the
> best tool aside, the assertion that Bayes filter were to intelligently
> decide is just crap. Bayes filters are making statistical decisions based
> on past training. "Maintenance free" doesn't work for them.
>   
ASSP's implementation of Paul Graham's algorithm is the most inefficient 
I know but it does more than bayesian statistics and that compensate for 
its incapacity at properly classify e-mails. In my case, I have a bunch 
of spamtraps addresses and ASSP learns from the e-mails it nets. Since 
spammers are sending tons of e-mails to every addresses they know, we 
can filter them out by simple comparison between the mails received by 
the spamtraps and the mails received by regular users. So, it does its 
own training even though a Bayes filter is not adequate considering how 
it performs in ASSP.

Beside, ASSP sits on the SMTP port. Therefore, it sees the mails sent by 
local users too and can whitelist the responses, preventing them from 
being filtered out and learning hams in the process. It can also learn 
from SMTP violations the spammers usually do and blacklist the offending 
senders very early and for every user of the server.

I don't believe ASSP is a great piece of software but it is the only one 
I know of that can do that and it does require very little or no action 
from the users it protect (especially no command line to run to learn 
more spam/ham).

>> By the way, I don't advise to let ASSP delete or reject blindly bayesian 
>> spams because, as you noticed, it doesn't take the whole mail into 
>> account and, in general, it does a poor job at filtering spams. It tends 
>> to have a high rate of false positives and you will loose good e-mails 
>> if you don't keep an eye on what it does.
>>     
>
> But then it seems to me that if you're using fetchmail or getmail, that
> integrating bogofilter, spamprobe or qsf in the mail system is more
> efficient than ASSP if it's prone to high resource use.
>
> At least bogofilter and SpamProbe are much faster than SpamAssassin's bayes
> mode. :-)
>   
Thanks for the advertisement :-) but my users are all windows 
(mal-)formatted and it is impossible to get them to run a linux command 
to classify their e-mails...

Frederic