fetchmail-users

[fetchmail-users] fetchmail gets stuck because of malformed spam

[Karel K. <cl...@tw...>]

I got some malformed spam which causes fetchmail to stop on this spam
and I am unable to download 162 waiting messages behind this spam.

This seems to be some kind of fetchmail DoS vulnerability - if someone
sends this mail to all fetchmail users in the world, they will be pretty
pissed off to have to log in to remote machine and erase the e-mail by
hand. And if he sends every hour, they will have to change to a
different program than fetchmail ;-)

clock@kestrel:~$ fetchmail
162 messages for clock at twin.jikos.cz.
reading message cl...@tw...:1 of 162 (765 header octets) fetchmail: SMTP error: 501 <agrode@%%DOMAIN%>: domain missing or malformed
gethostbyname failed for kestrel

Operating system gentoo linux
GCC 3.3.6
don't know how to get IMAP server greeting line
MDA: ESMTP Exim 4.50
 
This is fetchmail release 6.2.5.2+RPA+NTLM+SDPS+SSL+INET6+NLS
Fallback MDA: (none)
Linux kestrel 2.6.12-gentoo-r10 #2 Tue Oct 4 10:27:59 CEST 2005 i686 Intel(R) Pentium(R) M processor 1.50GHz GenuineIntel GNU/Linux
Taking options from command line and /home/clock/.fetchmailrc
Idfile is /home/clock/.fetchids
Fetchmail will forward misaddressed multidrop messages to clock.
Options for retrieving from cl...@tw...:
  True name of server is twin.jikos.cz.
  Protocol is IMAP.
  All available authentication methods will be tried.
  SSL encrypted sessions enabled.
  Server nonresponse timeout is 300 seconds (default).
  Default mailbox selected.
  Only new messages will be retrieved (--all off).
  Fetched messages will not be kept on the server (--keep off).
  Old messages will not be flushed before message retrieval (--flush off).
  Rewrite of server-local addresses is enabled (--norewrite off).
  Carriage-return stripping is disabled (stripcr off).
  Carriage-return forcing is disabled (forcecr off).
  Interpretation of Content-Transfer-Encoding is enabled (pass8bits off).
  MIME decoding is disabled (mimedecode off).
  Idle after poll is disabled (idle off).
  Nonempty Status lines will be kept (dropstatus off)
  Delivered-To lines will be kept (dropdelivered off)
  Fetch message size limit is 100 (--fetchsizelimit 100).
  Do binary search of UIDs during 9 out of 10 polls (--fastuidl 10).
  Messages will be SMTP-forwarded to: localhost (default)
  Single-drop mode: 1 local name(s) recognized.
  No UIDs saved from this host.
Options for retrieving from cl...@at...:
  True name of server is atrey.karlin.mff.cuni.cz.
  Protocol is IMAP.
  All available authentication methods will be tried.
  SSL encrypted sessions enabled.
  Server nonresponse timeout is 300 seconds (default).
  Default mailbox selected.
  Only new messages will be retrieved (--all off).
  Fetched messages will not be kept on the server (--keep off).
  Old messages will not be flushed before message retrieval (--flush off).
  Rewrite of server-local addresses is enabled (--norewrite off).
  Carriage-return stripping is disabled (stripcr off).
  Carriage-return forcing is disabled (forcecr off).
  Interpretation of Content-Transfer-Encoding is enabled (pass8bits off).
  MIME decoding is disabled (mimedecode off).
  Idle after poll is disabled (idle off).
  Nonempty Status lines will be kept (dropstatus off)
  Delivered-To lines will be kept (dropdelivered off)
  Fetch message size limit is 100 (--fetchsizelimit 100).
  Do binary search of UIDs during 9 out of 10 polls (--fastuidl 10).
  Messages will be SMTP-forwarded to: localhost (default)
  Single-drop mode: 1 local name(s) recognized.
  No UIDs saved from this host.
Options for retrieving from di...@po...:
  True name of server is pop.centrum.cz.
  Protocol is POP3.
  All available authentication methods will be tried.
  Server nonresponse timeout is 300 seconds (default).
  Default mailbox selected.
  Only new messages will be retrieved (--all off).
  Fetched messages will not be kept on the server (--keep off).
  Old messages will not be flushed before message retrieval (--flush off).
  Rewrite of server-local addresses is enabled (--norewrite off).
  Carriage-return stripping is disabled (stripcr off).
  Carriage-return forcing is disabled (forcecr off).
  Interpretation of Content-Transfer-Encoding is enabled (pass8bits off).
  MIME decoding is disabled (mimedecode off).
  Idle after poll is disabled (idle off).
  Nonempty Status lines will be kept (dropstatus off)
  Delivered-To lines will be kept (dropdelivered off)
  Fetch message size limit is 100 (--fetchsizelimit 100).
  Do binary search of UIDs during 9 out of 10 polls (--fastuidl 10).
  Messages will be SMTP-forwarded to: localhost (default)
  Single-drop mode: 1 local name(s) recognized.
  No UIDs saved from this host.
Options for retrieving from yc...@po...:
  True name of server is pop.centrum.cz.
  Protocol is POP3.
  All available authentication methods will be tried.
  Server nonresponse timeout is 300 seconds (default).
  Default mailbox selected.
  Only new messages will be retrieved (--all off).
  Fetched messages will not be kept on the server (--keep off).
  Old messages will not be flushed before message retrieval (--flush off).
  Rewrite of server-local addresses is enabled (--norewrite off).
  Carriage-return stripping is disabled (stripcr off).
  Carriage-return forcing is disabled (forcecr off).
  Interpretation of Content-Transfer-Encoding is enabled (pass8bits off).
  MIME decoding is disabled (mimedecode off).
  Idle after poll is disabled (idle off).
  Nonempty Status lines will be kept (dropstatus off)
  Delivered-To lines will be kept (dropdelivered off)
  Fetch message size limit is 100 (--fetchsizelimit 100).
  Do binary search of UIDs during 9 out of 10 polls (--fastuidl 10).
  Messages will be SMTP-forwarded to: localhost (default)
  Single-drop mode: 1 local name(s) recognized.
  No UIDs saved from this host.

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[Rob M. <rob...@gm...>]

On 26/10/05, Karel Kulhavy <cl...@tw...> wrote:
> I got some malformed spam which causes fetchmail to stop on this spam
> and I am unable to download 162 waiting messages behind this spam.
>
> This seems to be some kind of fetchmail DoS vulnerability - if someone
> sends this mail to all fetchmail users in the world, they will be pretty
> pissed off to have to log in to remote machine and erase the e-mail by
> hand. And if he sends every hour, they will have to change to a
> different program than fetchmail ;-)

I've seen a similar problem before, but usually fetchmail will skip
the offending mail and carry on.  One option may be to set 501 as one
of the anti-spam codes.

> don't know how to get IMAP server greeting line

telnet IMAPHOST 143

More useful than the config dump would be the contents of .fetchmailrc
and output of "fetchmail -v -v" for the problem email (along with the
matching log entries from your MTA).

--
                 Please keep list traffic on the list.
Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[Jakob H. <jh...@pl...>]

Karel Kulhavy wrote:

> reading message cl...@tw...:1 of 162 (765 header octets)
> fetchmail: SMTP error: 501 <agrode@%%DOMAIN%>: domain missing or malformed
> gethostbyname failed for kestrel

Fetchmail should go on retrieving the next message after a 501 from the
smtp server, but it seems to consider the failed gethostbyname() a fatal
error.
Looks more like a misconfiguration on your side. "kestrel" is your
hostname, right?

I personally would always recommend using fetchmail with the mda option
(for the usual pop/imap retrieval), so you won't have all these smtp
problems.
At least as long as you are not using multidrop, but that is not
recommended anyway.

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[Jakob H. <jh...@pl...>]

Karel Kulhavy wrote:


Please keep list traffic on the list.

>> I personally would always recommend using fetchmail with the mda option
> What is mda option?


It means that your mail is not delivered over smtp but to the local
delivery agent, which (usually) simply drops it in your inbox.

I use:

defaults
  mda "/usr/sbin/sendmail -i %T"
  is "jh" here

poll imap.web.de proto imap user x pass y fetchall ssl


For more information: man fetchmail

> What is multidrop?

The opposite of single-drop, which you are using.
For more information: man fetchmail

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[Thomas W. <to...@to...>]

> > or like this:
> >   mda "formail -s procmail -p"

> formail is unnecessary here, I think, as fetchmail calls the mda for every
> single message seperately.
But fetchmail doesn't generate a "From " line which is needed as a 
message separator in the mailbox. That's why formail is necessary.
Please check; maybe the "/usr/bin/procmail" example under --mda in 
the manual page is misleading here and should be changed as well.
Without formail, I couldn't produce a mailbox that is readable with "mail".

> > access to my other mails, I did not care to retrieve any of those
> > (using the webmail interface) which I should have sent here for bug
> > analysis, sorry.

> The output of fetchmail -v -v would have been a start. This could have
> also been an issue of the pop/imap server.
As I noted before (in a mail that unfortunately didn't make it on the 
list yet as I used the wrong sender address :( ) :
Whatever the issue, error handling should be reviewed and revised to 
never produce this kind of DoS situation, independent of actual error 
analysis.

Best regards,
Thomas

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[Jakob H. <jh...@pl...>]

Thomas Wolff wrote:
>> > or like this:
>> >   mda "formail -s procmail -p"
>> formail is unnecessary here, I think, as fetchmail calls the mda for
>> every single message seperately.
> But fetchmail doesn't generate a "From " line which is needed as a
> message separator in the mailbox. That's why formail is necessary.

Shouldn't that be done by procmail itself? Programs running an MDA
shouldn't care about the local mailbox format. I may be wrong about this,
it's long ago that I used procmail and mbox.

> Whatever the issue, error handling should be reviewed and revised to
> never produce this kind of DoS situation, independent of actual error
> analysis.

If it's a server issue, there's not much fetchmail can do about it. If
it's a fetchmail flaw, you are right of course.

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[<Tho...@si...>]

Jakob Hirsch wrote:

> I personally would always recommend using fetchmail with the mda option
> (for the usual pop/imap retrieval), so you won't have all these smtp
> problems.
> ...
> > What is mda option?
> It means that your mail is not delivered over smtp but to the local
> delivery agent, which (usually) simply drops it in your inbox.
> 
> I use:
> 
> defaults
>   mda "/usr/sbin/sendmail -i %T"
> ...

I have a personal setup that does not (cannot) rely on a working local 
mail environment (the machine is not normally intended for handling mails).
In this case, it might rather look like this:
  mda "formail >> ${MAIL-$HOME/Post/Inbox}"
or like this:
  mda "formail -s procmail -p"

So for me, there is no smtp involved in mail retrieval.
But I've actually had similar problems a while ago; fetchmail couldn't 
get past certain spam mails. I had to use a webmail interface to 
delete them to get any further. Unfortunately, as I needed quick 
access to my other mails, I did not care to retrieve any of those 
(using the webmail interface) which I should have sent here for bug 
analysis, sorry.

So I don't know either if the malformed parts raised the same problem 
as it occurred now. But in any case, I think fetchmail error handling 
should be carefully reviewed for any possible case of aborting the 
retrieval in order to avoid this DoS situation in the future.

Kind regards,
Thomas Wolff

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[Jakob H. <jh...@pl...>]

Tho...@si... wrote:

> In this case, it might rather look like this:
>   mda "formail >> ${MAIL-$HOME/Post/Inbox}"

This is a little dangerous, as there is no locking involved

> or like this:
>   mda "formail -s procmail -p"

formail is unnecessary here, I think, as fetchmail calls the mda for every
single message seperately.

> access to my other mails, I did not care to retrieve any of those
> (using the webmail interface) which I should have sent here for bug
> analysis, sorry.

The output of fetchmail -v -v would have been a start. This could have
also been an issue of the pop/imap server.

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[<Tho...@si...>]

I wrote:

> But I've actually had similar problems a while ago; fetchmail couldn't 
> get past certain spam mails. I had to use a webmail interface to 
> delete them to get any further. Unfortunately, as I needed quick 
> access to my other mails, I did not care to retrieve any of those 
> (using the webmail interface) which I should have sent here for bug 
> analysis, sorry.
As an additional information I just remembered that I had to use 
the webmail interface because Mozilla Thunderbird was not able to 
retrieve those mails, either, and got stuck the same way, 
when this problem occurred here a few months ago.

Kind regards,
Thomas Wolff

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[Jakob H. <jh...@pl...>]

Tho...@si... wrote:

> As an additional information I just remembered that I had to use
> the webmail interface because Mozilla Thunderbird was not able to
> retrieve those mails, either, and got stuck the same way,
> when this problem occurred here a few months ago.

So it is very likely that it was an pop/imap server issue, and fetchmail
cannot do much about it.
I have been running fetchmail for several years now and never had a single
case of it hanging because of some malformed message. That doesn't mean
there weren't any bugs, of course.

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[Thomas W. <to...@to...>]

I wrote:
> But I've actually had similar problems a while ago; fetchmail couldn't 
> get past certain spam mails. I had to use a webmail interface to 
> delete them to get any further. Unfortunately, as I needed quick 
> access to my other mails, I did not care to retrieve any of those 
> (using the webmail interface) which I should have sent here for bug 
> analysis, sorry.

mbarsalou (having a similar problem) wrote:
> Subject: [fetchmail-users] fetchmail stopping on spam
> ...
> This is purely anecdotal, however.


This problem occurred again yesterday and I have additional evidence now:

reading message pt8...@po...:18 of 43 (1873 octets) . flushed
reading message pt8...@po...:19 of 43 (1293 octets) fetchmail: incorrect header line found while scanning headers
fetchmail: message delimiter found while scanning headers
 flushed
fetchmail: client/server protocol error while fetching from pop.kundenserver.de
fetchmail: Query status=4 (PROTOCOL)

This was with fetchmail 6.2.3 and it stopped fetching mails at this point. 
There are two additional observations to note:
* Although fetchmail said "flushed" for all the previous mails (and 
  even the affected one, actually), it did not delete them from the 
  server.
* It left the affected message in the mailbox though (or at least 
  some initial part of the headers) (domain name changed manually):

Return-Path: <OCH...@no...>
Delivery-Date: Tue, 08 Nov 2005 17:25:36 +0100
Received: from pop.kundenserver.de [212.227.15.181]
	by localhost with POP3 (fetchmail-6.2.3)
	for root@localhost (single-drop); Thu, 10 Nov 2005 00:13:00 +0100 (CET)
Received: from [83.242.73.87] (helo=host-n2-73-87.telpol.net.pl)
	by mx.kundenserver.de (node=mxeu3) with ESMTP (Nemesis),
	id 0MKqIe-1EZWHr1aO4-0001NV for th...@xx...; Tue, 08 Nov 2005 17:25:31 +0100
Received: from peternixon.net [215.199.192.160


Then I switched to fetchmail 6.2.9rc7 and it fetched all mails as it 
should do, except the affected message, on which it still reported:
fetchmail: message delimiter found while scanning headers

What still concerns me, however, is that it discarded that message 
completely and wrote no trace of it to the mailbox, so messages with 
that kind of problem (whatever it may be) can not be checked or 
analysed anymore with fetchmail.


Kind regards,
Thomas Wolff

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[Rob M. <rob...@gm...>]

On 10/11/05, Thomas Wolff <to...@to...> wrote:
>
> Then I switched to fetchmail 6.2.9rc7 and it fetched all mails as it
> should do, except the affected message, on which it still reported:
> fetchmail: message delimiter found while scanning headers
>
> What still concerns me, however, is that it discarded that message
> completely and wrote no trace of it to the mailbox, so messages with
> that kind of problem (whatever it may be) can not be checked or
> analysed anymore with fetchmail.

The message is illegally formatted (in this case, has no blank line
between headers and body) and should never have been accepted by 1&1's
mail server (though from personal experience I'm not surprised). 
Fetchmail is doing the correct thing and discarding this email.

Previously it has been suggested that one option might be to wrap the
entire of the illegal mail and forward it to the postmaster.  Nobody
however was willing to put the effort into a patch to do this, and
frankly given that every occurence to date has been identified as spam
I'm not surprised.

I am however happy to see that the forthcoming 6.3.0 release will
handle this more gracefully than previous versions :)

--
                 Please keep list traffic on the list.
Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[Karel K. <cl...@tw...>]

On Thu, Oct 27, 2005 at 12:07:35PM +0200, Tho...@si... wrote:
> Jakob Hirsch wrote:

[...]

> So I don't know either if the malformed parts raised the same problem 
> as it occurred now. But in any case, I think fetchmail error handling 
> should be carefully reviewed for any possible case of aborting the 
> retrieval in order to avoid this DoS situation in the future.

Hehe one could send a malformed e-mail to this mailing list so all
people would get pissed - or write a bot which would send every hour
- that would make the bug to be solved promptly ;-)

CL<
> 
> Kind regards,
> Thomas Wolff

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[Rob M. <rob...@gm...>]

On 27/10/05, Karel Kulhavy <cl...@tw...> wrote:

> Hehe one could send a malformed e-mail to this mailing list so all
> people would get pissed - or write a bot which would send every hour
> - that would make the bug to be solved promptly ;-)

However, you'd have to be subscribed to do that, and if somebody were
stupid enough to do it then the following *would* happen promptly:

a) They'd be reported to their ISP
b) Their address would be unsubscribed (and I'd keep a note of the
source IP to ban them in future)

Worth noting too that the situation reported doesn't seem to by
typical.  Certainly on the occasions I've hit similar problems with
malformed email rejected by my SMTP server fetchmail happily carries
on fetching the other mail.  This suggests that the problem is a local
config issue.  Sadly without the OP providing the (much requested)
output of "fetchmail -v -v" it's impossible to do anything but
speculate.

--
                 Please keep list traffic on the list.
Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[Jakob H. <jh...@pl...>]

Karel Kulhavy wrote:

once again:
>> Please keep list traffic on the list.


> But I have mda option.

No, you don't, as your output of fetchmail -V says:

Messages will be SMTP-forwarded to: localhost (default)

That means that your MTA is used, which is fetchmail's default.
A MDA would procmail, maildrop etc., or Exim itself, but not via smtp.

With "mda" it would say something like

Messages will be delivered with "/usr/sbin/sendmail -i %T"

[fetchmail-users] Re: fetchmail gets stuck because of malformed spam

[Sunil S. <sh...@bo...>]

Quoting from Karel Kulhavy's mail on Wed, Oct 26, 2005 at 10:59:30AM +0200:
> I got some malformed spam which causes fetchmail to stop on this spam
> and I am unable to download 162 waiting messages behind this spam.
> 
> This seems to be some kind of fetchmail DoS vulnerability - if someone
> sends this mail to all fetchmail users in the world, they will be pretty
> pissed off to have to log in to remote machine and erase the e-mail by
> hand. And if he sends every hour, they will have to change to a
> different program than fetchmail ;-)

Could you send the output of "fetchmail -v" when the mail gets stuck?

My guess is that the mail gets stuck because fetchmail is unable to
send a bounce message.

-- 
Sunil Shetye.

Re: [fetchmail-users] fetchmail gets stuck because of malformed spam

[Jakob H. <jh...@pl...>]

Karel Kulhavy wrote:

>> That means that your MTA is used, which is fetchmail's default.
>> A MDA would procmail, maildrop etc., or Exim itself, but not via smtp.
> If I don't run Exim, fetchmail doesn't deliver. If I run Exim,

Which proves that you are _not_ using the MDA option and that you have no
clue about the difference MTA/MDA. Use your favorite search engine to find
one of the trillion websites explaining it.


PS: And stop mailing me personally, this is list traffic.