Menu
▾
▴
fetchmail-users
|
From: Antonio C. <pir...@ya...> - 2007-04-15 11:29:06
|
Hello,
I think fetchmail loses messages when a malformed header is present and
POP3 is used.
Put it shortly, if a message begins with (i.e., the first line of the
headers is) a line with only a number (e.g.
"000000000000000036333551"), the message is retrieved from the server,
deleted, but not forwarded to the local mail server (nor bounced back).
I don't know how did this line got there (the mail server which I am
downloading the mail from is not administered by me), but fetchmail
shouldn't lose the message anyway.
Here is a more detailed description of the problem, and some additional
information.
I run a Linux Fedora Core 6, with kernel 2.6.20.4.
I have the latest version of fetchmail (6.3.8). I built it from the
sources downloaded on Berlios, using the spec file from the fetchmail
6.3.7 package of Fedora Core.
The same problem happened with all the other versions I tried:
- fetchmail 6.3.6-2 (the rpm from the Fedora Core 6 updates)
- fetchmail 6.3.7-1 (the rpm from Fedora Core 6.92).
The message which gets deleted is a normal message, but, before the
first line there is a line with only a number (=> malformed header
line).
Here is a log from a telnet to port 110 of the POP3 server, followed by
a log of the subsequent fetchmail call in which I try to download the
same message.
Note: in this and the following example, I only masked:
- password
- server names and IP addresses
- email addresses of other people
- subject of the e-mail
- unique mail IDs and key fingerprint
- company name
[guido@localhost ~]$ telnet mailserver.doma.in 110
Trying 12.34.56.78...
Connected to mailserver.doma.in.
Escape character is '^]'.
+OK mailserver.doma.in Merak 8.9.1 POP3 Sun, 15 Apr 2007 03:05:09 +0200
<200...@ma...>
USER guido
+OK guido
PASS mypassword
+OK 330 messages (4548411) octets
TOP 1 1
+OK
000000000000000036333551
Received: from server.doma.in ([23.45.67.89])
by mailserver.doma.in (Merak 8.2.4) with ESMTP id DWH39514
for <gu...@ma...>; Fri, 19 Jan 2007 06:42:34 +0100
Received: from localhost (localhost [127.0.0.1])
by server.doma.in (Postfix) with ESMTP id 7E431219155;
Fri, 19 Jan 2007 06:42:51 +0100 (CET)
Received: from server.doma.in ([127.0.0.1])
by localhost (server.doma.in [127.0.0.1]) (amavisd-new, port 10025)
with LMTP id 25102-01-42; Fri, 19 Jan 2007 06:42:48 +0100 (CET)
Received: from whatever.site (whatever.domain.name [34.56.78.90])
by server.doma.in (Postfix) with SMTP id 6731B41793E;
Fri, 19 Jan 2007 06:42:39 +0100 (CET)
Received: from whatelse ([12.45.78.01])
by another.site (8.12.0/8.12.0) with SMTP id 4115586132701
for <gu...@do...>; Fri, 19 Jan 2007 06:42:56 +0100
Message-ID: <001801b73b95$1d34a0d3$006a1e14@whatelse>
From: Somebody <an...@in...dr>
To: gu...@do...
Subject: subject
Date: Fri, 19 Jan 2007 06:42:56 +0100
MIME-Version: 1.0
Content-Type: multipart/related;
type="multipart/alternative";
boundary="----=_NextPart_000_0013_01C73B95.0D34F9D0"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.1081
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3000
X-Virus-Scanned: amavisd-new at doma.in
X-Spam-Status: No, score=2.227 tagged_above=-10 required=4
tests=[BAYES_60=1,
EXTRA_MPART_TYPE=1.091, FORGED_RCVD_HELO=0.135, HTML_MESSAGE=0.001]
X-Spam-Score: 2.227
X-Spam-Level: **
This is a multi-part message in MIME format.
.
QUIT
+OK mailserver.doma.in closing connection
Connection closed by foreign host.
Here is the corresponding line from .fetchmailrc:
poll mailserver.doma.in proto POP3 auth password user guido with pass
mypassword is guido here fetchall forcecr fetchlimit 1
Note: if I change protocol (from POP3 to IMAP) I get a similar result.
Obviously the fetchmail log is different, but the message is deleted
from server and neither forwarded nor bounced.
Here is what I get when I run fetchmail:
[guido@localhost ~]$ fetchmail -v -v -v mailserver.doma.in
fetchmail: 6.3.8 querying mailserver.doma.in (protocol POP3) at Sun Apr
15 03:23:29 2007: poll started
Trying to connect to 12.34.56.78/110...connected.
fetchmail: POP3< +OK mailserver.doma.in Merak 8.9.1 POP3 Sun, 15 Apr
2007 03:22:48 +0200 <200...@ma...>
fetchmail: POP3> CAPA
fetchmail: POP3< +OK Capability list follows
fetchmail: POP3< TOP
fetchmail: POP3< USER
fetchmail: POP3< APOP
fetchmail: POP3< EXPIRE NEVER
fetchmail: POP3< UIDL
fetchmail: POP3< SASL CRAM-MD5 PLAIN LOGIN DIGEST-MD5
fetchmail: POP3< STLS
fetchmail: POP3< .
fetchmail: POP3> STLS
fetchmail: POP3< +OK Ready to start TLS
fetchmail: Issuer Organization: Foo
fetchmail: Issuer CommonName: mailserver.doma.in
fetchmail: Server CommonName: mailserver.doma.in
fetchmail: mailserver.doma.in key fingerprint:
AA:83:C0:04:2D:0B:FC:1E:65:1E:09:00:A0:2C:7D:77
fetchmail: Server certificate verification error: self signed
certificate
fetchmail: POP3> CAPA
fetchmail: POP3< +OK Capability list follows
fetchmail: POP3< TOP
fetchmail: POP3< USER
fetchmail: POP3< APOP
fetchmail: POP3< EXPIRE NEVER
fetchmail: POP3< UIDL
fetchmail: POP3< SASL CRAM-MD5 PLAIN LOGIN DIGEST-MD5
fetchmail: POP3< .
fetchmail: mailserver.doma.in: upgrade to TLS succeeded.
fetchmail: POP3> USER guido
fetchmail: POP3< +OK guido
fetchmail: POP3> PASS *
fetchmail: POP3< +OK 330 messages (4548411) octets
fetchmail: selecting or re-polling default folder
fetchmail: POP3> STAT
fetchmail: POP3< +OK 330 4548411
330 messages for guido at mailserver.doma.in (4548411 octets).
fetchmail: POP3> LIST 1
fetchmail: POP3< +OK 1 14009
fetchmail: POP3> RETR 1
fetchmail: POP3< +OK 14009 octets
reading message gu...@ma...:1 of 330 (14009 octets)
fetchmail: incorrect header line found while scanning headers
fetchmail: line: 000000000000000036333551
............ flushed
fetchmail: POP3> DELE 1
fetchmail: POP3< +OK Message deleted
fetchmail: fetchlimit 1 reached; 329 messages left on server
mailserver.doma.in account guido
fetchmail: POP3> QUIT
fetchmail: POP3< +OK mailserver.doma.in closing connection
fetchmail: 6.3.8 querying mailserver.doma.in (protocol POP3) at Sun Apr
15 03:23:31 2007: poll completed
fetchmail: not swapping UID lists, no UIDs seen this query
fetchmail: Query status=13 (MAXFETCH)
fetchmail: Deleting fetchids file.
fetchmail: normal termination, status 13
fetchmail: Deleting fetchids file.
Last, this is the output from fetchmail -V:
[guido@localhost ~]$ fetchmail -V -v -v -v mailserver.doma.in
This is fetchmail release
6.3.8+GSS+RPA+NTLM+SDPS+SSL+HESIOD+NLS+KRB4+KRB5.
Copyright (C) 2002, 2003 Eric S. Raymond
Copyright (C) 2004 Matthias Andree, Eric S. Raymond, Robert M. Funk,
Graham Wilson
Copyright (C) 2005-2006 Sunil Shetye
Copyright (C) 2005-2007 Matthias Andree
Fetchmail comes with ABSOLUTELY NO WARRANTY. This is free software, and
you
are welcome to redistribute it under certain conditions. For details,
please see the file COPYING in the source or documentation directory.
Fallback MDA: (none)
Linux localhost 2.6.20.4 #1 Mon Mar 26 03:39:10 CEST 2007 i686 i686
i386 GNU/Linux
Taking options from command line and /home/guido/.fetchmailrc
Idfile is /home/guido/.fetchids
Fetchmail will show progress dots even in logfiles.
Fetchmail will forward misaddressed multidrop messages to guido.
Fetchmail will direct error mail to the sender.
Options for retrieving from gu...@ma...:
True name of server is mailserver.doma.in.
This host will be queried when no host is specified.
Password = "mypassword".
Protocol is POP3 (using default port).
Password authentication will be forced.
Server nonresponse timeout is 300 seconds (default).
Default mailbox selected.
All messages will be retrieved (--all on).
Fetched messages will not be kept on the server (--keep off).
Old messages will not be flushed before message retrieval (--flush
off).
Oversized messages will not be flushed before message retrieval
(--limitflush off).
Rewrite of server-local addresses is enabled (--norewrite off).
Carriage-return stripping is disabled (stripcr off).
Carriage-return forcing is enabled (forcecr on).
Interpretation of Content-Transfer-Encoding is enabled (pass8bits
off).
MIME decoding is disabled (mimedecode off).
Idle after poll is disabled (idle off).
Nonempty Status lines will be kept (dropstatus off)
Delivered-To lines will be kept (dropdelivered off)
Received-message limit is 1 (--fetchlimit 1).
Fetch message size limit is 100 (--fetchsizelimit 100).
Do binary search of UIDs during 3 out of 4 polls (--fastuidl 4).
No SMTP message batch limit (--batchlimit 0).
No forced expunges (--expunge 0).
Messages will be SMTP-forwarded to: localhost (default)
Spam-blocking disabled
No pre-connection command.
No post-connection command.
Single-drop mode: 1 local name recognized.
guido
No interface requirement specified.
No monitor interface specified.
No plugin command specified.
No plugout command specified.
No UIDs saved from this host.
No poll trace information will be added to the Received header.
.
On localhost I have a mail server (postfix), and nothing is sent to it.
If I stop the mail server (so that on localhost:25 there is no
response), the fetchmail log remains the same, this means that
fetchmail does not even try to contact the local mail server.
If I try to download a different message without the numeric line at
the beginning, postfix is correctly called and the message is forwarded
to the "guido" local mailbox.
I hope I gave you enough information to correct the error, if you need
more please do not esitate asking.
Good bye,
Guido Villa
___________________________________
L'email della prossima generazione? Puoi averla con la nuova Yahoo! Mail: http://it.docs.yahoo.com/nowyoucan.html
|
|
From: Matthias A. <mat...@gm...> - 2007-04-15 11:42:47
|
Antonio Capanna <pir...@ya...> writes: > I think fetchmail loses messages when a malformed header is present and > POP3 is used. > Put it shortly, if a message begins with (i.e., the first line of the > headers is) a line with only a number (e.g. > "000000000000000036333551"), the message is retrieved from the server, > deleted, but not forwarded to the local mail server (nor bounced back). > I don't know how did this line got there (the mail server which I am > downloading the mail from is not administered by me), but fetchmail > shouldn't lose the message anyway. First of all, thanks for the very detailed report - it's very nice to see complete reports with all information. This is a known issue (I'm loathe to call it a fetchmail bug though, because basically, the server sends invalid data and it's the server that needs fixing), but I think Gentoo are actually patching that - I fear fixing it in 6.3 though as people might then see garbage and complain about it. I will fix this for the next major release though. > +OK > 000000000000000036333551 > Received: from server.doma.in ([23.45.67.89]) > by mailserver.doma.in (Merak 8.2.4) with ESMTP id DWH39514 > for <gu...@ma...>; Fri, 19 Jan 2007 06:42:34 > +0100 HTH. Best regards, -- Matthias Andree |
|
From: Antonio C. <pir...@ya...> - 2007-04-15 12:06:24
|
Matthias Andree <mat...@gm...> writes:
> First of all, thanks for the very detailed report - it's very nice to
> see complete reports with all information.
:) You're welcome
> This is a known issue (I'm loathe to call it a fetchmail bug though,
> because basically, the server sends invalid data and it's the server
> that needs fixing), but I think Gentoo are actually patching that - I
> fear fixing it in 6.3 though as people might then see garbage and
> complain about it.
Yes, the server sends invalid data, but IMHO fetchmail should NEVER
lose a message (except where stated in the documentation), i.e.:
- fetchmail understands the message -> it handles it and deletes it
- fetchmail does not understand the message -> it forgets about it but
does not delete it from the server. This means, it's true, that the
message will be downloaded and discarded again all the subsequent
times, but I think this is better than completely losing a message. If
I wasn't checking the fetchmail logs when I started it, I would have
lost hundreds of messages from my mail server (it seens that all the
messages on that server have this strange header line).
Another possibility is to delete it from the server but save it
somewhere else.
What do you think?
> I will fix this for the next major release though.
>
> > +OK
> > 000000000000000036333551
> > Received: from server.doma.in ([23.45.67.89])
> > by mailserver.doma.in (Merak 8.2.4) with ESMTP id DWH39514
> > for <gu...@ma...>; Fri, 19 Jan 2007 06:42:34
> > +0100
>
> HTH.
>
> Best regards,
>
> --
> Matthias Andree
Thank you very much for your prompt answer, I will have a look at the
Gentoo patch.
Kind Regards,
Guido
___________________________________
L'email della prossima generazione? Puoi averla con la nuova Yahoo! Mail: http://it.docs.yahoo.com/nowyoucan.html
|
|
From: Rob M. <rob...@gm...> - 2007-04-15 12:58:51
|
On 4/15/07, Antonio Capanna <pir...@ya...> wrote:
<---SNIP--->
> Another possibility is to delete it from the server but save it
> somewhere else.
>
> What do you think?
Some time back (last year probably) this topic came up and one
suggestion was that malformed messages should be stuck inside a
text/plain message part and forwarded to the postmaster.
The main reason being that fetchmail can't guarantee to identify
what's wrong (the last thread was related to there being no blank line
separating the headers from the body). By forwarding the entire
broken message to the postmaster then the postmaster can at least
decide what to do.
Note though that your local SMTP server would likely have refused
these messages, probably with a permanent failure due to their being
malformed. This would then have resulted in fetchmail deleting them,
so effectively fetchmail just cut the SMTP server out of the decision
loop :> (Neither is an ideal situation though, I agree)
--
Please keep list traffic on the list.
Rob MacGregor
Whoever fights monsters should see to it that in the process he
doesn't become a monster. Friedrich Nietzsche
|
|
From: Antonio C. <pir...@ya...> - 2007-04-15 13:26:19
|
> Rob MacGregor <rob...@gm...> writes:
> Note though that your local SMTP server would likely have refused
> these messages, probably with a permanent failure due to their being
> malformed. This would then have resulted in fetchmail deleting them,
> so effectively fetchmail just cut the SMTP server out of the decision
> loop :> (Neither is an ideal situation though, I agree)
:-)
Actually, I just made this test. I patchd transact.c and removed the
line that discards the messages in this case, and then I tried again.
It happens that my postfix+maildrop configuration accepts the message,
and rewrites a new (partial) set of headers for it. But this is just
postfix: probably, other SMTP servers would react differently.
> Please keep list traffic on the list.
Sorry, I clicked "reply to all" without paying attention.
> Rob MacGregor
Thanks,
good bye,
Guido
___________________________________
L'email della prossima generazione? Puoi averla con la nuova Yahoo! Mail: http://it.docs.yahoo.com/nowyoucan.html
|
|
From: Volker K. <lis...@pa...> - 2007-04-15 14:01:52
|
On Sun 15 Apr 2007 22:04:25 NZST +1200, Antonio Capanna wrote: > Yes, the server sends invalid data, but IMHO fetchmail should NEVER > lose a message (except where stated in the documentation), i.e.: > - fetchmail understands the message -> it handles it and deletes it > - fetchmail does not understand the message -> it forgets about it but > does not delete it from the server. This means, it's true, that the > message will be downloaded and discarded again all the subsequent > times, In this case it would also be necessary to send a warning somehow, because you're DoS'ing yourself: ignoring the wasted traffic, eventually your mailbox at your mail provider will be full and no more mail is accepted for you. Volker -- Volker Kuhlmann is list0570 with the domain in header http://volker.dnsalias.net/ Please do not CC list postings to me. |
|
From: Matthias A. <mat...@gm...> - 2007-04-15 14:27:42
|
(sorry Volker, this message was meant for the list, of course) On Sun, 15 Apr 2007, Volker Kuhlmann wrote: > On Sun 15 Apr 2007 22:04:25 NZST +1200, Antonio Capanna wrote: > > > Yes, the server sends invalid data, but IMHO fetchmail should NEVER > > lose a message (except where stated in the documentation), i.e.: > > - fetchmail understands the message -> it handles it and deletes it > > - fetchmail does not understand the message -> it forgets about it but > > does not delete it from the server. This means, it's true, that the > > message will be downloaded and discarded again all the subsequent > > times, > > In this case it would also be necessary to send a warning somehow, > because you're DoS'ing yourself: ignoring the wasted traffic, eventually > your mailbox at your mail provider will be full and no more mail is > accepted for you. Indeed. Well, a way in between would be to take junk at the beginning and prepend X-Fetchmail-Garbage-Before-Header: and consider the next invalid header the begin of the body and terminate the header there, put a message there and then continue - which might however corrupt MIME multiparts. Whatever fetchmail is doing at that time, someone will be upset... -- Matthias Andree |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X