fetchmail-devel

[fetchmail-devel] [BUG] fetchmail hangs during pop3 pull after a mail with a null char

[David G. <da...@dg...>]

First : fetchmail is great - thanks  :) 

I sent this to fetchmail-friends a while back and it was suggested that I send it to the devlist.
I joined and lurked - and finally got round to sending this in...


It's been working for years with these occasional hangs that have been
fixed by popping the bad messages and manually filing them. I finally
had a bad message arrive when I was in a position to debug!

Summary : fetchmail hangs during pop3 pull after a mail with a null char.

The mail with a null char is pulled OK but then rejected by local
Cyrus lmtp and bounced to postmaster via exim4.20
The next pop3 pull then fails.
I've made an effort to trace and I think the hang occurs due to a
double call to SMTP_ok which is empty the second time. I am pretty
sure the second call originates at sink.c line 1433.

in the config expunge 1 fixes the problem (which makes sense)

general config is pop3->lmtp->local Cyrus IMAP

So I went through the FAQ G3 points:
1. OS: Linux RedHat 7.3 kernel 2.6.6
2. gcc version 2.96 20000731 (Red Hat Linux 7.3 2.96-110)
3. below
4. forwarding to lmtp listener on cyrus 2.2.3 (bounce goes to SMTP
exim 4.2)
5. -v -v -f /etc/fetchmailrc
6. at the end

nb: SMTP_ok loop start comments are my trace.

Aug 17 17:33:01 willow fetchmail[13648]: 6.2.5 querying pop3.ukfsn.org
(protocol POP3) at Tue 17 Aug 2004 05:33:01 PM BST: poll started
Aug 17 17:33:01 willow fetchmail[13648]: POP3< +OK
<aa5...@po...>
Aug 17 17:33:01 willow fetchmail[13648]: POP3> CAPA^M
Aug 17 17:33:01 willow fetchmail[13648]: POP3< +OK Capability list follows
Aug 17 17:33:01 willow fetchmail[13648]: POP3< PIPELINING
Aug 17 17:33:01 willow fetchmail[13648]: POP3< TOP
Aug 17 17:33:01 willow fetchmail[13648]: POP3< USER
Aug 17 17:33:01 willow fetchmail[13648]: POP3< UIDL
Aug 17 17:33:01 willow fetchmail[13648]: POP3< STLS
Aug 17 17:33:01 willow fetchmail[13648]: POP3< .
Aug 17 17:33:01 willow fetchmail[13648]: POP3> USER dgreaves^M
Aug 17 17:33:01 willow fetchmail[13648]: POP3< +OK Tell me your password.
Aug 17 17:33:01 willow fetchmail[13648]: POP3> PASS *^M
Aug 17 17:33:02 willow fetchmail[13648]: POP3< +OK Welcome aboard! You
have 55 messages.
Aug 17 17:33:05 willow fetchmail[13648]: POP3> STAT
Aug 17 17:33:05 willow fetchmail[13648]: POP3< +OK 55 429607
Aug 17 17:33:05 willow fetchmail[13648]: 55 messages for dgreaves at
pop3.ukfsn.org (429607 octets).



fetchmailrc:
set syslog
set postmaster "da...@dg..."
set nobouncemail
set properties ""
#set daemon 180
set idfile /var/run/fetchmail.ids

# The ukfsn accounts
poll pop3.ukfsn.org with proto POP3 tracepolls
~    user 'dgreaves' there with password 'xxxxxxx' is
da...@dg... here options fetchall lmtp smtp
/var/imap/socket/lmtp expunge 5
~    antispam 571 550 501 554

<more user accounts removed>

here is output from
~  ./fetchmail -v -v -f /etc/fetchmailrc

Aug 17 17:43:33 willow fetchmail[13675]: 6.2.5 querying pop3.ukfsn.org
(protocol POP3) at Tue 17 Aug 2004 05:43:33 PM BST: poll started
Aug 17 17:43:33 willow fetchmail[13675]: POP3< +OK
<40b...@po...>
Aug 17 17:43:33 willow fetchmail[13675]: POP3> CAPA^M
Aug 17 17:43:33 willow fetchmail[13675]: POP3< +OK Capability list follows
Aug 17 17:43:34 willow fetchmail[13675]: POP3< PIPELINING
Aug 17 17:43:34 willow fetchmail[13675]: POP3< TOP
Aug 17 17:43:34 willow fetchmail[13675]: POP3< USER
Aug 17 17:43:34 willow fetchmail[13675]: POP3< UIDL
Aug 17 17:43:34 willow fetchmail[13675]: POP3< STLS
Aug 17 17:43:34 willow fetchmail[13675]: POP3< .
Aug 17 17:43:34 willow fetchmail[13675]: POP3> USER dgreaves^M
Aug 17 17:43:34 willow fetchmail[13675]: POP3< +OK Tell me your password.
Aug 17 17:43:34 willow fetchmail[13675]: POP3> PASS *^M
Aug 17 17:43:34 willow fetchmail[13675]: POP3< +OK Welcome aboard! You
have 33 messages.
Aug 17 17:43:37 willow fetchmail[13675]: selecting or re-polling
default folder
Aug 17 17:43:37 willow fetchmail[13675]: POP3> STAT
Aug 17 17:43:37 willow fetchmail[13675]: POP3< +OK 33 186252
Aug 17 17:43:37 willow fetchmail[13675]: 33 messages for dgreaves at
pop3.ukfsn.org (186252 octets).
Aug 17 17:43:37 willow fetchmail[13675]: POP3> LIST 1
#**********************************************Aug 17 17:43:37 willow
fetchmail[13675]: POP3< +OK 1 9060
Aug 17 17:43:37 willow fetchmail[13675]: POP3> RETR 1
Aug 17 17:43:37 willow fetchmail[13675]: POP3< +OK Message follows
Aug 17 17:43:37 willow fetchmail[13675]: reading message
dgr...@po...:1 of 33 (9060 octets)
Aug 17 17:43:37 willow fetchmail[13675]: About to rewrite Return-Path:
<linux-kernel-owner+lkml=40d...@vg...>^M
Rewritten version is Return-Path:
<linux-kernel-owner+lkml=40d...@vg...>^M
Aug 17 17:43:38 willow fetchmail[13675]: About to rewrite From:
"O.Sezer" <se...@tt...>^M Rewritten version is From:
"O.Sezer" <se...@tt...>^M
Aug 17 17:43:38 willow fetchmail[13675]: About to rewrite To:
lin...@vg...^M Rewritten version is To:
lin...@vg...^M
Aug 17 17:43:38 willow fetchmail[13675]: About to rewrite Cc:
mar...@cy...^M Rewritten version is Cc:
mar...@cy...^M
Aug 17 17:43:38 willow fetchmail[13675]: About to rewrite Sender:
lin...@vg...^M Rewritten version is Sender:
lin...@vg...^M
Aug 17 17:43:38 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:38 willow fetchmail[13675]: LMTP< 220 willow LMTP Cyrus
v2.2.3 ready
Aug 17 17:43:38 willow fetchmail[13675]: LMTP> LHLO localhost
Aug 17 17:43:38 willow fetchmail[13675]: SMTP< 250-willow
Aug 17 17:43:38 willow fetchmail[13675]: SMTP< 250-8BITMIME
Aug 17 17:43:38 willow fetchmail[13675]: SMTP< 250-ENHANCEDSTATUSCODES
Aug 17 17:43:38 willow fetchmail[13675]: SMTP< 250-PIPELINING
Aug 17 17:43:38 willow fetchmail[13675]: SMTP< 250-SIZE
Aug 17 17:43:38 willow fetchmail[13675]: SMTP< 250-AUTH EXTERNAL
Aug 17 17:43:38 willow fetchmail[13675]: SMTP< 250 IGNOREQUOTA
Aug 17 17:43:38 willow fetchmail[13675]: forwarding to
/var/imap/socket/lmtp
Aug 17 17:43:38 willow fetchmail[13675]: LMTP> MAIL
FROM:<linux-kernel-owner+lkml=40d...@vg...>
SIZE=9060
Aug 17 17:43:38 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:38 willow fetchmail[13675]: LMTP< 250 2.1.0 ok
Aug 17 17:43:38 willow fetchmail[13675]: LMTP> RCPT
TO:<da...@dg...>
Aug 17 17:43:38 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:38 willow fetchmail[13675]: LMTP< 250 2.1.5 ok
Aug 17 17:43:38 willow fetchmail[13675]: LMTP> DATA
Aug 17 17:43:38 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:38 willow fetchmail[13675]: LMTP< 354 go ahead
********************************************************************************************************************************************************************Aug
17 17:43:38 willow fetchmail[13675]: message dgr...@po...:1
was not the expected length (9317 actual != 9060 expected)
Aug 17 17:43:38 willow fetchmail[13675]: LMTP>. (EOM)
Aug 17 17:43:38 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:38 willow fetchmail[13675]: LMTP< 250 2.1.5 Ok
Aug 17 17:43:38 willow fetchmail[13675]:  flushed
Aug 17 17:43:38 willow fetchmail[13675]: POP3> DELE 1^M
Aug 17 17:43:38 willow fetchmail[13675]: POP3< +OK Done.
Aug 17 17:43:38 willow fetchmail[13675]: POP3> LIST 2
Aug 17 17:43:38 willow fetchmail[13675]: POP3< +OK 2 5098
Aug 17 17:43:38 willow fetchmail[13675]: POP3> RETR 2
Aug 17 17:43:39 willow fetchmail[13675]: POP3< +OK Message follows
Aug 17 17:43:39 willow fetchmail[13675]: reading message
dgr...@po...:2 of 33 (5098 octets)
Aug 17 17:43:39 willow fetchmail[13675]: About to rewrite Return-Path:
<linux-kernel-owner+lkml=40d...@vg...>^M
Rewritten version is Return-Path:
<linux-kernel-owner+lkml=40d...@vg...>^M
#****************************************************************Aug
17 17:43:39 willow fetchmail[13675]: About to rewrite From: Christoph
Hellwig <hc...@in...>^M Rewritten version is From: Christoph
Hellwig <hc...@in...>^M
Aug 17 17:43:39 willow fetchmail[13675]: About to rewrite To: Markus
Lidel <Mar...@sh...>^M Rewritten version is To:
Markus Lidel <Mar...@sh...>^M
Aug 17 17:43:39 willow fetchmail[13675]: About to rewrite Cc:
Christoph Hellwig <hc...@in...>,^M ^IWarren Togami
<wt...@re...>, lin...@vg...^M Rewritten version
is Cc: Christoph Hellwig <hc...@in...>,^M ^IWarren Togami
<wt...@re...>, lin...@vg...^M
Aug 17 17:43:39 willow fetchmail[13675]: About to rewrite Sender:
lin...@vg...^M Rewritten version is Sender:
lin...@vg...^M
Aug 17 17:43:39 willow fetchmail[13675]: forwarding to
/var/imap/socket/lmtp
Aug 17 17:43:39 willow fetchmail[13675]: LMTP> MAIL
FROM:<linux-kernel-owner+lkml=40d...@vg...>
SIZE=5098
Aug 17 17:43:39 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:39 willow fetchmail[13675]: LMTP< 250 2.1.0 ok
Aug 17 17:43:39 willow fetchmail[13675]: LMTP> RCPT
TO:<da...@dg...>
Aug 17 17:43:39 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:39 willow fetchmail[13675]: LMTP< 250 2.1.5 ok
Aug 17 17:43:39 willow fetchmail[13675]: LMTP> DATA
Aug 17 17:43:39 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:39 willow fetchmail[13675]: LMTP< 354 go ahead
Aug 17 17:43:39 willow fetchmail[13675]: message
dgr...@po...:2 was not the expected length (5209 actual !=
5098 expected)
Aug 17 17:43:39 willow fetchmail[13675]: LMTP>. (EOM)
Aug 17 17:43:39 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:39 willow fetchmail[13675]: LMTP< 250 2.1.5 Ok
Aug 17 17:43:39 willow fetchmail[13675]:  flushed
Aug 17 17:43:39 willow fetchmail[13675]: POP3> DELE 2^M
Aug 17 17:43:39 willow fetchmail[13675]: POP3< +OK Done.
Aug 17 17:43:39 willow fetchmail[13675]: POP3> LIST 3
Aug 17 17:43:39 willow fetchmail[13675]: POP3< +OK 3 2147
Aug 17 17:43:39 willow fetchmail[13675]: POP3> RETR 3
Aug 17 17:43:40 willow fetchmail[13675]: POP3< +OK Message follows
Aug 17 17:43:40 willow fetchmail[13675]: reading message
dgr...@po...:3 of 33 (2147 octets)
Aug 17 17:43:40 willow fetchmail[13675]: About to rewrite Return-Path:
<reiserfs-list-return-20355-david=dgr...@na...>^M
Rewritten version is Return-Path:
<reiserfs-list-return-20355-david=dgr...@na...>^M
#********#****************************Aug 17 17:43:40 willow
fetchmail[13675]: About to rewrite From: elliott
<aur...@se...>^M Rewritten version is From: elliott
<aur...@se...>^M
Aug 17 17:43:40 willow fetchmail[13675]: About to rewrite To:
rei...@na...^M Rewritten version is To:
rei...@na...^M
Aug 17 17:43:40 willow fetchmail[13675]: forwarding to
/var/imap/socket/lmtp
Aug 17 17:43:40 willow fetchmail[13675]: LMTP> MAIL
FROM:<reiserfs-list-return-20355-david=dgr...@na...>
BODY=8BITMIME SIZE=2147
Aug 17 17:43:40 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:40 willow fetchmail[13675]: LMTP< 250 2.1.0 ok
Aug 17 17:43:40 willow fetchmail[13675]: LMTP> RCPT
TO:<da...@dg...>
Aug 17 17:43:40 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:40 willow fetchmail[13675]: LMTP< 250 2.1.5 ok
Aug 17 17:43:40 willow fetchmail[13675]: LMTP> DATA
Aug 17 17:43:40 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:40 willow fetchmail[13675]: LMTP< 354 go ahead
Aug 17 17:43:40 willow fetchmail[13675]: message
dgr...@po...:3 was not the expected length (2194 actual !=
2147 expected)
Aug 17 17:43:40 willow fetchmail[13675]: LMTP>. (EOM)
Aug 17 17:43:40 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:40 willow fetchmail[13675]: LMTP< 554 5.6.0 Message
contains NUL characters
Aug 17 17:43:40 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:40 willow fetchmail[13675]: SMTP< 220 willow ESMTP Exim
4.20 Tue, 17 Aug 2004 17:43:40 +0100
Aug 17 17:43:40 willow fetchmail[13675]: SMTP> HELO localhost
Aug 17 17:43:40 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:40 willow fetchmail[13675]: SMTP< 250 willow Hello
[AdFSF0x5m0a0vDWoZf1oPXMI8ichzbi7] at localhost.localdomain [127.0.0.1]
Aug 17 17:43:40 willow fetchmail[13675]: SMTP> MAIL
FROM:<FET...@wi...>
Aug 17 17:43:40 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:40 willow fetchmail[13675]: SMTP< 250 OK
Aug 17 17:43:40 willow fetchmail[13675]: SMTP> RCPT
TO:<da...@dg...>
Aug 17 17:43:40 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:40 willow fetchmail[13675]: SMTP< 250 Accepted
Aug 17 17:43:40 willow fetchmail[13675]: SMTP> DATA
Aug 17 17:43:40 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:40 willow fetchmail[13675]: SMTP< 354 Enter message,
ending with "." on a line by itself
Aug 17 17:43:40 willow fetchmail[13675]: SMTP: (bounce-message body)
Aug 17 17:43:40 willow fetchmail[13675]: SMTP>. (EOM)
Aug 17 17:43:40 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:40 willow fetchmail[13675]: SMTP< 250 OK id=1Bx73k-0003Ya-FK
Aug 17 17:43:40 willow fetchmail[13675]: SMTP> QUIT
Aug 17 17:43:40 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:40 willow fetchmail[13675]: SMTP< 221 willow closing
connection
Aug 17 17:43:40 willow fetchmail[13675]:  flushed
Aug 17 17:43:40 willow fetchmail[13675]: POP3> DELE 3^M
Aug 17 17:43:40 willow fetchmail[13675]: POP3< +OK Done.
Aug 17 17:43:40 willow fetchmail[13675]: POP3> LIST 4
Aug 17 17:43:40 willow fetchmail[13675]: POP3< +OK 4 4406
Aug 17 17:43:40 willow fetchmail[13675]: POP3> RETR 4
Aug 17 17:43:40 willow fetchmail[13675]: POP3< +OK Message follows
Aug 17 17:43:40 willow fetchmail[13675]: reading message
dgr...@po...:4 of 33 (4406 octets)
Aug 17 17:43:41 willow fetchmail[13675]: About to rewrite Return-Path:
<net...@os...>^M Rewritten version is Return-Path:
<net...@os...>^M
Aug 17 17:43:41 willow fetchmail[13675]: About to rewrite From:
Wensong Zhang <we...@li...>^M Rewritten version is From:
Wensong Zhang <we...@li...>^M
Aug 17 17:43:41 willow fetchmail[13675]: About to rewrite To:
ne...@os...^M Rewritten version is To: ne...@os...^M
Aug 17 17:43:41 willow fetchmail[13675]: About to rewrite Cc: Julian
Anastasov <ja...@ss...>^M Rewritten version is Cc: Julian Anastasov
<ja...@ss...>^M
Aug 17 17:43:41 willow fetchmail[13675]: About to rewrite Sender:
net...@os...^M Rewritten version is Sender:
net...@os...^M
Aug 17 17:43:41 willow fetchmail[13675]: forwarding to
/var/imap/socket/lmtp
Aug 17 17:43:41 willow fetchmail[13675]: SMTP> MAIL
FROM:<net...@os...> SIZE=4406
Aug 17 17:43:41 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:41 willow fetchmail[13675]: SMTP< 250 2.1.0 ok
Aug 17 17:43:41 willow fetchmail[13675]: SMTP> RCPT
TO:<da...@dg...>
Aug 17 17:43:41 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:41 willow fetchmail[13675]: SMTP< 250 2.1.5 ok
Aug 17 17:43:41 willow fetchmail[13675]: SMTP> DATA
Aug 17 17:43:41 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:41 willow fetchmail[13675]: SMTP< 354 go ahead
*******************************************************Aug 17 17:43:41
willow fetchmail[13675]: message dgr...@po...:4 was not the
expected length (4533 actual != 4406 expected)
Aug 17 17:43:41 willow fetchmail[13675]: SMTP>. (EOM)
Aug 17 17:43:41 willow fetchmail[13675]: SMTP_ok loop start
Aug 17 17:43:41 willow fetchmail[13675]: SMTP< 250 2.1.5 Ok

## 5 minute hang

Aug 17 17:48:41 willow fetchmail[13675]: smtp listener protocol error 2
Aug 17 17:48:41 willow fetchmail[13675]:  not flushed
Aug 17 17:48:41 willow fetchmail[13675]: POP3> LIST 5
Aug 17 17:48:41 willow fetchmail[13675]: POP3< -ERR Client has been
idle for too long.
Aug 17 17:48:41 willow fetchmail[13675]: Client has been idle for too
long.
Aug 17 17:48:41 willow fetchmail[13675]: POP3> QUIT^M
Aug 17 17:48:41 willow fetchmail[13675]: client/server protocol error
while fetching from pop3.ukfsn.org
Aug 17 17:48:41 willow fetchmail[13675]: 6.2.5 querying pop3.ukfsn.org
(protocol POP3) at Tue 17 Aug 2004 05:48:41 PM BST: poll completed
Aug 17 17:48:41 willow fetchmail[13675]: Query status=4 (PROTOCOL)

# ./fetchmail -V -v -v -f /etc/fetchmailrc
This is fetchmail release 6.2.5+NLS
Fallback MDA: (none)
Linux willow 2.6.6 #1 Wed Jun 2 12:15:21 BST 2004 i586 unknown
Taking options from command line and /etc/fetchmailrc
Idfile is /var/run/fetchmail.ids
Progress messages will be logged via syslog
Fetchmail will show progress dots even in logfiles.
Fetchmail will forward misaddressed multidrop messages to
da...@dg....
Fetchmail will direct error mail to the postmaster.
Options for retrieving from dgr...@po...:
~  True name of server is pop3.ukfsn.org.
~  This host will be queried when no host is specified.
~  Password = "xxxxxx".
~  Protocol is POP3 (using default port).
~  All available authentication methods will be tried.
~  Server nonresponse timeout is 300 seconds (default).
~  Default mailbox selected.
~  All messages will be retrieved (--all on).
~  Fetched messages will not be kept on the server (--keep off).
~  Old messages will not be flushed before message retrieval (--flush
off).
~  Rewrite of server-local addresses is enabled (--norewrite off).
~  Carriage-return stripping is disabled (stripcr off).
~  Carriage-return forcing is disabled (forcecr off).
~  Interpretation of Content-Transfer-Encoding is enabled (pass8bits off).
~  MIME decoding is disabled (mimedecode off).
~  Idle after poll is disabled (idle off).
~  Nonempty Status lines will be kept (dropstatus off)
~  Delivered-To lines will be kept (dropdelivered off)
~  No received-message limit (--fetchlimit 0).
~  Fetch message size limit is 100 (--fetchsizelimit 100).
~  Do binary search of UIDs during 9 out of 10 polls (--fastuidl 10).
~  No SMTP message batch limit (--batchlimit 0).
~  Deletion interval between expunges forced to 5 (--expunge 5).
~  Messages will be LMTP-forwarded to: /var/imap/socket/lmtp
~  Recognized listener spam block responses are: 571 550 501 554
~  No pre-connection command.
~  No post-connection command.
~  Single-drop mode: 1 local name(s) recognized.
~        da...@dg...
~  No interface requirement specified.
~  No monitor interface specified.
~  No plugin command specified.
~  No plugout command specified.
~  1 UIDs saved.
~        0bcc1e7633bb91ec04fbf4e1505b377d
~  Poll trace information will be added to the Received header.

other account info removed

David Greaves

Re: [fetchmail-devel] [BUG] fetchmail hangs during pop3 pull after a mail with a null char

[Matthias A. <ma...@dt...>]

David Greaves <da...@dg...> writes:

> Summary : fetchmail hangs during pop3 pull after a mail with a null
> char.

Can you try the tarball from
http://home.pages.de/~mandree/tmp/fetchmail-6.2.6.tar.bz2 and see if the
problem persists? There have been several bugfixes since 6.2.5, the
noteworthy documented at http://decoy.wox.org/svn/fetchmail/trunk/NEWS -
just to make sure we aren't chasing a bug that's already fixed?

NOTE the tarball there isn't the final 6.2.6 release tarball but a
preliminary snapshot and may see further changes before being released
as 6.2.6.

Thanks in advance.

-- 
Matthias Andree

[fetchmail-devel] Assorted minor bugs

[Brian C. <B.C...@po...>]

On Mon, Nov 08, 2004 at 01:01:31AM +0100, Matthias Andree wrote:
> Can you try the tarball from
> http://home.pages.de/~mandree/tmp/fetchmail-6.2.6.tar.bz2

Thanks for making this interim release.

I reported some problems with 6.2.5 and posted patches to the old
fetchmail-friends list in April, but none appear to have been incorporated
into 6.2.6. The small patchset I keep is attached, and still applies cleanly
to 6.2.6.

The bugs were described more fully on fetchmail-friends, but in summary:

(1) There is a completely unnecessary sleep(3) after logging in. I cannot
see any justification for this; the comment in the code talks about mailbox
locking and tempfiles, but when the POP3 server has replied "+OK" to a
login, then it is by its own admission locked and ready to receive commands.

I am guessing this was a workaround for some ancient and broken server, but
I think it should be removed as it just wastes time and resources.

(2a) When talking to an SSL server, where the certificate name does not
match, then annoyingly a 'certificate name mismatch' error is given three
times. The patch changes it so that this warning is only given if you select
verbose or sslcertck, to be the same as the other certificate
warnings/errors.

(2b) When talking to an SSL server, honour the sslcertpath option even if
sslcertck is not set (currently it is ignored); this allows you to use
fetchmail -v to get useful information on the certificate chain, even if
it's not being strictly enforced using sslcertck.

(3) There is a bug where 'showdots' fails to work if there are two 'poll'
entries in .fetchmailrc; I spent a while chasing this down, and it turns out
to be an option parsing bug. Full description here:
http://lists.ccil.org/pipermail/fetchmail-friends/2004-April/008518.html

This URL also contains separately the patches which are combined in the
attached bundle.

You might want to change my fix for (1) though: instead of #ifdef BOLLOCKS
then just remove the sleep and the (incorrect) comment associated with it
:-)

Regards,

Brian.

Re: [fetchmail-devel] Assorted minor bugs

[Matthias A. <ma...@dt...>]

Brian Candler <B.C...@po...> writes:

> On Mon, Nov 08, 2004 at 01:01:31AM +0100, Matthias Andree wrote:
>> Can you try the tarball from
>> http://home.pages.de/~mandree/tmp/fetchmail-6.2.6.tar.bz2
>
> Thanks for making this interim release.

It's a snapshot, not a release. I deliberately let a tarball escape...

> The bugs were described more fully on fetchmail-friends, but in summary:
>
> (1) There is a completely unnecessary sleep(3) after logging in. I cannot
> see any justification for this; the comment in the code talks about mailbox
> locking and tempfiles, but when the POP3 server has replied "+OK" to a
> login, then it is by its own admission locked and ready to receive commands.
>
> I am guessing this was a workaround for some ancient and broken server, but
> I think it should be removed as it just wastes time and resources.

We'll see what happens. I've removed the sleep for now, it's been
annoying myself for years so much I patched it away, with no ill effect
on common servers.

> (2a) When talking to an SSL server, where the certificate name does not
> match, then annoyingly a 'certificate name mismatch' error is given three
> times. The patch changes it so that this warning is only given if you select
> verbose or sslcertck, to be the same as the other certificate
> warnings/errors.

*Shrug* I don't see how it's printed three times. It's printed once on
my computer. Anyways, I would not want to limit this warning to verbose
or strict mode or similar, it's an important information and needs to be
printed unconditionally. If it annoys the user, tough luck, fix the
configuration or let him complain to the server's admin.

> (2b) When talking to an SSL server, honour the sslcertpath option even if
> sslcertck is not set (currently it is ignored); this allows you to use
> fetchmail -v to get useful information on the certificate chain, even if
> it's not being strictly enforced using sslcertck.

Merged.

> (3) There is a bug where 'showdots' fails to work if there are two 'poll'
> entries in .fetchmailrc; I spent a while chasing this down, and it turns out
> to be an option parsing bug. Full description here:
> http://lists.ccil.org/pipermail/fetchmail-friends/2004-April/008518.html

Merged. Thanks for your efforts in analyzing and fixing this.

Thank you!

-- 
Matthias Andree

Re: [fetchmail-devel] Assorted minor bugs

[Brian C. <B.C...@po...>]

> *Shrug* I don't see how it's printed three times. It's printed once on
> my computer.

There's some more detailled output at
http://lists.ccil.org/pipermail/fetchmail-friends/2004-April/008516.html

Basically, each verify callback operation causes this warning to be shown,
and with fetchmail -v you get this in addition to the other verification
warning associated with that callback. So with fetchmail -v I get (snipped a
little):

fetchmail: Server CommonName mismatch: webmail.example.com != pop3.example.com
fetchmail: Warning: server certificate verification: unable to get local issuer certificate
fetchmail: Server CommonName mismatch: webmail.example.com != pop3.example.com
fetchmail: Warning: server certificate verification: certificate not trusted
fetchmail: Server CommonName mismatch: webmail.example.com != pop3.example.com
fetchmail: Warning: server certificate verification: unable to verify the first certificate

That's with 6.2.5, I've not tried your snapshot yet.

Now, if I run fetchmail without -v, then the intervening verification errors
are not shown, so I just see the CN mismatch errors three times. Perhaps you
only see them once because you have root certs installed in the correct
places, or maybe it's a difference in our versions of openssl.

> Anyways, I would not want to limit this warning to verbose
> or strict mode or similar, it's an important information and needs to be
> printed unconditionally.

Well, I disagree that this information is any more important than
"certificate not trusted" or "unable to verify signature" or the various
other errors that only fetchmail -v or fetchmail sslcertck will show.

In particular: fetchmail (without -v) will not generate any warning for a
self-signed certificate where the CN is correct. However, if you have a
valid signed certificate from a known CA, but the hostname does not match,
you get the warning (three times in my case).

Both are violations of the SSL trust model. In the latter case, you know
that the server admin did at least manage to persuade a known and trusted CA
to sign a certificate, albeit for a different hostname. Whereas, in the
first case you could be the subject of a man-in-the-middle attack, and you
get no warning at all.

That's why my preferred solution was to silence this message without -v or
sslcertck: because more serious messages are already silenced.

Regards,

Brian.

Re: [fetchmail-devel] Assorted minor bugs

[Matthias A. <ma...@dt...>]

Brian Candler <B.C...@po...> writes:

> There's some more detailled output at
> http://lists.ccil.org/pipermail/fetchmail-friends/2004-April/008516.html
>
> Basically, each verify callback operation causes this warning to be shown,
> and with fetchmail -v you get this in addition to the other verification
> warning associated with that callback. So with fetchmail -v I get (snipped a
> little):
>
> fetchmail: Server CommonName mismatch: webmail.example.com != pop3.example.com
> fetchmail: Warning: server certificate verification: unable to get local issuer certificate
> fetchmail: Server CommonName mismatch: webmail.example.com != pop3.example.com
> fetchmail: Warning: server certificate verification: certificate not trusted
> fetchmail: Server CommonName mismatch: webmail.example.com != pop3.example.com
> fetchmail: Warning: server certificate verification: unable to verify the first certificate
>
> That's with 6.2.5, I've not tried your snapshot yet.
>
> Now, if I run fetchmail without -v, then the intervening verification errors
> are not shown, so I just see the CN mismatch errors three times. Perhaps you
> only see them once because you have root certs installed in the correct
> places, or maybe it's a difference in our versions of openssl.
>
>> Anyways, I would not want to limit this warning to verbose
>> or strict mode or similar, it's an important information and needs to be
>> printed unconditionally.
>
> Well, I disagree that this information is any more important than
> "certificate not trusted" or "unable to verify signature" or the various
> other errors that only fetchmail -v or fetchmail sslcertck will show.
>
> In particular: fetchmail (without -v) will not generate any warning for a
> self-signed certificate where the CN is correct. However, if you have a
> valid signed certificate from a known CA, but the hostname does not match,
> you get the warning (three times in my case).
>
> Both are violations of the SSL trust model. In the latter case, you know
> that the server admin did at least manage to persuade a known and trusted CA
> to sign a certificate, albeit for a different hostname. Whereas, in the
> first case you could be the subject of a man-in-the-middle attack, and you
> get no warning at all.
>
> That's why my preferred solution was to silence this message without -v or
> sslcertck: because more serious messages are already silenced.

I see you have some understanding of the SSL certificate/trust model and
verification process.

May I ask you to come up with a small patch that fixes _all_ violations
of the trust model in a first round, so that we _unconditionally_ print
all warnings, self-signed certificates, common name mismatches and so
on. These _are_ important warnings, if they are shown only in verbose
mode, that is a serious bug. For this first round, all problems should
always print a warning regardless of verbosity or logging
settings. sslcertck should make any validation failure an error that
aborts the connection to that server and skips to the next.

A second step (longer-term than the next release), i. e. after the next
official release, should then make strict checking the default and offer
the user options to relax checking for the case when the configuration
(client or server) cannot be fixed on short notice.

The whole point of SSL is avoiding the credentials to be revealed to
anything but the "real" server, and lax checks that do not prevent MITM
attacks defeat the whole purpose of SSL. I'd think creating a false
sense of security is worse than not offering security.

-- 
Matthias Andree

Re: [fetchmail-devel] [BUG] fetchmail hangs during pop3 pull after a mail with a null char

[David G. <da...@dg...>]

many moons ago (8 Nov 04) Matthias Andree wrote:

>David Greaves <da...@dg...> writes:
>
>  
>
>>Summary : fetchmail hangs during pop3 pull after a mail with a null
>>char.
>>    
>>
>
>Can you try the tarball from
>http://home.pages.de/~mandree/tmp/fetchmail-6.2.6.tar.bz2 and see if the
>problem persists? There have been several bugfixes since 6.2.5, the
>noteworthy documented at http://decoy.wox.org/svn/fetchmail/trunk/NEWS -
>just to make sure we aren't chasing a bug that's already fixed?
>
>NOTE the tarball there isn't the final 6.2.6 release tarball but a
>preliminary snapshot and may see further changes before being released
>as 6.2.6.
>
>Thanks in advance.
>  
>
Hi again Matthias :)

This was the reason I was upgrading to 6.2.5.991
I've included my original message in case you don't have a copy to hand.

I have a message in my pop queue with a null char in it (which is what 
triggers this).

There's a log below.

I can work around the problem if I "expunge 1" - but that's nasty.

As I said in my original message, I think it was sink.c calling smtp_ok 
- have you any suggestions as to where/how to debug this?

David

log:
# /usr/bin/fetchmail -N -v -v --nosyslog -f /etc/fetchmailrc
fetchmail: starting fetchmail 6.2.5.991 daemon
fetchmail: 6.2.5.991 querying pop3.ukfsn.org (protocol POP3) at Wed 23 
Mar 2005 11:57:08 AM GMT: poll started
fetchmail: POP3< +OK <ce2...@po...>
fetchmail: POP3> CAPA
fetchmail: POP3< +OK Capability list follows
fetchmail: POP3< PIPELINING
fetchmail: POP3< TOP
fetchmail: POP3< USER
fetchmail: POP3< UIDL
fetchmail: POP3< STLS
fetchmail: POP3< .
fetchmail: POP3> USER dgreaves
fetchmail: POP3< +OK Tell me your password.
fetchmail: POP3> PASS *
fetchmail: POP3< +OK Welcome aboard! You have 22 messages.
fetchmail: selecting or re-polling default folder
fetchmail: POP3> STAT
fetchmail: POP3< +OK 22 105420
22 messages for dgreaves at pop3.ukfsn.org (105420 octets).
fetchmail: POP3> LIST 1
fetchmail: POP3< +OK 1 3982
fetchmail: POP3> RETR 1
fetchmail: POP3< +OK Message follows

<snip some good pop'ing>

fetchmail: LMTP< 250 2.1.5 Ok
 flushed
fetchmail: POP3> DELE 2
fetchmail: POP3< +OK Done.
fetchmail: POP3> LIST 3
fetchmail: POP3< +OK 3 2852
fetchmail: POP3> RETR 3
fetchmail: POP3< +OK Message follows
reading message dgr...@po...:3 of 22 (2852 octets)
About to rewrite Return-Path: <ca...@ab...>
Rewritten version is Return-Path: <ca...@ab...>

About to rewrite From: Susan M.Taylor <ca...@ab...>
Rewritten version is From: Susan M.Taylor <ca...@ab...>

About to rewrite To: 64....@dg...
Rewritten version is To: 64....@dg...

fetchmail: forwarding to /var/imap/socket/lmtp
fetchmail: LMTP> MAIL FROM:<ca...@ab...> SIZE=2852
fetchmail: LMTP< 250 2.1.0 ok
fetchmail: LMTP> RCPT TO:<da...@dg...>
fetchmail: LMTP< 250 2.1.5 ok
fetchmail: LMTP> DATA
fetchmail: LMTP< 354 go ahead
#******************************************.**************************fetchmail: 
message dgr...@po...:3 was not the expected length (2947 
actual != 2852 expected)
fetchmail: LMTP>. (EOM)
fetchmail: LMTP< 554 5.6.0 Message contains NUL characters
fetchmail: SMTP< 220 willow.dgreaves.com ESMTP Exim 4.20 Wed, 23 Mar 
2005 11:57:11 +0000
fetchmail: SMTP> HELO localhost
fetchmail: SMTP< 250 willow.dgreaves.com Hello 
[IbwiAqeOpcFJkOALoy6jCLP6WNxrOhWp] at localhost.localdomain [127.0.0.1]
fetchmail: SMTP> MAIL FROM:<>
fetchmail: SMTP< 250 OK
fetchmail: SMTP> RCPT TO:<da...@dg...>
fetchmail: SMTP< 250 Accepted
fetchmail: SMTP> DATA
fetchmail: SMTP< 354 Enter message, ending with "." on a line by itself
fetchmail: SMTP: (bounce-message body)
fetchmail: SMTP>. (EOM)
fetchmail: SMTP< 250 OK id=1DE4U3-0008M5-Lv
fetchmail: SMTP> QUIT
fetchmail: SMTP< 221 willow.dgreaves.com closing connection
 flushed
fetchmail: POP3> DELE 3
fetchmail: POP3< +OK Done.
fetchmail: POP3> LIST 4
fetchmail: POP3< +OK 4 3583
fetchmail: POP3> RETR 4
fetchmail: POP3< +OK Message follows
reading message dgr...@po...:4 of 22 (3583 octets)
About to rewrite Return-Path: <co...@gu...>
Rewritten version is Return-Path: <co...@gu...>

About to rewrite From: Colin Guthrie <co...@gu...>
Rewritten version is From: Colin Guthrie <co...@gu...>

About to rewrite To: David Greaves <da...@dg...>
Rewritten version is To: David Greaves <da...@dg...>

fetchmail: forwarding to /var/imap/socket/lmtp
fetchmail: SMTP> MAIL FROM:<co...@gu...> BODY=7BIT SIZE=3583
fetchmail: SMTP< 250 2.1.0 ok
fetchmail: SMTP> RCPT TO:<da...@dg...>
fetchmail: SMTP< 250 2.1.5 ok
fetchmail: SMTP> DATA
fetchmail: SMTP< 354 go ahead
#******************************.******************************fetchmail: 
message dgr...@po...:4 was not the expected length (3678 
actual != 3583 expected)
fetchmail: SMTP>. (EOM)
fetchmail: SMTP< 250 2.1.5 Ok
 
<at this point fetchmail hangs...>

fetchmail: smtp listener protocol error
 not flushed
fetchmail: POP3> LIST 5

<and the server go bored of waiting>

fetchmail: POP3< -ERR Client has been idle for too long.
fetchmail: Client has been idle for too long.
fetchmail: POP3> QUIT
fetchmail: client/server protocol error while fetching from pop3.ukfsn.org

Re: [fetchmail-devel] Assorted minor bugs

[Brian C. <B.C...@po...>]

> May I ask you to come up with a small patch that fixes _all_ violations
> of the trust model in a first round, so that we _unconditionally_ print
> all warnings, self-signed certificates, common name mismatches and so
> on. These _are_ important warnings, if they are shown only in verbose
> mode, that is a serious bug.

OK, that turns out to be pretty straightforward.

Note that when sslcertck is not set, our verify callback always returns 1
(OK) even if it given 0 (verify error). That's necessary to stop the SSL
connection being dropped, but it can result in the callback being called
multiple times, and and the same SSL error being displayed more than once.
For example, before fixing this, with a self-signed certificate I got:

fetchmail: Server certificate verification error: self signed certificate
fetchmail: Server certificate verification error: self signed certificate

I've fixed this by not displaying the same error message twice in
succession. I've also used the _depth0ck variable to make sure that the
once-off checking of CN and fingerprint really is done only once.

fetchmail -v still displays additional information from the certificate, as
it did before, but all errors should now be displayed even without -v.

Regards,

Brian.

Re: [fetchmail-devel] Assorted minor bugs

[Matthias A. <ma...@dt...>]

Brian Candler <B.C...@po...> writes:

> I've fixed this by not displaying the same error message twice in
> succession. I've also used the _depth0ck variable to make sure that the
> once-off checking of CN and fingerprint really is done only once.

Thanks a lot, your patch will be part of fetchmail 6.2.6. I tested it,
found it worked for me and then merged it. While I was at it, I found
that --sslfingerprint and --sslcertpath were missing from --help and
added that.

-- 
Matthias Andree

Re: [fetchmail-devel] Assorted minor bugs

[Brian C. <B.C...@po...>]

> A second step (longer-term than the next release), i. e. after the next
> official release, should then make strict checking the default and offer
> the user options to relax checking for the case when the configuration
> (client or server) cannot be fixed on short notice.

I agree, except that if you specify an explicit fingerprint that should also
make the certificate checking non-strict.

That is, if you *know* the public key of the end-point you're talking to,
then you don't care if it has been signed or by whom or when. With
--sslfingerprint you're effectively using the ssh model, rather than the ssl
model, for preventing man-in-the-middle attacks.

Regards,

Brian.

Re: [fetchmail-devel] Assorted minor bugs

[Matthias A. <ma...@dt...>]

Brian Candler <B.C...@po...> writes:

>> A second step (longer-term than the next release), i. e. after the next
>> official release, should then make strict checking the default and offer
>> the user options to relax checking for the case when the configuration
>> (client or server) cannot be fixed on short notice.
>
> I agree, except that if you specify an explicit fingerprint that should also
> make the certificate checking non-strict.

Right. We aren't currently doing that as I can see:

$ LANG=C fetchmail -d0 --sslcertck --sslcertpath /dev/null -v \
  --sslfingerprint 99:A9:55:D9:F5:51:F9:40:CC:A4:C6:26:A2:8E:46:14 XXXXXXX

fetchmail: 6.2.6 querying XXXXXXX (protocol POP3) at Wed Nov 10 01:41:02 2004: poll started
fetchmail: Issuer Organization: YYYYY
fetchmail: Issuer CommonName: Matthias Andree
fetchmail: Server CommonName: XXXXXXX
fetchmail: XXXXXXX key fingerprint: 99:A9:55:D9:F5:51:F9:40:CC:A4:C6:26:A2:8E:46:14
fetchmail: XXXXXXX fingerprints match.
fetchmail: Server certificate verification error: unable to get local issuer certificate
12402:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:842:
fetchmail: SSL connection failed.

> That is, if you *know* the public key of the end-point you're talking to,
> then you don't care if it has been signed or by whom or when. With
> --sslfingerprint you're effectively using the ssh model, rather than the ssl
> model, for preventing man-in-the-middle attacks.

Yup.

-- 
Matthias Andree

Re: [fetchmail-devel] [BUG] fetchmail hangs during pop3 pull after a mail with a null char

[Matthias A. <mat...@gm...>]

David Greaves <da...@dg...> wrote, in October 2004:

> It's been working for years with these occasional hangs that have been
> fixed by popping the bad messages and manually filing them. I finally
> had a bad message arrive when I was in a position to debug!
>
> Summary : fetchmail hangs during pop3 pull after a mail with a null char.
>
> The mail with a null char is pulled OK but then rejected by local
> Cyrus lmtp and bounced to postmaster via exim4.20
> The next pop3 pull then fails.
> I've made an effort to trace and I think the hang occurs due to a
> double call to SMTP_ok which is empty the second time. I am pretty
> sure the second call originates at sink.c line 1433.

David, could you try to reproduce the problem with fetchmail 6.3.2 and
see if the problem is gone?  The hang might have been related to LMTP
bugs or rather fetchmail 6.3.1 and older not properly talking SMTP any
more after having talked LMTP, so chances are the hang no longer occurs
with 6.3.2.

Thanks,

-- 
Matthias Andree

Re: [fetchmail-devel] [BUG] fetchmail hangs during pop3 pull after a mail with a null char

[David G. <da...@dg...>]

Matthias Andree wrote:

>David Greaves <da...@dg...> wrote, in October 2004:
>
>  
>
>>It's been working for years with these occasional hangs that have been
>>fixed by popping the bad messages and manually filing them. I finally
>>had a bad message arrive when I was in a position to debug!
>>
>>Summary : fetchmail hangs during pop3 pull after a mail with a null char.
>>
>>The mail with a null char is pulled OK but then rejected by local
>>Cyrus lmtp and bounced to postmaster via exim4.20
>>The next pop3 pull then fails.
>>I've made an effort to trace and I think the hang occurs due to a
>>double call to SMTP_ok which is empty the second time. I am pretty
>>sure the second call originates at sink.c line 1433.
>>    
>>
>
>David, could you try to reproduce the problem with fetchmail 6.3.2 and
>see if the problem is gone?  The hang might have been related to LMTP
>bugs or rather fetchmail 6.3.1 and older not properly talking SMTP any
>more after having talked LMTP, so chances are the hang no longer occurs
>with 6.3.2.
>
>Thanks,
>  
>
OK
I'm away for 2 weeks so I'll have to check when I return...

David

--