Menu
▾
▴
fetchmail-users
|
From: Jason W. <jas...@in...> - 2005-12-16 09:13:56
|
I initially raised this question on the fetchmail-friends list, but was wisely advised to test with 6.3.1-pre1 and bring the problem here. I am quite confident this is a protocol issue, but I don't know IMAP and Fetchmail well enough to debug it. For some reason, it appears that Fetchmail doesn't detect the final "OK" message from the server and simply times out after retrieving the headers. Using an MUA (e.g., muttng), I can successfully retrieve messages via IMAP from this server. For the moment, I have configured fetchmail to use pop3 instead as a work-around so I can read mail easily from this account. From syslog: Dec 16 18:40:02 jdc fetchmail[13415]: 6.3.1-pre1 querying mail.internode.on.net (protocol IMAP) at Fri Dec 16 18:40:02 2005: poll started Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< * OK internode.on.net bld-mail04 Ready Dec 16 18:40:03 jdc fetchmail[13415]: IMAP> A0001 CAPABILITY Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< * CAPABILITY IMAP4 IMAP4REV1 NAMESPACE QUOTA UIDPLUS IDLE XFLDDATA SURGEMAIL Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< A0001 OK CAPABILITY completed Dec 16 18:40:03 jdc fetchmail[13415]: Protocol identified as IMAP4 rev 1 Dec 16 18:40:03 jdc fetchmail[13415]: IMAP> A0002 LOGIN "jasonjgw" * Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< A0002 OK LOGIN completed Dec 16 18:40:03 jdc fetchmail[13415]: selecting or re-polling default folder Dec 16 18:40:03 jdc fetchmail[13415]: IMAP> A0003 SELECT "INBOX" Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< * 1 EXISTS Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< * 0 RECENT Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< * OK [UIDVALIDITY 1134522857] UIDs valid Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< * FLAGS (\Answered \Flagged \Deleted \Draft \Seen) Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Draft \Seen)] Limited Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< A0003 OK [READ-WRITE] SELECT completed Dec 16 18:40:03 jdc fetchmail[13415]: 1 message waiting after first poll Dec 16 18:40:03 jdc fetchmail[13415]: IMAP> A0004 EXPUNGE Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< A0004 OK EXPUNGE completed Dec 16 18:40:03 jdc fetchmail[13415]: 1 message waiting after expunge Dec 16 18:40:03 jdc fetchmail[13415]: 1 message for jasonjgw at mail.internode.on.net. Dec 16 18:40:03 jdc fetchmail[13415]: IMAP> A0005 FETCH 1 RFC822.SIZE Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< * 1 FETCH (UID 16 RFC822.SIZE 1447) Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< A0005 OK FETCH completed Dec 16 18:40:03 jdc fetchmail[13415]: IMAP> A0006 FETCH 1 RFC822.HEADER Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< * 1 FETCH (UID 16 RFC822.HEADER {1360} Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< Received: from beatrice.nipl.net (unverified [62.94.93.142]) Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< ^Iby mail.internode.on.net (SurgeMail 3.2f) with ESMTP id 4426396 Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< ^Ifor <jas...@in...>; Fri, 16 Dec 2005 18:09:13 +1030 (CDT) Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< Return-Path: <ja...@ni...> Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< Received: from localhost (localhost.localdomain [127.0.0.1]) Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< ^Iby beatrice.nipl.net (Postfix) with ESMTP id 0336EE8624 Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< ^Ifor <jas...@in...>; Fri, 16 Dec 2005 07:39:04 +0000 (UTC) Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< Received: from beatrice.nipl.net ([127.0.0.1]) Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< ^Iby localhost (beatrice.nipl.net [127.0.0.1]) (amavisd-new, port 10024) Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< ^Iwith ESMTP id 01175-07 for <jas...@in...>; Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< ^IFri, 16 Dec 2005 07:39:01 +0000 (UTC) Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< Received: by beatrice.nipl.net (Postfix, from userid 1010) Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< ^Iid 79CEBE8644; Fri, 16 Dec 2005 07:39:01 +0000 (UTC) Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< Date: Fri, 16 Dec 2005 18:39:00 +1100 Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< From: Jason White <ja...@ni...> Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< To: jas...@in... Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< Subject: Fetchmail test for 6.3.1 Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< Message-ID: <200...@ni...> Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< Mime-Version: 1.0 Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< Content-Type: text/plain; charset=us-ascii Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< Content-Disposition: inline Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< User-Agent: Mutt/1.5.9i Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< X-Virus-Scanned: by amavisd-new-20030616-p10 (Debian) at nipl.net Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< X-Rcpt-To: <jas...@in...> Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< X-Vpipe: Scanner said clean (/usr/local/clamav/sbin/vscand-clamav) Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< X-NotAscii: charset=us-ascii Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< X-IP-stats: Incoming Last 0, First 0, in=1, out=0, spam=0 Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< X-External-IP: 62.94.93.142 Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< ) Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< A0006 OK FETCH completed Dec 16 18:40:33 jdc fetchmail[13415]: timeout after 30 seconds waiting for server mail.internode.on.net. Dec 16 18:40:33 jdc fetchmail[13415]: socket error while fetching from jas...@ma... Dec 16 18:40:33 jdc fetchmail[13415]: 6.3.1-pre1 querying mail.internode.on.net (protocol IMAP) at Fri Dec 16 18:40:33 2005: poll completed Dec 16 18:40:33 jdc fetchmail[13415]: Query status=2 (SOCKET) Dec 16 18:40:33 jdc fetchmail[13415]: Deleting fetchids file. Dec 16 18:40:33 jdc fetchmail[13415]: sleeping at Fri Dec 16 18:40:33 2005 Configuration: # global options set daemon 60 set syslog # Server options (mail.internode.on.net) poll mail.internode.on.net protocol imap timeout 30 # User options (internode.on.net) username jasonjgw is jason here password secret fetchall |
|
From: Matthias A. <mat...@gm...> - 2005-12-16 17:48:11
|
Jason White <jas...@in...> writes:
> Dec 16 18:40:03 jdc fetchmail[13415]: IMAP> A0006 FETCH 1 RFC822.HEADER
> Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< * 1 FETCH (UID 16
> RFC822.HEADER {1360}
Well, this tells fetchmail that 1360 bytes (octets) will
follow. Since this works for me with a Dovecot upstream server, I wonder
if it's really fetchmail's fault or perhaps SurgeMail's.
Could I possibly get a test mail account on that server to find out
which software is wrong? If so, please send the IMAP server, mail
address (where I would send test messages), account and password or
other credentials off-list. The X-PGP-Key header mentions my GnuPG key
so encrypted communication is possible (and should be used).
If that is not possible, can you change your password, run ethereal,
tethereal or tcpdump -s5000 to capture to a file (the tcpdump filter
rule would be "host mail.internode.on.net and port 143"), run fetchmail
once, then change your password again and mail me the resulting tcpdump
or tethereal capture file? It contains the password, hence the hassle
with changing it. Looking at the tcpdump/tethereal dump should also give
us an idea which of the two sides miscounted the message size.
> Dec 16 18:40:33 jdc fetchmail[13415]: timeout after 30 seconds waiting for server mail.internode.on.net.
> Dec 16 18:40:33 jdc fetchmail[13415]: socket error while fetching from jas...@ma...
Thank you for retrying with 6.3.1-pre1 and reporting your problem.
--
Matthias Andree
|
|
From: Sunil S. <sh...@bo...> - 2005-12-16 17:59:21
|
Quoting from Jason White's mail on Fri, Dec 16, 2005:
> I am quite confident this is a protocol issue, but I don't know IMAP and
> Fetchmail well enough to debug it. For some reason, it appears that Fetchmail
> doesn't detect the final "OK" message from the server and simply times out
> after retrieving the headers. Using an MUA (e.g., muttng), I can successfully
> retrieve messages via IMAP from this server. For the moment, I have configured
> fetchmail to use pop3 instead as a work-around so I can read mail easily from
> this account.
Your IMAP server is also sending UID information in the response. It
needs to be checked if sending UIDs in response is valid or not.
> From syslog:
...
> Dec 16 18:40:03 jdc fetchmail[13415]: IMAP> A0005 FETCH 1 RFC822.SIZE
> Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< * 1 FETCH (UID 16 RFC822.SIZE 1447)
^^^^^^
> Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< A0005 OK FETCH completed
> Dec 16 18:40:03 jdc fetchmail[13415]: IMAP> A0006 FETCH 1 RFC822.HEADER
> Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< * 1 FETCH (UID 16 RFC822.HEADER {1360}
^^^^^^
...
> Dec 16 18:40:03 jdc fetchmail[13415]: IMAP< A0006 OK FETCH completed
fetchmail is not expecting UIDs in the response. Hence the protocol
error. Could you try this patch and tell me if it works for you? Note
that you may get more errors if UIDs are sent in further responses
(like RFC822.TEXT) which have not been reached yet.
Here's the patch:
Index: fetchmail/imap.c
===================================================================
--- fetchmail/imap.c (revision 4550)
+++ fetchmail/imap.c (working copy)
@@ -880,8 +880,18 @@
if (num >= first && num <= last)
sizes[num - first] = size;
else
- report(stderr, "Warning: ignoring bogus data for message sizes returned by the server.\n");
+ report(stderr,
+ GT_("Warning: ignoring bogus data for message sizes returned by the server.\n"));
}
+ /* some servers return UID information here */
+ else if (sscanf(buf, "* %u FETCH (UID %*s RFC822.SIZE %u)", &num, &size) == 2)
+ {
+ if (num >= first && num <= last)
+ sizes[num - first] = size;
+ else
+ report(stderr,
+ GT_("Warning: ignoring bogus data for message sizes returned by the server.\n"));
+ }
}
return(PS_SUCCESS);
@@ -948,8 +958,11 @@
if ((ok = gen_recv(sock, buf, sizeof(buf))))
return(ok);
ptr = skip_token(buf); /* either "* " or "AXXXX " */
- if (sscanf(ptr, "%d FETCH (%*s {%d}", &num, lenp) == 2)
+ if (sscanf(ptr, "%d FETCH (RFC822.HEADER {%d}", &num, lenp) == 2)
break;
+ /* some servers return UID information here */
+ else if (sscanf(ptr, "%d FETCH (UID %*s RFC822.HEADER {%d}", &num, lenp) == 2)
+ break;
/* try to recover from chronically fucked-up M$ Exchange servers */
else if (!strncmp(ptr, "NO", 2))
{
--
Sunil Shetye.
|
|
From: Jason W. <jas...@in...> - 2005-12-17 00:59:34
|
On Fri, Dec 16, 2005 at 05:01:58PM +0530, Sunil Shetye wrote:
> Your IMAP server is also sending UID information in the response. It
> needs to be checked if sending UIDs in response is valid or not.
Yes, I was wondering whether the server was conforming to the RFC.
>
> fetchmail is not expecting UIDs in the response. Hence the protocol
> error. Could you try this patch and tell me if it works for you? Note
> that you may get more errors if UIDs are sent in further responses
> (like RFC822.TEXT) which have not been reached yet.
With the patch applied, Fetchmail read and delivered the entire message. Here
is the log of the body portion:
Dec 17 10:38:26 jdc fetchmail[4404]: IMAP> A0007 FETCH 1 BODY.PEEK[TEXT]
Dec 17 10:38:26 jdc fetchmail[4404]: IMAP< * 1 FETCH (UID 17 BODY[TEXT] {94}
Dec 17 10:38:26 jdc fetchmail[4404]: (94 body octets)
Dec 17 10:38:26 jdc fetchmail[4404]: IMAP< )
Dec 17 10:38:26 jdc fetchmail[4404]: IMAP< A0007 OK FETCH completed
I can carry out whatever further testing you consider appropriate, either to
this or to any generalized version of the patch that you decide to create.
I haven't tried messages with attachments yet, but thought I should report the
initial results.
|
|
From: Sunil S. <sh...@bo...> - 2005-12-17 07:09:38
|
Quoting from Jason White's mail on Sat, Dec 17, 2005:
> On Fri, Dec 16, 2005 at 05:01:58PM +0530, Sunil Shetye wrote:
>
> > Your IMAP server is also sending UID information in the response. It
> > needs to be checked if sending UIDs in response is valid or not.
> Yes, I was wondering whether the server was conforming to the RFC.
> >
> > fetchmail is not expecting UIDs in the response. Hence the protocol
> > error. Could you try this patch and tell me if it works for you? Note
> > that you may get more errors if UIDs are sent in further responses
> > (like RFC822.TEXT) which have not been reached yet.
> With the patch applied, Fetchmail read and delivered the entire message. Here
> is the log of the body portion:
> Dec 17 10:38:26 jdc fetchmail[4404]: IMAP> A0007 FETCH 1 BODY.PEEK[TEXT]
> Dec 17 10:38:26 jdc fetchmail[4404]: IMAP< * 1 FETCH (UID 17 BODY[TEXT] {94}
> Dec 17 10:38:26 jdc fetchmail[4404]: (94 body octets)
> Dec 17 10:38:26 jdc fetchmail[4404]: IMAP< )
>
> Dec 17 10:38:26 jdc fetchmail[4404]: IMAP< A0007 OK FETCH completed
>
> I can carry out whatever further testing you consider appropriate, either to
> this or to any generalized version of the patch that you decide to create.
> I haven't tried messages with attachments yet, but thought I should report the
> initial results.
Thanks for your update. It looks like the code reading the response to
BODY[TEXT] is far more liberal. So, no further patches and tests are
required.
Here is the same patch with comments updated.
Matthias, please consider this patch for 6.3.1.
Index: fetchmail/imap.c
===================================================================
--- fetchmail/imap.c (revision 4550)
+++ fetchmail/imap.c (working copy)
@@ -880,8 +880,24 @@
if (num >= first && num <= last)
sizes[num - first] = size;
else
- report(stderr, "Warning: ignoring bogus data for message sizes returned by the server.\n");
+ report(stderr,
+ GT_("Warning: ignoring bogus data for message sizes returned by the server.\n"));
}
+ /* some servers (like mail.internode.on.net bld-mail04) return UID information here
+ *
+ * IMAP> A0005 FETCH 1 RFC822.SIZE
+ * IMAP< * 1 FETCH (UID 16 RFC822.SIZE 1447)
+ * IMAP< A0005 OK FETCH completed
+ *
+ */
+ else if (sscanf(buf, "* %u FETCH (UID %*s RFC822.SIZE %u)", &num, &size) == 2)
+ {
+ if (num >= first && num <= last)
+ sizes[num - first] = size;
+ else
+ report(stderr,
+ GT_("Warning: ignoring bogus data for message sizes returned by the server.\n"));
+ }
}
return(PS_SUCCESS);
@@ -948,8 +964,19 @@
if ((ok = gen_recv(sock, buf, sizeof(buf))))
return(ok);
ptr = skip_token(buf); /* either "* " or "AXXXX " */
- if (sscanf(ptr, "%d FETCH (%*s {%d}", &num, lenp) == 2)
+ if (sscanf(ptr, "%d FETCH (RFC822.HEADER {%d}", &num, lenp) == 2)
break;
+ /* some servers (like mail.internode.on.net bld-mail04) return UID information here
+ *
+ * IMAP> A0006 FETCH 1 RFC822.HEADER
+ * IMAP< * 1 FETCH (UID 16 RFC822.HEADER {1360}
+ * ...
+ * IMAP< )
+ * IMAP< A0006 OK FETCH completed
+ *
+ */
+ else if (sscanf(ptr, "%d FETCH (UID %*s RFC822.HEADER {%d}", &num, lenp) == 2)
+ break;
/* try to recover from chronically fucked-up M$ Exchange servers */
else if (!strncmp(ptr, "NO", 2))
{
--
Sunil Shetye.
|
|
From: Matthias A. <mat...@gm...> - 2005-12-17 15:52:34
|
Sunil Shetye <sh...@bo...> writes:
> Matthias, please consider this patch for 6.3.1.
I wonder if it's the right thing to do, or if the server of the day
might return the flags or some other information along with the size. I
had a quick glance at the standard, and I'm not at all sure if it's
allowed for the server to return excess information.
At any case, if a server does that, we're dead in the water again. I'd
rather use a bit more of a parser to take the fetch response apart.
If we choose not to do that, the patch can be simplified and use this
structure:
if (sscanf(s, "pattern", ...) == 2
|| sscanf(s, "other pattern", ...) == 2)
{
do whatever is appropriate
}
What do you think?
--
Matthias Andree
|
|
From: Jason W. <jas...@in...> - 2005-12-18 00:57:12
|
On Sat, Dec 17, 2005 at 03:52:30PM +0100, Matthias Andree wrote: > Sunil Shetye <sh...@bo...> writes: > > > Matthias, please consider this patch for 6.3.1. > > I wonder if it's the right thing to do, or if the server of the day > might return the flags or some other information along with the size. I > had a quick glance at the standard, and I'm not at all sure if it's > allowed for the server to return excess information. There is a statemenn section 2.2.2, but I am not sure whether it is meant to cover this case: " A client MUST be prepared to accept any server response at all times. This includes server data that was not requested. Server data SHOULD be recorded, so that the client can reference its recorded copy rather than sending a command to the server to request the data. In the case of certain server data, the data MUST be recorded. This topic is discussed in greater detail in the Server Responses section." I couldn't find any more specific guidance in section 7, but then I didn't read it carefully. Also, this could be read as implying only to entire responses and not to additional data included within responses that are requested. Note also that in Fetchmail, imap_fetch_body() loops until it finds the required response. > > At any case, if a server does that, we're dead in the water again. I'd > rather use a bit more of a parser to take the fetch response apart. I agree with this and would be interested in an interpretation of the RFC as well regarding whether this is expected of the client. At least it might be considered good practice; Fetchmail already contains code to deal with the eccentricities of various servers and it wouldn't hurt to generalize it here if you consider it appropriate to do so. |
|
From: Matthias A. <mat...@gm...> - 2005-12-18 11:31:54
|
Jason White <jas...@in...> writes: > On Sat, Dec 17, 2005 at 03:52:30PM +0100, Matthias Andree wrote: >> Sunil Shetye <sh...@bo...> writes: >> >> > Matthias, please consider this patch for 6.3.1. >> >> I wonder if it's the right thing to do, or if the server of the day >> might return the flags or some other information along with the size. I >> had a quick glance at the standard, and I'm not at all sure if it's >> allowed for the server to return excess information. > There is a statemenn section 2.2.2, but I am not sure whether it is meant to > cover this case: > " A client MUST be prepared to accept any server response at all times. > This includes server data that was not requested. Server data SHOULD > be recorded, so that the client can reference its recorded copy > rather than sending a command to the server to request the data. In > the case of certain server data, the data MUST be recorded. > > This topic is discussed in greater detail in the Server Responses > section." > > I couldn't find any more specific guidance in section 7, but then I didn't > read it carefully. Also, this could be read as implying only to entire > responses and not to additional data included within responses that are > requested. Well, I'll just use an equivalent modification of Sunil's patch for 6.3.1, because, after looking at the respective code, the entire IMAP response parser must be rewritten - for instance, to support the ALERT response code or * BYE. And this rewrite is stuff for 6.4.0. We'll just use the minimal change that fixes the problem in 6.3.1. I'll release 6.3.1-rc1 later, and I'd ask you to test again so that my changes haven't broken the fix. -- Matthias Andree |
|
From: Jason W. <jas...@in...> - 2005-12-19 04:26:48
|
On Sun, Dec 18, 2005 at 11:31:49AM +0100, Matthias Andree wrote: > Well, I'll just use an equivalent modification of Sunil's patch for > 6.3.1, because, after looking at the respective code, the entire IMAP > response parser must be rewritten - for instance, to support the ALERT > response code or * BYE. And this rewrite is stuff for 6.4.0. I'll volunteer to test it when the new parser is implemented. > > We'll just use the minimal change that fixes the problem in 6.3.1. I'll > release 6.3.1-rc1 later, and I'd ask you to test again so that my > changes haven't broken the fix. > The fix still works in 6.3.1-rc1. |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X