fetchmail-users — User questions and discussion

Re: [fetchmail-users] Re: RETR with UIDL

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> runs:
>
>     fetchmail -a -d 0 ; fetchmail
>
> to download all mails once and then start normal polling. New mails
> which have been downloaded by the first instance of fetchmail will not
> be downloaded again by the second instance due to this patch.

Ah, so it still has a use and fixes duplicate downloads.
I'll reword my NEWS entry and merge the patch. Thanks.

-- 
Matthias Andree

Re: [fetchmail-users] Re: RETR with UIDL

[Matthias A. <mat...@gm...>]

Jakob Hirsch <jh...@pl...> writes:

> Anyway, I really think fetchmail should by default simply retrieve all
> message, using standard MUA mechanisms: RETR and DELE; no TOP, UIDL or
> LAST. This makes sure that fetchmail works with as much servers as
> possible without any tweaking.

UIDL is a pretty standard MUA (used by major MUAs) mechanism for "keep
messages on server" setups, and it's very likely I'll make UIDL default
in future versions.

I hope we can also add the often-requested "delete after N days" that
ESR refused (probably because he was scared of the implications with the
long-winding UID code), but I'm not sure if time permits that for 6.4.X.

> If one wants 'keep', UIDL/RETR should be used, with fallback to
>LAST/RETR. And there should probably be an option to force one of UIDL
>or LAST (uidl/nouidl?) so the user can enfore client or server side
>tracking (with fetchmail bailing out if the requested option is not
>available on the server). The magic fetchmail does is nice and works in
>most cases, but final control should be left to the user.

The user should not need to care about protocol details before he runs
into a conflict so serious that fetchmail cannot resolve it on its own.

I share the view on RETR becoming default.

Server-side tracking is a constant source of troubles with respect to
interrupted connections, software faults, aborts, reboots, any other
things, and relying on POP3 server state beyond UIDL just doesn't work
well, so everything will be switched to client-side tracking.

I also think I'll be discontinuing obsolete commands such as "RPOP"
(insecure, removed with RFC-1460 in 1993), "LAST" (removed with RFC-1725
in 1994, inconsistent behavior with RSET) in 6.4.X.  fetchmail should be
talking RFC-1939 (POP3)/1734 (AUTH)/2449 (TLS)/2222 (SASL).

Massively slashing old protocols may cause me to call the next release
7.0.0 rather than 6.4.0 so people will be warned it's a major
upgrade. The exact roadmap for the next fetchmail release isn't fixed yet.

-- 
Matthias Andree

[fetchmail-users] Re: RETR with UIDL

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Sat, Mar 04, 2006:
> Hm - how will this avoid redownloads of messages when the line is faulty
> and the connection is torn down?
> 
> It will recognize it's downloading seen messages, but download them all
> the same.

The only practical use of this patch is if a user with this
configuration:

    set daemon 300
    poll server
	uidl
	no fetchall
	keep

runs:

    fetchmail -a -d 0 ; fetchmail

to download all mails once and then start normal polling. New mails
which have been downloaded by the first instance of fetchmail will not
be downloaded again by the second instance due to this patch.

Of course, an equivalent effect may be achieved by removing all the
relevant entries from the fetchids file or deleting the fetchids file.

This patch has no relevance to our basic topic under discussion.

-- 
Sunil Shetye.

Re: [fetchmail-users] Re: RETR with UIDL

[Jakob H. <jh...@pl...>]

Sunil Shetye wrote:

>> | clean up TOP vs. RETR but leave TOP as an option for the server scum
>> | that deletes message right after RETR, voiding client retries (Jakob
>> | Hirsch, 2006-01-03).

Note that the server i was referring to (MercuryP/NLM, version 1.48)
only deletes RETRieved mail after a QUIT, thereby not really violating
the RFC.

> Are these "scum" servers supporting "uidl"? Please note that the

Yes. But I can only speak for the server mentioned above.

Anyway, I really think fetchmail should by default simply retrieve all
message, using standard MUA mechanisms: RETR and DELE; no TOP, UIDL or
LAST. This makes sure that fetchmail works with as much servers as
possible without any tweaking.

If one wants 'keep', UIDL/RETR should be used, with fallback to
LAST/RETR. And there should probably be an option to force one of UIDL
or LAST (uidl/nouidl?) so the user can enfore client or server side
tracking (with fetchmail bailing out if the requested option is not
available on the server). The magic fetchmail does is nice and works in
most cases, but final control should be left to the user.

As for TOP, there are legit uses, so there should also an option to
enable it explicitly.

Oh, and all of this should be documented as comprehensible as possible,
of course. Anybody with loads of free time on the hand? :)

But as Matthias said, all of this is surely nothing for 6.3.

Re: [fetchmail-users] [PATCH] Re: RETR with UIDL

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Matthias Andree's mail on Fri, Mar 03, 2006:
>> > The advice being given to force fetchmail to avoid using "TOP" with
>> > these buggy servers is to enable the "fetchall" option. The FAQ is
>> > littered with this advice for various servers with variety of
>> > problems. The problems with "fetchall" option are:
>> >
>> > - "fetchall" cannot be used with "uidl"
>> 
>> Well, we can fix eventual incompatibilities between these two options
>> any time. 6.3.X is open for bug fixes.
>
> Here is a patch for this.

Hm - how will this avoid redownloads of messages when the line is faulty
and the connection is torn down?

It will recognize it's downloading seen messages, but download them all
the same.

-- 
Matthias Andree

[fetchmail-users] [PATCH] Re: RETR with UIDL

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Fri, Mar 03, 2006:
> > The advice being given to force fetchmail to avoid using "TOP" with
> > these buggy servers is to enable the "fetchall" option. The FAQ is
> > littered with this advice for various servers with variety of
> > problems. The problems with "fetchall" option are:
> >
> > - "fetchall" cannot be used with "uidl"
> 
> Well, we can fix eventual incompatibilities between these two options
> any time. 6.3.X is open for bug fixes.

Here is a patch for this.

================================================================
Index: fetchmail-6.3/pop3.c
===================================================================
--- fetchmail-6.3/pop3.c	(revision 4719)
+++ fetchmail-6.3/pop3.c	(working copy)
@@ -894,7 +894,7 @@
      */
     last = 0;
     *newp = -1;
-    if (*countp > 0 && !ctl->fetchall)
+    if (*countp > 0 && (!ctl->fetchall || ctl->server.uidl))
     {
 	int fastuidl;
 	char id [IDLEN+1];
@@ -902,6 +902,7 @@
 	/* should we do fast uidl this time? */
 	fastuidl = ctl->fastuidl;
 	if (*countp > 7 &&		/* linear search is better if there are few mails! */
+	    !ctl->fetchall &&		/* with fetchall, all uids are required */
 	    !ctl->flush &&		/* with flush, it is safer to disable fastuidl */
 	    NUM_NONZERO (fastuidl))
 	{
================================================================

> The best option would be to use UIDL. Still we'll not be forcing UIDL
> downloads to use RETR in 6.3.X. We can safely do that in 6.4.X: the
> 6.3.X manpage already has warnings in place.

...

> > I don't get this part. If the pop3 server is supporting UIDL and
> > fetchmail is running with the "uidl" option, why bother about the LAST
> > pointer and message flags?
> 
> Why bother? Because I promised I'd stop the ESR madness of gold versions
> and promised 6.3.X would be compatible with 6.3.0 when I released 6.3.0.

Ok.

-- 
Sunil Shetye.

Re: [fetchmail-users] Re: RETR with UIDL

[Rob F. <rf...@fu...>]

Matthias Andree wrote:
> Let's do 6.3.3 (thanks for your efforts in getting the IMAP NOOP bugs
> nailed BTW), and then fork 6.4.X and make all interesting changes and
> cleanups in particular in 6.4.X.

FWIW, I tend to agree with Matthias here.
Is it time for the long-awaited UIDL revamp yet?  :-)

-- 
==============================|   "A microscope locked in on one point
 Rob Funk <rf...@fu...> |Never sees what kind of room that it's in"
 http://www.funknet.net/rfunk |    -- Chris Mars, "Stuck in Rewind"

Re: [fetchmail-users] Re: RETR with UIDL

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Matthias Andree's mail on Fri, Mar 03, 2006:
>> > Please note that the change I am recommending is that fetchmail
>> > *should* use RETR with pop3+uidl servers. Other pop3 servers will not
>> > be affected by this change.
>> 
>> Yes, it's still 6.4.X stuff since it deliberately changes behavior for
>> no good reason.
>
> There is a good reason.
>
> The advice being given to force fetchmail to avoid using "TOP" with
> these buggy servers is to enable the "fetchall" option. The FAQ is
> littered with this advice for various servers with variety of
> problems. The problems with "fetchall" option are:
>
> - "fetchall" cannot be used with "uidl"

Well, we can fix eventual incompatibilities between these two options
any time. 6.3.X is open for bug fixes.

> - "fetchall" causes mails to be redownloaded if there is a line-hit
> while fetchmail is downloading mails. If there are consecutive
> line-hits, the same set of mails could be downloaded any number of
> times.

Yes, true. So use just uidl without fetchall on flakey lines. There is
little reason to add fetchall to uidl. We can decouple TOP/RETR from
fetchall in 6.4.X and default to RETR if we want. Not in 6.3.X.

Let's do 6.3.3 (thanks for your efforts in getting the IMAP NOOP bugs
nailed BTW), and then fork 6.4.X and make all interesting changes and
cleanups in particular in 6.4.X.

> To avoid the redownloading of mails, one has to now probably add the
> "expunge" option also. Finding the best value for "expunge" is a
> problem itself:
>
> - "expunge 1" is expensive. It will reconnect for every mail. Many
>   mailservers may not like this too. However, each mail will be
>   downloaded only once.
> - "expunge 2" is less expensive. Mailservers may still object to this.
>   Very few mails will be redownloaded.
> - "expunge 20" is relatively cheap. Mailservers may not object to
>   this. Some mails will get redownloaded. However, on a bad day, if
>   you do get consecutive line-hits, you will still end up with
>   multiple copies of mails.

The best option would be to use UIDL. Still we'll not be forcing UIDL
downloads to use RETR in 6.3.X. We can safely do that in 6.4.X: the
6.3.X manpage already has warnings in place.

>> fetchmail 6.3.2 already introduced the (incompatible) "use RETR with
>> Maillennium server" change because it was considered we'd rather screw
>> some message flags and a LAST pointer rather than truncate messages.
>
> I don't get this part. If the pop3 server is supporting UIDL and
> fetchmail is running with the "uidl" option, why bother about the LAST
> pointer and message flags?

Why bother? Because I promised I'd stop the ESR madness of gold versions
and promised 6.3.X would be compatible with 6.3.0 when I released 6.3.0.

> And, as has already been said, most standard interactive clients use
> "uidl" anyway. So, they also will not get affected.

This is not a valid reason to change behavior in incompatible ways in
the midst of a stable version. I'd wondered if I'd wanted to add the
Maillennium workaround in 6.3.X at all or delay it until 6.4.X. There'd
been loads of reasons to defer it, not the least being a documented
workaround.

> Jakob, please give more details about the Mercury server:
>
> - its signature line
> - its CAPA output
> - whether it supports UIDL
> - whether you are using it with "uidl"
> and finally,
> - your opinion on whether fetchmail should use "RETR" with "uidl".

I'm not accepting changes into 6.3.X that switch TOP to RETR except if
they fix security bugs, data corruption or other critical issues, so
please don't spend your time before we've opened 6.4.X for hacking.

Priorities for incompatible changes are:

1. Security
2. Critical bugs (data loss, crash, similar)
3. Compatibility within stable releases
4. Protocol compliance
5. Consistency
6. Anything else

-- 
Matthias Andree

[fetchmail-users] Re: RETR with UIDL

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Fri, Mar 03, 2006:
> > Please note that the change I am recommending is that fetchmail
> > *should* use RETR with pop3+uidl servers. Other pop3 servers will not
> > be affected by this change.
> 
> Yes, it's still 6.4.X stuff since it deliberately changes behavior for
> no good reason.

There is a good reason.

The advice being given to force fetchmail to avoid using "TOP" with
these buggy servers is to enable the "fetchall" option. The FAQ is
littered with this advice for various servers with variety of
problems. The problems with "fetchall" option are:

- "fetchall" cannot be used with "uidl"
- "fetchall" causes mails to be redownloaded if there is a line-hit
  while fetchmail is downloading mails. If there are consecutive
  line-hits, the same set of mails could be downloaded any number of
  times.

To avoid the redownloading of mails, one has to now probably add the
"expunge" option also. Finding the best value for "expunge" is a
problem itself:

- "expunge 1" is expensive. It will reconnect for every mail. Many
  mailservers may not like this too. However, each mail will be
  downloaded only once.
- "expunge 2" is less expensive. Mailservers may still object to this.
  Very few mails will be redownloaded.
- "expunge 20" is relatively cheap. Mailservers may not object to
  this. Some mails will get redownloaded. However, on a bad day, if
  you do get consecutive line-hits, you will still end up with
  multiple copies of mails.

In short, the user has to go for some sort of experimentation and use
a sub-optimal solution to get the mails(*).

The best solution (wherever possible) is to use "uidl" as mails get
downloaded exactly once. To use "uidl", one has to use "no fetchall".
With "no fetchall", fetchmail refuses to use "RETR".

> fetchmail 6.3.2 already introduced the (incompatible) "use RETR with
> Maillennium server" change because it was considered we'd rather screw
> some message flags and a LAST pointer rather than truncate messages.

I don't get this part. If the pop3 server is supporting UIDL and
fetchmail is running with the "uidl" option, why bother about the LAST
pointer and message flags? After all, even if the user is running
multiple fetchmails, all those fetchmails would be running with the
same configuration.

And, as has already been said, most standard interactive clients use
"uidl" anyway. So, they also will not get affected.

> Sort of. At least the ones that were reported do not delete
> messages if the connection is hung up without prior "QUIT" command.
> <http://lists.berlios.de/pipermail/fetchmail-users/2006-January/000220.html>

Jakob, please give more details about the Mercury server:

- its signature line
- its CAPA output
- whether it supports UIDL
- whether you are using it with "uidl"
and finally,
- your opinion on whether fetchmail should use "RETR" with "uidl".

-- 
Sunil Shetye.

(*) This need for experimentation is not documented now. Currently,
there is no entry in FAQ which says that if you are using "fetchall",
you should necessarily use say "expunge 10". FAQ O9 does talk of
"fetchall" and "expunge". But, it seems to be bad-mouthing "uidl" and
switching to IMAP4 is given as a better option than using "expunge".
However, this is a documentation issue and not related to the main
topic.

[fetchmail-users] [PATCH] Re: Bug in imap_idle() with mail servers not supporting "idle"

[Sunil S. <sh...@bo...>]

Quoting from Brendan Lynch's mail on Thu, Mar 02, 2006:
> However I believe the servers is behaving correctly according to the 
> IMAP spec: I have attached an explanation of the secondary problem I am 
> seeing (extra 28 s wait) that I sent to Casper; let me know if you need 
> more info.

Yes, I have now read RFC 3501 section 5.3. This behaviour is correct.

> The key point is that the imap_idle() code when 'faking' IDLE support 
> calls imap_ok() (at line 637) without having issued any tagged command:

...

> Since there is no tagged command outstanding the server will not (and 
> should not) ever send a tagged response.  If we set imap_ok to only 
> return after a tagged response we are guaranteeing it will always only 
> return after a timeout.

...

> This again first does a gen_transact() which results in sending "A0130 
> NOOP" at 17:00:49.  A "RECENT" update is already on its way form the 
> server, so we receive it before the "A0130 NOOP completed" at 17:00:49.  
> In imap_ok() we first process the "* 1 RECENT" notification at line 117, 
> setting recentcount, then (since we ARE waiting a tag) loop and process 
> the NOOP ok.  we then return to gen_transact(), which in turn returns to 
> imap_idle().
> 
> Since setting "recentcount" does not also change the stage, we fail the 
> test at line 633 of imap_idle() and continue to do an extra imap_ok() 
> which sets a 28 second timer at 17:00:49 and does a recv() which times 
> out at 17:01:17.  We then return from imap_idle() to imap_getrange() 
> line 679.  At this point recentcount is set and we exit the "while 
> (recentcount == 0 && do_idle)" loop.
> 
> The wait between 17:00:49 and 17:01:17 was unnecessary, as already had 
> our RECENT notification.  The second part of my fix causes imap_ok() to 
> change the stage to STAGE_FETCH on receipt of a RECENT notification, 
> just as it already did on receipt of a EXISTS notification. This causes 
> the if condition "if (ok != 0 || stage != STAGE_IDLE)" to be met at line 
> 633 of imap_idle() so it immediately exits and does not do the extra 
> unneeded imap_ok() call.

Yes, you have diagnosed the problem correctly and your patch is also
correct. Based on your patch, I have prepared a new patch which does
not modify imap_ok(), but cleans up the code in imap_idle(). Could you
test this patch and see if this patch is acceptable to you? This patch
is identical to the patch I had sent to Casper Gripenberg, except that
comments relating to compliance with RFCs have been updated.

-- 
Sunil Shetye.

Re: [fetchmail-users] Re: RETR with UIDL

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Matthias Andree's mail on Fri, Mar 03, 2006:
>> The TODO in SVN's trunk directory already mentions:
>> 
>> | clean up TOP vs. RETR but leave TOP as an option for the server scum
>> | that deletes message right after RETR, voiding client retries (Jakob
>> | Hirsch, 2006-01-03).
>
> Are these "scum" servers supporting "uidl"?

I don't know, but we need to assume they do since that's the worst case.

> Please note that the change I am recommending is that fetchmail
> *should* use RETR with pop3+uidl servers. Other pop3 servers will not
> be affected by this change.

Yes, it's still 6.4.X stuff since it deliberately changes behavior for
no good reason.

fetchmail 6.3.2 already introduced the (incompatible) "use RETR with
Maillennium server" change because it was considered we'd rather screw
some message flags and a LAST pointer rather than truncate messages.

> Are there any logs/bug reports about these "scum" servers?

Sort of. At least the ones that were reported do not delete
messages if the connection is hung up without prior "QUIT" command.
<http://lists.berlios.de/pipermail/fetchmail-users/2006-January/000220.html>

-- 
Matthias Andree

[fetchmail-users] Re: RETR with UIDL

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Fri, Mar 03, 2006:
> The TODO in SVN's trunk directory already mentions:
> 
> | clean up TOP vs. RETR but leave TOP as an option for the server scum
> | that deletes message right after RETR, voiding client retries (Jakob
> | Hirsch, 2006-01-03).

Are these "scum" servers supporting "uidl"? Please note that the
change I am recommending is that fetchmail *should* use RETR with
pop3+uidl servers. Other pop3 servers will not be affected by this
change.

Are there any logs/bug reports about these "scum" servers?

-- 
Sunil Shetye.

Re: [fetchmail-users] RETR with UIDL

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Jakob Hirsch's mail on Thu, Mar 02, 2006:
>> The difference is simply that fetchmail (ab)uses TOP by default to
>> retrieve mail, while usual MUAs use the standard mechanism of RETR.
>> Nevertheless, Comcast's POP3 server is plain broken if he allows TOP but
>> truncates messages with it.
>
> There is one case where fetchmail could use RETR instead of TOP. That
> is when "uidl" has been enabled.

Not in 6.3.X, as that would change semantics WRT other clients. The
assumption fetchmail makes is that "TOP" leaves seen flags (where
recorded in headers) and the "LAST" pointer (way obsolete) alone. Plus,
some b0rked servers apparently remove messages right after RETR (see
below).

Changing UIDL to imply "use RETR" is an option only for later versions,
although it looks sensible in itself -- compatibility matters here.

I don't like fetchmail's assumptions about undocumented behavior either
(which got us into this trouble), and it appears the price for not using
RETR is high, but changes like these are definitely 6.4.X stuff.

> fetchmail currently uses TOP. If the server is supporting "uidl", then
> it should not matter if fetchmail uses TOP or RETR as all the email
> clients

This is the thinko. Not all clients use UIDL.
Some (including fetchmail) still use LAST...

> Any votes for or against this?

Against.

The TODO in SVN's trunk directory already mentions:

| clean up TOP vs. RETR but leave TOP as an option for the server scum
| that deletes message right after RETR, voiding client retries (Jakob
| Hirsch, 2006-01-03).

-- 
Matthias Andree

[fetchmail-users] Re: Bug in imap_idle() with mail servers not supporting "idle"

[Brendan L. <bre...@ai...>]

I have been in communication with Casper; see my response to Matthias 
earlier in this thread.

Brendan

sh...@bo... wrote:

>I think this issue is similar to the one reported by Casper
>Gripenberg. Could you try this patch and report if it works for you?
>
>=========================================================================
>Index: fetchmail/imap.c
>===================================================================
>--- fetchmail/imap.c    (revision 4696)
>+++ fetchmail/imap.c    (working copy)
>@@ -633,11 +633,12 @@
>    if (ok != 0 || stage != STAGE_IDLE)
>        return(ok);
> 
>-   /* wait (briefly) for an unsolicited status update */
>-   ok = imap_ok(sock, NULL);
>-   /* again, this is new mail or an error */
>-   if (ok != PS_IDLETIMEOUT)
>-       return(ok);
>+   /* we used to wait for an unsolicited status update here with
>+    * imap_ok(). However, since no command has actually been
>+    * sent, there is no "tag" which can be used for comparison.
>+    * Just sleep instead */
>+   set_timeout(0);
>+   sleep(mytimeout);
>     }
> 
>     /* restore normal timeout value */
>=========================================================================
>
>  
>

Re: [fetchmail-users] Bug in imap_idle() with mail servers not supporting "idle"

[Brendan L. <bre...@ai...>]

I've seen the problem with a couple of upstream servers: I'll get you 
the id of my current server when I get home.

However I believe the servers is behaving correctly according to the 
IMAP spec: I have attached an explanation of the secondary problem I am 
seeing (extra 28 s wait) that I sent to Casper; let me know if you need 
more info.

I've put comments to your comments below.

Brendan

mat...@gm... wrote:

>Brendan Lynch <bre...@ai...> writes:
>
>  
>
>>This code works fine until a notification of new mail is received (a "*
>>1 EXISTS" message is received).  At this point normally one is in the
>>"imap_ok()" routine called from line 637, and this correctly receives
>>the notification message and parses it, updating the "count" variable.
>>However it does not the return to imap_idle() routine, but instead
>>reissues a recv call having set a much longer (300s) timeout and after
>>the 300s have expired it then returns an error causing fetchmail to drop
>>the IMAP connection (a main point of this idle code was to keep the
>>connection open for subsequent message retrieval).  Net result is that
>>delivery of mail is delayed by 5 minutes and the server connection is
>>dropped and reestablished each time a wait for mail occurs.
>>
>>The problem seems to be caused by the loop condition in imap_ok():
>>
>>   151         }
>>   152     } while
>>   153         (tag[0] != '\0' && strncmp(buf, tag, strlen(tag)));
>>   154
>>
>>This assumes that tag[0] will be set to '\0' if one is not waiting for a
>>tagged response.  In this case the code should not be waiting for a
>>tagged response (it is waiting for an unsolicited notification).
>>However the 'tag' global character array is set by the gen_transact() at
>>line 630 and is not cleared before the call to imap_ok() at line 637.
>>    
>>
>
>This is correct, an IMAP client is supposed to parse untagged responses
>until a tagged response is received. Trying with Dovecot and hacking
>fetchmail a bit so it doesn't recognize RECENT and uses the NOOP
>emulation code yields:
>
>...
>fetchmail: IMAP> A0010 NOOP
>fetchmail: IMAP< A0010 OK NOOP completed.
>fetchmail: IMAP> A0011 NOOP
>fetchmail: IMAP< * 1 EXISTS
>fetchmail: IMAP< * 1 RECENT
>fetchmail: IMAP< A0011 OK NOOP completed.
>fetchmail: IMAP> A0012 NOOP
>...
>
>So it waits for the tagged NOOP response, and this is a requirement so
>it actually picks up both the EXISTS and the RECENT responses of working
>servers. Servers that do not respond with a tagged response to a NOOP
>command are broken.
>  
>
See my explanation to Casper. 

The key point is that the imap_idle() code when 'faking' IDLE support 
calls imap_ok() (at line 637) without having issued any tagged command:

    625         /* when faking an idle, we can't assume the server will
    626          * send us the new messages out of the blue (RFC2060);
    627          * this timeout is potentially the delay before we notice
    628          * new mail (can be small since NOOP checking is cheap) */
    629         mytimeout = 28;
    630         ok = gen_transact(sock, "NOOP");
    631         /* if there's an error (not likely) or we just found 
mail (stage
    632          * has changed, timeout has also been restored), we're 
done */
    633         if (ok != 0 || stage != STAGE_IDLE)
    634             return(ok);
    635
    636         /* wait (briefly) for an unsolicited status update */
    637         ok = imap_ok(sock, NULL);
    638         /* again, this is new mail or an error */
    639         if (ok != PS_IDLETIMEOUT)
    640             return(ok);

Since there is no tagged command outstanding the server will not (and 
should not) ever send a tagged response.  If we set imap_ok to only 
return after a tagged response we are guaranteeing it will always only 
return after a timeout.

Worse, if the server does send an unsolicited EXISTS message during this 
(28s) imap_ok() wait, there is already-existing special-case code to 
handle an unsolicited EXITS message in imap_ok:

     93                 /*
     94                  * Nasty kluge to handle RFC2177 IDLE.  If we 
know we're idling
     95                  * we can't wait for the tag matching the IDLE; 
we have to tell the
     96                  * server the IDLE is finished by shipping back 
a DONE when we
     97                  * see an EXISTS.  Only after that will a tagged 
response be
     98                  * shipped.  The idling flag also gets cleared 
on a timeout.
     99                  */
    100                 if (stage == STAGE_IDLE)
    101                 {
    102                     /* If IDLE isn't supported, we were only 
sending NOOPs anyway. */
    103                     if (has_idle)
    104                     {
    105                         /* we do our own write and report here 
to disable tagging */
    106                         SockWrite(sock, "DONE\r\n", 6);
    107                         if (outlevel >= O_MONITOR)
    108                             report(stdout, "IMAP> DONE\n");
    109                     }
    110
    111                     mytimeout = saved_timeout;
    112                     stage = STAGE_FETCH;
    113                 }

The last 2 lines are executed even for the NOOP-emulated idle.  This 
causes us to  change the timeout to 5 minutes and set stage to 
STAGE_FETCH.  Then, if we are waiting for a tagged response, we loop 
around and wait for 5 minutes for a tagged response (which we will never 
see) and eventually return PS_SOCKET (since our wait times out and we 
are not in STAGE_IDLE) causing fetchmail to drop and reestablish the 
mail server connection.

>  
>
>>The fix is a very simple one-line change to imap_idle (2 lines with
>>comments):
>>
>>   630         ok = gen_transact(sock, "NOOP");
>>   631         /* if there's an error (not likely) or we just found mail
>>(stage
>>   632          * has changed, timeout has also been restored), we're
>>done */
>>   633         if (ok != 0 || stage != STAGE_IDLE)
>>   634             return(ok);
>>   635
>>+   636         /* clear tag so imap_ok does not expect tagged response */
>>+   637         tag[0]='\0';
>>   638         /* wait (briefly) for an unsolicited status update */
>>   639         ok = imap_ok(sock, NULL);
>>    
>>
>
>So this patch would sort of break the IMAP client because it would jump
>out of the loop before having read the reply. This requires more
>thought. Casper Gripenberg reported a similar problem, so perhaps some
>common upstream server software is the culprit (and he suggested a
>different fix, I'll have a look at that too).
>  
>

No, the client *is not* waiting for a reply; the reply was received and 
processed in the gen_transact() call at line 630; this deals with an 
imap_ok wait where no command has been issued.

>What software is your upstream server running?
>
>Can I see a "fetchmail -Nvvv --nosyslog" trace of a failing IMAP session
>with NOOP emulation?
>  
>

I'll set this up this evening and send you the trace and a timed strace 
(which is much more useful).  My mail to Caper includes an strace where 
the 'exit on unsolicited notification during imap_idle fix is in place, 
but the second 'RECENT' modification is not.

>  
>
>>A second, more minor problem is that getting a "* RECENT" notification
>>does not break a caller out of the imap_idle's imap_ok() call.  This
>>causes an extra 28second wait after being notified about a message
>>before it is actually received.
>>
>>Diffs for the complete set of changes against 6.3.2 are attached to this
>>email.
>>
>>I have seen this in fetchmail 6.2.5 and 6.3.2 on linux platforms, but
>>this problem should be generic to all platforms.
>>diff -Naur fetchmail-6.3.2/imap.c fetchmail-6.3.2a/imap.c
>>--- fetchmail-6.3.2/imap.c  2006-01-20 10:38:45.000000000 +0000
>>+++ fetchmail-6.3.2a/imap.c 2006-02-23 23:54:52.000000000 +0000
>>@@ -116,6 +116,17 @@
>>        {
>>        recentcount_ok = 1;
>>        recentcount = atoi(buf+2);
>>+       /*
>>+        * Kluge to handle IDLE simulation. If we are in STAGE_IDLE and
>>+        * we are simulating (has_idle unset) we need to alert calling
>>+        * imap_idle() to the fact that we have received a new "recent"
>>+        * alert in order to break the imap_idle() out of its wait.
>>+        */
>>+       if (!has_idle && stage == STAGE_IDLE)
>>+       {
>>+           mytimeout = saved_timeout;
>>+           stage = STAGE_FETCH;
>>+       }
>>        }
>>        else if (strstr(buf, " EXPUNGE"))
>>        {
>>    
>>
>
>This looks acceptable.
>
>How can a server mark a new message "RECENT" without also sending an
>"EXISTS"? I'm inclined to consider such behavior broken. I'm willing to
>apply this nonetheless as I don't think it would hurt anyone.
>
>  
>
It cannot, but the order it does them depends on the server; and also 
the server may send an EXISTS message without a RECENT message if a 
message is expunged.  the 'recentcount' is what is used to break us out 
of one of the loops; see my explanation to Casper.

Brendan

[fetchmail-users] RETR with UIDL [Was: Re: After reading the FAQ, a question about lost/mangled attachments]

[Sunil S. <sh...@bo...>]

Quoting from Jakob Hirsch's mail on Thu, Mar 02, 2006:
> The difference is simply that fetchmail (ab)uses TOP by default to
> retrieve mail, while usual MUAs use the standard mechanism of RETR.
> Nevertheless, Comcast's POP3 server is plain broken if he allows TOP but
> truncates messages with it.

There is one case where fetchmail could use RETR instead of TOP. That
is when "uidl" has been enabled.

In this configuration:

poll server
    protocol pop3
    no fetchall
    no keep
    uidl

fetchmail currently uses TOP. If the server is supporting "uidl", then
it should not matter if fetchmail uses TOP or RETR as all the email
clients will be using "uidl" anyway. So, in this case, fetchmail
should use "RETR" instead.

I am not aware if Comcast's POP3 server support "uidl" or not. If they
do, I think this change should be made.

Any votes for or against this?

This is probably the change required (somebody confirm this too):

<    peek_capable = !ctl->fetchall && (!ctl->keep || ctl->server.uidl);
>    peek_capable = !(ctl->fetchall || ctl->keep || ctl->server.uidl);

-- 
Sunil Shetye.

Re: [fetchmail-users] After reading the FAQ, a question about lost/mangled attachments

[Jakob H. <jh...@pl...>]

Peter N. Spotts wrote:

> Hmmm...this is interesting. It would imply that when email clients run
> pop3 and pull the email in directly, they already have a fetchall-like
> function active? Otherwise, why would all attachments of any size arrive

The difference is simply that fetchmail (ab)uses TOP by default to
retrieve mail, while usual MUAs use the standard mechanism of RETR.
Nevertheless, Comcast's POP3 server is plain broken if he allows TOP but
truncates messages with it.

Re: [fetchmail-users] After reading the FAQ, a question about lost/mangled attachments

[Ed W. <ew...@ew...>]

On Thu, Mar 02, 2006 at 07:47:31AM -0500, Peter N. Spotts wrote:
> Ed Wilts wrote:
> >
> >But Comcast *is* the problem.  I have this in my .fetchmailrc and it
> >works fine, with attachments.  
> >
> >Without the "fetchall", any attachments over 70k were mangled.
> >
> 
> Hmmm...this is interesting. It would imply that when email clients run 
> pop3 and pull the email in directly, they already have a fetchall-like 
> function active? 

Right.  fetchmail does things differently.  Comcast is broken but since
many clients simply work around broken servers, Comcast doesn't see a
need to fix anything.

        .../Ed

-- 
Ed Wilts, Mounds View, MN, USA
mailto:ew...@ew...

Re: [fetchmail-users] After reading the FAQ, a question about lost/mangled attachments

[Peter N. S. <ps...@al...>]

Ed Wilts wrote:
> 
> But Comcast *is* the problem.  I have this in my .fetchmailrc and it
> works fine, with attachments.  
> 
> poll mail.comcast.net
>         proto POP3
>         user 'ewilts'
>         password 'noneofyourbusiness!'
>         fetchall
> 
> Without the "fetchall", any attachments over 70k were mangled.
> 

Hmmm...this is interesting. It would imply that when email clients run 
pop3 and pull the email in directly, they already have a fetchall-like 
function active? Otherwise, why would all attachments of any size arrive 
whole to TB, Evolution, or SC via Comcast as pop3 but not via fetchmail? 
Both presumably are merely polling Comcast for new mail. Ah, it's a 
curious world out there!

Anyway, I've added the fetchall to my fetchmailrc file. Thanks so much 
for your help...

With best regards,

Pete


-- 

       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             Peter N. Spotts | Science Correspondent
                  The Christian Science Monitor
              One Norway Street, Boston  MA  02115
       Office: 617-450-2449 | Office in home: 508-520-3139
         Email: ps...@al... | www.csmonitor.com
	         Amateur-radio call - KC1JB
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

"The knack of flying is to throw yourself at the ground and miss."
					         -- Douglas Adams

Re: [fetchmail-users] After reading the FAQ, a question about lost/mangled attachments

[Ed W. <ew...@ew...>]

On Wed, Mar 01, 2006 at 08:58:52PM -0500, Peter N. Spotts wrote:
> After reading the FAQ, I'm hip to fetchmail's "neutrality" on whether an 
> incoming email has an attachment or not. So I'm hoping for some guidance 
> on where to look for the source of a recurring problem of randomly lost 
> or "damaged" pdf attachments. Typically (though not always), the pdfs 
> that come through unscathed are one-page, sent one per email. The lost 
> or damaged attachments tend to be larger pdf files and emails with 
> multiple attachments.
> 
> When I noticed the damaged attachments, I set up my clients for POP3 to 
> pull the mail directly from the ISP. Then I had the senders resend the 
> email. All the attachments arrived cleanly from ISP straight into email 
> client. So although my ISP is Comcast (I noted the Comcast caveats on 
> the FAQ page), Comcast does not seem to be the problem either. And I've 
> deactivated clamav, thinking it might not like the attachments for some 
> reason. But the attachments still come in damaged via the fetchmail et 
> al route.
> 
> So, if Comcast isn't the problem and fetchmail doesn't give a hoot if an 
> email bears an attachment or not, where else should I look?

But Comcast *is* the problem.  I have this in my .fetchmailrc and it
works fine, with attachments.  

poll mail.comcast.net
        proto POP3
        user 'ewilts'
        password 'noneofyourbusiness!'
        fetchall

Without the "fetchall", any attachments over 70k were mangled.

-- 
Ed Wilts, Mounds View, MN, USA
mailto:ew...@ew...

[fetchmail-users] After reading the FAQ, a question about lost/mangled attachments

[Peter N. S. <ps...@al...>]

Folks,

After reading the FAQ, I'm hip to fetchmail's "neutrality" on whether an 
incoming email has an attachment or not. So I'm hoping for some guidance 
on where to look for the source of a recurring problem of randomly lost 
or "damaged" pdf attachments. Typically (though not always), the pdfs 
that come through unscathed are one-page, sent one per email. The lost 
or damaged attachments tend to be larger pdf files and emails with 
multiple attachments.

I've been running fetchmail on SuSE 10.0 on my laptop, and until today 
(when I installed the latest version of fetchmail) I've been running 
6.2.X. Rather than setting up my email clients (variously, Thunderbird, 
Evolution, and Sylpheed-Claws) as POP3 to pull mail directly into the 
client, I prefer to use fetchmail and procmail to pull email in from my 
ISP and run it through spamassassin and clamav before dumping it into 
/var/spool/mail/~. From my side of the screen, the mail comes into the 
client fully processed; I don't have to wait for the email client to 
chunk away looking for spam and virus "on screen" after I've hit the Get 
New Messages button.

When I noticed the damaged attachments, I set up my clients for POP3 to 
pull the mail directly from the ISP. Then I had the senders resend the 
email. All the attachments arrived cleanly from ISP straight into email 
client. So although my ISP is Comcast (I noted the Comcast caveats on 
the FAQ page), Comcast does not seem to be the problem either. And I've 
deactivated clamav, thinking it might not like the attachments for some 
reason. But the attachments still come in damaged via the fetchmail et 
al route.

So, if Comcast isn't the problem and fetchmail doesn't give a hoot if an 
email bears an attachment or not, where else should I look?

Thanks for any help you can give...

With best regards,

Pete Spotts
-- 

       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             Peter N. Spotts | Science Correspondent
                  The Christian Science Monitor
              One Norway Street, Boston  MA  02115
       Office: 617-450-2449 | Office in home: 508-520-3139
         Email: ps...@al... | www.csmonitor.com
	         Amateur-radio call - KC1JB
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

"The knack of flying is to throw yourself at the ground and miss."
					         -- Douglas Adams

[fetchmail-users] Re: Bug in imap_idle() with mail servers not supporting "idle"

[Sunil S. <sh...@bo...>]

I think this issue is similar to the one reported by Casper
Gripenberg. Could you try this patch and report if it works for you?

=========================================================================
Index: fetchmail/imap.c
===================================================================
--- fetchmail/imap.c	(revision 4696)
+++ fetchmail/imap.c	(working copy)
@@ -633,11 +633,12 @@
 	if (ok != 0 || stage != STAGE_IDLE)
 	    return(ok);
 
-	/* wait (briefly) for an unsolicited status update */
-	ok = imap_ok(sock, NULL);
-	/* again, this is new mail or an error */
-	if (ok != PS_IDLETIMEOUT)
-	    return(ok);
+	/* we used to wait for an unsolicited status update here with
+	 * imap_ok(). However, since no command has actually been
+	 * sent, there is no "tag" which can be used for comparison.
+	 * Just sleep instead */
+	set_timeout(0);
+	sleep(mytimeout);
     }
 
     /* restore normal timeout value */
=========================================================================

-- 
Sunil Shetye.

Re: [fetchmail-users] Re: [fetchmail]2 Delivered-To: fields in email

[Matthias A. <mat...@gm...>]

Ed Wilts <ew...@ew...> writes:

> On Mon, Feb 27, 2006 at 08:00:34PM +0000, Rob MacGregor wrote:
>> > Delivered-To: myadoofa_at_adoofa.com
>> > Received: from pop3.zippymail.co.uk [81.201.128.18]
>> >  by localhost with IMAP (fetchmail-6.2.5)
>> 
>> Note that you're running an old, vulnerable, version of
>> fetchmail.  
>
> The version does not necessarily mean that the image is vulnerable.  Red
> Hat backports their security patches to versions that the original
> developers may update by releasing a new version (with possible other
> fixes, features, and bugs).  I expect that other distributions do the same.
> http://www.redhat.com/advice/speaks_backport.html

Even if that's the case and the vulnerability has been removed by a
backported fix, the version (6.2.5) is still old and no longer supported
by the current fetchmail maintainers on the fetchmail* mailing
lists. See if the problem reproduces with a newer version, if it does,
ask here; if it does not, ask your packager for support.

-- 
Matthias Andree

[fetchmail-users] Re: Bug in imap_idle() with mail servers not supporting "idle"

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Mon, Feb 27, 2006:
> Brendan Lynch <bre...@ai...> writes:
> Can I see a "fetchmail -Nvvv --nosyslog" trace of a failing IMAP session
> with NOOP emulation?
> 
> > A second, more minor problem is that getting a "* RECENT" notification
> > does not break a caller out of the imap_idle's imap_ok() call.  This
> > causes an extra 28second wait after being notified about a message
> > before it is actually received.
> >
> > Diffs for the complete set of changes against 6.3.2 are attached to this
> > email.
> >
> > I have seen this in fetchmail 6.2.5 and 6.3.2 on linux platforms, but
> > this problem should be generic to all platforms.
> > diff -Naur fetchmail-6.3.2/imap.c fetchmail-6.3.2a/imap.c
> > --- fetchmail-6.3.2/imap.c  2006-01-20 10:38:45.000000000 +0000
> > +++ fetchmail-6.3.2a/imap.c 2006-02-23 23:54:52.000000000 +0000
> > @@ -116,6 +116,17 @@
> >         {
> >         recentcount_ok = 1;
> >         recentcount = atoi(buf+2);
> > +       /*
> > +        * Kluge to handle IDLE simulation. If we are in STAGE_IDLE and
> > +        * we are simulating (has_idle unset) we need to alert calling
> > +        * imap_idle() to the fact that we have received a new "recent"
> > +        * alert in order to break the imap_idle() out of its wait.
> > +        */
> > +       if (!has_idle && stage == STAGE_IDLE)
> > +       {
> > +           mytimeout = saved_timeout;
> > +           stage = STAGE_FETCH;
> > +       }
> >         }
> >         else if (strstr(buf, " EXPUNGE"))
> >         {
> 
> This looks acceptable.
> 
> How can a server mark a new message "RECENT" without also sending an
> "EXISTS"? I'm inclined to consider such behavior broken. I'm willing to
> apply this nonetheless as I don't think it would hurt anyone.

I think, it would still be advisable to first get the output of
"fetchmail -Nvv --nosyslog" before accepting the patch. It is possible
that the problem lies elsewhere.

-- 
Sunil Shetye.

[fetchmail-users] Re: [fetchmail]2 Delivered-To: fields in email

[Ed W. <ew...@ew...>]

On Mon, Feb 27, 2006 at 08:00:34PM +0000, Rob MacGregor wrote:
> > Delivered-To: myadoofa_at_adoofa.com
> > Received: from pop3.zippymail.co.uk [81.201.128.18]
> >  by localhost with IMAP (fetchmail-6.2.5)
> 
> Note that you're running an old, vulnerable, version of
> fetchmail.  

The version does not necessarily mean that the image is vulnerable.  Red
Hat backports their security patches to versions that the original
developers may update by releasing a new version (with possible other
fixes, features, and bugs).  I expect that other distributions do the same.
http://www.redhat.com/advice/speaks_backport.html

        .../Ed

-- 
Ed Wilts, Mounds View, MN, USA
mailto:ew...@ew...