fetchmail-users — User questions and discussion

[fetchmail-users] Re: Bug in imap_idle() with mail servers not supporting "idle"

[Brendan L. <bre...@ai...>]

I have been in communication with Casper; see my response to Matthias 
earlier in this thread.

Brendan

sh...@bo... wrote:

>I think this issue is similar to the one reported by Casper
>Gripenberg. Could you try this patch and report if it works for you?
>
>=========================================================================
>Index: fetchmail/imap.c
>===================================================================
>--- fetchmail/imap.c    (revision 4696)
>+++ fetchmail/imap.c    (working copy)
>@@ -633,11 +633,12 @@
>    if (ok != 0 || stage != STAGE_IDLE)
>        return(ok);
> 
>-   /* wait (briefly) for an unsolicited status update */
>-   ok = imap_ok(sock, NULL);
>-   /* again, this is new mail or an error */
>-   if (ok != PS_IDLETIMEOUT)
>-       return(ok);
>+   /* we used to wait for an unsolicited status update here with
>+    * imap_ok(). However, since no command has actually been
>+    * sent, there is no "tag" which can be used for comparison.
>+    * Just sleep instead */
>+   set_timeout(0);
>+   sleep(mytimeout);
>     }
> 
>     /* restore normal timeout value */
>=========================================================================
>
>  
>

Re: [fetchmail-users] Bug in imap_idle() with mail servers not supporting "idle"

[Brendan L. <bre...@ai...>]

I've seen the problem with a couple of upstream servers: I'll get you 
the id of my current server when I get home.

However I believe the servers is behaving correctly according to the 
IMAP spec: I have attached an explanation of the secondary problem I am 
seeing (extra 28 s wait) that I sent to Casper; let me know if you need 
more info.

I've put comments to your comments below.

Brendan

mat...@gm... wrote:

>Brendan Lynch <bre...@ai...> writes:
>
>  
>
>>This code works fine until a notification of new mail is received (a "*
>>1 EXISTS" message is received).  At this point normally one is in the
>>"imap_ok()" routine called from line 637, and this correctly receives
>>the notification message and parses it, updating the "count" variable.
>>However it does not the return to imap_idle() routine, but instead
>>reissues a recv call having set a much longer (300s) timeout and after
>>the 300s have expired it then returns an error causing fetchmail to drop
>>the IMAP connection (a main point of this idle code was to keep the
>>connection open for subsequent message retrieval).  Net result is that
>>delivery of mail is delayed by 5 minutes and the server connection is
>>dropped and reestablished each time a wait for mail occurs.
>>
>>The problem seems to be caused by the loop condition in imap_ok():
>>
>>   151         }
>>   152     } while
>>   153         (tag[0] != '\0' && strncmp(buf, tag, strlen(tag)));
>>   154
>>
>>This assumes that tag[0] will be set to '\0' if one is not waiting for a
>>tagged response.  In this case the code should not be waiting for a
>>tagged response (it is waiting for an unsolicited notification).
>>However the 'tag' global character array is set by the gen_transact() at
>>line 630 and is not cleared before the call to imap_ok() at line 637.
>>    
>>
>
>This is correct, an IMAP client is supposed to parse untagged responses
>until a tagged response is received. Trying with Dovecot and hacking
>fetchmail a bit so it doesn't recognize RECENT and uses the NOOP
>emulation code yields:
>
>...
>fetchmail: IMAP> A0010 NOOP
>fetchmail: IMAP< A0010 OK NOOP completed.
>fetchmail: IMAP> A0011 NOOP
>fetchmail: IMAP< * 1 EXISTS
>fetchmail: IMAP< * 1 RECENT
>fetchmail: IMAP< A0011 OK NOOP completed.
>fetchmail: IMAP> A0012 NOOP
>...
>
>So it waits for the tagged NOOP response, and this is a requirement so
>it actually picks up both the EXISTS and the RECENT responses of working
>servers. Servers that do not respond with a tagged response to a NOOP
>command are broken.
>  
>
See my explanation to Casper. 

The key point is that the imap_idle() code when 'faking' IDLE support 
calls imap_ok() (at line 637) without having issued any tagged command:

    625         /* when faking an idle, we can't assume the server will
    626          * send us the new messages out of the blue (RFC2060);
    627          * this timeout is potentially the delay before we notice
    628          * new mail (can be small since NOOP checking is cheap) */
    629         mytimeout = 28;
    630         ok = gen_transact(sock, "NOOP");
    631         /* if there's an error (not likely) or we just found 
mail (stage
    632          * has changed, timeout has also been restored), we're 
done */
    633         if (ok != 0 || stage != STAGE_IDLE)
    634             return(ok);
    635
    636         /* wait (briefly) for an unsolicited status update */
    637         ok = imap_ok(sock, NULL);
    638         /* again, this is new mail or an error */
    639         if (ok != PS_IDLETIMEOUT)
    640             return(ok);

Since there is no tagged command outstanding the server will not (and 
should not) ever send a tagged response.  If we set imap_ok to only 
return after a tagged response we are guaranteeing it will always only 
return after a timeout.

Worse, if the server does send an unsolicited EXISTS message during this 
(28s) imap_ok() wait, there is already-existing special-case code to 
handle an unsolicited EXITS message in imap_ok:

     93                 /*
     94                  * Nasty kluge to handle RFC2177 IDLE.  If we 
know we're idling
     95                  * we can't wait for the tag matching the IDLE; 
we have to tell the
     96                  * server the IDLE is finished by shipping back 
a DONE when we
     97                  * see an EXISTS.  Only after that will a tagged 
response be
     98                  * shipped.  The idling flag also gets cleared 
on a timeout.
     99                  */
    100                 if (stage == STAGE_IDLE)
    101                 {
    102                     /* If IDLE isn't supported, we were only 
sending NOOPs anyway. */
    103                     if (has_idle)
    104                     {
    105                         /* we do our own write and report here 
to disable tagging */
    106                         SockWrite(sock, "DONE\r\n", 6);
    107                         if (outlevel >= O_MONITOR)
    108                             report(stdout, "IMAP> DONE\n");
    109                     }
    110
    111                     mytimeout = saved_timeout;
    112                     stage = STAGE_FETCH;
    113                 }

The last 2 lines are executed even for the NOOP-emulated idle.  This 
causes us to  change the timeout to 5 minutes and set stage to 
STAGE_FETCH.  Then, if we are waiting for a tagged response, we loop 
around and wait for 5 minutes for a tagged response (which we will never 
see) and eventually return PS_SOCKET (since our wait times out and we 
are not in STAGE_IDLE) causing fetchmail to drop and reestablish the 
mail server connection.

>  
>
>>The fix is a very simple one-line change to imap_idle (2 lines with
>>comments):
>>
>>   630         ok = gen_transact(sock, "NOOP");
>>   631         /* if there's an error (not likely) or we just found mail
>>(stage
>>   632          * has changed, timeout has also been restored), we're
>>done */
>>   633         if (ok != 0 || stage != STAGE_IDLE)
>>   634             return(ok);
>>   635
>>+   636         /* clear tag so imap_ok does not expect tagged response */
>>+   637         tag[0]='\0';
>>   638         /* wait (briefly) for an unsolicited status update */
>>   639         ok = imap_ok(sock, NULL);
>>    
>>
>
>So this patch would sort of break the IMAP client because it would jump
>out of the loop before having read the reply. This requires more
>thought. Casper Gripenberg reported a similar problem, so perhaps some
>common upstream server software is the culprit (and he suggested a
>different fix, I'll have a look at that too).
>  
>

No, the client *is not* waiting for a reply; the reply was received and 
processed in the gen_transact() call at line 630; this deals with an 
imap_ok wait where no command has been issued.

>What software is your upstream server running?
>
>Can I see a "fetchmail -Nvvv --nosyslog" trace of a failing IMAP session
>with NOOP emulation?
>  
>

I'll set this up this evening and send you the trace and a timed strace 
(which is much more useful).  My mail to Caper includes an strace where 
the 'exit on unsolicited notification during imap_idle fix is in place, 
but the second 'RECENT' modification is not.

>  
>
>>A second, more minor problem is that getting a "* RECENT" notification
>>does not break a caller out of the imap_idle's imap_ok() call.  This
>>causes an extra 28second wait after being notified about a message
>>before it is actually received.
>>
>>Diffs for the complete set of changes against 6.3.2 are attached to this
>>email.
>>
>>I have seen this in fetchmail 6.2.5 and 6.3.2 on linux platforms, but
>>this problem should be generic to all platforms.
>>diff -Naur fetchmail-6.3.2/imap.c fetchmail-6.3.2a/imap.c
>>--- fetchmail-6.3.2/imap.c  2006-01-20 10:38:45.000000000 +0000
>>+++ fetchmail-6.3.2a/imap.c 2006-02-23 23:54:52.000000000 +0000
>>@@ -116,6 +116,17 @@
>>        {
>>        recentcount_ok = 1;
>>        recentcount = atoi(buf+2);
>>+       /*
>>+        * Kluge to handle IDLE simulation. If we are in STAGE_IDLE and
>>+        * we are simulating (has_idle unset) we need to alert calling
>>+        * imap_idle() to the fact that we have received a new "recent"
>>+        * alert in order to break the imap_idle() out of its wait.
>>+        */
>>+       if (!has_idle && stage == STAGE_IDLE)
>>+       {
>>+           mytimeout = saved_timeout;
>>+           stage = STAGE_FETCH;
>>+       }
>>        }
>>        else if (strstr(buf, " EXPUNGE"))
>>        {
>>    
>>
>
>This looks acceptable.
>
>How can a server mark a new message "RECENT" without also sending an
>"EXISTS"? I'm inclined to consider such behavior broken. I'm willing to
>apply this nonetheless as I don't think it would hurt anyone.
>
>  
>
It cannot, but the order it does them depends on the server; and also 
the server may send an EXISTS message without a RECENT message if a 
message is expunged.  the 'recentcount' is what is used to break us out 
of one of the loops; see my explanation to Casper.

Brendan

[fetchmail-users] RETR with UIDL [Was: Re: After reading the FAQ, a question about lost/mangled attachments]

[Sunil S. <sh...@bo...>]

Quoting from Jakob Hirsch's mail on Thu, Mar 02, 2006:
> The difference is simply that fetchmail (ab)uses TOP by default to
> retrieve mail, while usual MUAs use the standard mechanism of RETR.
> Nevertheless, Comcast's POP3 server is plain broken if he allows TOP but
> truncates messages with it.

There is one case where fetchmail could use RETR instead of TOP. That
is when "uidl" has been enabled.

In this configuration:

poll server
    protocol pop3
    no fetchall
    no keep
    uidl

fetchmail currently uses TOP. If the server is supporting "uidl", then
it should not matter if fetchmail uses TOP or RETR as all the email
clients will be using "uidl" anyway. So, in this case, fetchmail
should use "RETR" instead.

I am not aware if Comcast's POP3 server support "uidl" or not. If they
do, I think this change should be made.

Any votes for or against this?

This is probably the change required (somebody confirm this too):

<    peek_capable = !ctl->fetchall && (!ctl->keep || ctl->server.uidl);
>    peek_capable = !(ctl->fetchall || ctl->keep || ctl->server.uidl);

-- 
Sunil Shetye.

Re: [fetchmail-users] After reading the FAQ, a question about lost/mangled attachments

[Jakob H. <jh...@pl...>]

Peter N. Spotts wrote:

> Hmmm...this is interesting. It would imply that when email clients run
> pop3 and pull the email in directly, they already have a fetchall-like
> function active? Otherwise, why would all attachments of any size arrive

The difference is simply that fetchmail (ab)uses TOP by default to
retrieve mail, while usual MUAs use the standard mechanism of RETR.
Nevertheless, Comcast's POP3 server is plain broken if he allows TOP but
truncates messages with it.

Re: [fetchmail-users] After reading the FAQ, a question about lost/mangled attachments

[Ed W. <ew...@ew...>]

On Thu, Mar 02, 2006 at 07:47:31AM -0500, Peter N. Spotts wrote:
> Ed Wilts wrote:
> >
> >But Comcast *is* the problem.  I have this in my .fetchmailrc and it
> >works fine, with attachments.  
> >
> >Without the "fetchall", any attachments over 70k were mangled.
> >
> 
> Hmmm...this is interesting. It would imply that when email clients run 
> pop3 and pull the email in directly, they already have a fetchall-like 
> function active? 

Right.  fetchmail does things differently.  Comcast is broken but since
many clients simply work around broken servers, Comcast doesn't see a
need to fix anything.

        .../Ed

-- 
Ed Wilts, Mounds View, MN, USA
mailto:ew...@ew...

Re: [fetchmail-users] After reading the FAQ, a question about lost/mangled attachments

[Peter N. S. <ps...@al...>]

Ed Wilts wrote:
> 
> But Comcast *is* the problem.  I have this in my .fetchmailrc and it
> works fine, with attachments.  
> 
> poll mail.comcast.net
>         proto POP3
>         user 'ewilts'
>         password 'noneofyourbusiness!'
>         fetchall
> 
> Without the "fetchall", any attachments over 70k were mangled.
> 

Hmmm...this is interesting. It would imply that when email clients run 
pop3 and pull the email in directly, they already have a fetchall-like 
function active? Otherwise, why would all attachments of any size arrive 
whole to TB, Evolution, or SC via Comcast as pop3 but not via fetchmail? 
Both presumably are merely polling Comcast for new mail. Ah, it's a 
curious world out there!

Anyway, I've added the fetchall to my fetchmailrc file. Thanks so much 
for your help...

With best regards,

Pete


-- 

       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             Peter N. Spotts | Science Correspondent
                  The Christian Science Monitor
              One Norway Street, Boston  MA  02115
       Office: 617-450-2449 | Office in home: 508-520-3139
         Email: ps...@al... | www.csmonitor.com
	         Amateur-radio call - KC1JB
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

"The knack of flying is to throw yourself at the ground and miss."
					         -- Douglas Adams

Re: [fetchmail-users] After reading the FAQ, a question about lost/mangled attachments

[Ed W. <ew...@ew...>]

On Wed, Mar 01, 2006 at 08:58:52PM -0500, Peter N. Spotts wrote:
> After reading the FAQ, I'm hip to fetchmail's "neutrality" on whether an 
> incoming email has an attachment or not. So I'm hoping for some guidance 
> on where to look for the source of a recurring problem of randomly lost 
> or "damaged" pdf attachments. Typically (though not always), the pdfs 
> that come through unscathed are one-page, sent one per email. The lost 
> or damaged attachments tend to be larger pdf files and emails with 
> multiple attachments.
> 
> When I noticed the damaged attachments, I set up my clients for POP3 to 
> pull the mail directly from the ISP. Then I had the senders resend the 
> email. All the attachments arrived cleanly from ISP straight into email 
> client. So although my ISP is Comcast (I noted the Comcast caveats on 
> the FAQ page), Comcast does not seem to be the problem either. And I've 
> deactivated clamav, thinking it might not like the attachments for some 
> reason. But the attachments still come in damaged via the fetchmail et 
> al route.
> 
> So, if Comcast isn't the problem and fetchmail doesn't give a hoot if an 
> email bears an attachment or not, where else should I look?

But Comcast *is* the problem.  I have this in my .fetchmailrc and it
works fine, with attachments.  

poll mail.comcast.net
        proto POP3
        user 'ewilts'
        password 'noneofyourbusiness!'
        fetchall

Without the "fetchall", any attachments over 70k were mangled.

-- 
Ed Wilts, Mounds View, MN, USA
mailto:ew...@ew...

[fetchmail-users] After reading the FAQ, a question about lost/mangled attachments

[Peter N. S. <ps...@al...>]

Folks,

After reading the FAQ, I'm hip to fetchmail's "neutrality" on whether an 
incoming email has an attachment or not. So I'm hoping for some guidance 
on where to look for the source of a recurring problem of randomly lost 
or "damaged" pdf attachments. Typically (though not always), the pdfs 
that come through unscathed are one-page, sent one per email. The lost 
or damaged attachments tend to be larger pdf files and emails with 
multiple attachments.

I've been running fetchmail on SuSE 10.0 on my laptop, and until today 
(when I installed the latest version of fetchmail) I've been running 
6.2.X. Rather than setting up my email clients (variously, Thunderbird, 
Evolution, and Sylpheed-Claws) as POP3 to pull mail directly into the 
client, I prefer to use fetchmail and procmail to pull email in from my 
ISP and run it through spamassassin and clamav before dumping it into 
/var/spool/mail/~. From my side of the screen, the mail comes into the 
client fully processed; I don't have to wait for the email client to 
chunk away looking for spam and virus "on screen" after I've hit the Get 
New Messages button.

When I noticed the damaged attachments, I set up my clients for POP3 to 
pull the mail directly from the ISP. Then I had the senders resend the 
email. All the attachments arrived cleanly from ISP straight into email 
client. So although my ISP is Comcast (I noted the Comcast caveats on 
the FAQ page), Comcast does not seem to be the problem either. And I've 
deactivated clamav, thinking it might not like the attachments for some 
reason. But the attachments still come in damaged via the fetchmail et 
al route.

So, if Comcast isn't the problem and fetchmail doesn't give a hoot if an 
email bears an attachment or not, where else should I look?

Thanks for any help you can give...

With best regards,

Pete Spotts
-- 

       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             Peter N. Spotts | Science Correspondent
                  The Christian Science Monitor
              One Norway Street, Boston  MA  02115
       Office: 617-450-2449 | Office in home: 508-520-3139
         Email: ps...@al... | www.csmonitor.com
	         Amateur-radio call - KC1JB
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

"The knack of flying is to throw yourself at the ground and miss."
					         -- Douglas Adams

[fetchmail-users] Re: Bug in imap_idle() with mail servers not supporting "idle"

[Sunil S. <sh...@bo...>]

I think this issue is similar to the one reported by Casper
Gripenberg. Could you try this patch and report if it works for you?

=========================================================================
Index: fetchmail/imap.c
===================================================================
--- fetchmail/imap.c	(revision 4696)
+++ fetchmail/imap.c	(working copy)
@@ -633,11 +633,12 @@
 	if (ok != 0 || stage != STAGE_IDLE)
 	    return(ok);
 
-	/* wait (briefly) for an unsolicited status update */
-	ok = imap_ok(sock, NULL);
-	/* again, this is new mail or an error */
-	if (ok != PS_IDLETIMEOUT)
-	    return(ok);
+	/* we used to wait for an unsolicited status update here with
+	 * imap_ok(). However, since no command has actually been
+	 * sent, there is no "tag" which can be used for comparison.
+	 * Just sleep instead */
+	set_timeout(0);
+	sleep(mytimeout);
     }
 
     /* restore normal timeout value */
=========================================================================

-- 
Sunil Shetye.

Re: [fetchmail-users] Re: [fetchmail]2 Delivered-To: fields in email

[Matthias A. <mat...@gm...>]

Ed Wilts <ew...@ew...> writes:

> On Mon, Feb 27, 2006 at 08:00:34PM +0000, Rob MacGregor wrote:
>> > Delivered-To: myadoofa_at_adoofa.com
>> > Received: from pop3.zippymail.co.uk [81.201.128.18]
>> >  by localhost with IMAP (fetchmail-6.2.5)
>> 
>> Note that you're running an old, vulnerable, version of
>> fetchmail.  
>
> The version does not necessarily mean that the image is vulnerable.  Red
> Hat backports their security patches to versions that the original
> developers may update by releasing a new version (with possible other
> fixes, features, and bugs).  I expect that other distributions do the same.
> http://www.redhat.com/advice/speaks_backport.html

Even if that's the case and the vulnerability has been removed by a
backported fix, the version (6.2.5) is still old and no longer supported
by the current fetchmail maintainers on the fetchmail* mailing
lists. See if the problem reproduces with a newer version, if it does,
ask here; if it does not, ask your packager for support.

-- 
Matthias Andree

[fetchmail-users] Re: Bug in imap_idle() with mail servers not supporting "idle"

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Mon, Feb 27, 2006:
> Brendan Lynch <bre...@ai...> writes:
> Can I see a "fetchmail -Nvvv --nosyslog" trace of a failing IMAP session
> with NOOP emulation?
> 
> > A second, more minor problem is that getting a "* RECENT" notification
> > does not break a caller out of the imap_idle's imap_ok() call.  This
> > causes an extra 28second wait after being notified about a message
> > before it is actually received.
> >
> > Diffs for the complete set of changes against 6.3.2 are attached to this
> > email.
> >
> > I have seen this in fetchmail 6.2.5 and 6.3.2 on linux platforms, but
> > this problem should be generic to all platforms.
> > diff -Naur fetchmail-6.3.2/imap.c fetchmail-6.3.2a/imap.c
> > --- fetchmail-6.3.2/imap.c  2006-01-20 10:38:45.000000000 +0000
> > +++ fetchmail-6.3.2a/imap.c 2006-02-23 23:54:52.000000000 +0000
> > @@ -116,6 +116,17 @@
> >         {
> >         recentcount_ok = 1;
> >         recentcount = atoi(buf+2);
> > +       /*
> > +        * Kluge to handle IDLE simulation. If we are in STAGE_IDLE and
> > +        * we are simulating (has_idle unset) we need to alert calling
> > +        * imap_idle() to the fact that we have received a new "recent"
> > +        * alert in order to break the imap_idle() out of its wait.
> > +        */
> > +       if (!has_idle && stage == STAGE_IDLE)
> > +       {
> > +           mytimeout = saved_timeout;
> > +           stage = STAGE_FETCH;
> > +       }
> >         }
> >         else if (strstr(buf, " EXPUNGE"))
> >         {
> 
> This looks acceptable.
> 
> How can a server mark a new message "RECENT" without also sending an
> "EXISTS"? I'm inclined to consider such behavior broken. I'm willing to
> apply this nonetheless as I don't think it would hurt anyone.

I think, it would still be advisable to first get the output of
"fetchmail -Nvv --nosyslog" before accepting the patch. It is possible
that the problem lies elsewhere.

-- 
Sunil Shetye.

[fetchmail-users] Re: [fetchmail]2 Delivered-To: fields in email

[Ed W. <ew...@ew...>]

On Mon, Feb 27, 2006 at 08:00:34PM +0000, Rob MacGregor wrote:
> > Delivered-To: myadoofa_at_adoofa.com
> > Received: from pop3.zippymail.co.uk [81.201.128.18]
> >  by localhost with IMAP (fetchmail-6.2.5)
> 
> Note that you're running an old, vulnerable, version of
> fetchmail.  

The version does not necessarily mean that the image is vulnerable.  Red
Hat backports their security patches to versions that the original
developers may update by releasing a new version (with possible other
fixes, features, and bugs).  I expect that other distributions do the same.
http://www.redhat.com/advice/speaks_backport.html

        .../Ed

-- 
Ed Wilts, Mounds View, MN, USA
mailto:ew...@ew...

Re: [fetchmail-users] Bug in imap_idle() with mail servers not supporting "idle"

[Matthias A. <mat...@gm...>]

Brendan Lynch <bre...@ai...> writes:

> This code works fine until a notification of new mail is received (a "*
> 1 EXISTS" message is received).  At this point normally one is in the
> "imap_ok()" routine called from line 637, and this correctly receives
> the notification message and parses it, updating the "count" variable.
> However it does not the return to imap_idle() routine, but instead
> reissues a recv call having set a much longer (300s) timeout and after
> the 300s have expired it then returns an error causing fetchmail to drop
> the IMAP connection (a main point of this idle code was to keep the
> connection open for subsequent message retrieval).  Net result is that
> delivery of mail is delayed by 5 minutes and the server connection is
> dropped and reestablished each time a wait for mail occurs.
>
> The problem seems to be caused by the loop condition in imap_ok():
>
>    151         }
>    152     } while
>    153         (tag[0] != '\0' && strncmp(buf, tag, strlen(tag)));
>    154
>
> This assumes that tag[0] will be set to '\0' if one is not waiting for a
> tagged response.  In this case the code should not be waiting for a
> tagged response (it is waiting for an unsolicited notification).
> However the 'tag' global character array is set by the gen_transact() at
> line 630 and is not cleared before the call to imap_ok() at line 637.

This is correct, an IMAP client is supposed to parse untagged responses
until a tagged response is received. Trying with Dovecot and hacking
fetchmail a bit so it doesn't recognize RECENT and uses the NOOP
emulation code yields:

...
fetchmail: IMAP> A0010 NOOP
fetchmail: IMAP< A0010 OK NOOP completed.
fetchmail: IMAP> A0011 NOOP
fetchmail: IMAP< * 1 EXISTS
fetchmail: IMAP< * 1 RECENT
fetchmail: IMAP< A0011 OK NOOP completed.
fetchmail: IMAP> A0012 NOOP
...

So it waits for the tagged NOOP response, and this is a requirement so
it actually picks up both the EXISTS and the RECENT responses of working
servers. Servers that do not respond with a tagged response to a NOOP
command are broken.

> The fix is a very simple one-line change to imap_idle (2 lines with
> comments):
>
>    630         ok = gen_transact(sock, "NOOP");
>    631         /* if there's an error (not likely) or we just found mail
> (stage
>    632          * has changed, timeout has also been restored), we're
> done */
>    633         if (ok != 0 || stage != STAGE_IDLE)
>    634             return(ok);
>    635
> +   636         /* clear tag so imap_ok does not expect tagged response */
> +   637         tag[0]='\0';
>    638         /* wait (briefly) for an unsolicited status update */
>    639         ok = imap_ok(sock, NULL);

So this patch would sort of break the IMAP client because it would jump
out of the loop before having read the reply. This requires more
thought. Casper Gripenberg reported a similar problem, so perhaps some
common upstream server software is the culprit (and he suggested a
different fix, I'll have a look at that too).

What software is your upstream server running?

Can I see a "fetchmail -Nvvv --nosyslog" trace of a failing IMAP session
with NOOP emulation?

> A second, more minor problem is that getting a "* RECENT" notification
> does not break a caller out of the imap_idle's imap_ok() call.  This
> causes an extra 28second wait after being notified about a message
> before it is actually received.
>
> Diffs for the complete set of changes against 6.3.2 are attached to this
> email.
>
> I have seen this in fetchmail 6.2.5 and 6.3.2 on linux platforms, but
> this problem should be generic to all platforms.
> diff -Naur fetchmail-6.3.2/imap.c fetchmail-6.3.2a/imap.c
> --- fetchmail-6.3.2/imap.c  2006-01-20 10:38:45.000000000 +0000
> +++ fetchmail-6.3.2a/imap.c 2006-02-23 23:54:52.000000000 +0000
> @@ -116,6 +116,17 @@
>         {
>         recentcount_ok = 1;
>         recentcount = atoi(buf+2);
> +       /*
> +        * Kluge to handle IDLE simulation. If we are in STAGE_IDLE and
> +        * we are simulating (has_idle unset) we need to alert calling
> +        * imap_idle() to the fact that we have received a new "recent"
> +        * alert in order to break the imap_idle() out of its wait.
> +        */
> +       if (!has_idle && stage == STAGE_IDLE)
> +       {
> +           mytimeout = saved_timeout;
> +           stage = STAGE_FETCH;
> +       }
>         }
>         else if (strstr(buf, " EXPUNGE"))
>         {

This looks acceptable.

How can a server mark a new message "RECENT" without also sending an
"EXISTS"? I'm inclined to consider such behavior broken. I'm willing to
apply this nonetheless as I don't think it would hurt anyone.

-- 
Matthias Andree

[fetchmail-users] Re: Fetchmail

[Sunil S. <sh...@bo...>]

Hello Indika,

Quoting from Indika Mendis's mail on Mon, Feb 27, 2006:
>    My  name  is  Indika  and  using  fetchmail in my linux mail server to
>    download  mails.  It  is working properly...  but I have to do a small
>    thing....  Our  staff  members are receiving some unwanted emails with
>    several  subjects.  Is  there  any  configuration  that we can do with
>    fetchmail  or  any  other  thing  in linux servers....  I want to stop
>    these mail by filtering subject wise... pl help me to solve this prob.

This is a fetchmail FAQ entry:

<http://fetchmail.berlios.de/fetchmail-FAQ.html#G4>

For filtering by subject, use procmail or maildrop. For that, you will
have to find out what programs are installed on your system. Also, you
will have to analyse the output of "fetchmail -v".

Here is one sample setup:

Question                                             Possible answer
--------------------------------------------------------------------
What SMTP servers are installed on my system?        qmail, sendmail
What mail filtering programs are installed?          maildrop, procmail
Where is fetchmail forwarding my mails to?           SMTP server
What SMTP server is running on my system?            sendmail
Is my SMTP server also delivering mails directly?    no
Who is delivering my mails?                          procmail
What spam filtering programs are installed?          spamassassin

In this setup, procmail has already been enabled. So, it is the
question of just writing the procmailrc. Details of setting up the
procmailrc (per-user as well as globally) can be found in procmail
documentation.

If you are interesting in spam filtering, you can enable that too in
the rc file of the mail-filtering program.

If you have more queries, please subscribe to the user mailling list.
You can find more details of the lists at
<http://fetchmail.berlios.de/>. After subscribing, send the output of
"fetchmail -V" and "fetchmail -v" to the user mailing list.

-- 
Sunil Shetye.

[fetchmail-users] Bug in imap_idle() with mail servers not supporting "idle"

[Brendan L. <bre...@ai...>]

I've run across what I believe to be a bug in imap_idle() on servers not 
supporting the IDLE extension.  There is code in imap_idle() that tries 
to simulate and idle() call with frequent NOOP calls:

    624     } else {  /* no idle support, fake it */
    625         /* when faking an idle, we can't assume the server will
    626          * send us the new messages out of the blue (RFC2060);
    627          * this timeout is potentially the delay before we notice
    628          * new mail (can be small since NOOP checking is cheap) */
    629         mytimeout = 28;
    630         ok = gen_transact(sock, "NOOP");
    631         /* if there's an error (not likely) or we just found 
mail (stage
    632          * has changed, timeout has also been restored), we're 
done */
    633         if (ok != 0 || stage != STAGE_IDLE)
    634             return(ok);
    635
    636         /* wait (briefly) for an unsolicited status update */
    637         ok = imap_ok(sock, NULL);
    638         /* again, this is new mail or an error */
    639         if (ok != PS_IDLETIMEOUT)
    640             return(ok);
    641     }

This code works fine until a notification of new mail is received (a "* 
1 EXISTS" message is received).  At this point normally one is in the 
"imap_ok()" routine called from line 637, and this correctly receives 
the notification message and parses it, updating the "count" variable.  
However it does not the return to imap_idle() routine, but instead 
reissues a recv call having set a much longer (300s) timeout and after 
the 300s have expired it then returns an error causing fetchmail to drop 
the IMAP connection (a main point of this idle code was to keep the 
connection open for subsequent message retrieval).  Net result is that 
delivery of mail is delayed by 5 minutes and the server connection is 
dropped and reestablished each time a wait for mail occurs.

The problem seems to be caused by the loop condition in imap_ok():

    151         }
    152     } while
    153         (tag[0] != '\0' && strncmp(buf, tag, strlen(tag)));
    154

This assumes that tag[0] will be set to '\0' if one is not waiting for a 
tagged response.  In this case the code should not be waiting for a 
tagged response (it is waiting for an unsolicited notification).  
However the 'tag' global character array is set by the gen_transact() at 
line 630 and is not cleared before the call to imap_ok() at line 637.

The fix is a very simple one-line change to imap_idle (2 lines with 
comments):

    630         ok = gen_transact(sock, "NOOP");
    631         /* if there's an error (not likely) or we just found 
mail (stage
    632          * has changed, timeout has also been restored), we're 
done */
    633         if (ok != 0 || stage != STAGE_IDLE)
    634             return(ok);
    635
+   636         /* clear tag so imap_ok does not expect tagged response */
+   637         tag[0]='\0';
    638         /* wait (briefly) for an unsolicited status update */
    639         ok = imap_ok(sock, NULL);

A second, more minor problem is that getting a "* RECENT" notification 
does not break a caller out of the imap_idle's imap_ok() call.  This 
causes an extra 28second wait after being notified about a message 
before it is actually received.

Diffs for the complete set of changes against 6.3.2 are attached to this 
email.

I have seen this in fetchmail 6.2.5 and 6.3.2 on linux platforms, but 
this problem should be generic to all platforms.

Re: [fetchmail-users] Incorrect header line and lost mails

[Matthias A. <mat...@gm...>]

Frederic Marchal <fre...@wo...> writes:

> Well, I'm a C programmer and I may be able to help but I'm mainly a
> windows programmer (yes, I know, shame on me :-) ) and I'm not yet all
> too familiar with the Linux programming and especially with the patch
> mechanism and none at all with the diff tool.

The workflow is:

1. create a copy of all files you need to edit, for instance, 
   copy foo.c to foo.c.orig

   alternatively, copy the whole directory tree recursively

2. edit until you're happy with the code

3. test the edits (compile, install, run)

4. generate a unified (preferred) or context patch.  Not all diff
   utilities support unified output, but all are to support context
   format, as mandated by the relevant standard, IEEE Std 1003.1-2001.

   Plain ed-style patches (unfortunately still the default in the diff
   utility) are impractical so they are not usually accepted.

The diff tool is quite simple actually, if you created file copies:

diff -u OLDFILE NEWFILE >PATCHFILE (unified diff, preferred)

diff -c OLDFILE NEWFILE >PATCHFILE (context diff, alternative)

If you copied the whole directory before the edits:

diff -u -r BACKUP_DIR EDITED_DIR >PATCHFILE    (or -c instead of -u)

If you added new files to EDITED_DIR, you'd use -N but I don't think
we'll need this here.

diff, used either way, will produce a file (PATCHFILE) that you'd attach
to a message and mail to me (or Rob, or preferably the -devel@ list) or
upload to the BerliOS patch tracker.

If I have the choice, I prefer the -devel@ list. That's public, archived
and easiest for me since I don't need to fetch anything, but get it
pushed into my inbox.


The patch utility does the reverse and will usually just be used like
this, it derives the file names from the PATCHFILE.

patch <PATCHFILE

perhaps with -d (to change directory) or -p (to strip leading path
components from the paths shown in the patch file; used if patch cannot
find the files).

> Nevertheless, I have identified the core of the problem (its the call
> to readheaders in drive.c and the response to PS_REFUSED). I still
> have to check what happens if an invalid FROM or TO header is passed
> after readheaders but I think I can fix the program.

That would be great.

Basically they should not matter that much. fetchmail has some
deprecated code to guess from To:/Cc: or similar, but that is an
unreliable concept and doesn't deserve attention, From: shouldn't matter
as long as Return-Path: is given (and even then, From: shouldn't
matter). Missing or corrupted Return-Path: headers should be treated as
though the message had contained "Return-Path: <>".

> I'll also need some direction about the proper way of doing it for a
> good integration in the current source.

The bigger concern is that fetchmail shouldn't be emitting headers it
knows are broken, so that the next hop (the MTA or MDA) does not get
confused. I wonder if fetchmail should prefix the broken lines with
X-Fetchmail-Escaped-Broken-Header: or something similar.

-- 
Matthias Andree

Re: [fetchmail-users] Incorrect header line and lost mails

[Frederic M. <fre...@wo...>]

Matthias Andree wrote:
> Rob MacGregor <rob...@gm...> writes:
>
>   
>> On 2/13/06, Frederic Marchal <fre...@wo...> wrote:
>>     
>>> Hello,
>>>
>>> I found 15 lines like this in my log this morning and 3 more earlier
>>> last week:
>>>
>>> fetchmail[1226]: incorrect header line found while scanning headers
>>>       
>> You'll find threads about this in the list archive.  I don't remember
>> the outcome, but I've a feeling that 6.3.x handles these (illegal)
>> messages differently.
>>     
>
> I don't recall having a fix for this particular issue, but 6.3.2
> generally has some dozens of bugfixes worth having anyways. I'll put
> this on my 6.3.3 TODO, but don't think it will appear within the next
> three weeks (unless someone else sends a patch). Don't hold your breath,
> I'm busy ATM.
>   

Well, I'm a C programmer and I may be able to help but I'm mainly a 
windows programmer (yes, I know, shame on me :-) ) and I'm not yet all 
too familiar with the Linux programming and especially with the patch 
mechanism and none at all with the diff tool. Nevertheless, I have 
identified the core of the problem (its the call to readheaders in 
drive.c and the response to PS_REFUSED). I still have to check what 
happens if an invalid FROM or TO header is passed after readheaders but 
I think I can fix the program. I'll also need some direction about the 
proper way of doing it for a good integration in the current source.

If you think it is reasonable, we can discuss this further on the 
fetchmail-devel list.

Frederic

Re: [fetchmail-users] Incorrect header line and lost mails

[Frederic M. <fre...@wo...>]

Rob MacGregor wrote:
> On 2/13/06, Frederic Marchal <fre...@wo...> wrote:
>   
>> Hello,
>>
>> I found 15 lines like this in my log this morning and 3 more earlier
>> last week:
>>
>> fetchmail[1226]: incorrect header line found while scanning headers
>>     
>
> You'll find threads about this in the list archive.  I don't remember
> the outcome, but I've a feeling that 6.3.x handles these (illegal)
> messages differently.
>
> It would be worth giving the latest version a try and see if that helps.
>   
I read the source code of 6.3.2 and the handling of those illegal mails 
is still exactly the same. Those e-mails are discarded and the 
"incorrect header line" message is written to the log.

Frederic


WOW Company or its affiliates do not accept legal responsibility 
for the contents of this message. The views or opinions 
presented are solely those of the author and do not necessarily 
represent those of WOW Company or any of its affiliates.

No contract will be established with WOW Company other than 
in writing (letter) and with the explicit authorization of the director 
of WOW Company.

Re: [fetchmail-users] Incorrect header line and lost mails

[Matthias A. <mat...@gm...>]

Rob MacGregor <rob...@gm...> writes:

> On 2/13/06, Frederic Marchal <fre...@wo...> wrote:
>> Hello,
>>
>> I found 15 lines like this in my log this morning and 3 more earlier
>> last week:
>>
>> fetchmail[1226]: incorrect header line found while scanning headers
>
> You'll find threads about this in the list archive.  I don't remember
> the outcome, but I've a feeling that 6.3.x handles these (illegal)
> messages differently.

I don't recall having a fix for this particular issue, but 6.3.2
generally has some dozens of bugfixes worth having anyways. I'll put
this on my 6.3.3 TODO, but don't think it will appear within the next
three weeks (unless someone else sends a patch). Don't hold your breath,
I'm busy ATM.

-- 
Matthias Andree

Re: [fetchmail-users] Incorrect header line and lost mails

[Rob M. <rob...@gm...>]

On 2/13/06, Frederic Marchal <fre...@wo...> wrote:
> Hello,
>
> I found 15 lines like this in my log this morning and 3 more earlier
> last week:
>
> fetchmail[1226]: incorrect header line found while scanning headers

You'll find threads about this in the list archive.  I don't remember
the outcome, but I've a feeling that 6.3.x handles these (illegal)
messages differently.

It would be worth giving the latest version a try and see if that helps.

--
                 Please keep list traffic on the list.
Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

[fetchmail-users] Incorrect header line and lost mails

[Frederic M. <fre...@wo...>]

Hello,

I found 15 lines like this in my log this morning and 3 more earlier 
last week:

fetchmail[1226]: incorrect header line found while scanning headers


I read the source of fetchmail to learn more about it and I found out 
that any header containing at least one line without a colon and not 
starting with a space is considered illegal as per rfc822 and send to 
oblivion without any other consideration (at least none I could find).

Alas, it turns out that one of the three e-mails I could recover was 
definitely not a spam... I agree it was not fully rfc822 compliant but 
it was a good e-mail according to *my* standards and it could have been 
delivered since the invalid line was a very long TO header from hotmail 
(not rfc822 compliant either) wrapped by my ISP  !

I agree that the sin was in the original e-mail, but is it really the 
purpose of fetchmail to enforce rfc822 so drastically ? With such a 
name, I would expect it to fetch the e-mails and leave any 
"counter-measure" to a program designed for such a task (or at least to 
have an option to drop or accept invalid e-mails).

Now, I'm reluctant to patch and recompile the program myself. Is it 
possible to configure fetchmail to save a copy of every mail in a local 
folder before it is processed ? Could that be achieved with the --plugin 
option and how ?

I'm using fetchmail 6.2.5 as provided by the stable branch of debian.

Frederic







WOW Company or its affiliates do not accept legal responsibility 
for the contents of this message. The views or opinions 
presented are solely those of the author and do not necessarily 
represent those of WOW Company or any of its affiliates.

No contract will be established with WOW Company other than 
in writing (letter) and with the explicit authorization of the director 
of WOW Company.

[fetchmail-users] patch 6.3.2.1 for fetchmail 6.3.2 fixes SEGFAULT with .netrc files

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings,

Craig Leres identified a new bug in fetchmail 6.3.2 (that now tries to
erase the .netrc passwords before freeing its memory) that causes
fetchmail to crash right after reading the .netrc file.

Craig also provided a patch that I have accepted into the SVN
repository that will appear in fetchmail 6.3.3. I am including the patch
here (if your mailer is not GnuPG enabled, you need to manually edit all
lines that start with "- -" so that they start with "-" - i. e.  remove
the first minus and blank character on those lines).

I recommend to add this patch as an interim fix to all distribution
packages and on all sites that wish to use .netrc files.

The patch can also be downloaded:

http://download.berlios.de/fetchmail/patch-6.3.2.1-fix-netrc-SIGSEGV.diff

My GnuPG signature for the patch is available at:

http://download.berlios.de/fetchmail/patch-6.3.2.1-fix-netrc-SIGSEGV.diff.asc

Here is the patch:

.......................................................................
Craig Leres identified a problem that makes fetchmail 6.3.2 (only this
version) crash if the .netrc file does not contain a password for a
particular account.

This patch is mostly Craig Leres' work has been committed to the
SVN repository and should be applied to fetchmail 6.3.2 on all sites
that plan to use netrc files:

Index: netrc.c
===================================================================
- --- netrc.c	(Revision 4683)
+++ netrc.c	(Revision 4684)
@@ -314,8 +314,10 @@
 free_netrc(netrc_entry *a) {
     while(a) {
 	netrc_entry *n = a->next;
- -	memset(a->password, 0x55, strlen(a->password));
- -	xfree(a->password);
+	if (a->password != NULL) {
+		memset(a->password, 0x55, strlen(a->password));
+		free(a->password);
+	}
 	xfree(a->login);
 	xfree(a->host);
 	xfree(a);

Sorry for the inconvenience.  -- Matthias Andree, 2006-01-30
.......................................................................

Regards,

- -- 
Matthias Andree
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFD3dmkvmGDOQUufZURAv1+AKDYf5zB++Dyj6buzKS0Fz6W9B70bQCglnYI
F7gplc9LV+Ixh88mq0DSFNI=
=4UM8
-----END PGP SIGNATURE-----

[fetchmail-users] Re: [fetchmail-users] Re: [fetchmail-users] Re: [fetchmail-users] fetchmail doesn´t fetching

[Rob M. <rob...@gm...>]

Please:

1) Keep traffic on the list
2) Don't top post


On 26/01/06, Alex Moreno <al3...@gm...> wrote:
> ok, sorry, i thougth it was -V
>
> -------
> trantor:~# fetchmail -v
> fetchmail: no mailservers have been specified.
> trantor:~# fetchmail -v -v -v
> fetchmail: no mailservers have been specified.
> -------
>
>  is that an error?

Yes.  Try specifying the path to your configuration file:

fetchmail -v -v -v -f /path/to/fetchmailrc

--
                 Please keep list traffic on the list.
Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

[fetchmail-users] Re: [fetchmail-users] Re: [fetchmail-users] fetchmail doesn´t fetching

[Rob M. <rob...@gm...>]

On 26/01/06, Alex Moreno <al3...@gm...> wrote:
> smtp works ok. I´m using qmail some time ago and now, after installing mutt
> i can send mails without problems... The problem appears trying to read the
> incoming mails by pop3... fetchmail is not writing my mails on ~/Mail or any
> other known directory...

Once more, before I give up...

Output of "fetchmail -v -v -v" please.

--
                 Please keep list traffic on the list.
Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] fetchmail doesn´t fetching

[Matthias A. <mat...@gm...>]

On Thu, 26 Jan 2006, Alex Moreno wrote:

> smtp works ok. I´m using qmail some time ago and now, after installing mutt
> i can send mails without problems... The problem appears trying to read the
> incoming mails by pop3... fetchmail is not writing my mails on ~/Mail or any
> other known directory...

Please do not top-post.  http://www.netmeister.org/news/learn2quote.html
applies to mailing lists, too.

Fetchmail DOES NOT (as of 6.3.X and older versions) deliver to mailboxes
or maildirs.  Fetchmail hands mail off to your SMTP server (qmail in
your case), which is responsible for delivery.

Check your qmail configuration (its mail log, qmail-showctl
output, qmail-qread and similar) to find out where your mail is stuck or
going to. Double-check if qmail knows what your computer is called
(/var/qmail/control/me, .../locals).


Side note: qmail turned out over the years to be a defective and
unmaintained software package, and rather complicated to set up. My
/personal/ favorite alternatives at this time are Postfix + Dovecot.
Others may be happy with Courier-IMAP, popa3d, Exim, Courier as a whole,
or other systems. (<http://www.postfix.org/>, <http://dovecot.org/>)

Details on qmail flaws are given in
<http://home.pages.de/~mandree/qmail-bugs.html>.

-- 
Matthias Andree