fetchmail-users — User questions and discussion

Re: [fetchmail-users] [PATCH] Re: Bug in imap_idle() with mail servers not supporting "idle"

[Matthias A. <mat...@gm...>]

Brendan Lynch <bre...@ai...> writes:

> That seem to be intentional first-pass behaviour (i.e. not really the general
> Idle polling case).  Look at these lines from imap_getrange():
>
>     732          * We should have an expunge here to
>     733          * a) avoid fetching deleted mails during 'fetchall'
>     734          * b) getting a wrong count of mails during 'no fetchall'
>     735          */
>     736         if (!check_only && !ctl->keep && count > 0)
>     737         {
>     738             ok = internal_expunge(sock);
>     739             if (ok)
>
> This generates the EXPUNGE you see.  However this only happens for the first
> message, since this is the "else" case of the if statement:
>
>     672     if (pass > 1)
>     673     {
>     674         /* deleted mails have already been expunged by
>     675          * end_mailbox_poll().
>
> and will only be taken before the first message has been seen. After the first
> message has been seen no EXPUNGE is sent (and the comment suggests why the
> code does not do an EXPUNGE in this case).

Well, yes, but I think it's pointless to run EXPUNGE in this particular
context. We might only expunge the deletions of another process, and
that's probably not what fetchmail should be doing.

I'll probably merge Sunil's patch tomorrow.

-- 
Matthias Andree

Re: [fetchmail-users] [PATCH] Re: Bug in imap_idle() with mail servers not supporting "idle"

[Brendan L. <bre...@ai...>]

That seem to be intentional first-pass behaviour (i.e. not really the 
general Idle polling case).  Look at these lines from imap_getrange():

    732          * We should have an expunge here to
    733          * a) avoid fetching deleted mails during 'fetchall'
    734          * b) getting a wrong count of mails during 'no fetchall'
    735          */
    736         if (!check_only && !ctl->keep && count > 0)
    737         {
    738             ok = internal_expunge(sock);
    739             if (ok)

This generates the EXPUNGE you see.  However this only happens for the 
first message, since this is the "else" case of the if statement:

    672     if (pass > 1)
    673     {
    674         /* deleted mails have already been expunged by
    675          * end_mailbox_poll().

and will only be taken before the first message has been seen. After the 
first message has been seen no EXPUNGE is sent (and the comment suggests 
why the code does not do an EXPUNGE in this case).

A trace shows exactly this behaviour:

First time:

    18:00:55 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0,
    0}}, NULL) = 018:00:55 setitimer(ITIMER_REAL, {it_interval={0, 0},
    it_value={28, 0}}, NULL) = 0
    18:00:55 recv(4, 0xbfea66fb, 8192, MSG_PEEK) = ? ERESTARTSYS (To be
    restarted)
    18:01:23 --- SIGALRM (Alarm clock) @ 0 (0) ---
    18:01:23 sigreturn()                    = ? (mask now [])
    18:01:23 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0,
    0}}, NULL) = 018:01:23 setitimer(ITIMER_REAL, {it_interval={0, 0},
    it_value={0, 0}}, NULL) = 0
    18:01:23 write(4, "A0013 NOOP\r\n", 12) = 12
    18:01:23 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={300,
    0}}, NULL) = 0
    18:01:23 recv(4, "A0013 OK NOOP completed\r\n", 8192, MSG_PEEK) = 25
    18:01:23 read(4, "A0013 OK NOOP completed\r\n", 25) = 25
    18:01:23 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0,
    0}}, NULL) = 018:01:23 setitimer(ITIMER_REAL, {it_interval={0, 0},
    it_value={28, 0}}, NULL) = 0
    18:01:23 recv(4, "* 1 EXISTS\r\n", 8192, MSG_PEEK) = 12
    18:01:47 read(4, "* 1 EXISTS\r\n", 12)  = 12
    18:01:47 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0,
    0}}, NULL) = 018:01:47 setitimer(ITIMER_REAL, {it_interval={0, 0},
    it_value={0, 0}}, NULL) = 0
    18:01:47 write(4, "A0014 EXPUNGE\r\n", 15) = 15
    18:01:47 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={300,
    0}}, NULL) = 0
    18:01:47 recv(4, "* 1 RECENT\r\nA0014 OK EXPUNGE completed\r\n",
    8192, MSG_PEEK) = 40
    18:01:47 read(4, "* 1 RECENT\r\n", 12)  = 12
    18:01:47 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0,
    0}}, NULL) = 018:01:47 setitimer(ITIMER_REAL, {it_interval={0, 0},
    it_value={300, 0}}, NULL) = 0
    18:01:47 recv(4, "A0014 OK EXPUNGE completed\r\n", 8192, MSG_PEEK) = 28
    18:01:47 read(4, "A0014 OK EXPUNGE completed\r\n", 28) = 2818:01:47
    setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
    18:01:47 write(4, "A0015 FETCH 1 RFC822.SIZE\r\n", 27) = 27
    18:01:47 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={300,
    0}}, NULL) = 0
    18:01:47 recv(4, "* 1 FETCH (RFC822.SIZE 7640)\r\n", 8192, MSG_PEEK)
    = 30

Later times:

    18:03:12 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={28,
    0}}, NULL) = 0
    18:03:12 recv(4, 0xbfea66fb, 8192, MSG_PEEK) = ? ERESTARTSYS (To be
    restarted)
    18:03:40 --- SIGALRM (Alarm clock) @ 0 (0) ---
    18:03:40 sigreturn()                    = ? (mask now [])
    18:03:40 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0,
    0}}, NULL) = 018:03:40 setitimer(ITIMER_REAL, {it_interval={0, 0},
    it_value={0, 0}}, NULL) = 0 18:03:40 write(4, "A0024 NOOP\r\n", 12) = 12
    18:03:40 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={300,
    0}}, NULL) = 0
    18:03:40 recv(4, "A0024 OK NOOP completed\r\n", 8192, MSG_PEEK) = 25
    18:03:40 read(4, "A0024 OK NOOP completed\r\n", 25) = 25
    18:03:40 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0,
    0}}, NULL) = 018:03:40 setitimer(ITIMER_REAL, {it_interval={0, 0},
    it_value={28, 0}}, NULL) = 0
    18:03:40 recv(4, "* 1 EXISTS\r\n", 8192, MSG_PEEK) = 12
    18:03:49 read(4, "* 1 EXISTS\r\n", 12)  = 12
    18:03:49 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0,
    0}}, NULL) = 018:03:49 setitimer(ITIMER_REAL, {it_interval={0, 0},
    it_value={0, 0}}, NULL) = 018:03:49 write(4, "A0025 NOOP\r\n", 12) = 12
    18:03:49 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={300,
    0}}, NULL) = 0
    18:03:49 recv(4, "* 1 RECENT\r\nA0025 OK NOOP completed\r\n", 8192,
    MSG_PEEK) = 37
    18:03:49 read(4, "* 1 RECENT\r\n", 12)  = 12
    18:03:49 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0,
    0}}, NULL) = 018:03:49 setitimer(ITIMER_REAL, {it_interval={0, 0},
    it_value={300, 0}}, NULL) = 0
    18:03:49 recv(4, "A0025 OK NOOP completed\r\n", 8192, MSG_PEEK) = 25
    18:03:49 read(4, "A0025 OK NOOP completed\r\n", 25) = 25
    18:03:49 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0,
    0}}, NULL) = 018:03:49 setitimer(ITIMER_REAL, {it_interval={0, 0},
    it_value={0, 0}}, NULL) = 018:03:49 write(4, "A0026 FETCH 1
    RFC822.SIZE\r\n", 27) = 27

After the first RECEIVE further RECEIVES are followed by a FETCH without 
intervening EXPUNGE.

Brendan

mat...@gm... wrote:

>I've committed the patch. One thing makes me wonder though:
>
>fetchmail: IMAP> A0004 NOOP
>fetchmail: IMAP< A0004 OK NOOP completed.
>fetchmail: IMAP> A0005 NOOP
>fetchmail: IMAP< * 1 EXISTS
>fetchmail: IMAP< * 1 RECENT
>fetchmail: IMAP< A0005 OK NOOP completed.
>fetchmail: IMAP> A0006 EXPUNGE
>fetchmail: IMAP< A0006 OK Expunge completed.
>
>Why does fetchmail perform an EXPUNGE after NOOP?
>Accident or intent?
>
>  
>

Re: [fetchmail-users] O'Reilly DevCenter weighs in on fetchmail

[Rob F. <rf...@fu...>]

Nico Golde wrote:
> > | *  fetchmail, but who uses fetchmail anymore?
>
> What do they use? ;-)
> Kmail builtin pop3 support? ;)

Kmail, Evolution, Thunderbird, or maybe just Gmail.

-- 
==============================| "A slice of life isn't the whole cake
 Rob Funk <rf...@fu...> | One tooth will never make a full grin"
 http://www.funknet.net/rfunk |    -- Chris Mars, "Stuck in Rewind"

Re: [fetchmail-users] O'Reilly DevCenter weighs in on fetchmail

[Nico G. <ni...@ng...>]

Hi,
* Rob Funk <rf...@fu...> [2006-03-11 17:57]:
[...] 
> | *  fetchmail, but who uses fetchmail anymore?
> [...]

What do they use? ;-)
Kmail builtin pop3 support? ;)
Regards Nico
-- 
Nico Golde - JAB: ni...@ja... | GPG: 0x73647CFF
http://www.ngolde.de | http://www.muttng.org | http://grml.org
Forget about that mouse with 3/4/5 buttons -
gimme a keyboard with 103/104/105 keys!

[fetchmail-users] O'Reilly DevCenter weighs in on fetchmail

[Rob F. <rf...@fu...>]

I couldn't help being amused at this....

http://www.linuxdevcenter.com/lpt/a/6502
In an article on "Fine-Tuning Kubuntu", we have:
| Other services are optional, so start them at boot only
| if you know you need them:
| *  atd, for scheduled batch jobs.
| *  bluez-utils, for Bluetooth devices.
| *  bootlogd, not necessary, as syskslogd handles bootlogs.
| *  cupsys, necessary for printing.
| *  evms, logical volume manager. Don't use this unless you have logical
|    volumes configured. 
| *  fetchmail, but who uses fetchmail anymore?
[...]

-- 
==============================|   "A microscope locked in on one point
 Rob Funk <rf...@fu...> |Never sees what kind of room that it's in"
 http://www.funknet.net/rfunk |    -- Chris Mars, "Stuck in Rewind"

[fetchmail-users] [PATCH-2] Re: Bug in imap_idle() with mail servers not supporting "idle"

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Fri, Mar 10, 2006:
> I've committed the patch. One thing makes me wonder though:
> 
> fetchmail: IMAP> A0004 NOOP
> fetchmail: IMAP< A0004 OK NOOP completed.
> fetchmail: IMAP> A0005 NOOP
> fetchmail: IMAP< * 1 EXISTS
> fetchmail: IMAP< * 1 RECENT
> fetchmail: IMAP< A0005 OK NOOP completed.
> fetchmail: IMAP> A0006 EXPUNGE
> fetchmail: IMAP< A0006 OK Expunge completed.
> 
> Why does fetchmail perform an EXPUNGE after NOOP?
> Accident or intent?

Looks like an accident.

The original intention is to do an EXPUNGE immediately after SELECT.
In this case, since no mails were found after SELECT, the EXPUNGE was
sent after the NOOP(s). This EXPUNGE after NOOP is pointless and
confusing. This patch should remove this.

======================================================================
Index: fetchmail-6.3/imap.c
===================================================================
--- fetchmail-6.3/imap.c	(revision 4730)
+++ fetchmail-6.3/imap.c	(working copy)
@@ -737,16 +737,6 @@
 				    "%d messages waiting after first poll\n",
 				    count), count);
 
-	/* no messages?  then we may need to idle until we get some */
-	while (count == 0 && do_idle) {
-	    ok = imap_idle(sock);
-	    if (ok)
-	    {
-		report(stderr, GT_("re-poll failed\n"));
-		return(ok);
-	    }
-	}
-
 	/*
 	 * We should have an expunge here to
 	 * a) avoid fetching deleted mails during 'fetchall'
@@ -765,6 +755,23 @@
 					"%d messages waiting after expunge\n",
 					count), count);
 	}
+
+	if (count == 0 && do_idle)
+	{
+	    /* no messages?  then we may need to idle until we get some */
+	    while (count == 0) {
+		ok = imap_idle(sock);
+		if (ok)
+		{
+		    report(stderr, GT_("re-poll failed\n"));
+		    return(ok);
+		}
+	    }
+	    if (outlevel >= O_DEBUG)
+		report(stdout, ngettext("%d message waiting after re-poll\n",
+					"%d messages waiting after re-poll\n",
+					count), count);
+	}
     }
 
     *countp = count;
======================================================================

-- 
Sunil Shetye.

[fetchmail-users] fetchmail 6.3.3.rc1 release candidate

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings,

collecting six weeks worth of bugfixes, I have uploaded a release
candidate for fetchmail 6.3.3 to http://mandree.home.pages.de/fetchmail/

Please test and see if it fixed the bug(s) you reported recently.

My thanks go to Sunil Shetye for lots of the fixes.

Changes in fetchmail 6.3.3.rc1 (from 6.3.2):

# BUG FIXES:
* Do not attempt to overwrite the netrc password if none has been specified.
  This fixes a segmentation fault bug introduced into 6.3.2.
  Fixes BerliOS bug #6234.  BerliOS patch #804 by Craig Leres.
  The patch, as accepted into fetchmail, was available separately from
  <http://download.berlios.de/fetchmail/patch-6.3.2.1-fix-netrc-SIGSEGV.diff>
* Handle other clients concurrently accessing IMAP mailboxes better.
  Fetchmail quits the poll if the EXPUNGE count does not match expectations, and
  servers not updating RECENT counts after EXPUNGE are handled in a better way.
  (Patch by Sunil Shetye.)
* fetchmail no longer replaces the local user ID for an empty envelope sender
  when using the proprietary SDPS extension for POP3. Fixes Debian Bug#353575,
  reported by Roger Lynn.
* fetchmail no longer prints empty lines in verbose mode when using syslog.
* fetchmail no longer prints UID lists in verbose mode when using syslog.
* POP3: fetchmail can now use UIDL in fetchall keep mode, to avoid re-fetching
  the same messages again when the fetchall keyword is removed. Patch by
  Sunil Shetye. For details, please see
  <http://lists.berlios.de/pipermail/fetchmail-users/2006-March/000308.html>
* IMAP: fix hangs in NOOP-based IDLE emulation. Reported by Casper Gripenberg
  and Brendan Lynch, fix by Sunil Shetye (his patch was merged) and Brendan
  Lynch.
* ./configure --quiet is now quieter (no SSL and fallback-related output).

# CHANGES:
* --idle can now be specified on the command line, too.
* --fetchall is now supported on the command-line.

# DOCUMENTATION:
* "ssl" is a user option rather than a server option. Patch by Nico Golde.
  Fixes Debian Bug#354661, reported by Keith Hellman.
* The manual page now suggests "--" before the addresses in the sendmail MDA
  example, for safety.
* The FAQ item X9, Domino IMAP omits Content-Transfer-Encoding header, was
  added. Information provided by Anthony Kim on the fetchmail-friends list
  in March 2006.
* Credit Chris Boyle with the NOOP emulation code for IDLE in fetchmail 6.2.4.
  Eric forgot to credit Chris, thanks to Sunil Shetye for providing these links:
  http://lists.ccil.org/pipermail/fetchmail-friends/2003-July/007705.html
  http://lists.ccil.org/pipermail/fetchmail-friends/2003-July/007713.html

Happy fetchmailing,
Matthias Andree
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFEEV7GvmGDOQUufZURAjl5AJ9fBOAS+WO+hOQFM6AFPcEW+opMUwCfZQlv
c8JaKWTGJHrK29QHTkgDFcw=
=VIjl
-----END PGP SIGNATURE-----

Re: [fetchmail-users] [PATCH] Re: Bug in imap_idle() with mail servers not supporting "idle"

[Matthias A. <mat...@gm...>]

I've committed the patch. One thing makes me wonder though:

fetchmail: IMAP> A0004 NOOP
fetchmail: IMAP< A0004 OK NOOP completed.
fetchmail: IMAP> A0005 NOOP
fetchmail: IMAP< * 1 EXISTS
fetchmail: IMAP< * 1 RECENT
fetchmail: IMAP< A0005 OK NOOP completed.
fetchmail: IMAP> A0006 EXPUNGE
fetchmail: IMAP< A0006 OK Expunge completed.

Why does fetchmail perform an EXPUNGE after NOOP?
Accident or intent?

-- 
Matthias Andree

Re: [fetchmail-users] After reading the FAQ, a question about lost/mangled attachments

[Matthias A. <mat...@gm...>]

Peter N. Spotts wrote:

> My apologies for the tardy reply. I'll remove the fetchall command from
> my rc file and see how things work. I should have said earlier that I
> had just updated to 6.3.2 that morning but hadn't tried it yet...I just
> had this urge to write something after several weeks of
> frustration!  ;-)

Well, I hope that your frustration ended with the installation of 6.3.2 and
you'll find out that next time you run it.

Otherwise, let me know quick so the fix can become part of the 6.3.3 release.

Thanks,
Matthias

Re: [fetchmail-users] After reading the FAQ, a question about lost/mangled attachments

[Peter N. S. <ps...@al...>]

On Sat, 2006-03-04 at 19:31 +0100, Matthias Andree wrote:
> "Peter N. Spotts" <ps...@al...> writes:
> 
> > I've been running fetchmail on SuSE 10.0 on my laptop, and until today
> > (when I installed the latest version of fetchmail) I've been running
> > 6.2.X.
> 
> [...]
> 
> > So although my ISP is Comcast (I noted the Comcast caveats on
> > the FAQ page), Comcast does not seem to be the problem either.
> 
> That would be news.
> 
> fetchmail, beginning with version 6.3.2, recognizes Comcast's broken
> servers ("Maillennium POP3/PROXY server") and disables the problematic
> use of the TOP command and uses RETR instead - so updating to 6.3.2
> should have fixed all known Comcast problems.
> 
> -- 
> Matthias Andree

Matthias,

My apologies for the tardy reply. I'll remove the fetchall command from
my rc file and see how things work. I should have said earlier that I
had just updated to 6.3.2 that morning but hadn't tried it yet...I just
had this urge to write something after several weeks of
frustration!  ;-)

With best regards,

Pete
-- 

      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            Peter N. Spotts | Science Correspondent
                 The Christian Science Monitor
             One Norway Street, Boston  MA  02115
      Office: 617-450-2449 | Office in home: 508-520-3139
        Email: ps...@al... | www.csmonitor.com
                 Amateur-radio call - KC1JB
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

"The knack of flying is to throw yourself at the ground and miss."
                                                 -- Douglas Adams

[fetchmail-users] Re: [PATCH] Re: Bug in imap_idle() with mail servers not supporting "idle"

[Brendan L. <bre...@ai...>]

Your patch works too.  I cannot say the code is clearer than my fix, but 
you are no doubt a lot more familiar with this code than I am.  The key 
fix (beyond not waiting for a tagged response when no tagged command had 
been issued) in your fix is at line 595, where you do not attempt to 
enter the pseudo-idle wait if a RECENT message has been received 
(recentcount != 0).

Thanks,

Brendan

sh...@bo... wrote:

>Quoting from Brendan Lynch's mail on Thu, Mar 02, 2006:
>  
>
>>However I believe the servers is behaving correctly according to the 
>>IMAP spec: I have attached an explanation of the secondary problem I am 
>>seeing (extra 28 s wait) that I sent to Casper; let me know if you need 
>>more info.
>>    
>>
>
>Yes, I have now read RFC 3501 section 5.3. This behaviour is correct.
>
>  
>
>>The key point is that the imap_idle() code when 'faking' IDLE support 
>>calls imap_ok() (at line 637) without having issued any tagged command:
>>    
>>
>
>...
>
>  
>
>>Since there is no tagged command outstanding the server will not (and 
>>should not) ever send a tagged response.  If we set imap_ok to only 
>>return after a tagged response we are guaranteeing it will always only 
>>return after a timeout.
>>    
>>
>
>...
>
>  
>
>>This again first does a gen_transact() which results in sending "A0130 
>>NOOP" at 17:00:49.  A "RECENT" update is already on its way form the 
>>server, so we receive it before the "A0130 NOOP completed" at 17:00:49.  
>>In imap_ok() we first process the "* 1 RECENT" notification at line 117, 
>>setting recentcount, then (since we ARE waiting a tag) loop and process 
>>the NOOP ok.  we then return to gen_transact(), which in turn returns to 
>>imap_idle().
>>
>>Since setting "recentcount" does not also change the stage, we fail the 
>>test at line 633 of imap_idle() and continue to do an extra imap_ok() 
>>which sets a 28 second timer at 17:00:49 and does a recv() which times 
>>out at 17:01:17.  We then return from imap_idle() to imap_getrange() 
>>line 679.  At this point recentcount is set and we exit the "while 
>>(recentcount == 0 && do_idle)" loop.
>>
>>The wait between 17:00:49 and 17:01:17 was unnecessary, as already had 
>>our RECENT notification.  The second part of my fix causes imap_ok() to 
>>change the stage to STAGE_FETCH on receipt of a RECENT notification, 
>>just as it already did on receipt of a EXISTS notification. This causes 
>>the if condition "if (ok != 0 || stage != STAGE_IDLE)" to be met at line 
>>633 of imap_idle() so it immediately exits and does not do the extra 
>>unneeded imap_ok() call.
>>    
>>
>
>Yes, you have diagnosed the problem correctly and your patch is also
>correct. Based on your patch, I have prepared a new patch which does
>not modify imap_ok(), but cleans up the code in imap_idle(). Could you
>test this patch and see if this patch is acceptable to you? This patch
>is identical to the patch I had sent to Casper Gripenberg, except that
>comments relating to compliance with RFCs have been updated.
>
>  
>
>------------------------------------------------------------------------
>
>Index: fetchmail-6.3/imap.c
>===================================================================
>--- fetchmail-6.3/imap.c   (revision 4705)
>+++ fetchmail-6.3/imap.c   (working copy)
>@@ -621,7 +621,6 @@
> {
>     int ok;
> 
>-    stage = STAGE_IDLE;
>     saved_timeout = mytimeout;
> 
>     if (has_idle) {
>@@ -629,6 +628,7 @@
>    * at least every 28 minutes:
>    * (the server may have an inactivity timeout) */
>   mytimeout = 1680; /* 28 min */
>+  stage = STAGE_IDLE;
>   /* enter IDLE mode */
>   ok = gen_transact(sock, "IDLE");
> 
>@@ -637,37 +637,43 @@
>       SockWrite(sock, "DONE\r\n", 6);
>       if (outlevel >= O_MONITOR)
>       report(stdout, "IMAP> DONE\n");
>-  } else
>-      /* not idle timeout */
>-      return ok;
>+      /* reset stage and timeout here: we are not idling any more */
>+      mytimeout = saved_timeout;
>+      stage = STAGE_FETCH;
>+      /* get OK IDLE message */
>+      ok = imap_ok(sock, NULL);
>+  }
>     } else {  /* no idle support, fake it */
>-  /* when faking an idle, we can't assume the server will
>-   * send us the new messages out of the blue (RFC2060);
>-   * this timeout is potentially the delay before we notice
>-   * new mail (can be small since NOOP checking is cheap) */
>-  mytimeout = 28;
>+  /* Note: stage and timeout have not been changed here as NOOP
>+   * does not idle */
>   ok = gen_transact(sock, "NOOP");
>-  /* if there's an error (not likely) or we just found mail (stage 
>-   * has changed, timeout has also been restored), we're done */
>-  if (ok != 0 || stage != STAGE_IDLE)
>-      return(ok);
> 
>-  /* wait (briefly) for an unsolicited status update */
>-  ok = imap_ok(sock, NULL);
>-  /* again, this is new mail or an error */
>-  if (ok != PS_IDLETIMEOUT)
>-      return(ok);
>+  /* no error, but no new mail either */
>+  if (ok == PS_SUCCESS && recentcount == 0)
>+  {
>+      /* There are some servers who do send new mail
>+       * notification out of the blue. This is in compliance
>+       * with RFC 2060 section 5.3. Wait for that with a low
>+       * timeout */
>+      mytimeout = 28;
>+      stage = STAGE_IDLE;
>+      /* We are waiting for notification; no tag needed */
>+      tag[0] = '\0';
>+      /* wait (briefly) for an unsolicited status update */
>+      ok = imap_ok(sock, NULL);
>+      if (ok == PS_IDLETIMEOUT) {
>+      /* no notification came; ok */
>+      ok = PS_SUCCESS;
>+      }
>+  }
>     }
> 
>     /* restore normal timeout value */
>+    set_timeout(0);
>     mytimeout = saved_timeout;
>     stage = STAGE_FETCH;
> 
>-    /* get OK IDLE message */
>-    if (has_idle)
>-        return imap_ok(sock, NULL);
>-
>-    return PS_SUCCESS;
>+    return(ok);
> }
> 
> static int imap_getrange(int sock, 
>  
>

Re: [fetchmail-users] After reading the FAQ, a question about lost/mangled attachments

[Matthias A. <mat...@gm...>]

"Peter N. Spotts" <ps...@al...> writes:

> I've been running fetchmail on SuSE 10.0 on my laptop, and until today
> (when I installed the latest version of fetchmail) I've been running
> 6.2.X.

[...]

> So although my ISP is Comcast (I noted the Comcast caveats on
> the FAQ page), Comcast does not seem to be the problem either.

That would be news.

fetchmail, beginning with version 6.3.2, recognizes Comcast's broken
servers ("Maillennium POP3/PROXY server") and disables the problematic
use of the TOP command and uses RETR instead - so updating to 6.3.2
should have fixed all known Comcast problems.

-- 
Matthias Andree

Re: [fetchmail-users] Re: RETR with UIDL

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> runs:
>
>     fetchmail -a -d 0 ; fetchmail
>
> to download all mails once and then start normal polling. New mails
> which have been downloaded by the first instance of fetchmail will not
> be downloaded again by the second instance due to this patch.

Ah, so it still has a use and fixes duplicate downloads.
I'll reword my NEWS entry and merge the patch. Thanks.

-- 
Matthias Andree

Re: [fetchmail-users] Re: RETR with UIDL

[Matthias A. <mat...@gm...>]

Jakob Hirsch <jh...@pl...> writes:

> Anyway, I really think fetchmail should by default simply retrieve all
> message, using standard MUA mechanisms: RETR and DELE; no TOP, UIDL or
> LAST. This makes sure that fetchmail works with as much servers as
> possible without any tweaking.

UIDL is a pretty standard MUA (used by major MUAs) mechanism for "keep
messages on server" setups, and it's very likely I'll make UIDL default
in future versions.

I hope we can also add the often-requested "delete after N days" that
ESR refused (probably because he was scared of the implications with the
long-winding UID code), but I'm not sure if time permits that for 6.4.X.

> If one wants 'keep', UIDL/RETR should be used, with fallback to
>LAST/RETR. And there should probably be an option to force one of UIDL
>or LAST (uidl/nouidl?) so the user can enfore client or server side
>tracking (with fetchmail bailing out if the requested option is not
>available on the server). The magic fetchmail does is nice and works in
>most cases, but final control should be left to the user.

The user should not need to care about protocol details before he runs
into a conflict so serious that fetchmail cannot resolve it on its own.

I share the view on RETR becoming default.

Server-side tracking is a constant source of troubles with respect to
interrupted connections, software faults, aborts, reboots, any other
things, and relying on POP3 server state beyond UIDL just doesn't work
well, so everything will be switched to client-side tracking.

I also think I'll be discontinuing obsolete commands such as "RPOP"
(insecure, removed with RFC-1460 in 1993), "LAST" (removed with RFC-1725
in 1994, inconsistent behavior with RSET) in 6.4.X.  fetchmail should be
talking RFC-1939 (POP3)/1734 (AUTH)/2449 (TLS)/2222 (SASL).

Massively slashing old protocols may cause me to call the next release
7.0.0 rather than 6.4.0 so people will be warned it's a major
upgrade. The exact roadmap for the next fetchmail release isn't fixed yet.

-- 
Matthias Andree

[fetchmail-users] Re: RETR with UIDL

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Sat, Mar 04, 2006:
> Hm - how will this avoid redownloads of messages when the line is faulty
> and the connection is torn down?
> 
> It will recognize it's downloading seen messages, but download them all
> the same.

The only practical use of this patch is if a user with this
configuration:

    set daemon 300
    poll server
	uidl
	no fetchall
	keep

runs:

    fetchmail -a -d 0 ; fetchmail

to download all mails once and then start normal polling. New mails
which have been downloaded by the first instance of fetchmail will not
be downloaded again by the second instance due to this patch.

Of course, an equivalent effect may be achieved by removing all the
relevant entries from the fetchids file or deleting the fetchids file.

This patch has no relevance to our basic topic under discussion.

-- 
Sunil Shetye.

Re: [fetchmail-users] Re: RETR with UIDL

[Jakob H. <jh...@pl...>]

Sunil Shetye wrote:

>> | clean up TOP vs. RETR but leave TOP as an option for the server scum
>> | that deletes message right after RETR, voiding client retries (Jakob
>> | Hirsch, 2006-01-03).

Note that the server i was referring to (MercuryP/NLM, version 1.48)
only deletes RETRieved mail after a QUIT, thereby not really violating
the RFC.

> Are these "scum" servers supporting "uidl"? Please note that the

Yes. But I can only speak for the server mentioned above.

Anyway, I really think fetchmail should by default simply retrieve all
message, using standard MUA mechanisms: RETR and DELE; no TOP, UIDL or
LAST. This makes sure that fetchmail works with as much servers as
possible without any tweaking.

If one wants 'keep', UIDL/RETR should be used, with fallback to
LAST/RETR. And there should probably be an option to force one of UIDL
or LAST (uidl/nouidl?) so the user can enfore client or server side
tracking (with fetchmail bailing out if the requested option is not
available on the server). The magic fetchmail does is nice and works in
most cases, but final control should be left to the user.

As for TOP, there are legit uses, so there should also an option to
enable it explicitly.

Oh, and all of this should be documented as comprehensible as possible,
of course. Anybody with loads of free time on the hand? :)

But as Matthias said, all of this is surely nothing for 6.3.

Re: [fetchmail-users] [PATCH] Re: RETR with UIDL

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Matthias Andree's mail on Fri, Mar 03, 2006:
>> > The advice being given to force fetchmail to avoid using "TOP" with
>> > these buggy servers is to enable the "fetchall" option. The FAQ is
>> > littered with this advice for various servers with variety of
>> > problems. The problems with "fetchall" option are:
>> >
>> > - "fetchall" cannot be used with "uidl"
>> 
>> Well, we can fix eventual incompatibilities between these two options
>> any time. 6.3.X is open for bug fixes.
>
> Here is a patch for this.

Hm - how will this avoid redownloads of messages when the line is faulty
and the connection is torn down?

It will recognize it's downloading seen messages, but download them all
the same.

-- 
Matthias Andree

[fetchmail-users] [PATCH] Re: RETR with UIDL

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Fri, Mar 03, 2006:
> > The advice being given to force fetchmail to avoid using "TOP" with
> > these buggy servers is to enable the "fetchall" option. The FAQ is
> > littered with this advice for various servers with variety of
> > problems. The problems with "fetchall" option are:
> >
> > - "fetchall" cannot be used with "uidl"
> 
> Well, we can fix eventual incompatibilities between these two options
> any time. 6.3.X is open for bug fixes.

Here is a patch for this.

================================================================
Index: fetchmail-6.3/pop3.c
===================================================================
--- fetchmail-6.3/pop3.c	(revision 4719)
+++ fetchmail-6.3/pop3.c	(working copy)
@@ -894,7 +894,7 @@
      */
     last = 0;
     *newp = -1;
-    if (*countp > 0 && !ctl->fetchall)
+    if (*countp > 0 && (!ctl->fetchall || ctl->server.uidl))
     {
 	int fastuidl;
 	char id [IDLEN+1];
@@ -902,6 +902,7 @@
 	/* should we do fast uidl this time? */
 	fastuidl = ctl->fastuidl;
 	if (*countp > 7 &&		/* linear search is better if there are few mails! */
+	    !ctl->fetchall &&		/* with fetchall, all uids are required */
 	    !ctl->flush &&		/* with flush, it is safer to disable fastuidl */
 	    NUM_NONZERO (fastuidl))
 	{
================================================================

> The best option would be to use UIDL. Still we'll not be forcing UIDL
> downloads to use RETR in 6.3.X. We can safely do that in 6.4.X: the
> 6.3.X manpage already has warnings in place.

...

> > I don't get this part. If the pop3 server is supporting UIDL and
> > fetchmail is running with the "uidl" option, why bother about the LAST
> > pointer and message flags?
> 
> Why bother? Because I promised I'd stop the ESR madness of gold versions
> and promised 6.3.X would be compatible with 6.3.0 when I released 6.3.0.

Ok.

-- 
Sunil Shetye.

Re: [fetchmail-users] Re: RETR with UIDL

[Rob F. <rf...@fu...>]

Matthias Andree wrote:
> Let's do 6.3.3 (thanks for your efforts in getting the IMAP NOOP bugs
> nailed BTW), and then fork 6.4.X and make all interesting changes and
> cleanups in particular in 6.4.X.

FWIW, I tend to agree with Matthias here.
Is it time for the long-awaited UIDL revamp yet?  :-)

-- 
==============================|   "A microscope locked in on one point
 Rob Funk <rf...@fu...> |Never sees what kind of room that it's in"
 http://www.funknet.net/rfunk |    -- Chris Mars, "Stuck in Rewind"

Re: [fetchmail-users] Re: RETR with UIDL

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Matthias Andree's mail on Fri, Mar 03, 2006:
>> > Please note that the change I am recommending is that fetchmail
>> > *should* use RETR with pop3+uidl servers. Other pop3 servers will not
>> > be affected by this change.
>> 
>> Yes, it's still 6.4.X stuff since it deliberately changes behavior for
>> no good reason.
>
> There is a good reason.
>
> The advice being given to force fetchmail to avoid using "TOP" with
> these buggy servers is to enable the "fetchall" option. The FAQ is
> littered with this advice for various servers with variety of
> problems. The problems with "fetchall" option are:
>
> - "fetchall" cannot be used with "uidl"

Well, we can fix eventual incompatibilities between these two options
any time. 6.3.X is open for bug fixes.

> - "fetchall" causes mails to be redownloaded if there is a line-hit
> while fetchmail is downloading mails. If there are consecutive
> line-hits, the same set of mails could be downloaded any number of
> times.

Yes, true. So use just uidl without fetchall on flakey lines. There is
little reason to add fetchall to uidl. We can decouple TOP/RETR from
fetchall in 6.4.X and default to RETR if we want. Not in 6.3.X.

Let's do 6.3.3 (thanks for your efforts in getting the IMAP NOOP bugs
nailed BTW), and then fork 6.4.X and make all interesting changes and
cleanups in particular in 6.4.X.

> To avoid the redownloading of mails, one has to now probably add the
> "expunge" option also. Finding the best value for "expunge" is a
> problem itself:
>
> - "expunge 1" is expensive. It will reconnect for every mail. Many
>   mailservers may not like this too. However, each mail will be
>   downloaded only once.
> - "expunge 2" is less expensive. Mailservers may still object to this.
>   Very few mails will be redownloaded.
> - "expunge 20" is relatively cheap. Mailservers may not object to
>   this. Some mails will get redownloaded. However, on a bad day, if
>   you do get consecutive line-hits, you will still end up with
>   multiple copies of mails.

The best option would be to use UIDL. Still we'll not be forcing UIDL
downloads to use RETR in 6.3.X. We can safely do that in 6.4.X: the
6.3.X manpage already has warnings in place.

>> fetchmail 6.3.2 already introduced the (incompatible) "use RETR with
>> Maillennium server" change because it was considered we'd rather screw
>> some message flags and a LAST pointer rather than truncate messages.
>
> I don't get this part. If the pop3 server is supporting UIDL and
> fetchmail is running with the "uidl" option, why bother about the LAST
> pointer and message flags?

Why bother? Because I promised I'd stop the ESR madness of gold versions
and promised 6.3.X would be compatible with 6.3.0 when I released 6.3.0.

> And, as has already been said, most standard interactive clients use
> "uidl" anyway. So, they also will not get affected.

This is not a valid reason to change behavior in incompatible ways in
the midst of a stable version. I'd wondered if I'd wanted to add the
Maillennium workaround in 6.3.X at all or delay it until 6.4.X. There'd
been loads of reasons to defer it, not the least being a documented
workaround.

> Jakob, please give more details about the Mercury server:
>
> - its signature line
> - its CAPA output
> - whether it supports UIDL
> - whether you are using it with "uidl"
> and finally,
> - your opinion on whether fetchmail should use "RETR" with "uidl".

I'm not accepting changes into 6.3.X that switch TOP to RETR except if
they fix security bugs, data corruption or other critical issues, so
please don't spend your time before we've opened 6.4.X for hacking.

Priorities for incompatible changes are:

1. Security
2. Critical bugs (data loss, crash, similar)
3. Compatibility within stable releases
4. Protocol compliance
5. Consistency
6. Anything else

-- 
Matthias Andree

[fetchmail-users] Re: RETR with UIDL

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Fri, Mar 03, 2006:
> > Please note that the change I am recommending is that fetchmail
> > *should* use RETR with pop3+uidl servers. Other pop3 servers will not
> > be affected by this change.
> 
> Yes, it's still 6.4.X stuff since it deliberately changes behavior for
> no good reason.

There is a good reason.

The advice being given to force fetchmail to avoid using "TOP" with
these buggy servers is to enable the "fetchall" option. The FAQ is
littered with this advice for various servers with variety of
problems. The problems with "fetchall" option are:

- "fetchall" cannot be used with "uidl"
- "fetchall" causes mails to be redownloaded if there is a line-hit
  while fetchmail is downloading mails. If there are consecutive
  line-hits, the same set of mails could be downloaded any number of
  times.

To avoid the redownloading of mails, one has to now probably add the
"expunge" option also. Finding the best value for "expunge" is a
problem itself:

- "expunge 1" is expensive. It will reconnect for every mail. Many
  mailservers may not like this too. However, each mail will be
  downloaded only once.
- "expunge 2" is less expensive. Mailservers may still object to this.
  Very few mails will be redownloaded.
- "expunge 20" is relatively cheap. Mailservers may not object to
  this. Some mails will get redownloaded. However, on a bad day, if
  you do get consecutive line-hits, you will still end up with
  multiple copies of mails.

In short, the user has to go for some sort of experimentation and use
a sub-optimal solution to get the mails(*).

The best solution (wherever possible) is to use "uidl" as mails get
downloaded exactly once. To use "uidl", one has to use "no fetchall".
With "no fetchall", fetchmail refuses to use "RETR".

> fetchmail 6.3.2 already introduced the (incompatible) "use RETR with
> Maillennium server" change because it was considered we'd rather screw
> some message flags and a LAST pointer rather than truncate messages.

I don't get this part. If the pop3 server is supporting UIDL and
fetchmail is running with the "uidl" option, why bother about the LAST
pointer and message flags? After all, even if the user is running
multiple fetchmails, all those fetchmails would be running with the
same configuration.

And, as has already been said, most standard interactive clients use
"uidl" anyway. So, they also will not get affected.

> Sort of. At least the ones that were reported do not delete
> messages if the connection is hung up without prior "QUIT" command.
> <http://lists.berlios.de/pipermail/fetchmail-users/2006-January/000220.html>

Jakob, please give more details about the Mercury server:

- its signature line
- its CAPA output
- whether it supports UIDL
- whether you are using it with "uidl"
and finally,
- your opinion on whether fetchmail should use "RETR" with "uidl".

-- 
Sunil Shetye.

(*) This need for experimentation is not documented now. Currently,
there is no entry in FAQ which says that if you are using "fetchall",
you should necessarily use say "expunge 10". FAQ O9 does talk of
"fetchall" and "expunge". But, it seems to be bad-mouthing "uidl" and
switching to IMAP4 is given as a better option than using "expunge".
However, this is a documentation issue and not related to the main
topic.

[fetchmail-users] [PATCH] Re: Bug in imap_idle() with mail servers not supporting "idle"

[Sunil S. <sh...@bo...>]

Quoting from Brendan Lynch's mail on Thu, Mar 02, 2006:
> However I believe the servers is behaving correctly according to the 
> IMAP spec: I have attached an explanation of the secondary problem I am 
> seeing (extra 28 s wait) that I sent to Casper; let me know if you need 
> more info.

Yes, I have now read RFC 3501 section 5.3. This behaviour is correct.

> The key point is that the imap_idle() code when 'faking' IDLE support 
> calls imap_ok() (at line 637) without having issued any tagged command:

...

> Since there is no tagged command outstanding the server will not (and 
> should not) ever send a tagged response.  If we set imap_ok to only 
> return after a tagged response we are guaranteeing it will always only 
> return after a timeout.

...

> This again first does a gen_transact() which results in sending "A0130 
> NOOP" at 17:00:49.  A "RECENT" update is already on its way form the 
> server, so we receive it before the "A0130 NOOP completed" at 17:00:49.  
> In imap_ok() we first process the "* 1 RECENT" notification at line 117, 
> setting recentcount, then (since we ARE waiting a tag) loop and process 
> the NOOP ok.  we then return to gen_transact(), which in turn returns to 
> imap_idle().
> 
> Since setting "recentcount" does not also change the stage, we fail the 
> test at line 633 of imap_idle() and continue to do an extra imap_ok() 
> which sets a 28 second timer at 17:00:49 and does a recv() which times 
> out at 17:01:17.  We then return from imap_idle() to imap_getrange() 
> line 679.  At this point recentcount is set and we exit the "while 
> (recentcount == 0 && do_idle)" loop.
> 
> The wait between 17:00:49 and 17:01:17 was unnecessary, as already had 
> our RECENT notification.  The second part of my fix causes imap_ok() to 
> change the stage to STAGE_FETCH on receipt of a RECENT notification, 
> just as it already did on receipt of a EXISTS notification. This causes 
> the if condition "if (ok != 0 || stage != STAGE_IDLE)" to be met at line 
> 633 of imap_idle() so it immediately exits and does not do the extra 
> unneeded imap_ok() call.

Yes, you have diagnosed the problem correctly and your patch is also
correct. Based on your patch, I have prepared a new patch which does
not modify imap_ok(), but cleans up the code in imap_idle(). Could you
test this patch and see if this patch is acceptable to you? This patch
is identical to the patch I had sent to Casper Gripenberg, except that
comments relating to compliance with RFCs have been updated.

-- 
Sunil Shetye.

Re: [fetchmail-users] Re: RETR with UIDL

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Matthias Andree's mail on Fri, Mar 03, 2006:
>> The TODO in SVN's trunk directory already mentions:
>> 
>> | clean up TOP vs. RETR but leave TOP as an option for the server scum
>> | that deletes message right after RETR, voiding client retries (Jakob
>> | Hirsch, 2006-01-03).
>
> Are these "scum" servers supporting "uidl"?

I don't know, but we need to assume they do since that's the worst case.

> Please note that the change I am recommending is that fetchmail
> *should* use RETR with pop3+uidl servers. Other pop3 servers will not
> be affected by this change.

Yes, it's still 6.4.X stuff since it deliberately changes behavior for
no good reason.

fetchmail 6.3.2 already introduced the (incompatible) "use RETR with
Maillennium server" change because it was considered we'd rather screw
some message flags and a LAST pointer rather than truncate messages.

> Are there any logs/bug reports about these "scum" servers?

Sort of. At least the ones that were reported do not delete
messages if the connection is hung up without prior "QUIT" command.
<http://lists.berlios.de/pipermail/fetchmail-users/2006-January/000220.html>

-- 
Matthias Andree

[fetchmail-users] Re: RETR with UIDL

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Fri, Mar 03, 2006:
> The TODO in SVN's trunk directory already mentions:
> 
> | clean up TOP vs. RETR but leave TOP as an option for the server scum
> | that deletes message right after RETR, voiding client retries (Jakob
> | Hirsch, 2006-01-03).

Are these "scum" servers supporting "uidl"? Please note that the
change I am recommending is that fetchmail *should* use RETR with
pop3+uidl servers. Other pop3 servers will not be affected by this
change.

Are there any logs/bug reports about these "scum" servers?

-- 
Sunil Shetye.

Re: [fetchmail-users] RETR with UIDL

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Jakob Hirsch's mail on Thu, Mar 02, 2006:
>> The difference is simply that fetchmail (ab)uses TOP by default to
>> retrieve mail, while usual MUAs use the standard mechanism of RETR.
>> Nevertheless, Comcast's POP3 server is plain broken if he allows TOP but
>> truncates messages with it.
>
> There is one case where fetchmail could use RETR instead of TOP. That
> is when "uidl" has been enabled.

Not in 6.3.X, as that would change semantics WRT other clients. The
assumption fetchmail makes is that "TOP" leaves seen flags (where
recorded in headers) and the "LAST" pointer (way obsolete) alone. Plus,
some b0rked servers apparently remove messages right after RETR (see
below).

Changing UIDL to imply "use RETR" is an option only for later versions,
although it looks sensible in itself -- compatibility matters here.

I don't like fetchmail's assumptions about undocumented behavior either
(which got us into this trouble), and it appears the price for not using
RETR is high, but changes like these are definitely 6.4.X stuff.

> fetchmail currently uses TOP. If the server is supporting "uidl", then
> it should not matter if fetchmail uses TOP or RETR as all the email
> clients

This is the thinko. Not all clients use UIDL.
Some (including fetchmail) still use LAST...

> Any votes for or against this?

Against.

The TODO in SVN's trunk directory already mentions:

| clean up TOP vs. RETR but leave TOP as an option for the server scum
| that deletes message right after RETR, voiding client retries (Jakob
| Hirsch, 2006-01-03).

-- 
Matthias Andree