fetchmail-devel — Development related discussion

[fetchmail-devel] New PO Template file for `fetchmail'

[Translation P. R. <tra...@ir...>]

Hello, gentle maintainer. This is a message from the Translation
Project robot.

A new PO Template file, for programs using the textual domain
`fetchmail', has just been made available to language teams for
translation, and a copy is available as:
>    http://www.iro.umontreal.ca/translation/domains/POT/fetchmail-6.3.4-rc1.pot

The file should soon be made available in mirror sites as:

>    ftp://ftp.unex.es/pub/gnu-i18n/po/domains/POT/fetchmail-6.3.4-rc1.pot
>    http://translation.sf.net/domains/POT/fetchmail-6.3.4-rc1.pot


In your releases, this file is usually found as `po/fetchmail.pot'.
It is created or updated automatically at `make dist' time, or
whenever you run `make update-po' in the `po/' subdirectory.  Whenever
you have a distribution ready which holds a newer PO Template, please
send the URL of this distribution to the address below.  The
distribution could be a pretest or a snapshot, it does not even have
to compile.  This is to be used by translators, when they need to get
some translation context from your sources.

Within the Translation Project, each PO Template file should have
different version numbers, but since it is not OK to have two
different distributions having same version numbers, this is not a
problem in practice.
Here is the URL information which has just been provided to translators for
your package.  Please inform the translation coordinator, at the address
given below, if the information does not appear to be adequate or current:

>    http://www.dt.e-technik.uni-dortmund.de/~ma/fetchmail/fetchmail-6.3.4-rc1.tar.gz

Translated PO files will later be automatically e-mailed to you.

Thanks for your collaboration,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                mailto:tra...@ir...

[fetchmail-devel] [Bug #6985] authentication error

[<ad...@be...>]

Bug #6985, was updated on 2006-Apr-04 18:39
Here is a current snapshot of the bug.

Project: Community Fetchmail
Category: None
Status: Open
Resolution: None
Bug Group: None
Priority: 5
Submitted by: geoffleach
Assigned to : none
Summary: authentication error

Details: On switching POP3 servers, what worked previously now gets an authentication error.  Some details. We're dealing with a satellite ISP here. What used to be Direcway is now Hughes.net. POP3 worked fine previously. Access to the account, using the same system name and password, from IE works with no problems at all. Presumably the POP3 server is using some scheme that IE knows about, but not Fetchmail.

Any ideas?

6.3.2 querying mail.hughes.net (protocol POP3) at Sat 01 Apr 2006 08:40:06 AM PST: poll started
POP3< +OK POP3 PROXY server ready (7.2. 066) <42E...@n0...>
POP3> CAPA
POP3< +OK Capability list follows
POP3< TOP
POP3< RESP-CODES
POP3< USER
POP3< SASL CRAM-MD5 DIGEST-MD5
POP3< PIPELINING
POP3< UIDL
POP3< .
POP3> AUTH CRAM-MD5
POP3< + PDFGQkNCN0FCMjEzNDk0ODI3N0JFMkM3MUY0MUE5RkJEOUUxNjc4NDRAbjA0OS5zYzAuY3AubmV0Pg==
POP3> Z2VvZmZAaHVnaGVzLm5ldCBlYjhlNzEzMDVkNDgxZTc5MjNhZThiZjFiNTYzM2NmNA==
POP3< -ERR proxy authentication failed proxy authentication failed
POP3> USER ge...@hu... Unknown login or authentication error on ge...@hu...@mail.hughes.net.criticalpath.net
    socket error while fetching from ge...@hu...@mail.hughes.net
6.3.2 querying mail.hughes.net (protocol POP3) at Sat 01 Apr 2006 08:40:12 AM PST: poll completed
Query status=2 (SOCKET)


For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=6985&group_id=1824

Re: [fetchmail-devel] [PATCHes] Misc. bug fixes

[Matthias A. <mat...@gm...>]

Miloslav Trmac <mi...@re...> writes:

> +old_libs=$LIBS

This should be old_LIBS="$LIBS" with capitalized LIBS part in the
variable name, in order to match...

> +for lib in '' -lresolv; do
> +    if test -z "$lib"; then
> +       AC_MSG_CHECKING([for res_search in libc])
> +    else
> +       AC_MSG_CHECKING([for res_search in $lib])
> +    fi
> +    LIBS="$old_LIBS $lib"

...this part      ^^^^

Else we'd lose whatever was in LIBS at the time (for instance -lsocket
-lnsl on Solaris, where I found the problem).

-- 
Matthias Andree

Re: [fetchmail-devel] [PATCHes] Misc. bug fixes

[Matthias A. <mat...@gm...>]

Miloslav Trmac <mi...@re...> writes:

> Hello,
> The warning cleanups have allowed me to notice bugs on some more exotic
> architectures which were previously lost in the noise:
>
> - in several places the *printf format string doesn't agree with the
>   parameter types
> - glibc doesn't export res_* on architectures with newer ABI, only
>   __res_* and defines macros in <resolv.h>, thus configure would not
>   detect res_* presence.
>
> The attached patches fix these problems.

Merged for 6.3.4, thanks.

-- 
Matthias Andree

[fetchmail-devel] [PATCHes] Misc. bug fixes

[Miloslav T. <mi...@re...>]

Hello,
The warning cleanups have allowed me to notice bugs on some more exotic
architectures which were previously lost in the noise:

- in several places the *printf format string doesn't agree with the
  parameter types
- glibc doesn't export res_* on architectures with newer ABI, only
  __res_* and defines macros in <resolv.h>, thus configure would not
  detect res_* presence.

The attached patches fix these problems.
Thanks,
	Mirek

[fetchmail-devel] fetchmail 6.3.3 stable release fixes several bugs

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

(Resending announcement with corrected Subject.)

Greetings,

I am announcing the release of fetchmail 6.3.3. This new stable version
of fetchmail fixes several bugs, among them two crashes. For details,
please see below.

This is a recommended upgrade for all users of any previous fetchmail versions.

The software is available from:
<https://developer.berlios.de/project/showfiles.php?group_id=1824&release_id=8784>

The fetchmail home pages is:
<http://www.fetchmail.info/>  or  <http://fetchmail.berlios.de/>

These are the relevant changes in 6.3.3 since 6.3.2,
unless otherwise noted, changes to this release were made by Matthias Andree:

# DEPRECATED FEATURES AND MAJOR INCOMPATIBLE CHANGE ADVANCE WARNINGS
* The "protocol auto" default inside fetchmail may be removed from a future
  fetchmail release. Explicit configuration of the protocol is recommended.

# KNOWN BUGS:
* Sun Workshop 6 (SPARC) is known to miscompile the lexer in 64-bit mode.
  Either compile 32-bit code or use GCC to compile 64-bit fetchmail.
  Note that fetchmail doesn't take advantage of 64-bit code anyways,
  so compiling 32-bit SPARC code should be fine.

# BUG FIXES:
* SEGFAULT: Do not attempt to overwrite the netrc password if none has been
  specified.  This fixes a segmentation fault bug introduced into 6.3.2.
  Fixes BerliOS bug #6234.  BerliOS patch #804 by Craig Leres.
  The patch, as accepted into fetchmail, was available separately from
  <http://download.berlios.de/fetchmail/patch-6.3.2.1-fix-netrc-SIGSEGV.diff>
* SEGFAULT: Work around C libraries that return a NULL in getaddrinfo()'s
  ai_canonname record, to avoid a segfault. Affects for instance FreeBSD 4.10,
  4.11 and 5.3 when dotted quads are given as server names.
  Analysis and fix by Vladimir Olegovich Ravodin (Владимир Олегович Раводин).
* IMAP: fix hangs in NOOP-based IDLE emulation. Reported by Casper Gripenberg
  and Brendan Lynch, fix by Sunil Shetye (his patch was merged) and Brendan Lynch.
* IMAP: Handle other clients concurrently accessing IMAP mailboxes better.
  Fetchmail quits the poll if the EXPUNGE count does not match expectations, and
  servers not updating RECENT counts after EXPUNGE are handled in a better way.
  (Patch by Sunil Shetye.)
* IMAP: Stop sending EXPUNGE after NOOP-idling (patch by Sunil Shetye).
* POP3: fetchmail can now use UIDL in fetchall keep mode, to avoid re-fetching
  the same messages again when the fetchall keyword is removed. Patch by
  Sunil Shetye. For details, please see
  <http://lists.berlios.de/pipermail/fetchmail-users/2006-March/000308.html>
* LMTP: fix bug in LMTP port validation (patch by Miloslav Trmac).
* SDPS: fetchmail no longer replaces the local user ID for an empty envelope
  sender when using the proprietary SDPS extension for POP3.
  Fixes Debian Bug#353575, reported by Roger Lynn.
* SDPS: Warn and disable SDPS if POP3 is disabled to avoid compilation errors.
* fetchmail no longer prints empty lines in verbose mode when using syslog.
* fetchmail no longer prints UID lists in verbose mode when using syslog.
* ./configure --quiet is now quieter (no SSL and fallback-related output).
* Miloslav Trmac's patch (with minor changes) to fix char * sign consistency,
  unused arguments and variables.
* More signedness, unused argument/variable and other warning fixes.

# CHANGES:
* --idle can now be specified on the command line, too.
* --fetchall is now supported on the command-line.
* POP3: Lower default fastuidl span to 4 (i. e. every 4th run fetches the
  whole UIDL list), patch by Sunil Shetye.

# DOCUMENTATION:
* "ssl" is a user option rather than a server option. Patch by Nico Golde.
  Fixes Debian Bug#354661, reported by Keith Hellman.
* The manual page now suggests "--" before the addresses in the sendmail MDA
  example, for safety.
* The FAQ item X9, Domino IMAP omits Content-Transfer-Encoding header, was
  added. Information provided by Anthony Kim on the fetchmail-friends list
  in March 2006.
* Credit Chris Boyle with the NOOP emulation code for IDLE in fetchmail 6.2.4.
  Eric forgot to credit Chris, thanks to Sunil Shetye for providing these links:
  http://lists.ccil.org/pipermail/fetchmail-friends/2003-July/007705.html
  http://lists.ccil.org/pipermail/fetchmail-friends/2003-July/007713.html
* Added a section about RETR vs. TOP to the manual page.
* Changed section/subsection levels in some areas.

Regards,

- -- 
Matthias Andree
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFELGsyvmGDOQUufZURApZZAJ4zs8yobFhbIr8mYv1WPkuiGUPs5gCcDm8l
iXmVUzfH0MJAGZIkbelGbLA=
=UdCS
-----END PGP SIGNATURE-----

[fetchmail-devel] fetchmail 6.3.2 stable release fixes several bugs

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings,

I am announcing the release of fetchmail 6.3.3. This new stable version
of fetchmail fixes several bugs, among them two crashes. For details,
please see below.

This is a recommended upgrade for all users of any previous fetchmail versions.

The software is available from:
<https://developer.berlios.de/project/showfiles.php?group_id=1824&release_id=8784>

The fetchmail home pages is:
<http://www.fetchmail.info/>  or  <http://fetchmail.berlios.de/>

These are the relevant changes in 6.3.3 since 6.3.2,
unless otherwise noted, changes to this release were made by Matthias Andree:

# DEPRECATED FEATURES AND MAJOR INCOMPATIBLE CHANGE ADVANCE WARNINGS
* The "protocol auto" default inside fetchmail may be removed from a future
  fetchmail release. Explicit configuration of the protocol is recommended.

# KNOWN BUGS:
* Sun Workshop 6 (SPARC) is known to miscompile the lexer in 64-bit mode.
  Either compile 32-bit code or use GCC to compile 64-bit fetchmail.
  Note that fetchmail doesn't take advantage of 64-bit code anyways,
  so compiling 32-bit SPARC code should be fine.

# BUG FIXES:
* SEGFAULT: Do not attempt to overwrite the netrc password if none has been
  specified.  This fixes a segmentation fault bug introduced into 6.3.2.
  Fixes BerliOS bug #6234.  BerliOS patch #804 by Craig Leres.
  The patch, as accepted into fetchmail, was available separately from
  <http://download.berlios.de/fetchmail/patch-6.3.2.1-fix-netrc-SIGSEGV.diff>
* SEGFAULT: Work around C libraries that return a NULL in getaddrinfo()'s
  ai_canonname record, to avoid a segfault. Affects for instance FreeBSD 4.10,
  4.11 and 5.3 when dotted quads are given as server names.
  Analysis and fix by Vladimir Olegovich Ravodin (Владимир Олегович Раводин).
* IMAP: fix hangs in NOOP-based IDLE emulation. Reported by Casper Gripenberg
  and Brendan Lynch, fix by Sunil Shetye (his patch was merged) and Brendan Lynch.
* IMAP: Handle other clients concurrently accessing IMAP mailboxes better.
  Fetchmail quits the poll if the EXPUNGE count does not match expectations, and
  servers not updating RECENT counts after EXPUNGE are handled in a better way.
  (Patch by Sunil Shetye.)
* IMAP: Stop sending EXPUNGE after NOOP-idling (patch by Sunil Shetye).
* POP3: fetchmail can now use UIDL in fetchall keep mode, to avoid re-fetching
  the same messages again when the fetchall keyword is removed. Patch by
  Sunil Shetye. For details, please see
  <http://lists.berlios.de/pipermail/fetchmail-users/2006-March/000308.html>
* LMTP: fix bug in LMTP port validation (patch by Miloslav Trmac).
* SDPS: fetchmail no longer replaces the local user ID for an empty envelope
  sender when using the proprietary SDPS extension for POP3.
  Fixes Debian Bug#353575, reported by Roger Lynn.
* SDPS: Warn and disable SDPS if POP3 is disabled to avoid compilation errors.
* fetchmail no longer prints empty lines in verbose mode when using syslog.
* fetchmail no longer prints UID lists in verbose mode when using syslog.
* ./configure --quiet is now quieter (no SSL and fallback-related output).
* Miloslav Trmac's patch (with minor changes) to fix char * sign consistency,
  unused arguments and variables.
* More signedness, unused argument/variable and other warning fixes.

# CHANGES:
* --idle can now be specified on the command line, too.
* --fetchall is now supported on the command-line.
* POP3: Lower default fastuidl span to 4 (i. e. every 4th run fetches the
  whole UIDL list), patch by Sunil Shetye.

# DOCUMENTATION:
* "ssl" is a user option rather than a server option. Patch by Nico Golde.
  Fixes Debian Bug#354661, reported by Keith Hellman.
* The manual page now suggests "--" before the addresses in the sendmail MDA
  example, for safety.
* The FAQ item X9, Domino IMAP omits Content-Transfer-Encoding header, was
  added. Information provided by Anthony Kim on the fetchmail-friends list
  in March 2006.
* Credit Chris Boyle with the NOOP emulation code for IDLE in fetchmail 6.2.4.
  Eric forgot to credit Chris, thanks to Sunil Shetye for providing these links:
  http://lists.ccil.org/pipermail/fetchmail-friends/2003-July/007705.html
  http://lists.ccil.org/pipermail/fetchmail-friends/2003-July/007713.html
* Added a section about RETR vs. TOP to the manual page.
* Changed section/subsection levels in some areas.

Regards,

- -- 
Matthias Andree
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFELGb1vmGDOQUufZURAg0HAKCQREup4O65GbS0yhJTJAFJ8nSNpwCfaFpT
IQtgYLQEWocf/VW0FjaCIz0=
=uCKv
-----END PGP SIGNATURE-----

[fetchmail-devel] Re: Incorrect header line and lost mails

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Fri, Mar 17, 2006:
> > fetchmail, by default, uses first IMAP and then POP3 (protocol auto).
> 
> Should we deprecate this "protocol" in fetchmail proper so we can
> remove it in 6.4.0?  I have a strong itch to do just that.
> 
> Somebody got any objections?

What should be the new default protocol be then, imap or pop3?

Yes, protocol auto should be removed as it is confusing.

> > - If your mailserver is delivering to multiple folders, choose IMAP.
> >   
> > - If you want to download all mails and not leave a copy on the
> > server, choose IMAP.
> 
> In --all --nokeep setups, POP3 is usually simpler - any particular
> reason why you suggest IMAP here?

If the connection to the mailserver breaks for any reason, mails get
downloaded again in POP3. The only POP3 setup which downloads mails
exactly once and deletes the downloaded mails is:

poll mailserver
    protocol pop3
    uidl         # required for unique download
    no fetchall  # required for unique download
    no keep
    flush        # to directly delete mails downloaded before the previous socket error

Unfortunately, this setup is not documented in the FAQ anywhere. I was
hoping to cover it in the proposed FAQ section:

    C#. Ok, I choose POP3. What next?

Due to the overhead of "uidl", I would prefer suggesting imap here.

> > - If you intend to keep mails on the server, check if your server
> >   supports UIDL. If it does, choose POP3 and enable the "uidl" option.
> >   With POP3 and UIDL, it is even possible to access the mailbox
> >   through multiple e-mail clients without affecting fetchmail.
> >
> > - If you intend to keep mails on the server, but your server does not
> >   support UIDL, choose IMAP. However, you should not access the
> 
> Make that "MUST NOT" access...

Ok.

> >   mailbox through other e-mail clients as fetchmail will not be
> >   able to download all new mails.
> >
> > - If you are having a slow connection to the mailserver and/or expect
> >   frequent socket errors from the mailserver, choose IMAP and do not
> >   leave a copy on the mailserver.
> 
> This, too, is something POP3+UIDL should handle.

On a slow connection, I feel that the download and delete setup works
the best. Yes, POP3+UIDL can definitely handle it, but I think that
IMAP can do it in the minimum amount of traffic. However, I think the
wording of my point is wrong. Here is how it should be:

==========================================================================
- If you are having a slow connection to the mailserver and/or expect
  frequent socket errors from the mailserver, you should consider not
  leaving a copy on the mailserver. This is because leaving a copy on
  the mailserver adds a small but significant amount of traffic from
  the mailserver in every poll.

  If you still prefer leaving a copy on the mailserver, choose POP3
  with the "uidl" option. If you prefer not to leave a copy on the
  mailserver, choose IMAP.
==========================================================================

> >> My basic idea is: say we have u UIDLs stored, and the server has m
> >> messages. Then, send "UIDL" u+1 "UIDL" u+2 ... "UIDL" m pipelined
> >> (non-blocking) (or, on future RANGE capable POP3 servers, "UIDL" u+1 "-"
> >> m), and if ANY seen message is in that range, do slow UIDL. No binary
> >> search with many round trips. Of course, this needs to track deletions
> >> properly and if another client client deletes messages, needs to back
> >> down and fetch the whole list. Perhaps sanity checking the last known
> >> UIDL helps detecting third-party deletions on the assumption that the
> >> likelyhood of the server recycling a UID is low.

This sanity checking part is indeed correct.

Pipelining will, of course, require a lot of work. Also, getting the
full UIDL list might actually turn out to be a lot faster than
pipelining if there are many new mails.

I still have reservations about fetching the whole UIDL list
immediately after detecting the mismatch. If it is postponed to the
next poll, it should be ok.

-- 
Sunil Shetye.

Re: [fetchmail-devel] Re: Incorrect header line and lost mails

[Jakob H. <jh...@pl...>]

Matthias Andree wrote:

>> fetchmail, by default, uses first IMAP and then POP3 (protocol auto).
> Should we deprecate this "protocol" in fetchmail proper so we can
> remove it in 6.4.0?  I have a strong itch to do just that.

I second this. I'm really not a friend of such overly automagic. If
someone doesn't know which protocol to use (or even which protocol his
server uses and to find out), fetchmail is not the right tool.

Re: [fetchmail-devel] Re: Incorrect header line and lost mails

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Apart from the issue of fixing, what is missing is documentation on
> this. There is no place where it is written what configuration should
> a user use to derive the best mileage from fetchmail. Maybe, some FAQ
> entry like:

Looks good.

> =========================================================================
> C#. What protocol should I choose with my mailserver? My mailserver
> supports both IMAP and POP3.
>
> fetchmail, by default, uses first IMAP and then POP3 (protocol auto).

Should we deprecate this "protocol" in fetchmail proper so we can
remove it in 6.4.0?  I have a strong itch to do just that.

Somebody got any objections?

fetchmailconf could still probe for protocols (but needs to be told
about SSL) -- and actually, someone should overhaul it (and rework the
ergonomic properties. The current layout of the Tk interface is pretty
unergonomic and in some places astonishing for the user.)

> This doesn't work when you are keeping mails on the mailserver or when
> there are errors (like socket errors or deliver errors) after a few
> mails have been downloaded.
>
> So, as a first step, you should necessarily choose between IMAP and
> POP3.
>
> - If your mailserver is delivering to multiple folders, choose IMAP.
>   
> - If you want to download all mails and not leave a copy on the
> server, choose IMAP.

In --all --nokeep setups, POP3 is usually simpler - any particular
reason why you suggest IMAP here?

> - If you intend to keep mails on the server, check if your server
>   supports UIDL. If it does, choose POP3 and enable the "uidl" option.
>   With POP3 and UIDL, it is even possible to access the mailbox
>   through multiple e-mail clients without affecting fetchmail.
>
> - If you intend to keep mails on the server, but your server does not
>   support UIDL, choose IMAP. However, you should not access the

Make that "MUST NOT" access...

>   mailbox through other e-mail clients as fetchmail will not be
>   able to download all new mails.
>
> - If you are having a slow connection to the mailserver and/or expect
>   frequent socket errors from the mailserver, choose IMAP and do not
>   leave a copy on the mailserver.

This, too, is something POP3+UIDL should handle.

>> My basic idea is: say we have u UIDLs stored, and the server has m
>> messages. Then, send "UIDL" u+1 "UIDL" u+2 ... "UIDL" m pipelined
>> (non-blocking) (or, on future RANGE capable POP3 servers, "UIDL" u+1 "-"
>> m), and if ANY seen message is in that range, do slow UIDL. No binary
>> search with many round trips. Of course, this needs to track deletions
>> properly and if another client client deletes messages, needs to back
>> down and fetch the whole list. Perhaps sanity checking the last known
>> UIDL helps detecting third-party deletions on the assumption that the
>> likelyhood of the server recycling a UID is low.
>
> This might not work with "no keep" when fetchmail is not sure if the
> mails marked by fetchmail for deletion have actually been expunged or
> not.

But fetchmail knows if it has seen "+OK" or rather EPIPE in response to
a QUIT command, and in no other case than having successfully sent
"QUIT" can it assume the messages have been expunged.

-- 
Matthias Andree

[fetchmail-devel] Re: Incorrect header line and lost mails

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Tue, Mar 14, 2006:
> > Imagine the worst-case scenario: Frederic's mailbox has 2000 mails on
> > this server and it has to be accessed on a slow dialup line which is
> > prone to disconnection. Even if the assumption doesn't work, it will
> > still download a few mails. Going back to linear search will probably
> > fetch even less mails due to the overheads.
> 
> If it confuses the user, it needs to be fixed. If the connection is too
> flakey to regularly provide the full UIDL connection (which will usually
> compress well with V.42bis or MNP5), then the user should re-think if he
> needs "keep messages on server" setups. Usually there are better
> alternatives, such as *copying* new messages from INBOX to a second IMAP
> folder which is then downloaded in nokeep mode, so that there is no need
> to download UID lists.

Apart from the issue of fixing, what is missing is documentation on
this. There is no place where it is written what configuration should
a user use to derive the best mileage from fetchmail. Maybe, some FAQ
entry like:

=========================================================================
C#. What protocol should I choose with my mailserver? My mailserver
supports both IMAP and POP3.

fetchmail, by default, uses first IMAP and then POP3 (protocol auto).
This doesn't work when you are keeping mails on the mailserver or when
there are errors (like socket errors or deliver errors) after a few
mails have been downloaded.

So, as a first step, you should necessarily choose between IMAP and
POP3.

- If your mailserver is delivering to multiple folders, choose IMAP.
  
- If you want to download all mails and not leave a copy on the
  server, choose IMAP.

- If you intend to keep mails on the server, check if your server
  supports UIDL. If it does, choose POP3 and enable the "uidl" option.
  With POP3 and UIDL, it is even possible to access the mailbox
  through multiple e-mail clients without affecting fetchmail.

- If you intend to keep mails on the server, but your server does not
  support UIDL, choose IMAP. However, you should not access the
  mailbox through other e-mail clients as fetchmail will not be
  able to download all new mails.

- If you are having a slow connection to the mailserver and/or expect
  frequent socket errors from the mailserver, choose IMAP and do not
  leave a copy on the mailserver.

C#. Ok, I choose IMAP. What next?

...

C#. Ok, I choose POP3. What next?

...
=========================================================================

> > The question is not just of keeping in memory or database. The
> > question is: What should be done if a mismatch is found? If the only
> > solution is to go back to downloading all UIDs at one go, then I would
> > prefer continuing with the incorrect assumption.
> 
> Well, I'd rather download full UID lists to get rid of bogus
> assumptions. Missing messages because of false UIDL assumptions delays
> messages far more than connection drops (which are usually noticed
> quickly) does, and that causes certainly more reports than long UIDL
> downloads.
> 
> A RANGES extension for POP3 that allowed UIDL n- or UIDL n-m would help
> a lot.

If this extension is being used, using it is the better option.

> My basic idea is: say we have u UIDLs stored, and the server has m
> messages. Then, send "UIDL" u+1 "UIDL" u+2 ... "UIDL" m pipelined
> (non-blocking) (or, on future RANGE capable POP3 servers, "UIDL" u+1 "-"
> m), and if ANY seen message is in that range, do slow UIDL. No binary
> search with many round trips. Of course, this needs to track deletions
> properly and if another client client deletes messages, needs to back
> down and fetch the whole list. Perhaps sanity checking the last known
> UIDL helps detecting third-party deletions on the assumption that the
> likelyhood of the server recycling a UID is low.

This might not work with "no keep" when fetchmail is not sure if the
mails marked by fetchmail for deletion have actually been expunged or
not.

-- 
Sunil Shetye.

Re: [fetchmail-devel] [PATCH] Re: Incorrect header line and lost mails

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Matthias Andree's mail on Tue, Mar 14, 2006:
>> > One option is to change the default to a lower value from the current
>> > value of 10. The lowest practical value is 2, which will use linear
>> > search in alternate polls.
>> 
>> The algorithm tries to find the first unused UIDL with a binary search,
>> on the assumption that all new messages were at the end of the
>> list. This doesn't work for Fr?d?ric's server and others.
>
> Imagine the worst-case scenario: Frederic's mailbox has 2000 mails on
> this server and it has to be accessed on a slow dialup line which is
> prone to disconnection. Even if the assumption doesn't work, it will
> still download a few mails. Going back to linear search will probably
> fetch even less mails due to the overheads.

If it confuses the user, it needs to be fixed. If the connection is too
flakey to regularly provide the full UIDL connection (which will usually
compress well with V.42bis or MNP5), then the user should re-think if he
needs "keep messages on server" setups. Usually there are better
alternatives, such as *copying* new messages from INBOX to a second IMAP
folder which is then downloaded in nokeep mode, so that there is no need
to download UID lists.

> The question is not just of keeping in memory or database. The
> question is: What should be done if a mismatch is found? If the only
> solution is to go back to downloading all UIDs at one go, then I would
> prefer continuing with the incorrect assumption.

Well, I'd rather download full UID lists to get rid of bogus
assumptions. Missing messages because of false UIDL assumptions delays
messages far more than connection drops (which are usually noticed
quickly) does, and that causes certainly more reports than long UIDL
downloads.

A RANGES extension for POP3 that allowed UIDL n- or UIDL n-m would help
a lot.

>> Well. Should I lower the default to 4 for the nonce or wait for a patch? :)
>
> Here it is:

I thought about a sanity checking patch that detects when slowuidl is
needed, but I'm taking this for the nonce, too.

My basic idea is: say we have u UIDLs stored, and the server has m
messages. Then, send "UIDL" u+1 "UIDL" u+2 ... "UIDL" m pipelined
(non-blocking) (or, on future RANGE capable POP3 servers, "UIDL" u+1 "-"
m), and if ANY seen message is in that range, do slow UIDL. No binary
search with many round trips. Of course, this needs to track deletions
properly and if another client client deletes messages, needs to back
down and fetch the whole list. Perhaps sanity checking the last known
UIDL helps detecting third-party deletions on the assumption that the
likelyhood of the server recycling a UID is low.

-- 
Matthias Andree

[fetchmail-devel] [PATCH] Re: Incorrect header line and lost mails

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Tue, Mar 14, 2006:
> > One option is to change the default to a lower value from the current
> > value of 10. The lowest practical value is 2, which will use linear
> > search in alternate polls.
> 
> The algorithm tries to find the first unused UIDL with a binary search,
> on the assumption that all new messages were at the end of the
> list. This doesn't work for Fr?d?ric's server and others.

Imagine the worst-case scenario: Frederic's mailbox has 2000 mails on
this server and it has to be accessed on a slow dialup line which is
prone to disconnection. Even if the assumption doesn't work, it will
still download a few mails. Going back to linear search will probably
fetch even less mails due to the overheads.

> > Other solutions are also possible. For example, keeping a map of UID
> > and message number in memory (with corrections for mails deleted after
> > logout).
> 
> Aren't we keeping those in memory anyways? If so, such a check wouldn't
> hurt and serve those users whose servers create messages out of order.
> 
> In the long run, we'd probably better keep them in a database on disk to
> keep the memory footprint down.

The question is not just of keeping in memory or database. The
question is: What should be done if a mismatch is found? If the only
solution is to go back to downloading all UIDs at one go, then I would
prefer continuing with the incorrect assumption.

The fight is not just between linear vs. binary search. The basic idea
of fastuidl is to avoid downloading all UIDs at one go.

> Well. Should I lower the default to 4 for the nonce or wait for a patch? :)

Here it is:

===============================================================================
Index: fetchmail-6.3/fetchmailconf.py
===================================================================
--- fetchmail-6.3/fetchmailconf.py	(revision 4739)
+++ fetchmail-6.3/fetchmailconf.py	(working copy)
@@ -249,7 +249,7 @@
 	self.warnings = 3600	# Size warning interval (see tunable.h)
 	self.fetchlimit = 0	# Max messages fetched per batch
 	self.fetchsizelimit = 100	# Max message sizes fetched per transaction
-	self.fastuidl = 10	# Do fast uidl 9 out of 10 times
+	self.fastuidl = 4	# Do fast uidl 3 out of 4 times
 	self.batchlimit = 0	# Max message forwarded per batch
 	self.expunge = 0	# Interval between expunges (IMAP)
 	self.ssl = 0		# Enable Seccure Socket Layer
Index: fetchmail-6.3/fetchmail.man
===================================================================
--- fetchmail-6.3/fetchmail.man	(revision 4739)
+++ fetchmail-6.3/fetchmail.man	(working copy)
@@ -615,7 +615,7 @@
 once followed by binary searches in 'n-1' polls if 'n' is greater than
 1; binary search is always used if 'n' is 1; linear search is always
 used if 'n' is 0. In non-daemon mode, binary search is used if 'n' is
-1; otherwise linear search is used.
+1; otherwise linear search is used. The default value of 'n' is 4.
 This option works with POP3 only.
 .TP
 .B \-e <count> | \-\-expunge <count>
Index: fetchmail-6.3/fetchmail.c
===================================================================
--- fetchmail-6.3/fetchmail.c	(revision 4739)
+++ fetchmail-6.3/fetchmail.c	(working copy)
@@ -970,7 +970,7 @@
     def_opts.remotename = user;
     def_opts.listener = SMTP_MODE;
     def_opts.fetchsizelimit = 100;
-    def_opts.fastuidl = 10;
+    def_opts.fastuidl = 4;
 
     /* get the location of rcfile */
     rcfiledir[0] = 0;
===============================================================================

-- 
Sunil Shetye.

Re: [fetchmail-devel] Re: Incorrect header line and lost mails

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Matthias Andree's mail on Tue, Mar 07, 2006:
>> what should we do WRT fastuidl?  We seem to need sanity checks to disable
>> the fastuidl search for servers such as Fr?d?ric's.
>
> One option is to change the default to a lower value from the current
> value of 10. The lowest practical value is 2, which will use linear
> search in alternate polls.

The algorithm tries to find the first unused UIDL with a binary search,
on the assumption that all new messages were at the end of the
list. This doesn't work for Frédéric's server and others.

> Other solutions are also possible. For example, keeping a map of UID
> and message number in memory (with corrections for mails deleted after
> logout).

Aren't we keeping those in memory anyways? If so, such a check wouldn't
hurt and serve those users whose servers create messages out of order.

In the long run, we'd probably better keep them in a database on disk to
keep the memory footprint down.

> Note that Frederic has claimed that messages with low UID never get
> downloaded. This is only partially correct. In his case, messages with
> low UID never get downloaded whenever a binary search is performed.
> However, a linear search is performed once in every ten polls
> (starting with the first poll) and these messages which were missed
> out earlier will get downloaded in this poll with linear search.

Well. Should I lower the default to 4 for the nonce or wait for a patch? :)

-- 
Matthias Andree

Re: [fetchmail-devel] [PATCHes] Misc. cleanups

[Matthias A. <mat...@gm...>]

Miloslav Trmac <mi...@re...> writes:

> - fetchmail-signed.patch: Avoid implicit conversions between
>   char * and unsigned char *; they are not defined by ISO C and gcc
>   currently warns about them.  Most places were changed to use "char *",
>   or "void *" where general binary data is handled.

The report.c patch appears wrong, it has to be n >= 0 rather than n < 0,
and needs to be changed in both branches of each of the two #ifdef.

What is the reason for the (int32) cast in kerberos.c near line 220?

> - fetchmail-comparisons.patch: Change variable types or add explicit
>   casts to make all casts use types with consistent type signedness,
>   for clarity
> - fetchmail-clean.patch: Remove or (void) out unused parameters, avoid
>   assignments directly in if ().
>
> - fetchmail-lmtp.patch: A real bug detected by the compiler, "," has
>   lower priority than "||".

Other than that, I merged your patches.

Thanks a lot.

-- 
Matthias Andree

Re: [fetchmail-devel] [PATCH] Allow to specify pid dir independent of home dir

[Matthias A. <mat...@gm...>]

On Tue, 14 Mar 2006, Héctor García wrote:

> El mar, 14-03-2006 a las 00:09 +0100, Matthias Andree escribió:
> > This patch lacks documentation (fetchmail.man), and is not needed.
> > 
> I'll write the docu if there is any chance to integrate this change.

If it's a command line and rcfile option there is.

> If there is a chance, I'll go into write documentation and maybe
> converting it also into a command line option.

I'd rather use that (and a rcfile option).
Users are often confused by environment variables.

-- 
Matthias Andree

Re: [fetchmail-devel] [PATCH] Allow to specify pid dir independent of home dir

[Héctor G. <he...@de...>]

El mar, 14-03-2006 a las 00:09 +0100, Matthias Andree escribió:
> This patch lacks documentation (fetchmail.man), and is not needed.
> 
I'll write the docu if there is any chance to integrate this change.

> You can use FETCHMAILHOME and then override the UIDL file location with
> --idfile /var/lib/fetchmail/foo-uids.
> 
The problem still is that if I want to place the pid
on /var/run/fetchmail I have to tell fetchmail that fetchmail's home is
there.
The problem on doing this, is that I cannot have a .ssh dir on my
fetchmail home with ssh keys, since the dirs at /var/run/fetchmail are
cleaned on every boot.
So far we fix it placing fetchmail's home at /var/lib/fetchmail but then
we have a problem when the machines dies and don't clean the .pid file.

> Remember that the user running fetchmail needs write access
> (create/rename files) to the directory.
> 
> I'll reconsider it for 6.4.0 (it's on my TODO),
> but don't rely on it being applied.
> 
If there is a chance, I'll go into write documentation and maybe
converting it also into a command line option.

Regards,

Héctor

Re: [fetchmail-devel] [PATCH] Allow to specify pid dir independent of home dir

[Matthias A. <mat...@gm...>]

Héctor García <he...@de...> writes:

> This patch would allow to specify the pid dir independent of the home
> dir.
> This would allow to have fetchmail-UIDL-cache file on /var/lib/fetchmail
> and pid file on /var/run/fetchmail
>
> This is needed when you run a global fetchmail with non-root user (as we
> do on Debian)
>
> So far this patch allows to specify pid dir with env var FETCHMAILPIDDIR

This patch lacks documentation (fetchmail.man), and is not needed.

You can use FETCHMAILHOME and then override the UIDL file location with
--idfile /var/lib/fetchmail/foo-uids.

Remember that the user running fetchmail needs write access
(create/rename files) to the directory.

I'll reconsider it for 6.4.0 (it's on my TODO),
but don't rely on it being applied.

-- 
Matthias Andree

Re: [fetchmail-devel] [PATCHes] Misc. cleanups

[Matthias A. <mat...@gm...>]

Nico Golde <ni...@ng...> writes:

> Hi,
> * Miloslav Trmac <mi...@re...> [2006-03-12 01:25]:
> [...] 
>> -    if (retval = krb5_cc_get_principal(context, ccdef, &client)) {
>> +    if ((retval = krb5_cc_get_principal(context, ccdef, &client)) != 0) {
>
> [...] 
> What is your point here? I can't see a real reason to do 
> this because its the same.

Yes, but it causes GCC warnings and who says it shouldn't have been
retval == ... ? I'll see if I take Mirek's patch or just write

       retval = krb5..;
       if (retval) { ... }

when reviewing all of it. 

BTW, Mirek, thanks for your detail work.

The warnings have been annoying me for ages but I didn't have the time
to fix them.

-- 
Matthias Andree

[fetchmail-devel] Re: Misc. cleanups

[Sunil S. <sh...@bo...>]

Quoting from Nico Golde's mail on Sun, Mar 12, 2006:
> * Miloslav Trmac <mi...@re...> [2006-03-12 01:25]:
> [...] 
> > -    if (retval = krb5_cc_get_principal(context, ccdef, &client)) {
> > +    if ((retval = krb5_cc_get_principal(context, ccdef, &client)) != 0) {
> 
> [...] 
> What is your point here? I can't see a real reason to do 
> this because its the same.

The C construct

    if (var = expression) { ... }

is valid, but is also a common typo. The actual intention may be to
write it as:

    if (var == expression) { ... }

To emphasise that it is not a typo, it is preferred to write the first
expression above in one of the following formats:

    if ((var = expression) != 0) { ... }
    if ((var = expression)) { ... }

So, it's the same, but it's different.

-- 
Sunil Shetye.

[fetchmail-devel] [PATCH] Allow to specify pid dir independent of home dir

[Héctor G. <he...@de...>]

Hi,

This patch would allow to specify the pid dir independent of the home
dir.
This would allow to have fetchmail-UIDL-cache file on /var/lib/fetchmail
and pid file on /var/run/fetchmail

This is needed when you run a global fetchmail with non-root user (as we
do on Debian)

So far this patch allows to specify pid dir with env var FETCHMAILPIDDIR

Regards,

Héctor

Re: [fetchmail-devel] [PATCHes] Misc. cleanups

[Miloslav T. <mi...@re...>]

Nico Golde napsal(a):
> Hi,
> * Miloslav Trmac <mi...@re...> [2006-03-12 01:25]:
> [...] 
>> -    if (retval = krb5_cc_get_principal(context, ccdef, &client)) {
>> +    if ((retval = krb5_cc_get_principal(context, ccdef, &client)) != 0) {
> 
> [...] 
> What is your point here? I can't see a real reason to do 
> this because its the same.
No real change, just avoiding gcc warnings to better see the warnings
that detect real bugs.
	Mirek

Re: [fetchmail-devel] [PATCHes] Misc. cleanups

[Nico G. <ni...@ng...>]

Hi,
* Miloslav Trmac <mi...@re...> [2006-03-12 01:25]:
[...] 
> -    if (retval = krb5_cc_get_principal(context, ccdef, &client)) {
> +    if ((retval = krb5_cc_get_principal(context, ccdef, &client)) != 0) {

[...] 
What is your point here? I can't see a real reason to do 
this because its the same.
Regards Nico
-- 
Nico Golde - JAB: ni...@ja... | GPG: 0x73647CFF
http://www.ngolde.de | http://www.muttng.org | http://grml.org
Forget about that mouse with 3/4/5 buttons -
gimme a keyboard with 103/104/105 keys!

[fetchmail-devel] [PATCHes] Misc. cleanups

[Miloslav T. <mi...@re...>]

Hello,
as usual I have choosen the perfect time for cleanup patches :)

The patches avoid most warnings with gcc -Wall -Wextra:
- fetchmail-signed.patch: Avoid implicit conversions between
  char * and unsigned char *; they are not defined by ISO C and gcc
  currently warns about them.  Most places were changed to use "char *",
  or "void *" where general binary data is handled.
- fetchmail-comparisons.patch: Change variable types or add explicit
  casts to make all casts use types with consistent type signedness,
  for clarity
- fetchmail-clean.patch: Remove or (void) out unused parameters, avoid
  assignments directly in if ().

- fetchmail-lmtp.patch: A real bug detected by the compiler, "," has
  lower priority than "||".

Thanks,
	Mirek

Re: [fetchmail-devel] Re: Incorrect header line and lost mails

[Frederic M. <fre...@wo...>]

Sunil Shetye wrote:
> Quoting from Matthias Andree's mail on Tue, Mar 07, 2006:
>   
>> what should we do WRT fastuidl?  We seem to need sanity checks to disable
>> the fastuidl search for servers such as Fr?d?ric's.
>>     
>
> One option is to change the default to a lower value from the current
> value of 10. The lowest practical value is 2, which will use linear
> search in alternate polls.
>
> A default of 4 should be alright. It will use linear search in the
> first poll, followed by binary search in the next three polls,
> followed by linear search in the next poll, and so on.
>
> Other solutions are also possible. For example, keeping a map of UID
> and message number in memory (with corrections for mails deleted after
> logout). Then, when there is a mismatch in the message number expected
> and the message number received from the remote server for any one
> mail, a linear search should be forced in the next(*) poll. This would
> be equivalent to setting fastuidl to 2 if this mismatch happens in
> every poll.
>
> Note that Frederic has claimed that messages with low UID never get
> downloaded. This is only partially correct. In his case, messages with
> low UID never get downloaded whenever a binary search is performed.
> However, a linear search is performed once in every ten polls
> (starting with the first poll) and these messages which were missed
> out earlier will get downloaded in this poll with linear search.
>
>   
Thank you for the clarification ! I was wondering why none of my user 
had complained about any lost e-mail. We receive a lot of spam but I 
would have been surprised if all the lost e-mails were only spam :-)

A suitable solution would be to have an entry in the FAQ with the 
settings required for a smooth operation with Mercury. If I understand 
it right, the only bad effect is a long delay (up to 10 times the poll 
delay) in the delivery of some e-mails. It isn't that bad.

BTW, I reported this sorting feature and the long lines wrapping problem 
to the Mercury mailing list but I don't expect any answer. It is closed 
source and with only one developer. David Harris seems to be busy fixing 
other more urgent problems in his mail client Pegasus Mail.

Frederic