fetchmail-devel — Development related discussion

[fetchmail-devel] fetchmail 6.3.2 stable release with security relevant bugfixes

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings,

I am announcing the release of fetchmail 6.3.2.

This release fixes a denial of service bug/fetchmail crash after sending
a bounce, adds a Maillennium (Comcast) workaround and fixes other bugs.
(The security announcement will be mailed separately.)

This is a recommended upgrade for all users of any previous fetchmail versions.

The software is available from:
<https://developer.berlios.de/project/showfiles.php?group_id=1824&release_id=8784>

These are the relevant changes in 6.3.2 since 6.3.1,
unless otherwise noted, changes to this release were made by Matthias Andree.

# SECURITY FIX IN THIS RELEASE
* CVE-2006-0321: Fix segfault or bus error after bouncing a message.  This bug
  was introduced into 6.3.0 when removing alloca(); it caused fetchmail to free
  random memory.  Reported by Nathaniel W. Turner, Debian Bug#348747.
  See fetchmail-SA-2006-01.txt

# DEPRECATED FEATURES AND MAJOR INCOMPATIBLE CHANGE ADVANCE WARNINGS
* The --enable-fallback (fall back to MDA if MTA unavailable) may be removed
  from a future fetchmail release.

# INCOMPATIBLE CHANGES
* Automatically disable the POP3 TOP command if the greeting string contains
  "Maillennium POP3/PROXY server", which is used by comcast and known to
  truncate messages after 80 kByte. Fall back to RETR, and complain if we had
  used TOP otherwise (the warning is printed only once per server in daemon
  mode).  Suggested by Ed Wilts.
  *Note* that this means messages are marked read on these servers, which is a
  deviation from how 6.3.1 behaved, but we have no alternative, comcast haven't
  fixed this bug in years.  Preventing the loss of the remainder of the message
  justifies this incompatible fix.
* fetchmail, since 6.3.0, requires write permission to the directory holding the
  idfile. See the amendment in the 6.3.0 MAJOR INCOMPATIBLE CHANGES section
  below for details. The manual page was updated.

# CHANGES RELEVANT TO PACKAGERS
* The outdated BUGS document was removed from the distribution.
* Added fetchmail-SA-2006-01.txt to the distribution.

# BUG FIXES
* SMTP/LMTP cleanup to fix these two bugs:
  - switch back to SMTP after having tried LMTP hosts (multiple smtphost hosts)
  - switch back to LMTP after sending a bounce.
  The patch removes the global state variable that was the root of this problem.
  Patch by Sunil Shetye. (MA)
* Don't complain about fetchall keep in --configdump mode. Bug introduced in
  6.3.0.
* fetchmailconf.py: Fix novice help for Poll interval and fetchall.
  Reported by Justin Pryzby, Debian Bug #344978.
* Some verbose output disappeared in debug mode. Adding further -v options would
  alternate between verbose and debug mode. debug mode now comprises all verbose
  output, and adding more -v options does not switch back from debug to verbose
  mode.
* fetchmail.man: Fix accented characters in Héctor García's name. Merged from
  downstream debian/patches/01_man_page.dpatch.
* Add missing --help text for "--sslcertck" option.
* fetchmailconf.py: Accept --help and --version.
* fetchmail --version  now prints the copyright notice.
* don't complain about READ-ONLY IMAP folders in --fetchall --keep mode.
  Reported Alexander Zangerl, Debian Bug#348964.
* the RPM .spec file now generates a -debuginfo package on newer RPM versions.

Regards,

- -- 
Matthias Andree
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFD04UzvmGDOQUufZURApk+AJ46zXfGAd0jHsbxziZ7JQpPugjJKwCeM2xW
DSAxh7uWlnM7Teolv4wF+BE=
=hrZ0
-----END PGP SIGNATURE-----

[fetchmail-devel] New Polish PO file for `fetchmail'

[Translation P. R. <tra...@ir...>]

Hello, gentle maintainer. This is a message from the Translation
Project robot.

A revised PO file, for programs using the textual domain `fetchmail',
has been submitted by the team of translators taking care of the
Polish language.  This particular file, along with all other PO files
pertaining to the same textual domain, is available as:
>    http://www.iro.umontreal.ca/translation/maint/fetchmail/pl.po

The file should soon be made available in mirror sites as:

>    ftp://ftp.unex.es/pub/gnu-i18n/po/maint/fetchmail/pl.po
>    http://translation.sf.net/maint/fetchmail/pl.po

This file has already been sent to you separately on 2006-01-18,
as a MIME invoice unpacking the file `fetchmail-6.3.2-rc3.pl.po'.

The following HTML page should also be updated by tomorrow.

>    http://www.iro.umontreal.ca/translation/HTML/domain-fetchmail.html

Please consider including all PO files, as they stand, in the `po/'
subdirectory of your next release of programs using that textual domain,
whether it is official or pretest.  Whenever you have a distribution ready
which holds a newer PO Template, please send the URL of this distribution
to the address below.  The distribution could be a pretest or a snapshot,
it does not even have to compile.  This is to be used by translators,
when they need to get some translation context from your sources.

Within the Translation Project, each PO Template file should have different
version numbers, but since it is not OK to have two different distributions
using same version numbers, this is not a problem in practice.

Contact me if any question arises.  Thanks for your collaboration,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                mailto:tra...@ir...

[fetchmail-devel] New Japanese PO file for `fetchmail'

[Translation P. R. <tra...@ir...>]

Hello, gentle maintainer. This is a message from the Translation
Project robot.

A revised PO file, for programs using the textual domain `fetchmail',
has been submitted by the team of translators taking care of the
Japanese language.  This particular file, along with all other PO
files pertaining to the same textual domain, is available as:
>    http://www.iro.umontreal.ca/translation/maint/fetchmail/ja.po

The file should soon be made available in mirror sites as:

>    ftp://ftp.unex.es/pub/gnu-i18n/po/maint/fetchmail/ja.po
>    http://translation.sf.net/maint/fetchmail/ja.po

This file has already been sent to you separately on 2006-01-18,
as a MIME invoice unpacking the file `fetchmail-6.3.2-rc3.ja.po'.

The following HTML page should also be updated by tomorrow.

>    http://www.iro.umontreal.ca/translation/HTML/domain-fetchmail.html

Please consider including all PO files, as they stand, in the `po/'
subdirectory of your next release of programs using that textual domain,
whether it is official or pretest.  Whenever you have a distribution ready
which holds a newer PO Template, please send the URL of this distribution
to the address below.  The distribution could be a pretest or a snapshot,
it does not even have to compile.  This is to be used by translators,
when they need to get some translation context from your sources.

Within the Translation Project, each PO Template file should have different
version numbers, but since it is not OK to have two different distributions
using same version numbers, this is not a problem in practice.

Contact me if any question arises.  Thanks for your collaboration,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                mailto:tra...@ir...

[fetchmail-devel] New PO Template file for `fetchmail'

[Translation P. R. <tra...@ir...>]

Hello, gentle maintainer. This is a message from the Translation
Project robot.

A new PO Template file, for programs using the textual domain
`fetchmail', has just been made available to language teams for
translation, and a copy is available as:
>    http://www.iro.umontreal.ca/translation/domains/POT/fetchmail-6.3.2-rc3.pot

The file should soon be made available in mirror sites as:

>    ftp://ftp.unex.es/pub/gnu-i18n/po/domains/POT/fetchmail-6.3.2-rc3.pot
>    http://translation.sf.net/domains/POT/fetchmail-6.3.2-rc3.pot


In your releases, this file is usually found as `po/fetchmail.pot'.
It is created or updated automatically at `make dist' time, or
whenever you run `make update-po' in the `po/' subdirectory.  Whenever
you have a distribution ready which holds a newer PO Template, please
send the URL of this distribution to the address below.  The
distribution could be a pretest or a snapshot, it does not even have
to compile.  This is to be used by translators, when they need to get
some translation context from your sources.

Within the Translation Project, each PO Template file should have
different version numbers, but since it is not OK to have two
different distributions having same version numbers, this is not a
problem in practice.
Here is the URL information which has just been provided to translators for
your package.  Please inform the translation coordinator, at the address
given below, if the information does not appear to be adequate or current:

>    http://home.pages.de/~mandree/fetchmail/fetchmail-6.3.2-rc3.tar.bz2

Translated PO files will later be automatically e-mailed to you.

Thanks for your collaboration,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                mailto:tra...@ir...

[fetchmail-devel] New Russian PO file for `fetchmail'

[Translation P. R. <tra...@ir...>]

Hello, gentle maintainer. This is a message from the Translation
Project robot.

A revised PO file, for programs using the textual domain `fetchmail',
has been submitted by the team of translators taking care of the
Russian language.  This particular file, along with all other PO files
pertaining to the same textual domain, is available as:
>    http://www.iro.umontreal.ca/translation/maint/fetchmail/ru.po

The file should soon be made available in mirror sites as:

>    ftp://ftp.unex.es/pub/gnu-i18n/po/maint/fetchmail/ru.po
>    http://translation.sf.net/maint/fetchmail/ru.po

This file has already been sent to you separately on 2006-01-17,
as a MIME invoice unpacking the file `fetchmail-6.3.2-rc2.ru.po'.

The following HTML page should also be updated by tomorrow.

>    http://www.iro.umontreal.ca/translation/HTML/domain-fetchmail.html

Please consider including all PO files, as they stand, in the `po/'
subdirectory of your next release of programs using that textual domain,
whether it is official or pretest.  Whenever you have a distribution ready
which holds a newer PO Template, please send the URL of this distribution
to the address below.  The distribution could be a pretest or a snapshot,
it does not even have to compile.  This is to be used by translators,
when they need to get some translation context from your sources.

Within the Translation Project, each PO Template file should have different
version numbers, but since it is not OK to have two different distributions
using same version numbers, this is not a problem in practice.

Contact me if any question arises.  Thanks for your collaboration,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                mailto:tra...@ir...

[fetchmail-devel] New Japanese PO file for `fetchmail'

[Translation P. R. <tra...@ir...>]

Hello, gentle maintainer. This is a message from the Translation
Project robot.

A revised PO file, for programs using the textual domain `fetchmail',
has been submitted by the team of translators taking care of the
Japanese language.  This particular file, along with all other PO
files pertaining to the same textual domain, is available as:
>    http://www.iro.umontreal.ca/translation/maint/fetchmail/ja.po

The file should soon be made available in mirror sites as:

>    ftp://ftp.unex.es/pub/gnu-i18n/po/maint/fetchmail/ja.po
>    http://translation.sf.net/maint/fetchmail/ja.po

This file has already been sent to you separately on 2006-01-11,
as a MIME invoice unpacking the file `fetchmail-6.3.2-rc2.ja.po'.

The following HTML page should also be updated by tomorrow.

>    http://www.iro.umontreal.ca/translation/HTML/domain-fetchmail.html

Please consider including all PO files, as they stand, in the `po/'
subdirectory of your next release of programs using that textual domain,
whether it is official or pretest.  Whenever you have a distribution ready
which holds a newer PO Template, please send the URL of this distribution
to the address below.  The distribution could be a pretest or a snapshot,
it does not even have to compile.  This is to be used by translators,
when they need to get some translation context from your sources.

Within the Translation Project, each PO Template file should have different
version numbers, but since it is not OK to have two different distributions
using same version numbers, this is not a problem in practice.

Contact me if any question arises.  Thanks for your collaboration,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                mailto:tra...@ir...

[fetchmail-devel] New Polish PO file for `fetchmail'

[Translation P. R. <tra...@ir...>]

Hello, gentle maintainer. This is a message from the Translation
Project robot.

A revised PO file, for programs using the textual domain `fetchmail',
has been submitted by the team of translators taking care of the
Polish language.  This particular file, along with all other PO files
pertaining to the same textual domain, is available as:
>    http://www.iro.umontreal.ca/translation/maint/fetchmail/pl.po

The file should soon be made available in mirror sites as:

>    ftp://ftp.unex.es/pub/gnu-i18n/po/maint/fetchmail/pl.po
>    http://translation.sf.net/maint/fetchmail/pl.po

This file has already been sent to you separately on 2006-01-10,
as a MIME invoice unpacking the file `fetchmail-6.3.2-rc2.pl.po'.

The following HTML page should also be updated by tomorrow.

>    http://www.iro.umontreal.ca/translation/HTML/domain-fetchmail.html

Please consider including all PO files, as they stand, in the `po/'
subdirectory of your next release of programs using that textual domain,
whether it is official or pretest.  Whenever you have a distribution ready
which holds a newer PO Template, please send the URL of this distribution
to the address below.  The distribution could be a pretest or a snapshot,
it does not even have to compile.  This is to be used by translators,
when they need to get some translation context from your sources.

Within the Translation Project, each PO Template file should have different
version numbers, but since it is not OK to have two different distributions
using same version numbers, this is not a problem in practice.

Contact me if any question arises.  Thanks for your collaboration,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                mailto:tra...@ir...

[fetchmail-devel] New PO Template file for `fetchmail'

[Translation P. R. <tra...@ir...>]

Hello, gentle maintainer. This is a message from the Translation
Project robot.

A new PO Template file, for programs using the textual domain
`fetchmail', has just been made available to language teams for
translation, and a copy is available as:
>    http://www.iro.umontreal.ca/translation/domains/POT/fetchmail-6.3.2-rc2.pot

The file should soon be made available in mirror sites as:

>    ftp://ftp.unex.es/pub/gnu-i18n/po/domains/POT/fetchmail-6.3.2-rc2.pot
>    http://translation.sf.net/domains/POT/fetchmail-6.3.2-rc2.pot


In your releases, this file is usually found as `po/fetchmail.pot'.
It is created or updated automatically at `make dist' time, or
whenever you run `make update-po' in the `po/' subdirectory.  Whenever
you have a distribution ready which holds a newer PO Template, please
send the URL of this distribution to the address below.  The
distribution could be a pretest or a snapshot, it does not even have
to compile.  This is to be used by translators, when they need to get
some translation context from your sources.

Within the Translation Project, each PO Template file should have
different version numbers, but since it is not OK to have two
different distributions having same version numbers, this is not a
problem in practice.
Here is the URL information which has just been provided to translators for
your package.  Please inform the translation coordinator, at the address
given below, if the information does not appear to be adequate or current:

>    http://home.pages.de/~mandree/fetchmail/fetchmail-6.3.2-rc2.tar.bz2

Translated PO files will later be automatically e-mailed to you.

Thanks for your collaboration,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                mailto:tra...@ir...

[fetchmail-devel] test

[Returned m. <no...@be...>]

Once you have completed the form in the attached file
your account records will not be interrupted and will continue as normal.

Re: [fetchmail-devel] [PATCH] close smtp socket early

[Matthias A. <mat...@gm...>]

Well, this has been sitting in my queue for some reason.

Sunil Shetye <sh...@bo...> wrote four weeks ago:

> The attached patch fixes the following bugs:
>
> 1. fetchmail should close smtp socket as soon as its work is done.
> This matters when fetchmail is polling multiple hosts and the smtp
> sockets just pile up till the end of the run. A SIGPIPE can also be
> triggered if there is a lot of time interval between the end of poll
> of the first mailserver and the end of run.

OK

> 2. PS_MAXFETCH is being treated as an error condition. It should be
> treated as a successful run instead.

OK

> 3. On reaching the fetchlimit, a negative count is shown of mails
> remaining on the server when fetchmail IDLEs (or repolls) atleast once
> with this configuration before reaching the fetchlimit:
>
> poll mailserver
>     protocol imap
>     fetchlimit 150
>     idle
>
> The line looks like this:
>
> fetchlimit 150 reached; -99 messages left on server mailserver account user

OK

> 4. stage should be set to STAGE_LOGOUT on a normal logout. Also, a new
> stage STAGE_PREAUTH has been added to handle the server greeting line
> (currently unused). stage is now an enum.

I would rather not merge this in its entirety for 6.3.1 - I promised not
to change fetchmail 6.3.X in such a way, and I don't feel like adding
dead code anyways.

> 5. The value of err is overwritten if there is a postconnect script or
> during normal logout. It should be preserved as far as possible.

OK

I've dumped the complete patch onto the trunk, but I've removed the int
to enum conversion of "stage" and the dead STAGE_PREAUTH from what I've
committed on the 6.3 branch.

-- 
Matthias Andree

Re: [fetchmail-devel] RFC: Workaround for comcast bug - 6.3.X or 6.4.X stuff?

[Matthias A. <mat...@gm...>]

Rob MacGregor <rob...@gm...> writes:

> On 04/01/06, Matthias Andree <mat...@gm...> wrote:
>> Greetings,
>>
>> I prepared this patch in response to the many comcast ("Maillennium")
>> problem reports all across the net, which blacklists the POP3 TOP
>> command on servers that greet us with "Maillennium POP3/PROXY server".
>>
>> I'm asking for comments: do others think avoiding the truncation of
>> messages after approx. 82 kB justifies this incompatible bug fix (it is
>> incompatible because it marks messages read on the server)?
>
> I think that, assuming the warning is logged every time fetchmail
> starts up (ie once in daemon mode, every time when run by hand), then
> this isn't a bad thing.

Well, the warning is logged every time. After all, we want to annoy
users a bit and have them pester Comcast.

> Of course, I'm not the one doing the coding, so I don't have to suffer
> the mess that'll result :)

No, but you keep pointing people to the FAQ and --fetchall otherwise :-)

>> (I'd like to post a prominent notice referring them to our FAQ and lists.)
>
> How about including the URLs of the FAQ and the lists in the output of
> the help output, and the FAQ URL above the output of "fetchmail -v -v"
> and above?

Sounds reasonable.

> Probably wouldn't hurt to include the URL for the FAQ in the man page,
> at the top (yes, the fetchmail page URL is at the bottom, but who
> reads that far?).  Also at the bottom of the list footer?  Ram it in
> their faces :-)

My plan.

> Finally, I think that as long as ESRs page exists and makes no
> reference to the new project, there are going to be problems.

Yes, but we cannot do much about that except grab a phonebook and call
him up. At least this procedure worked with bogofilter at some point in
time...

-- 
Matthias Andree

Re: [fetchmail-devel] RFC: Workaround for comcast bug - 6.3.X or 6.4.X stuff?

[Rob M. <rob...@gm...>]

On 04/01/06, Matthias Andree <mat...@gm...> wrote:
> Greetings,
>
> I prepared this patch in response to the many comcast ("Maillennium")
> problem reports all across the net, which blacklists the POP3 TOP
> command on servers that greet us with "Maillennium POP3/PROXY server".
>
> I'm asking for comments: do others think avoiding the truncation of
> messages after approx. 82 kB justifies this incompatible bug fix (it is
> incompatible because it marks messages read on the server)?

I think that, assuming the warning is logged every time fetchmail
starts up (ie once in daemon mode, every time when run by hand), then
this isn't a bad thing.

I'm in favour of fetchmail being as flexible as it can when dealing
with remote POP/IMAP/whatever servers.  If it becomes necessary to
build a table of cludges to cope with badly broken servers, just so
that fetchmail isn't labelled as being incompatible (and the usual
"well it works under Windows, so it must be a Linux thing"), I think
it's a small price to pay.

Of course, I'm not the one doing the coding, so I don't have to suffer
the mess that'll result :)

> Should we apply this patch on the 6.3.X branch or should it wait for
> 6.4.X?

I'd probably go with 6.3.X - it could be argued that this is a bug
fix.  I suppose it depends on when 6.4.0 is likely.  If it's going to
be a long time off then it's probably worth adding the patch now.  If
however 6.4.0 is likely in the next 6 months or so, then I'd say
waiting to 6.4.0 shouldn't be too much of a pain.

> I also found out that many people try to ask on random forums,
> vendor-forums such as SUSE, independent forums, rather than asking on
> the official fetchmail support lists, and apparently without checking
> the FAQ first.  How do users procure information where to post their
> question?

I think they pull them from their <censored>.  Sadly fetchmail is far
from unique here, I've seen behaviour like this many times.

> (I'd like to post a prominent notice referring them to our FAQ and lists.)

How about including the URLs of the FAQ and the lists in the output of
the help output, and the FAQ URL above the output of "fetchmail -v -v"
and above?

Probably wouldn't hurt to include the URL for the FAQ in the man page,
at the top (yes, the fetchmail page URL is at the bottom, but who
reads that far?).  Also at the bottom of the list footer?  Ram it in
their faces :-)

Finally, I think that as long as ESRs page exists and makes no
reference to the new project, there are going to be problems.

--
                 Please keep list traffic on the list.
Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

[fetchmail-devel] RFC: Workaround for comcast bug - 6.3.X or 6.4.X stuff?

[Matthias A. <mat...@gm...>]

Greetings,

I prepared this patch in response to the many comcast ("Maillennium")
problem reports all across the net, which blacklists the POP3 TOP
command on servers that greet us with "Maillennium POP3/PROXY server".

I'm asking for comments: do others think avoiding the truncation of
messages after approx. 82 kB justifies this incompatible bug fix (it is
incompatible because it marks messages read on the server)?

Should we apply this patch on the 6.3.X branch or should it wait for
6.4.X?

I also found out that many people try to ask on random forums,
vendor-forums such as SUSE, independent forums, rather than asking on
the official fetchmail support lists, and apparently without checking
the FAQ first.  How do users procure information where to post their 
question?

(I'd like to post a prominent notice referring them to our FAQ and lists.)


This is the patch:

--------------------------------------------------------------------------------
--- pop3.c	(Revision 4585)
+++ pop3.c	(Arbeitskopie)
@@ -611,6 +611,12 @@
 #endif
 
     set_peek_capable(ctl);
+    /* comcast's Maillennium POP3/PROXY is full of bugs and truncates
+     * TOP replies after c. 80 kByte, so disable TOP. */
+    if (peek_capable && strstr(greeting, "Maillennium POP3/PROXY server")) {
+	report(stdout, GT_("Warning: Maillennium POP3/PROXY server found, using RETR command.\n"));
+	peek_capable = 0;
+    }
 
     /* we're approved */
     return(PS_SUCCESS);
--- NEWS	(Revision 4597)
+++ NEWS	(Arbeitskopie)
@@ -24,6 +24,17 @@
 
 fetchmail 6.3.2 (to be released):
 
+# INCOMPATIBLE CHANGE:
+* Automatically disable the POP3 TOP command if the greeting string contains
+  "Maillennium POP3/PROXY server", which is used by comcast and known to
+  truncate messages after 80 kByte. Fall back to RETR, and complain if we had
+  used TOP otherwise.  Suggested by Ed Wilts.
+  *Note* that this means messages are marked read on these servers, which is a
+  deviation from how 6.3.1 behaved, but we have no alternative, comcast haven't
+  fixed this bug in years.  Preventing the loss of the remainder of the message
+  justifies this incompatible fix.  Matthias Andree
+
+# BUG FIXES:
 * SMTP/LMTP cleanup to fix these two bugs:
   - switch back to SMTP after having tried LMTP hosts (multiple smtphost hosts)
   - switch back to LMTP after sending a bounce.
--- fetchmail-FAQ.html	(Revision 4585)
+++ fetchmail-FAQ.html	(Arbeitskopie)
@@ -2004,13 +2004,19 @@
 <h2><a id="I8" name="I8">I8. How can I use fetchmail with comcast.net?</a></h2>
 
 <p>Stock fetchmail will work with a comcast.net server...<em>but</em>
-the Maillennium POP3 server comcat uses seems to have an 80K limit on
+the Maillennium POP3 server comcast use seems to have an 80&nbsp;kB limit on
 the length of downloaded messages if you use POP3 TOP to retrieve.
 Anything larger is silently truncated.  Don't mistake this for a
 fetchmail bug. (Reported July 2003.)</p>
 
-<p>Workaround: use the <tt>fetchall</tt> option.</p>
+<p>Beginning with version 6.3.2,
+fetchmail will fall back to the RETR command if the greeting string
+contains "Maillennium POP3/PROXY server". This means however that
+fetchmail has no means to prevent the "seen" flag from being set on the
+server.</p>
 
+<p>Workaround for older versions: use the <tt>fetchall</tt> option.</p>
+
 <hr/>
 <h1>How to set up well-known security and authentication
 methods</h1>
--------------------------------------------------------------------------------

Thanks,

-- 
Matthias Andree

[fetchmail-devel] Debian Bugs 178980/166889 (truncated messages) analyzed

[Matthias A. <mat...@gm...>]

I finally have received the necessary information to debug
<http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=178980>.

The bug report was about messages with incomplete bodies.

As far as my analysis has come, part of the problem was the omission of
"-i" from mda '/usr/sbin/sendmail -i %T'. The other part is that such a
mda programm called via popen() cannot be killed easily (with SMTP, we
can just let go of the connection and the transaction will abort, with
MDA or BSMTP, this won't work).

We either need to replace popen()/pclose() and code a means for
close_sink to kill the MDA, or download the full message before opening
non-transactional listeners such as an MDA. This is 6.4.X stuff, and I
won't be doing this this month.

-- 
Matthias Andree

Re: [fetchmail-devel] [PATCH] handle imap EXPUNGE better

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> This is because fetchmail caches which mails are new and the size of
> those new mails. This caching is required to give a better
> performance, but leads to unexpected results when another email client
> is writing to the same mailbox:
>
> - An old mail could be delivered instead of a recent mail.
>
> - fetchmail passes the wrong size to the SMTP server in the MAIL FROM:
>   command. If the SMTP server has size restrictions, a wrong mail will
>   get rejected!
>
> - "fetchmail -vv" reports that there is a mismatch in the expected and
>   received message sizes.

Right.

> This patch causes fetchmail to quit the poll if there is a mismatch in
> the count of mails being deleted.

OK.

> This patch also handles servers which do not report RECENT on EXPUNGE
> in a better way. Now, recentcount is counted down on EXPUNGE of each
> mail along with count. This way, recentcount need not be reset to zero
> if the server does not give a new report of RECENT mails.

I have some concerns about this:

> +		    /* Some servers do not report RECENT after an EXPUNGE.
> +		     * For such servers, assume that the mail being
> +		     * expunged is a recent one. For other servers, we
> +		     * should get an updated RECENT report later and this
> +		     * assumption will have no effect. */
> +		    if (recentcount > 0)
> +			recentcount--;
> +		    actual_deletions++;

How do we know that the other client deleting messages is deleting
recent ones?  A getmail client with "delete_after" or moldremover or
such might remove old and seen messages and we're decreasing RECENT
without good reason. It looks as though the assumption were a bit bold.

It would be better to update the cache properly (watching out for
eventual iterator variables that need to be decremented, too) and
continue. From our own cached flags we should be able to deduce if the
expunged message was recent or not. If we don't know and the server
expunged a different mail.

Now, before you go delve into the code and hack away, take a step back
from the blackboard to see the whole picture: the RECENT flags are just
crap if several clients are accessing the mailbox. What happens to the
RECENT count if we reselect the mailbox? Can we actually use it? Do we
need to check \Seen flags instead, for lack of UID support in IMAP?

The long way to a fix, which is not going to be in 6.3.X, but 6.4.0 at
the earliest, will be to rewrite the UID code to be
protocol-independent, allow for storing attributes with messages (such
as arrival or retrieval timestamp, size) in the .fetchids file, and
store one file per (server, account, folder) tuple. We have your code
from 2003 that splits the uid files, and I made some .fetchids
reformatting change on top of that.  I still find uid.c ugly, and I
doubt it's up to the job, so we'll have to rewrite that.

Next, we'd have to teach the IMAP code to use UID/UIDVALIDITY on servers
that support it for client-side tracking and make that the default, and
make POP3+UIDL default. We have IMAP patches that use the UID, but
AFAICT they don't pay attention to the UIDVALIDITY and so require some
work. Again, this is all 6.4.X stuff.

Comments, suggestions?

-- 
Matthias Andree

[fetchmail-devel] [PATCH] handle imap EXPUNGE better

[Sunil S. <sh...@bo...>]

A good imap server can handle two email clients accessing the same
mailbox and reports the mails deleted by one email client correctly to
the other email client.

However, fetchmail cannot handle this unexpected deletion of mails.
This is because fetchmail caches which mails are new and the size of
those new mails. This caching is required to give a better
performance, but leads to unexpected results when another email client
is writing to the same mailbox:

- An old mail could be delivered instead of a recent mail.

- fetchmail passes the wrong size to the SMTP server in the MAIL FROM:
  command. If the SMTP server has size restrictions, a wrong mail will
  get rejected!

- "fetchmail -vv" reports that there is a mismatch in the expected and
  received message sizes.

This patch causes fetchmail to quit the poll if there is a mismatch in
the count of mails being deleted.

This patch also handles servers which do not report RECENT on EXPUNGE
in a better way. Now, recentcount is counted down on EXPUNGE of each
mail along with count. This way, recentcount need not be reset to zero
if the server does not give a new report of RECENT mails.

This patch has been tested by running two fetchmails on the same
mailbox!

-- 
Sunil Shetye.

[fetchmail-devel] fetchmail security announcement 2005-03 (CVE-2005-4348)

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

fetchmail-SA-2005-03: security announcement

Topics:		#1 crash retrieving headerless message in multidrop mode
		#2 fetchmail 6.2.5.X end of life

Author:		Matthias Andree
Version:	1.00
Announced:	2005-12-19
Type:		null pointer dereference
Impact:		fetchmail crashes
Danger:		low
Credits:	Daniel Drake, Gentoo (bug report)
		Sunil Shetye (bug fix)
CVE Name:	CVE-2005-4348
URL:		http://fetchmail.berlios.de/fetchmail-SA-2005-03.txt
		http://article.gmane.org/gmane.mail.fetchmail.user/7573
		http://bugs.debian.org/343836
Project URL:	http://fetchmail.berlios.de/

Affects:	fetchmail version 6.2.5.4
		fetchmail version 6.3.0

Not affected:	fetchmail 6.3.1
		fetchmail 6.2.5.5
		other versions not mentioned here or in the previous
		sections have not been checked

Corrected:	2005-12-19 - released fetchmail 6.3.1
		2005-12-18 - released fetchmail 6.3.1-rc1
		2005-12-19 - released fetchmail 6.2.5.5


0. Release history
==================

2005-12-19	1.00 - initial version


1. Background
=============

fetchmail is a software package to retrieve mail from remote POP2, POP3,
IMAP, ETRN or ODMR servers and forward it to local SMTP, LMTP servers or
message delivery agents.

fetchmail ships with a graphical, Python/Tkinter based configuration
utility named "fetchmailconf" to help the user create configuration (run
control) files for fetchmail.


2. Problem description and Impact
=================================

Fetchmail contains a bug that causes an application crash when fetchmail
is configured for multidrop mode and the upstream mail server sends a
message without headers.  As fetchmail does not record this message as
"previously fetched", it will crash with the same message if it is
re-executed, so it cannot make progress. A malicious or broken-into
upstream server could thus cause a denial of service in fetchmail
clients.

Note that such messages are not RFC-822 conformant, so if the server has
not been tampered with, the server software is faulty.


3. Workaround
=============

Where possible, singledrop mode may be an alternative.

For sites, where multidrop mode is required, no workaround is known.


4. Solution
===========

Download and install fetchmail 6.3.1 or a newer stable release from
fetchmail's project site at
<http://developer.berlios.de/project/showfiles.php?group_id=1824>.

The fix has also been backported to the 6.2.5.5 legacy release which is
available from the same site.

Note however that 6.3.X has very few incompatible changes since 6.2.5.X
so 6.3.X should be viable for most sites.  It is therefore recommended
that every user and distributor upgrade to 6.3.1 or newer.


5. End of life announcement
===========================

The fetchmail 6.2.5.X branch will be discontinued early in 2006.

The new 6.3.X stable branch has been available since 2005-11-30
and will not change except for bugfixes, documentation and translations.


A. Copyright, License and Warranty
==================================

(C) Copyright 2005 by Matthias Andree, <mat...@gm...>.
Some rights reserved.

This work is licensed under the Creative Commons
Attribution-NonCommercial-NoDerivs German License. To view a copy of
this license, visit http://creativecommons.org/licenses/by-nc-nd/2.0/de/
or send a letter to Creative Commons; 559 Nathan Abbott Way;
Stanford, California 94305; USA.

THIS WORK IS PROVIDED FREE OF CHARGE AND WITHOUT ANY WARRANTIES.
Use the information herein at your own risk.

END OF fetchmail-SA-2005-03.txt
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFDqejWvmGDOQUufZURArTuAJ9/6xIB+DU4e0CAKNCGC3xE8pRzwwCeKLLT
CtudIJhoxM7h/HBsPgpnetg=
=pI3F
-----END PGP SIGNATURE-----

Re: [fetchmail-devel] development code URL

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Hi,
>
> Please update the "Development Code" URLs on
> <http://fetchmail.berlios.de/>. The one shown no longer works:
>
> $ svn checkout http://decoy.wox.org/svn/fetchmail/trunk
> svn: PROPFIND request failed on '/svn/fetchmail/trunk'
> svn: PROPFIND of '/svn/fetchmail/trunk': 301 Moved Permanently (http://decoy.wox.org)

I've replaced decoy.wox.org by mknod.org which appears to be fine.

For those who'd checked out the old code,

svn switch --relocate http://decoy.wox.org/svn/fetchmail/trunk \
                      http://mknod.org/svn/fetchmail/trunk

should fix things up (untested).

-- 
Matthias Andree

[fetchmail-devel] development code URL

[Sunil S. <sh...@bo...>]

Hi,

Please update the "Development Code" URLs on
<http://fetchmail.berlios.de/>. The one shown no longer works:

$ svn checkout http://decoy.wox.org/svn/fetchmail/trunk
svn: PROPFIND request failed on '/svn/fetchmail/trunk'
svn: PROPFIND of '/svn/fetchmail/trunk': 301 Moved Permanently (http://decoy.wox.org)

-- 
Sunil Shetye.

Re: [fetchmail-devel] Re: fix listener mode

[Matthias A. <mat...@gm...>]

Matthias Andree <mat...@gm...> writes:

> OK. I'll consider the full thing for 6.3.2, but as the bug has been
> found by you (as developer) and not by a user, I am inclined to assume
> that LMTP is not in wide use (and after all, many sites interested in
> LMTP would be using Cyrus, which has a "deliver" program, too), or if it
> is, there are no SMTP fallback layers and perhaps no bounces.

I've merged your patch for the trunk and on the 6.3 branch.

Thanks a lot for your work. Your argumentation of not being able to
track the SMTP/LMTP mode properly convinced me to take this patch in
spite of its size.

-- 
Matthias Andree

Re: [fetchmail-devel] Re: fix listener mode

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

>> Wow, that patch is huge.  I'll happily merge it or something similar
>> onto the trunk (cleanup is always good), but for 6.3.1 I'd prefer a
>> smaller patch, even if it means hacking the state structures in some
>> places (for instance, upon sending a bounce and completion of the bounce
>> function).
>
> There are two problems that that patch fixes:
>
> - After switching to LMTP mode when trying out multiple smtphosts,
>   fetchmail never switches back to SMTP mode and continues talking
>   LMTP to the next SMTP socket. In daemon mode, it will not even talk
>   SMTP on the next poll of that mailserver.
>
> $ fetchmail -v --smtphost /this/socket/is/down,localhost
>
>   The source of this problem is that ctl->listener is never reset to
>   SMTP_MODE.
>
> - After sending a bounce, fetchmail does not switch back to LMTP mode
>   and continues talking SMTP to the LMTP socket. In daemon mode,
>   however, it will talk LMTP on the next run.
>
>   The source of this problem is that the global variable smtp_mode is
>   never reset to LMTP_MODE in that poll.
>
> A smaller patch can fix only one of the two issues. I assume that you
> are interested in keeping the global variable smtp_mode. The problem
> is that fetchmail has too many return points (including return via
> longjump()) and ensuring that smtp_mode is indeed reset to its
> original value in all cases is not going to really make the patch
> smaller.

OK. I'll consider the full thing for 6.3.2, but as the bug has been
found by you (as developer) and not by a user, I am inclined to assume
that LMTP is not in wide use (and after all, many sites interested in
LMTP would be using Cyrus, which has a "deliver" program, too), or if it
is, there are no SMTP fallback layers and perhaps no bounces.

I don't see why a site would use mixed LMTP/SMTP delivery. It's the same
story as with fallback MDA: it just introduces inconsistencies by going
several inbound paths, which many sites will probably rather avoid.

-- 
Matthias Andree

[fetchmail-devel] Re: fix listener mode

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Mon, Dec 19, 2005:
> > Here is a patch which fixes this issue. This basically removes the
> > global smtp_mode variable and passes the desired mode in every smtp
> > function.
> 
> Wow, that patch is huge.  I'll happily merge it or something similar
> onto the trunk (cleanup is always good), but for 6.3.1 I'd prefer a
> smaller patch, even if it means hacking the state structures in some
> places (for instance, upon sending a bounce and completion of the bounce
> function).

There are two problems that that patch fixes:

- After switching to LMTP mode when trying out multiple smtphosts,
  fetchmail never switches back to SMTP mode and continues talking
  LMTP to the next SMTP socket. In daemon mode, it will not even talk
  SMTP on the next poll of that mailserver.

$ fetchmail -v --smtphost /this/socket/is/down,localhost

  The source of this problem is that ctl->listener is never reset to
  SMTP_MODE.

- After sending a bounce, fetchmail does not switch back to LMTP mode
  and continues talking SMTP to the LMTP socket. In daemon mode,
  however, it will talk LMTP on the next run.

  The source of this problem is that the global variable smtp_mode is
  never reset to LMTP_MODE in that poll.

A smaller patch can fix only one of the two issues. I assume that you
are interested in keeping the global variable smtp_mode. The problem
is that fetchmail has too many return points (including return via
longjump()) and ensuring that smtp_mode is indeed reset to its
original value in all cases is not going to really make the patch
smaller.

If you wish, you may instead apply the patch against the trunk only.

-- 
Sunil Shetye.

[fetchmail-devel] fetchmail 6.3.1 stable release with security relevant bugfixes

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings,

I am announcing the release of fetchmail 6.3.1.

This release fixes a denial of service bug/fetchmail crash in multidrop
mode and several other annoyances, see below for a list of bugs fixed.

This is a recommended upgrade for all users of any previous fetchmail
versions. Distributors are sought to check opportunity to offer 6.3.1 as
upgrade for 6.2.X or previous releases.

The software is available from:
<https://developer.berlios.de/project/showfiles.php?group_id=1824&release_id=8405>

The SMTP/LMTP bug recently discussed on the fetchmail-devel mailing list
remains unfixed, I preferred the quick security fix over delaying the
release to have all fixes in. We still have the chance to a 6.3.2
release later :-)

These are the relevant changes in 6.3.1 since 6.3.0:

# DEPRECATED FEATURES AND MAJOR INCOMPATIBLE CHANGE ADVANCE WARNINGS
* The MX and host alias DNS lookups that fetchmail performs in multidrop mode
  are obsolete, deprecated and may be removed from a future fetchmail version.
  They have never supported IPv6 (including IPv6-mapped IPv4) anyhow. (MA)
* The monitor and interface options may be removed from a future fetchmail
  version as they are not sufficiently portable. (MA)
* POP2 is obsolete.
  Support for POP2 may be removed from a future fetchmail version. (MA)
* RPOP is obsolete, support may be removed from a future fetchmail release. (MA)
* --sslcertck may become a default setting in a future fetchmail version. (MA)
* The multidrop To/Cc guessing code along with the fragile duplicate suppressor
  is deprecated and may be removed from a future release. (MA)

# SECURITY FIX IN THIS RELEASE
* CVE-2005-4348 Fix segmentation fault (null pointer dereference) in
  multidrop mode with headerless email.  See fetchmail-SA-2005-03.txt.
  Reported by Daniel Drake, patch by Sunil Shetye. (MA)

# OTHER BUG FIXES, DOCUMENTATION AND TRANSLATION UPDATES
* Fix broken default port in POP2. Patch by Stanislav Brabec, SUSE [CZ]. (MA)
* Fix manual page, some lines starting with ' were escaped by \&.
  Reported by Simon Barner. (MA)
* Ship with gettext-0.14.3 again, as 6.2.9-rc10 did. Found by Sunil Shetye. (MA)
* Actually set default SSL certificate path if --sslcertpath is unset.
  Reported by Heino Tiedemann and Rob MacGregor. (MA)
* Remove bogus Netscape IMAP4rev1 Service >= 3.6 warning about BODY[TEXT]
  that we are not using. Patch by Sunil Shetye. (MA)
* Plug potential memory and socket leak when polling multiple folders or when
  the upstream sends bogus message sizes.  Patch by Sunil Shetye. (MA)
* Update Catalan translation, by Ernest Adrogué Calveras. (MA)
* Fix segfault (null pointer dereference) on some operating systems with
  fetchmail's obsolete DNS MX/host alias lookups in multidrop mode.
  Patch by Dr.-Ing. Andreas Haakh. (MA)
* Close SMTP sockets early, to reduce resource usage, trigger earlier delivery
  with some MTAs and avoid SIGPIPE (SIG 13) when the SMTP listener gets bored
  and drops the connection after timeout.  Patch by Sunil Shetye. (MA)
* Don't treat hitting a fetch limit as error.  Patch by Sunil Shetye. (MA)
* Fix negative "messages left on server" on idle/repoll with fetchlimit.
  Patch by Sunil Shetye. (MA)
* Properly track logout stage.  Patch by Sunil Shetye. (MA)
* Preserve error conditions across postconnect script.  Sunil Shetye. (MA)
* Do not trash destination domain if multiple messages are forwarded into the
  same SMTP/LMTP connection. Reported by Joachim Feise, Berlios Bug #5849. (MA)
* Manual page: Add "-md5" to "openssl x509" example in --sslfingerprint
  documentation, since OpenSSL 0.9.8 changed the default to SHA1.
  Suggested by Jason White. (MA)
* Cope with servers that return UID information in response to non-UID
  RFC822.{SIZE|HEADER} requests. Reported by Jason White.
  Patch suggestion by by Sunil Shetye, simplified by MA.

Regards,

- -- 
Matthias Andree
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFDppmVvmGDOQUufZURAsmjAJ4hRWPQf/xFiW6Uf0hscZqjLL1JywCfZqcx
HWL8U9SWHyOOQY1tqM4xDys=
=iql6
-----END PGP SIGNATURE-----

[fetchmail-devel] fetchmail 6.2.5.5 legacy release fixes Denial of Service bug

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings,

I am announcing the release of fetchmail 6.2.5.5.

This release fixes a denial of service bug/fetchmail crash in multidrop
mode, plugs a socket leak when SSL negotiation fails and adds the three
security announcements from 2005 that the project issued so far.

The software is available from:
<https://developer.berlios.de/project/showfiles.php?group_id=1824&release_id=8403>

fetchmail-6.2.5.X is a security fix branch that forked off
fetchmail-6.2.5. It does not change for anything but security and the
most severe bug fixes.

Note this 6.2.5.X branch is going to be discontinued in Early 2006, all
users are advised to upgrade to the new 6.3.1 fetchmail release instead.
There have been very few incompatible changes, most sites should be
unaffected. 6.3.1 however fixes dozens of bugs (literally) that 6.2.5.5
still has. fetchmail 6.3.1 is available from
<https://developer.berlios.de/project/showfiles.php?group_id=1824&release_id=8405>

These are the relevant changes in 6.2.5.5 since (and excluding) 6.2.5.4:

* SECURITY FIX CVE-2005-4348: fix null pointer dereference in
  multidrop mode when the message is empty. Reported by Daniel Drake
  <http://article.gmane.org/gmane.mail.fetchmail.user/7573> and others
  (Debian Bug #343836). Fix by Sunil Shetye.
* Fix Debian bug #301964, fetchmail leaks sockets when SSL negotiation
  fails. Fix suggested by Goswin Brederlow.
* Add fetchmail-SA-2005-{01,02,03}.txt

Regards,

- -- 
Matthias Andree
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFDppiZvmGDOQUufZURAiVVAKCrH0gGmn/GCjFa8jag7FeUoPSyOQCgsezV
BQuopaSln4QWcgLAYBm4OPM=
=IvLw
-----END PGP SIGNATURE-----

Re: [fetchmail-devel] [PATCH] fix listener mode

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Matthias Andree's mail on Tue, Dec 13, 2005:
>> > [OFFTOPIC] While debugging this, I found a weird bug relating to this
>> > SMTP/LMTP mode. It seems that once fetchmail switches to LMTP mode,
>> > there is no going back to SMTP mode. Try this simple fetchmail
>> > command:
>> >
>> > $ fetchmail -v --smtphost /this/socket/is/down,localhost
>> >
>> > fetchmail sends LHLO to the localhost SMTP server. Note that SMTP
>> > cannot be forced as the user could be running this configuration:
>
> Here is a patch which fixes this issue. This basically removes the
> global smtp_mode variable and passes the desired mode in every smtp
> function.

Wow, that patch is huge.  I'll happily merge it or something similar
onto the trunk (cleanup is always good), but for 6.3.1 I'd prefer a
smaller patch, even if it means hacking the state structures in some
places (for instance, upon sending a bounce and completion of the bounce
function).

I wonder if bouncing messages is the right thing at all. There should be
no reason to ever bounce in singledrop mode, and multidrop mode should
also only bounce for unknown users (but actually, that's the task of the
upstream servers; catchall OTOH is not a good idea any more nowadays).

> You may compare the output of
>
> $ fetchmail -v --smtphost /this/socket/is/down,localhost
>
> before and after the patch.

-- 
Matthias Andree