fetchmail-devel — Development related discussion

[fetchmail-devel] fetchmail 6.3.18-pre2 snapshot should fix Exchange authentication

[Matthias A. <mat...@gm...>]

Greetings,

I have been working on the "Exchange authentication failures" problem a
bit, and seem to have found a bug where fetchmail goes out of
synchronization, and mistakes the previous GSSAPI authentication failure
for a failure of the password authentication.

Fetchmail 6.3.18-pre2 is supposed to fix this bug.  To do that, please:

1. download and unpack the fetchmail tarball (URLs below)
2. cd to to the unpacked directory
3. type: patch -p1 <contrib/patch-for-debugging
4. ./configure and install as usual
5. run fetchmail with these additional options:
   --auth any -vvvd0 --nodetach --nosyslog
6. if the transcript you get contains a "AUTH GSSAPI" attempt, please
   send me the log off-list. You may need to edit my address manually
   depending on mailer.

PLEASE HELP: If you can offer access to test servers that I can send a
short test mail to and then log into to retrieve that test message -
particularly Exchange 2007 or Exchange 2010 is desired, but others
besides Cyrus IMAP and Dovecot are also welcome - please let me know.

PLEASE HELP: fetchmail needs translators for the program strings. Some
languages (such as those shown below) are in quite good shape, but
others are lacking a bit.  Translation information can be found at
<http://translationproject.org/domain/fetchmail.html>

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

DOWNLOAD this beta software from:
<http://home.pages.de/~mandree/fetchmail/>

The repository can be browsed at and cloned from:
<http://gitorious.org/fetchmail/fetchmail>

Git (the software used to keep the fetchmail source code version
controlled) information is at: <http://git-scm.com/>

CHANGES since the previous formal release of fetchmail listed below.
Unless otherwise noted, the changes were made by Matthias Andree:

# SECURITY IMPROVEMENTS TO DEFANG X.509 CERTIFICATE ABUSE
* Fetchmail now only accepts wildcard certificate common names and subject
  alternative names if they start with "*.". Previous versions would accept
  wildcards even if no period followed immediately.
* Fetchmail now disallows wildcards in certificates to match domain literals
  (such as 10.9.8.7), or wildcards in domain literals ("*.168.23.23").
  The test is overly picky and triggers if the pattern (after skipping the
  initial wildcard "*") or domain consists solely of digits and dots, and thus
  matches more than needed.
* Fetchmail now disallows wildcarding top-level domains.

# BUG FIXES
* Fetchmail would warn about insecure SSL/TLS connections even if a matching
  --sslfingerprint was specified. This is an omission from an SSL usability
  change made in 6.3.17.  Fixes Debian Bug#580796 reported by Roland Stigge.
* Fetchmail 6.3.15, 6.3.16, and 6.3.17 would pick up libmd5 to obtain MD5*
  functions, as an effect of an undocumented Solaris MD5 fix.
  This fails if, for instance, libmd5.so was installed on other operating
  systems as part of libwww on machines where long isn't 32-bits. Fixes Gentoo
  Bug #319283, reported - including the hint to libwww - by Karl Hakimian.
  Side effect: fetchmail will now use -lmd on Solaris rather than -lmd5.
* Fetchmail will no longer print connection attempts and errors for one host
  in "silent" and "normal" logging modes, unless all connections fail. This
  should reduce irritation around refused-connection logging if services are
  only on an IPv4 socket if the host also supports IPv6. Often observed as
  connections refused to ::1/25 when the subsequent connection to 127.0.0.1/25
  then - silently - succeeds.  Fetchmail, unless in verbose mode, will collect
  all connect errors and only report them if all of them fail.
* Fetchmail will now apply timeouts to the authentication stage. This stage
  encompasses STARTTLS/STLS negotiation in IMAP/POP3.
  Reported missing by Thomas Jarosch.
* Fetchmail will not try GSSAPI authentication automatically unless it has GSS
  credentials. This avoids getting servers such as Exchange 2007 wedged if
  GSSAPI authentication fails.  Reported by Patrick Rynhart, Debian Bug #568455,
  and Alan Murrell, to the fetchmail-users list.
  Note that if GSSAPI fails for other reasons, you can use the --auth option to
  work around that.
* Fetchmail now parses response to "FETCH n:m RFC822.SIZE" and "FETCH n
  RFC822.HEADER" in a more flexible manner. (Sunil Shetye)
* Fetchmail now cancels GSSAPI authentication properly when encountering GSS
  errors. It now sends an asterisk on a line by its own, as required in SASL.
    This should fix protocol synchronization issues that cause Authentication
  failure, particularly with Exchange 2007 and Exchange 2010 servers, when
  Kerberos authentication was offered by the server and attempted by fetchmail.

# CHANGES
* When encountering incorrect headers, fetchmail will refer to the bad-header
  option in the manpage. BerliOS Bug #17272, change suggested by Björn Voigt.
* Fetchmail now decodes and reports GSSAPI status codes upon errors.

# TRANSLATION UPDATES
  [zh_CN] Chinese/simplified (Ji Zheng-Yu)
  [cs]    Czech (Petr Pisar)
  [fr]    French (Frédéric Marchal)
  [de]    German
  [it]    Italian (Vincenzo Campanella)
  [ja]    Japanese (Takeshi Hamasaki)
  [pl]    Polish (Jakub Bogusz)

# KNOWN BUGS AND WORKAROUNDS:
  (this section floats upwards through the NEWS file so it stays with the
  current release information - however, it was stuck with 6.3.8 for a while)
* fetchmail does not handle messages without Message-ID header well
  (See sourceforge.net bug #780933)
* BSMTP is mostly untested and errors can cause corrupt output.
* Sun Workshop 6 (SPARC) is known to miscompile the configuration file lexer in
  64-bit mode.  Either compile 32-bit code or use GCC to compile 64-bit
  fetchmail.  Note that fetchmail doesn't take advantage of 64-bit code,
  so compiling 32-bit SPARC code should not cause any difficulties.
* fetchmail does not track pending deletes over crashes
* the command line interface is sometimes a bit stubborn, for instance,
  fetchmail -s doesn't work with a daemon running
* Linux may return duplicates of an IP address in some circumstances if no or
  no global IPv6 addresses are configured. (No workaround. Ubuntu Bug#582585,
  Novell Bug#606980.)

-- 
Matthias Andree

[fetchmail-devel] New Chinese (simplified) PO file for 'fetchmail' (version 6.3.18-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Chinese (simplified) team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/zh_CN.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

Re: [fetchmail-devel] [fetchmail-users] fetchmail 6.3.18-pre1 snapshot up for testing

[Matthias A. <mat...@gm...>]

Am 03.09.2010, 06:38 Uhr, schrieb grarpamp:

>>  # SECURITY IMPROVEMENTS TO DEFANG X.509 CERTIFICATE ABUSE
>
> As a general note, should the degree of lint checking potentially impact
> CA/self/unsigned certs with various parametes that clueless operators  
> might
> be using... if the user specifies the md5 and/or sha1 fingerprint, along  
> with a
> future 'accept_fingerprint' option, that cert should be accepted despite  
> said
> lint. Every so often someones business requirement forces them to use/do
> silly things with certs :)

I believe the changes made are quite conservative. The new code is at  
<http://gitorious.org/fetchmail/fetchmail/blobs/master/x509_name_match.c>  
and it's documented what I'm trying to refuse. Please raise any concerns  
you find. :)

-- 
Matthias Andree

Re: [fetchmail-devel] [fetchmail-users] fetchmail 6.3.18-pre1 snapshot up for testing

[grarpamp <gra...@gm...>]

>  # SECURITY IMPROVEMENTS TO DEFANG X.509 CERTIFICATE ABUSE

As a general note, should the degree of lint checking potentially impact
CA/self/unsigned certs with various parametes that clueless operators might
be using... if the user specifies the md5 and/or sha1 fingerprint, along with a
future 'accept_fingerprint' option, that cert should be accepted despite said
lint. Every so often someones business requirement forces them to use/do
silly things with certs :)

[fetchmail-devel] New French PO file for 'fetchmail' (version 6.3.18-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the French team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/fr.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Italian PO file for 'fetchmail' (version 6.3.18-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Italian team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/it.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Polish PO file for 'fetchmail' (version 6.3.18-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Polish team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/pl.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Czech PO file for 'fetchmail' (version 6.3.18-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Czech team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/cs.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Japanese PO file for 'fetchmail' (version 6.3.18-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Japanese team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/ja.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New template for 'fetchmail' made available

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.  (If you have
any questions, send them to <coo...@tr...>.)

A new POT file for textual domain 'fetchmail' has been made available
to the language teams for translation.  It is archived as:

    http://translationproject.org/POT-files/fetchmail-6.3.18-pre1.pot

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

Below is the URL which has been provided to the translators of your
package.  Please inform the translation coordinator, at the address
at the bottom, if this information is not current:

    http://www.dt.e-technik.tu-dortmund.de/~ma/fetchmail/fetchmail-6.3.18-pre1.tar.bz2

We can arrange things so that translated PO files are automatically e-mailed
to you when they arrive.  Ask at the address below if you want this.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] fetchmail 6.3.18-pre1 snapshot up for testing

[Matthias A. <mat...@gm...>]

Greetings,

#1 I am asking interested fetchmail users to test a preview release of
fetchmail. Change details are below.

#2 Also, if you can offer access to test servers that I can send a short
test mail to and then log into to retrieve that test message -
particularly Exchange 2007 is desired, but others besides Cyrus IMAP and
Dovecot are also welcome - please let me know.

#3 Finally, fetchmail needs translators for the program strings. Some
languages (such as those shown below) are in quite good shape, but
others are lacking a bit.

Translation information at
<http://translationproject.org/domain/fetchmail.html>

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

DOWNLOAD this beta software from:
<http://home.pages.de/~mandree/fetchmail/>

The repository can be browsed at and cloned from:
<http://gitorious.org/fetchmail/fetchmail>

Git (the software used to keep the fetchmail source code version
controlled) information is at: <http://git-scm.com/>

CHANGES since the previous formal release of fetchmail listed below.
Unless otherwise noted, the changes were made by Matthias Andree:


# SECURITY IMPROVEMENTS TO DEFANG X.509 CERTIFICATE ABUSE
* Fetchmail now only accepts wildcard certificate common names and subject
  alternative names if they start with "*.". Previous versions would accept
  wildcards even if no period followed immediately.
* Fetchmail now disallows wildcards in certificates to match domain literals
  (such as 10.9.8.7), or wildcards in domain literals ("*.168.23.23").
  The test is overly picky and triggers if the pattern (after skipping the
  initial wildcard "*") or domain consist solely of digits and dots and matches
  more than needed.
* Fetchmail now disallows wildcarding top-level domains.

# BUG FIXES
* Fetchmail would warn about insecure SSL/TLS connections even if a matching
  --sslfingerprint was specified. This is an omission from an SSL usability
  change made in 6.3.17.  Fixes Debian Bug#580796 reported by Roland Stigge.
* Fetchmail 6.3.15, 6.3.16, and 6.3.17 would pick up libmd5 to obtain MD5*
  functions, as an effect of an undocumented Solaris MD5 fix.
  This fails if, for instance, libmd5.so was installed on other operating
  systems as part of libwww on machines where long isn't 32-bits. Fixes Gentoo
  Bug #319283, reported - including the hint to libwww - by Karl Hakimian.
  Side effect: fetchmail will now use -lmd on Solaris rather than -lmd5.
* Fetchmail will no longer print connection attempts and errors for one host
  in "silent" and "normal" logging modes, unless all connections fail. This
  should reduce irritation around refused-connection logging if services are
  only on an IPv4 socket if the host also supports IPv6. Often observed as
  connections refused to ::1/25 when the subsequent connection to 127.0.0.1/25
  then - silently - succeeds.  Fetchmail, unless in verbose mode, will collect
  all connect errors and only report them if all of them fail.
* Fetchmail will now apply timeouts to the authentication stage. This stage
  encompasses STARTTLS/STLS negotiation in IMAP/POP3.
  Reported missing by Thomas Jarosch.
* Fetchmail will not try GSSAPI authentication automatically unless it has GSS
  credentials. This avoids getting servers such as Exchange 2007 wedged if
  GSSAPI authentication fails.  Reported by Patrick Rynhart, Debian Bug #568455,
  and Alan Murrell, to the fetchmail-users list.
  Note that if GSSAPI fails for other reasons, you can use the --auth option to
  work around that.
* Fetchmail now parses response to "FETCH n:m RFC822.SIZE" and "FETCH n
  RFC822.HEADER" in a more flexible manner. (Sunil Shetye)

# CHANGES
* When encountering incorrect headers, fetchmail will refer to the bad-header
  option in the manpage. BerliOS Bug #17272, change suggested by Björn Voigt.
* Fetchmail now decodes and reports GSSAPI status codes upon errors.

# KNOWN BUGS AND WORKAROUNDS:
  (this section floats upwards through the NEWS file so it stays with the
  current release information - however, it was stuck with 6.3.8 for a while)
* fetchmail does not handle messages without Message-ID header well
  (See sourceforge.net bug #780933)
* BSMTP is mostly untested and errors can cause corrupt output.
* Sun Workshop 6 (SPARC) is known to miscompile the configuration file lexer in
  64-bit mode.  Either compile 32-bit code or use GCC to compile 64-bit
  fetchmail.  Note that fetchmail doesn't take advantage of 64-bit code,
  so compiling 32-bit SPARC code should not cause any difficulties.
* fetchmail does not track pending deletes over crashes
* the command line interface is sometimes a bit stubborn, for instance,
  fetchmail -s doesn't work with a daemon running
* Linux may return duplicates of an IP address in some circumstances if no or
  no global IPv6 addresses are configured. (No workaround. Ubuntu Bug#582585,
  Novell Bug#606980.)

-- 
Matthias Andree

Re: [fetchmail-devel] Slow handling of large fetchid-list?

[Matthias A. <mat...@gm...>]

Am 27.08.2010 17:29, schrieb Robert Linden:
> Hello fetchmail-devel!
> 
> I would like to get a confirmation that the following fetchmail-behaviour is
> normal/the way it is implemented, or if I might be doing something wrong:

Confirmed as suboptimal fetchmail -6.3.X behaviour, due to use of inadequate
data structures, namely linked lists, as you've already figured out.

> I noticed that fetchmail becomes very slow when I have had a lot of
> mails downloaded, i.e. when the fetchid-file grows very large.
> The numbers for one run (with no new mail) look roughly like this:
> 
>     10 lines:  0.5 sec
> 30.000 lines:  5.0 sec
> 45.000 lines: 10.0 sec
> 60.000 lines: 20.0 sec
> 90.000 lines: 70.0 sec
> 
> At first glance (at the code) it seems that the handling of this list is
> suboptimal, with traversal of linked lists and stuff like that. Also it is
> written [completely] a lot.
> 
> Is this over-proportional time to be expected and has anybody seen it
> too, or are long id-lists working fine for everone but me?

Over-proportional time (over the number of messages or UIDs stored) is
unavoidable the way fetchmail is currently designed, however we can do better
than what we're doing currently.

In fact, Rainer Weikusat has contributed new code that gets complexity down from
O(n^2) to O(n log n) with P(atricia)-Tries. We don't get better than O(n log n)
without major changes to the overall design, and I'm not sure going further is
worth the effort. In practice however, O(n log n) is quite manageable, and it
feels much faster and wastes much less CPU.

While not exact and lacking constant factors and constant overhead, this gives a
rough idea of how the current and new code effort would scale in a simulation.

Left column is the number of messages kept, right two columns are computational
complexity compared, roughly.

                     n               current                   new

                    10                   100                  33.2
                    30                   900                 147.2
                   100                 10000                 664.4
                   300                 90000                2468.6
                  1000               1000000                9965.8
                  3000               9000000               34652.2
                 10000             100000000              132877.1
                 30000             900000000              446180.2
                100000           10000000000             1660964

See the list archives:
<http://lists.berlios.de/pipermail/fetchmail-devel/2010-June/001359.html>

I plan to merge this for fetchmail 6.4 (which please consider a working name and
not necessarily the final release version, I may call it 6.5 or 7.0 depending on
other changes I make).

> Apologies if this is the wrong list for such a question.

No need to apologize, you've picked the right list.

> PS: just FYI, I use the same id-file for several different accounts, so my
> first workaround will be to split it up, so that each account has it's own
> list - but still I wonder if this is a common problem.

It will help only a tiny bit, because fetchmail splits the .fetchids file into
several lists internally. Not worth the effort IMO.

-- 
Matthias Andree

[fetchmail-devel] Slow handling of large fetchid-list?

[Robert L. <r.l...@ta...>]

Hello fetchmail-devel!

I would like to get a confirmation that the following fetchmail-behaviour is
normal/the way it is implemented, or if I might be doing something wrong:

I noticed that fetchmail becomes very slow when I have had a lot of
mails downloaded, i.e. when the fetchid-file grows very large.
The numbers for one run (with no new mail) look roughly like this:

    10 lines:  0.5 sec
30.000 lines:  5.0 sec
45.000 lines: 10.0 sec
60.000 lines: 20.0 sec
90.000 lines: 70.0 sec

At first glance (at the code) it seems that the handling of this list is
suboptimal, with traversal of linked lists and stuff like that. Also it is
written [completely] a lot.

Is this over-proportional time to be expected and has anybody seen it
too, or are long id-lists working fine for everone but me?

Apologies if this is the wrong list for such a question.

Thanks a lot,
rob

PS: just FYI, I use the same id-file for several different accounts, so my
first workaround will be to split it up, so that each account has it's own
list - but still I wonder if this is a common problem.

-- 
tarent Gesellschaft für Softwareentwicklung und IT-Beratung mbH
Geschäftsführer: Boris Esser, Elmar Geese
HRB AG Bonn 5168 - USt-ID (VAT): DE122264941

Heilsbachstraße 24,  53123 Bonn,   Telefon: +49 228 52675-0
Thiemannstraße 36 a, 12059 Berlin, Telefon: +49 30 5682943-30
Internet: http://www.tarent.de/  • Telefax: +49 228 52675-25

Re: [fetchmail-devel] timeout not working for TLS

[Matthias A. <mat...@gm...>]

Am 28.07.2010, 15:44 Uhr, schrieb Thomas Jarosch:

> On Wednesday, 28. July 2010 15:31:19 Matthias Andree wrote:
>> > This morning I had a better idea: Enable core dumps and kill the task
>> > via "kill -11" if it hangs again. Then we'll know for sure where it's
>> > stuck.
>>
>> Don't. fetchmail in most modes suppresses writing core files with
>> setrlimit(), to avoid passwords hitting the disk outside the .netrc or
>> .fetchmailrc files.
>
> I just patched that out :o) Before pushing the "new" version to the
> productive system, I've verified the coredump gives a useful backtrace.
>
>> Instead, if fetchmail hangs, just attach gdb with "gdb
>> /usr/bin/fetchmail-unstripped 12345", where fetchmail-unstripped is an
>> executable compiled with -g or better -ggdb3 option, and installed  
>> before
>> the run, without running strip and without adding -s to the install
>> command line.
>
> No gdb on the productive box. I've just installed the unstripped version
> (took me 30min to discover the -s switch in the compile statement.  
> Whoops).
> Now we need to wait a month or so... thanks for your suggestions!

Hi Thomas,

Well... perhaps not.

The attached patch sets the timeout for the getauth() stage (which entails  
STARTTLS-like negotiation). Please try that too and see if you get timeout  
reports while fetchmail tries to negotiate TLS. It should.

Thanks for the report and offered help in debugging. Would be good if you  
could report back in a month or so :-)

Best regards

-- 
Matthias Andree

Re: [fetchmail-devel] timeout not working for TLS

[Matthias A. <mat...@gm...>]

Am 28.07.2010 19:04, schrieb R P Herrold:
> On Wed, 28 Jul 2010, Matthias Andree wrote:
> 
>> Don't. fetchmail in most modes suppresses writing core files 
>> with setrlimit(), to avoid passwords hitting the disk 
>> outside the .netrc or .fetchmailrc files.
> 
> goodness -- I recall reporting that matter to ESR and getting 
> the patch, probably a decade ago

Hi there,

just for our edification, with grep & git gui blame, it took me only a couple of
seconds to figure this out:

It's been 12 years minus 5 days that ESR committed this code to fetchmail.c:

>     /*
>      * Before getting passwords, disable core dumps unless -v -d0 mode is on.
>      * Core dumps could otherwise contain passwords to be scavenged by a
>      * cracker.
>      */
>     if (outlevel < O_VERBOSE || run.poll_interval > 0)
>     {
>         struct rlimit corelimit;
>         corelimit.rlim_cur = 0;
>         corelimit.rlim_max = 0;
>         setrlimit(RLIMIT_CORE, &corelimit);
>     }

(I'm pasting from a future development branch that has lost the #ifdef
HAVE_SETRLIMIT guards.)

Now, after two repository conversions (CVS->SVN and SVN->Git),
we can still figure when he did that:

> commit 1587e4153763fab493acf2deee9028e24e1da57f
> Author: Eric S. Raymond <es...@th...>
> Date:   Sun Aug 2 16:30:25 1998 +0000
> 
>     Improved security.
> 
>     svn path=/trunk/; revision=2032

>From OLDNEWS:

> fetchmail-4.5.5 (Mon Aug  3 16:08:14 EDT 1998), 15286 lines:
...
> * Added setrlimit call to inhibit core dumps unless debugging is on.
...

This also states how Thomas can enable core dumps: always run with -vd0 (which
spams the logs or cron output quite a bit). I had - long ago - read there was
such code, but lacked the time to dig deeper earlier today.

Now that I got this pointer, here we go... :)

Best regards

-- 
Matthias Andree

[fetchmail-devel] timeout not working for TLS

[R P H. <he...@ow...>]

On Wed, 28 Jul 2010, Matthias Andree wrote:

> Don't. fetchmail in most modes suppresses writing core files 
> with setrlimit(), to avoid passwords hitting the disk 
> outside the .netrc or .fetchmailrc files.

goodness -- I recall reporting that matter to ESR and getting 
the patch, probably a decade ago

-- Russ herrold

Re: [fetchmail-devel] timeout not working for TLS

[Thomas J. <tho...@in...>]

On Wednesday, 28. July 2010 15:31:19 Matthias Andree wrote:
> > This morning I had a better idea: Enable core dumps and kill the task
> > via "kill -11" if it hangs again. Then we'll know for sure where it's
> > stuck.
> 
> Don't. fetchmail in most modes suppresses writing core files with
> setrlimit(), to avoid passwords hitting the disk outside the .netrc or
> .fetchmailrc files.

I just patched that out :o) Before pushing the "new" version to the 
productive system, I've verified the coredump gives a useful backtrace.

> Instead, if fetchmail hangs, just attach gdb with "gdb
> /usr/bin/fetchmail-unstripped 12345", where fetchmail-unstripped is an
> executable compiled with -g or better -ggdb3 option, and installed before
> the run, without running strip and without adding -s to the install
> command line.

No gdb on the productive box. I've just installed the unstripped version 
(took me 30min to discover the -s switch in the compile statement. Whoops).
Now we need to wait a month or so... thanks for your suggestions!

Cheers,
Thomas

Re: [fetchmail-devel] timeout not working for TLS

[Matthias A. <mat...@gm...>]

Am 28.07.2010 11:31, schrieb Thomas Jarosch:
> On Tuesday, 27. July 2010 16:56:52 Matthias Andree wrote:
>> > I recently had two cases of hanging fetchmail processes
>> > during TLS negotiation with two different servers.
>> 
>> Might be a missing call to set_timeout() around the POP3 STLS/getauth()
>> related methods.  Try wrapped SSL ("ssl" option) as a workaround, it
>> might help, and if so, help debugging.
> 
> The issue only appears every other month or so. I also thought about adding 
> debug output to set_timeout() so we could trace the last set timeout,
> though this will only give a close approximation where that problem is.
> (or none at all if the code mostly uses the same timeout values)
> 
> This morning I had a better idea: Enable core dumps and kill the task via 
> "kill -11" if it hangs again. Then we'll know for sure where it's stuck.

Don't. fetchmail in most modes suppresses writing core files with setrlimit(),
to avoid passwords hitting the disk outside the .netrc or .fetchmailrc files.

Instead, if fetchmail hangs, just attach gdb with "gdb
/usr/bin/fetchmail-unstripped 12345", where fetchmail-unstripped is an
executable compiled with -g or better -ggdb3 option, and installed before the
run, without running strip and without adding -s to the install command line.

Then, once gdb has attached to the hanging process, "backtrace full" will
provide the necessary debug output. Inside gdb you can issue "signal 2" (SIGINT)
and "continue" to have fetchmail terminate the run in an orderly manner.

> I prefer not to add the workaround for now
> as I'd like to trace the real issue.

:)

-- 
Matthias Andree

Re: [fetchmail-devel] timeout not working for TLS

[Thomas J. <tho...@in...>]

On Tuesday, 27. July 2010 16:56:52 Matthias Andree wrote:
> > I recently had two cases of hanging fetchmail processes
> > during TLS negotiation with two different servers.
> 
> Might be a missing call to set_timeout() around the POP3 STLS/getauth()
> related methods.  Try wrapped SSL ("ssl" option) as a workaround, it
> might help, and if so, help debugging.

The issue only appears every other month or so. I also thought about adding 
debug output to set_timeout() so we could trace the last set timeout,
though this will only give a close approximation where that problem is.
(or none at all if the code mostly uses the same timeout values)

This morning I had a better idea: Enable core dumps and kill the task via 
"kill -11" if it hangs again. Then we'll know for sure where it's stuck.

I prefer not to add the workaround for now
as I'd like to trace the real issue.

Thanks,
Thomas

Re: [fetchmail-devel] timeout not working for TLS

[Matthias A. <mat...@gm...>]

Am 27.07.2010 15:21, schrieb Thomas Jarosch:

> I recently had two cases of hanging fetchmail processes
> during TLS negotiation with two different servers.

Might be a missing call to set_timeout() around the POP3 STLS/getauth() related
methods.  Try wrapped SSL ("ssl" option) as a workaround, it might help, and if
so, help debugging.

-- 
Matthias Andree

[fetchmail-devel] timeout not working for TLS

[Thomas J. <tho...@in...>]

Hello,

I recently had two cases of hanging fetchmail processes
during TLS negotiation with two different servers.

The config looks like this:
-----------------
defaults proto pop3
set logfile /var/log/fetchmail
set postmaster "postmaster"
set showdots
poll "server" timeout 90 user "USER" password "PASS" to "something" fetchall
...
poll "server" timeout 90 user "USERx" password "PASS" to "otherX" fetchall
-----------------


I was able to grab a log of a stalled fetchmail process:
----------------------------------------------
fetchmail: 6.3.17 querying pop.1und1.com (protocol POP3) at Sat, 24 Jul 2010 
04:57:01 +0200 (CEST): poll started
fetchmail: Trying to connect to 212.227.15.161/110...connected.
fetchmail: POP3< +OK POP server ready H mimap4
fetchmail: POP3> CAPA
fetchmail: POP3< +OK Capability list follows
fetchmail: POP3< TOP
fetchmail: POP3< USER
fetchmail: POP3< UIDL
fetchmail: POP3< STLS
fetchmail: POP3< IMPLEMENTATION trinity
fetchmail: POP3< .
fetchmail: POP3> STLS
fetchmail: POP3< +OK Begin TLS negotiation
---------- fetchmail hangs ------------------

Any idea why the timeout of 90 seconds is not triggered?

Maybe it's stuck in the openssl code and the timeout is not working?
The openssl version in use is 0.9.8o.

Best regards,
Thomas Jarosch

[fetchmail-devel] [Bug #17295] Ferchmail provides insufficient log information

[<ad...@be...>]

Bug #17295, was updated on 2010-Jun-27 01:42
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Open
Resolution: None
Bug Group: None
Priority: 2
Submitted by: gtozzi
Assigned to : none
Summary: Ferchmail provides insufficient log information

Details: Hello,

First of all, I wish to thank you all for sharing with us this very useful software.

I'm running fetchmail 6.3.17+GSS+NTLM+SDPS+SSL+NLS+KRB5 in daemon mode to check a bunch of accounts.
The log goes to syslog. Everything works fine but the log files are too inaccurate IMHO. Every line should ALWAYS include the name of the interested account.

Example. Now I have:
Jun 27 01:33:32 localhost fetchmail[18328]: Attenzione: si continua malgrado la connessione non sia sicura (si raccomanda l'utilizzo di --sslcertck)
Jun 27 01:33:43 localhost fetchmail[18328]: Errore di verifica del certificato del server: self signed certificate
Jun 27 01:33:43 localhost fetchmail[18328]: Questo significa che la radice che firma il certificato (emesso per /C=US/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=dime148.dizinc.com/emailAddress=ss...@di...) non è nella posizione dei certificati CA fidati o che c_rehash deve essere eseguito sulla directory del certificato. Per maggiori dettagli consultare le pagine man a proposito di --sslcertpath e di --sslcertfile.

This is not su useful, because I can't easily establish which account is triggering this error. I was expection instead:
Jun 27 01:33:32 localhost fetchmail[18328]: USERNAME@SERVERNAME: Attenzione: si continua malgrado la connessione non sia sicura (si raccomanda l'utilizzo di --sslcertck)
Jun 27 01:33:43 localhost fetchmail[18328]: USERNAME@SERVERNAME: Errore di verifica del certificato del server: self signed certificate
Jun 27 01:33:43 localhost fetchmail[18328]: USERNAME@SERVERNAME: Questo significa che la radice che firma il certificato (emesso per /C=US/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=dime148.dizinc.com/emailAddress=ss...@di...) non è nella posizione dei certificati CA fidati o che c_rehash deve essere eseguito sulla directory del certificato. Per maggiori dettagli consultare le pagine man a proposito di --sslcertpath e di --sslcertfile.


Follow-Ups:

Date: 2010-Jun-27 14:55
By: m-a

Comment:
Thanks. This is actually a feature request - it will be a while before this can be implemented.
-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=17295&group_id=1824

[fetchmail-devel] [Bug #17295] Ferchmail provides insufficient log information

[<ad...@be...>]

Bug #17295, was updated on 2010-Jun-26 15:42
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Open
Resolution: None
Bug Group: None
Priority: 5
Submitted by: gtozzi
Assigned to : none
Summary: Ferchmail provides insufficient log information

Details: Hello,

First of all, I wish to thank you all for sharing with us this very useful software.

I'm running fetchmail 6.3.17+GSS+NTLM+SDPS+SSL+NLS+KRB5 in daemon mode to check a bunch of accounts.
The log goes to syslog. Everything works fine but the log files are too inaccurate IMHO. Every line should ALWAYS include the name of the interested account.

Example. Now I have:
Jun 27 01:33:32 localhost fetchmail[18328]: Attenzione: si continua malgrado la connessione non sia sicura (si raccomanda l'utilizzo di --sslcertck)
Jun 27 01:33:43 localhost fetchmail[18328]: Errore di verifica del certificato del server: self signed certificate
Jun 27 01:33:43 localhost fetchmail[18328]: Questo significa che la radice che firma il certificato (emesso per /C=US/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=dime148.dizinc.com/emailAddress=ss...@di...) non è nella posizione dei certificati CA fidati o che c_rehash deve essere eseguito sulla directory del certificato. Per maggiori dettagli consultare le pagine man a proposito di --sslcertpath e di --sslcertfile.

This is not su useful, because I can't easily establish which account is triggering this error. I was expection instead:
Jun 27 01:33:32 localhost fetchmail[18328]: USERNAME@SERVERNAME: Attenzione: si continua malgrado la connessione non sia sicura (si raccomanda l'utilizzo di --sslcertck)
Jun 27 01:33:43 localhost fetchmail[18328]: USERNAME@SERVERNAME: Errore di verifica del certificato del server: self signed certificate
Jun 27 01:33:43 localhost fetchmail[18328]: USERNAME@SERVERNAME: Questo significa che la radice che firma il certificato (emesso per /C=US/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=dime148.dizinc.com/emailAddress=ss...@di...) non è nella posizione dei certificati CA fidati o che c_rehash deve essere eseguito sulla directory del certificato. Per maggiori dettagli consultare le pagine man a proposito di --sslcertpath e di --sslcertfile.


For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=17295&group_id=1824

Re: [fetchmail-devel] [fetchmail-users] poblems with pop3 and expunge

[Matthias A. <mat...@gm...>]

Am 25.06.2010, 02:58 Uhr, schrieb Lorenzo Milesi:

>> The configuration is contradictory. "keep" prevents deletions
>> altogether -
>> and then there will be nothing to expunge.
>
> I don't think so.
> I run with keep in daemon mode, to keep mail on the server. Once in a  
> while I execute with -F, to free up server.
>
> In fact, removing the "keep" and running, doesn't seem to expunge  
> anything.

--flush isn't the same as omitting --keep, and these are not opposites.

Unfortunately, this is not too obvious from the documentation, and it  
might be (I'll have to dig deeper in the source code to figure, which  
won't happen today) that --expunge isn't taken into account with --flush.

If someone else wants to take a look and possibly send a patch, feel free.

Relevant code is in driver.c's fetch_messages() near the check for  
ctl->flush, code available at <http://gitorious.org/fetchmail/fetchmail/>,  
on the master branch.

-- 
Matthias Andree

[fetchmail-devel] [Bug #17272] Fetchmail does not fetch some Web.DE mails

[<ad...@be...>]

Bug #17272, was updated on 2010-Jun-17 14:17
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Closed
Resolution: Fixed
Bug Group: None
Priority: 3
Submitted by: bjoernv
Assigned to : m-a
Summary: Fetchmail does not fetch some Web.DE mails

Details: Fetchmail is not able to fetch some automatically generated Web.DE mails.

Especially the so called "POP3 Reports" could not be fetched. These mails where generated by the German Freemail provider Web.DE if the user has some suspect mails in his folders which where not delivered by POP3. 

After some debugging I found, that these mails contain a header with some small problems. First the line "This is a multi-part message in MIME format." is part of the header. Second the header and the body is not divided by a single new line, but by a Space character followed by a new line. 

Fetchmails output and a Sample mail is appended. 

The problem was tested with Fetchmail 6.3.11 from openSUSE 11.2 and with the current GIT version.

The mails from Web.DE are badly formatted. Anyway I think, that Fetchmail should be more tolerant with these mails. Normal MUAs like Seamonkey and Thunderbird can deal with the Web.DE mails. 

Follow-Ups:

Date: 2010-Jun-17 22:29
By: m-a

Comment:
was fixed in 6.3.15 - will consider changing the error message
-------------------------------------------------------

Date: 2010-Jun-17 22:21
By: bjoernv

Comment:
Using "--bad-header accept" helps to solve the issue with Web.DE. Fetchmail corrects the incorrect header with this option (Fetchmail inserts a newline before "This is a multi-part message in MIME format.").

So the problem is solved for me. But may be, it's a good idea to add a hint after the error messages like

"reading message XX...@po...:3 of 8 (10352 octets) fetchmail: incorrect header line found while scanning headers"

Something link ("(try --bad-header accept)") would be helpful.

-------------------------------------------------------

Date: 2010-Jun-17 21:01
By: m-a

Comment:
Please 
- complain to web.de to have them generate their messages in proper format
- concurrently retest with fetchmail 6.3.17 and use "--bad-header accept" as command line option and follow up to report if this works.
-------------------------------------------------------

Date: 2010-Jun-17 14:51
By: bjoernv

Comment:
Couldn't I attach files here?

The sample Web.DE "POP3 Report" can be found here. If your browser shows extra newlines please download the file and open it with an good editor. The extra newlines are could be some "CR" characters:

http://user.cs.tu-berlin.de/~bjoern/fetchmail/fetchmail-web.de-message.log.txt

The sample Fetchmail log (from Fetchmail Git-Master on 2010-06-17) is here:

http://user.cs.tu-berlin.de/~bjoern/fetchmail/fetchmail-git-master-web.de.log.txt
-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=17272&group_id=1824