fetchmail-devel — Development related discussion

Re: [fetchmail-devel] fetchmail 6.3.20-pre1 release candidate should fix STARTTLS hangs

[Thomas J. <tho...@in...>]

Hello Matthias,

On Monday, 23. May 2011 21:00:27 Matthias Andree wrote:
> DOWNLOAD this beta software from:
> <http://home.pages.de/~mandree/fetchmail/>

Small side note: I had trouble downloading the .tar.bz2 version,
the tar.xz version downloaded just fine.

> # SECURITY FIXES
> * Fetchmail's socket timeout handling was incomplete.  Network outages in
> the wrong phase of a communication, combined with unlucky operating
> systems and their defaults, could cause fetchmail to hang for extended
> amounts of time. Freezes for beyond a week were reported by Thomas
> Jarosch. Fetchmail sets UNIX- and Internet-domain socket send and
> receive timeouts now. This fixes a hang during STARTTLS negotiation
> reported by Thomas Jarosch.

The timeout works fine, I've tested every step of the POP3 protocol 
communication until we triggered the bug the last time.

I think I've seen a kind of unrelated bug: If you let the TLS negotation 
time out on the "server side", fetchmail will proceed to send the user name 
before shutting down: "USER xyz". Bug in the state machine?

> # CHANGES
> * fetchmail now supports an environment variable to suppress marking

Out of curiosity, why is this an environment variable
instead of a configuration option?

Cheers,
Thomas

Re: [fetchmail-devel] timeout not working for TLS

[Matthias A. <mat...@gm...>]

Am 28.04.2011 17:28, schrieb Matthias Andree:

> Sorry for the incomplete fix in 6.3.18, and thanks for the report.  Now
> I need to make sure I catch all similar bugs before releasing the next
> version, and I need to update and re-issue the corresponding CVE (or a
> new one, need to check with the gurus) and security announcement.
> 
> The fix needs a thorough analysis of the code.  Note that I've already
> queued patches to ditch SSLv2 support, I need to reconsider that, or
> making that an option so that distributors can go ahead and update their
> 6.3.ancient version to 6.3.20 without major incompatibilities.
> 
> I'll get back to this.
> 
> Thanks again -- I do appreciate bug reports that are easy to reproduce :-)

Thomas,

6.3.20-pre1 should fix that - please test and report back (see the
separate announcement).

It took a while since I chose to set SO_SNDTIMEO/SO_RCVTIMEO, a BSD
socket-level timeout feature, and also SO_KEEPALIVE, to detect crashed
TCP connections (although that can take 2 hours and more than 11 minutes
on some operating systems to trigger - instead you'll usually get a
socket error).

I need particular test reports of --idle mode.

Best regards,
Matthias

[fetchmail-devel] fetchmail 6.3.20-pre1 release candidate should fix STARTTLS hangs

[Matthias A. <mat...@gm...>]

Greetings,

Sunil has worked on handling large IMAP mailboxes.
I have been working on the STARTTLS hang problem reported by Thomas
Jarosch, which should be fixed now, along with assorted minor
protocol nits that were picked.  I am not sure if I want/need a CVE for
a denial of service that is OS dependent.  Opinions solicited.

Please test fetchmail 6.3.20-pre1 on your operating system.
To do that, please:

1. download and unpack the fetchmail tarball (URLs below)
2. cd to to the unpacked directory
3. ./configure and install as usual
4. run fetchmail with these additional options:
   --auth any -vvvd0 --nodetach --nosyslog
5. report success or failure to the list or me personally.

PLEASE HELP: If you can offer access to test servers that I can send a
short test mail to and then log into to retrieve that test message -
particularly Exchange 2007 or Exchange 2010 is desired, but others
besides Cyrus IMAP and Dovecot are also welcome - please let me know.

PLEASE HELP: fetchmail needs translators for the program strings. Some
languages (such as those shown below) are in quite good shape, but
others are lacking a bit.  Translation information can be found at
<http://translationproject.org/domain/fetchmail.html>

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

DOWNLOAD this beta software from:
<http://home.pages.de/~mandree/fetchmail/>

The repository can be browsed at and cloned from:
<http://gitorious.org/fetchmail/fetchmail> - the branch is "legacy_63".

Git (the software used to keep the fetchmail source code version
controlled) information is at: <http://git-scm.com/>

CHANGES since the previous formal release of fetchmail listed below.
Unless otherwise noted, the changes were made by Matthias Andree:

# SECURITY FIXES
* Fetchmail's socket timeout handling was incomplete.  Network outages in the
  wrong phase of a communication, combined with unlucky operating systems and
  their defaults, could cause fetchmail to hang for extended amounts of time.
  Freezes for beyond a week were reported by Thomas Jarosch. Fetchmail sets
  UNIX- and Internet-domain socket send and receive timeouts now.
  This fixes a hang during STARTTLS negotiation reported by Thomas Jarosch.

# CHANGES
* fetchmail now always uses its own MD5 implementation.  The library and header
  variants are too diverse, and we've been bitten before -- and configure
  complains noisily on Cyrus-SASL's RFC1321 md5.h.
* fetchmail now supports an environment variable to suppress marking deleted
  messages as seen at the same time, FETCHMAIL_IMAP_DELETED_REMAINS_UNSEEN.
  See the manual page for details. Requested by Jonathan Buschmann.
* fetchmail sets Internet domain sockets to "keepalive" mode now. Note that
  there is no portable way to configure actual timeouts for this mode, and some
  systems only support a system-wide timeout setting.

# BUG FIXES
* Call strlen() only once when removing CRLF from a line. (Sunil Shetye)
* Do not search for UNSEEN messages in ranges. Usually, there are very few new
  messages and most of the range searches result in nothing. Instead, split the
  long response to make the IMAP driver think that there are multiple lines of
  response. (Sunil Shetye)
* Do not print "skipping message" for old messages even in verbose mode. If
  there are too many old messages, the logs just get filled without any real
  activity. (Sunil Shetye) (suggested by Yunfan Jiang)

# TRANSLATION UPDATES
  [de]    German (Matthias Andree)
  [ja]    Japanese (Takeshi Hamasaki)

-- 
Matthias Andree

[fetchmail-devel] Gitorious branch reshuffling

[Matthias A. <mat...@gm...>]

Greetings,

I am in the process of changing the branches in the Git repository
around.  I am planning to do 6.3.20 as a last 6.3.X release, to fix
timeouts and collect other accumulated fixes.

6.4.0 will drop some cruft, require - conservatively - a bit more modern
systems (though those compliant to 2001+ standards should be good),
accelerates POP3 and should fix socket timeout handling a bit more
aggressively.

7.0.0 is future work, has started, but requires major overhauls for
maintainability and more radical changes. Likely to diverge from 6.X
to the point that code merges become impossible.


This is the current shape:

-------------------------------------------------------------------------
Git name of     Future     Future
BRANCH		Releases   Directions
-------------------------------------------------------------------------
legacy_63	6.3.X	   to be frozen after 6.3.20, except security
(new)                      or grave bugs, last to support ancient OSes
-------------------------------------------------------------------------
master          6.4.X      feature remove branch, drops SSLv2, IMAP2,
                           POP2, RPOP, and more, speeds up POP3,
                           interim branch for fixes with less ballast,
                           likely the last in C (without C++), requires
                           newer operating system and C compiler
                           (SUSv3 and C89 should do for now, may
                           require C99 soonish).

                           I have back-merged anything not C++-related
                           from the "next" branch.
-------------------------------------------------------------------------
common-6x       none       branch to collect changes shared between
(new)                      legacy_63 and master, "common ancestor"
-------------------------------------------------------------------------
next            7.0.X      development/major overhaul branch,
                           converted to C++, to be rebased onto master
                           soon (if possible, haven't tried)
-------------------------------------------------------------------------

I'll send another notice once the "next" branch has been rebased onto
master, or if I give up on that plan.

Note that those who have pulled from Git in the past few hours may need
to discard the past few changes on upstream/master and re-pull.

Also note that you may need to "make distclean" and re-configure. To
obtain the current ./configure options you'd used, run "./config.status
--version" and look at the first few lines of output.

-- 
Matthias Andree

Re: [fetchmail-devel] [PATCH] how to make fetchmail not to check the mail with seen flag of my gmail?

[Sunil S. <sun...@ro...>]

Hi,
----- Original Message -----

> I think the real issue is not related to IMAP at all. What Yunfan Jiang may be 
> trying to say is that when fetchmail -v runs without syslog, there are too many 
> skipping messages without any real protocol level activity. Note that this is 
> true for both IMAP and POP3. This is also true for fetchmail -v -v.
> 
> Say, there are 20,000 mails with just one new mail. When a user runs 
> "fetchmail -v --nosyslog" or "fetchmail -v -v", the user 
> sees some protocol level activity at the start. This could be "SEARCH 
> UNSEEN" for IMAP or "UIDL" for POP3. From that activity, it 
> becomes clear that there is just one new mail.
> 
> What follows this activity is a series of "skipping message n" lines 
> from 1 to 19,999. These lines do not make any sense if all the user is trying to 
> do is to look at the protocol level logs for the 20,000th mail. In fact, if the 
> output is to the terminal, these 19,999 lines without any real IMAP/POP3 
> activity can lead to a slow scrolling. There is also the risk of hitting the 
> timeout at the mailserver since no real activity has occurred on the mailserver 
> end while those 19,999 lines are getting printed.
> 
> For IMAP, the logs could look like:
> 
> IMAP> A0010 SEARCH UNSEEN
> IMAP< * SEARCH 20000
> IMAP< A0010 OK SEARCH completed
> skipping message ...@...:1 (not flushed)
> skipping message ...@...:2 (not flushed)
> skipping message ...@...:3 (not flushed)
> ...
> skipping message ...@...:19997 (not flushed)
> skipping message ...@...:19998 (not flushed)
> skipping message ...@...:19999 (not flushed)
> IMAP> A0011 FETCH 20000 RFC822.SIZE
> 
> The real solution would be to remove all the "skipping message n (not 
> flushed)" messages if they are not related to any protocol activity as they 
> do not add any real value. The patch attached should do the trick. Please check 
> and evaluate. Note that I have not tested for POP3 with UIDL as yet.


I am sending the patch to the devel list to check if the patch is getting removed here.

Sorry for the unnecessary traffic.

-- 
Sunil Shetye.

Re: [fetchmail-devel] [PATCH] fetchmail fix coordination and future planning

[Matthias A. <mat...@gm...>]

Am 04.05.2011 12:50, schrieb Sunil Shetye:
> ----- Original Message -----
> 
>> And it went unnoticed possibly because you use different compiler flags
>> (I routinely compile with -pedantic -Wall and often also with -Wextra or
>> -W, and I routinely also try Intel C++ and Clang, and I take care to
>> have at least -O1 or usually -O2 set -- my current computer is fast
>> enough and compiles fetchmail (after configure) in < 2.5 s real time.)
> 
> Is there a way of enabling those CFLAGS flags as well as the configure flags by default? Every time I switch machines and checkout fetchmail again (which is rare), I seem to miss out setting the same set of flags again.

Hm. These are compiler-specific, so we'd have to test explicitly.

> I have enabled the warnings now. I have also added -Wformat=2 and got some warnings.
> 
> Attached is a minor patch to fix those warnings.

Good catch, although I have the nasty feeling I've fixed similar bugs in
other .c files before. I wonder how these slipped past my attention...

Re: [fetchmail-devel] [PATCH] fetchmail fix coordination and future planning

[Sunil S. <sun...@ro...>]

----- Original Message -----

> And it went unnoticed possibly because you use different compiler flags
> (I routinely compile with -pedantic -Wall and often also with -Wextra or
> -W, and I routinely also try Intel C++ and Clang, and I take care to
> have at least -O1 or usually -O2 set -- my current computer is fast
> enough and compiles fetchmail (after configure) in < 2.5 s real time.)

Is there a way of enabling those CFLAGS flags as well as the configure flags by default? Every time I switch machines and checkout fetchmail again (which is rare), I seem to miss out setting the same set of flags again.

I have enabled the warnings now. I have also added -Wformat=2 and got some warnings.

Attached is a minor patch to fix those warnings.

-- 
Sunil Shetye.

Re: [fetchmail-devel] fetchmail fix coordination and future planning

[Matthias A. <mat...@gm...>]

Am 04.05.2011 08:46, schrieb Sunil Shetye:

> Okay. Note that the prototype was just copied from gen_recv() and a parameter was added later.

Oops :) I've fixed the transact.c documentation for Doxygen, and fixed
some unsafe macro expansions (missing parentheses) along the way.

Re: [fetchmail-devel] fetchmail fix coordination and future planning

[Matthias A. <mat...@gm...>]

On 04.05.2011 08:46, Sunil Shetye replied:

>>     Fixup: remove unused variables.
> 
> How did I miss that?

Hi Sunil,

As fetchmail 6.3.X is C89-style, variable declarations are far away from
their use, so missing "first/last" was easy. Add to that the humongous
text length of some code parts...

And it went unnoticed possibly because you use different compiler flags
(I routinely compile with -pedantic -Wall and often also with -Wextra or
-W, and I routinely also try Intel C++ and Clang, and I take care to
have at least -O1 or usually -O2 set -- my current computer is fast
enough and compiles fetchmail (after configure) in < 2.5 s real time.)

> Thanks for the review.

Thanks for the counter-review.

Best regards
Matthias

Re: [fetchmail-devel] fetchmail fix coordination and future planning

[Sunil S. <sun...@ro...>]

----- Original Message -----

> Observations:
> 
> - the patch makes an undocumented assumption that the buffer for a
> second call to gen_recv_split is always large enough to copy the
> remainder. If it isn't, you lose data.  Doesn't happen in fetchmail now,
> but is a maintenance concern, so I'll add an assert() for now.
> 
> - you don't guard the prefix length - I've added that.
> 
> - you don't check if the data-to-be-cached fits into the rs structure -
> I've added that.

Good. All required.

> - /** foo */ isn't a typo, but a marker for Doxygen so it actually looks
> at the comment for documentation, and extracts it. Also note the
> ordering of these comments versus the comma; alternatively you can write
> /**< foo */ to document the PREVIOUS argument in a prototype.

Okay. Note that the prototype was just copied from gen_recv() and a parameter was added later.

>     Fixup: match prefix caseblind, add some guards, streamline phase
> handling.

Required.

>     Add a few asserts to catch abuse, and use strlcpy/strlcat rather than
>     snprintf because some implementations of the latter are unsuitable for
>     detecting buffer exhaustion.

Required.

>     Fixup: remove unused variables.

How did I miss that?

Thanks for the review.

-- 
Sunil Shetye.

Re: [fetchmail-devel] fetchmail fix coordination and future planning

[Matthias A. <mat...@gm...>]

Am 04.05.2011 01:14, schrieb Matthias Andree:

> I will take your patch in spite of my concerns, and test it.

Observations:

- the patch makes an undocumented assumption that the buffer for a
second call to gen_recv_split is always large enough to copy the
remainder. If it isn't, you lose data.  Doesn't happen in fetchmail now,
but is a maintenance concern, so I'll add an assert() for now.

- you don't guard the prefix length - I've added that.

- you don't check if the data-to-be-cached fits into the rs structure -
I've added that.

- /** foo */ isn't a typo, but a marker for Doxygen so it actually looks
at the comment for documentation, and extracts it. Also note the
ordering of these comments versus the comma; alternatively you can write
/**< foo */ to document the PREVIOUS argument in a prototype.

My changes have been pushed out to the Git master branch, please check
my two Fixup commits:

commit 646f36e1c1369fd65c0d641cae48fa4425613462
Author: Matthias Andree <mat...@gm...>
Date:   Wed May 4 02:02:30 2011 +0200

    Fixup: match prefix caseblind, add some guards, streamline phase
handling.

    Add a few asserts to catch abuse, and use strlcpy/strlcat rather than
    snprintf because some implementations of the latter are unsuitable for
    detecting buffer exhaustion.

M       transact.c

commit d7d43f53e1da9d5c57961ea26fefa609de1e30e7
Author: Matthias Andree <mat...@gm...>
Date:   Wed May 4 01:58:46 2011 +0200

    Fixup: remove unused variables.

M       imap.c

Re: [fetchmail-devel] fetchmail fix coordination and future planning

[Matthias A. <mat...@gm...>]

Am 03.05.2011 20:25, schrieb Sunil Shetye:
> Hi Matthias,
> 
> 
> 
> ----- Original Message -----
>> As so the second (with _split and caching), I wonder if it's the right
>> approach. It looks a bit as though we're curing the symptoms instead of
>> the cause.  The cause is a buffer too small to handle the response, and
>> the cure would be to make the buffer dynamic and possibly resize it if
>> we don't get a \r?\n in it.
>>
>>
>> A dynamic buffer could also save us the strcpy(argbuf, buf) copy which,
>> IMO, is quite wasteful.
> 
> I had given a thought for implementing dynamic buffers. However, some issues came to my mind:
> 
> - What if the buffer required is very long? For example, an extreme response like:
> 
> IMAP< * SEARCH 1 2 3 4 5 6 ... 999995 999996 999997 99998 99999 100000
> 
> will require a dynamic buffer of size 575k. Is it okay to allocate such a huge buffer?

Possibly yes, although some limit would be in order.

Do we really need to optimize 6.3.X with obtrusive changes?  This will
make merging between master and next harder.

> - Who will free the buffer? Obviously, it will have to be statically allocated in order to free it later.
> 
> - How long will it remain? Is it safe to free it immediately on the next call?

> In short, without a proper memory manager, it may not be safe to just allocate a dynamic buffer of such a huge size.

I'll sidestep these for now.

> Note that the strcpy(argbuf, buf) can still be avoided by directly passing argbuf (if available) to gen_recv().

OK. It's a separate microoptimization anyways, not worth discussing.
Someone will do it ;-)

> Also, note that that patch is a bug fix. The current method range search for UNSEEN mails is too slow for huge mailboxes. So, a fix for this issue is required for 6.3.X.

It may be painful, but if it works, it's not a bug unless the
documentation promises otherwise (which it does not).

For a 25,000 message mailbox with 1% unseen, the amount of SEARCH
transactions seems to be reasonably low.  I think POP3 + uidl + keep in
that situation would burn something between 3 and 30 seconds CPU time on
a Phenom II 2.5 GHz CPU w/ GCC 4.5 compiled code while chewing on UIDLs.
The fix was contributed by Rainer Weikusat and is in the next branch and
in 7.0.0-alpha2.

>> What I propose is to have three branches, details below the list:
>>
>> - one for 6.3.X (new name to be devised), for a final release and
>> possibly later critical/security fixes
>>
>> - one for a new 6.4.X or 7.X.X branch (master) where such fixes as the
>> IMAP search ranges can be made
> 
> I think, it is necessary to fix the range search issue in 6.3.X itself as fetchmail is now unusable with IMAP when the mailbox is huge and keep is on.

Try POP3 UIDL on a mailbox with the same message (86000) count and let
me know if you still think IMAP is "unusable".  Hint: check what Rainer
and I have been benchmarking when he contributed the PATRICIA
implementation.  I've avoided it in 6.3.X because I have portability
concerns.

I will take your patch in spite of my concerns, and test it.

I suggest that you check the manual pages of strlcpy and strlcat, these
are alternatives to snprintf(buf, siz, "%s%s", s1, s2).  glibc is one of
the major distros lacking it, but we ship replacements.  Also note that
<strings.h> has strcasecmp and, more usefully, strncasecmp, and we have
our replacements for OSes lacking them, too.

Re: [fetchmail-devel] fetchmail fix coordination and future planning (was: [PATCH] imap: introduce imapsearchmax parameter)

[Sunil S. <sun...@ro...>]

Hi Matthias,



----- Original Message -----
> As so the second (with _split and caching), I wonder if it's the right
> approach. It looks a bit as though we're curing the symptoms instead of
> the cause.  The cause is a buffer too small to handle the response, and
> the cure would be to make the buffer dynamic and possibly resize it if
> we don't get a \r?\n in it.
> 
> 
> A dynamic buffer could also save us the strcpy(argbuf, buf) copy which,
> IMO, is quite wasteful.

I had given a thought for implementing dynamic buffers. However, some issues came to my mind:

- What if the buffer required is very long? For example, an extreme response like:

IMAP< * SEARCH 1 2 3 4 5 6 ... 999995 999996 999997 99998 99999 100000

will require a dynamic buffer of size 575k. Is it okay to allocate such a huge buffer?

- Who will free the buffer? Obviously, it will have to be statically allocated in order to free it later.

- How long will it remain? Is it safe to free it immediately on the next call?

In short, without a proper memory manager, it may not be safe to just allocate a dynamic buffer of such a huge size.


Note that the strcpy(argbuf, buf) can still be avoided by directly passing argbuf (if available) to gen_recv().


Also, note that that patch is a bug fix. The current method range search for UNSEEN mails is too slow for huge mailboxes. So, a fix for this issue is required for 6.3.X.


> What I propose is to have three branches, details below the list:
> 
> - one for 6.3.X (new name to be devised), for a final release and
> possibly later critical/security fixes
> 
> - one for a new 6.4.X or 7.X.X branch (master) where such fixes as the
> IMAP search ranges can be made

I think, it is necessary to fix the range search issue in 6.3.X itself as fetchmail is now unusable with IMAP when the mailbox is huge and keep is on.

-- 
Sunil Shetye.

[fetchmail-devel] fetchmail fix coordination and future planning (was: [PATCH] imap: introduce imapsearchmax parameter)

[Matthias A. <mat...@gm...>]

Am 03.05.2011 06:05, schrieb Sunil Shetye:
> Hi,
> 
> 
> ----- Original Message -----
>> So, at the IMAP protocol level, we have run out of options.
>>
>> I will check if the fetchmail parsing can be improved directly in 
>> fetchmail itself.
> 
> Attached are the patches which should solve this issue.
> 
> The first patch has nothing to do with this issue. It only removes excess calls to strlen().
> 
> The second patch works by sending a "SEARCH UNSEEN" command and reading the response in batches (when required).
> 
> Andrea, please test the patches for your mailbox.
> 
> Matthias, please evaluate the patches for incorporation in fetchmail.

Hi Sunil,

the first patch is easy, accepted, thank you.


As so the second (with _split and caching), I wonder if it's the right
approach. It looks a bit as though we're curing the symptoms instead of
the cause.  The cause is a buffer too small to handle the response, and
the cure would be to make the buffer dynamic and possibly resize it if
we don't get a \r?\n in it.


A dynamic buffer could also save us the strcpy(argbuf, buf) copy which,
IMO, is quite wasteful.


What I propose is to have three branches, details below the list:

- one for 6.3.X (new name to be devised), for a final release and
possibly later critical/security fixes

- one for a new 6.4.X or 7.X.X branch (master) where such fixes as the
IMAP search ranges can be made

- one for the future 7.X.X or 8.X.X branch (next) for radical changes
such as the C++ migration and major structural changes.


Details:

1 - create a side branch from master for 6.3.X, revert the SSLv2
removal, fix STARTTLS timeouts (the code is almost completed) and clear
up the MD5 mess.  Release 6.3.20 from the side branch.


2 - master advances to 7.0.X, sporting feature removals from the "next"
branch (7.0.0-alpha2) such as IMAP2, POP2, SSLv2, support for pre-SUSv3
systems and other cleanups, but not the C++ migration.  Revamp the
socket handling, using dynamic buffers, use Rainer's patch to speed up
POP3 UID handling with PATRICIA data structures (this gives us the edge
over getmail ;-))

  - The SSL handling is not done properly, but needs to be.
Look at the _ssl_ctx and all that stuff, we're wasting humongous amounts
of static memory (FD_SETSIZE is 1024 on my system, and sizeof(SSL) is
512, so we waste 512 kibiB of RAM just for SSL support) just because the
socket code sticks to simple ints, rather than using a proper structure
of our own.



3 - next advances to 8.0.X including a C++ migration, with major overhaul:

  - I have started converting the code (on the "next" branch in Git) to
C++ with Standard Template Library and Boost.  Boost is, for the biggest
part, a headers-only library, so it doesn't create additional run-time
dependencies, and it is peer-reviewed by the C++ gurus and often the
basis for future standards <http://www.boost.org/>. IMAPCapa* is an
example of how compact the code is.

    C++ features such as the string type, or STL containers such as
vector, map, set, list/deque/stack; or Boost functions for intervals,
will greatly simplify the code.  As will exceptions.


  - I plan to support UID-based fetches in IMAP too, so we can fetch
from READ-ONLY folders and need no longer tamper with \Seen flags.

    In that case, UIDs are strictly monotonically increasing (unless
UIDVALIDITY has changed), so we know for certain where we need to start
the SEARCH or FETCH, too, so we avoid empty fetches. (Note Boost's
Interval type!)


  - I plan to support pipelined operation for common protocols to avoid
the roundtrip delays incurred through lock-step behaviour.  It's not
possible everywhere (for instance, when doing a binary search for POP3
UIDLs), but should help maxing out the wires or WiFi even in a single
thread once we get to downloading.


  - I want to clean up the code to an extent where we can ultimately
fetch from several accounts in parallel.


Opinions, comments, remarks, concerns, ideas are solicited.

Thanks.

Best regards,
Matthias

Re: [fetchmail-devel] [PATCH] imap: introduce imapsearchmax parameter

[Sunil S. <sun...@ro...>]

Hi,


----- Original Message -----
> So, at the IMAP protocol level, we have run out of options.
> 
> I will check if the fetchmail parsing can be improved directly in 
> fetchmail itself.

Attached are the patches which should solve this issue.

The first patch has nothing to do with this issue. It only removes excess calls to strlen().

The second patch works by sending a "SEARCH UNSEEN" command and reading the response in batches (when required).

Andrea, please test the patches for your mailbox.

Matthias, please evaluate the patches for incorporation in fetchmail.

-- 
Sunil Shetye.

Re: [fetchmail-devel] [PATCH] imap: introduce imapsearchmax parameter

[Matthias A. <mat...@gm...>]

Am 23.04.2011 01:28, schrieb Andrea Righi:
> To retrieve new messages from a IMAP folder we use the IMAP SEARCH
> command, but we limit the messages processed by the query to 1000.
> This limit is hard-coded and it cannot be changed at runtime.
> 
> This value is not the best option in some cases, especially for large
> folders, because we need to submit a query each 1000 messages.
> 
> The following patch introduces a new option (imapsearchmax) that can be
> used to change this limit at runtime.

Note that Exchange 2010 may also need a server-side fix then. If we're
not up in the 2000+ range we apparently don't trigger it:

http://support.microsoft.com/kb/2494798/en-us

Re: [fetchmail-devel] imap: introduce imapsearchmax parameter

[Sunil S. <sun...@ro...>]

--- On Fri, 29/4/11, Andrea Righi <rig...@gm...> wrote:
> > > > 1. Parse the response to the SELECT command:

...

> A0001 SELECT "INBOX"
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted
> \Seen \*)]
> * OK [UIDVALIDITY 2]
> * 518299 EXISTS
> * 0 RECENT
> * OK [UIDNEXT 652168]
> A0001 OK [READ-WRITE] INBOX selected. (Success)

Parsing will not give the unseen count information for you. Also, I 
realised later that this method will not work when 'idle' is enabled.

> > > > 2. Send a STATUS command

This may not work correctly on open folders.

> > > > 3. Send an extended SEARCH command

This is not supported by your server.

So, at the IMAP protocol level, we have run out of options.

I will check if the fetchmail parsing can be improved directly in 
fetchmail itself.

The following is a recap of the problem:

==========================================================================
fetchmail gets the unseen message list using the SEARCH command. There is 
no standard mechanism to get the count and range of unseen messages.

fetchmail currently reads the response to the SEARCH command in a buffer 
of size MSGBUFSIZE=8k. In a normal case when there are only a few new 
mails, the response fits nicely in this buffer:

IMAP> A0010 SEARCH UNSEEN
IMAP< * SEARCH 499998 499999 500000
IMAP< A0010 OK SEARCH completed

However, when there are too many unseen mails, the buffer response gets 
truncated (possibly in the middle of a number).

IMAP> A0011 SEARCH UNSEEN
IMAP< * SEARCH 490001 490002 490003 ... 499998 499999 500000
IMAP< A0011 OK SEARCH completed

IMAP_SEARCH_MAX=1k (which in reality should be a function of MSGBUFSIZE 
and count) tries to avoid the truncation by searching for unseen mail in 
batches.

IMAP> A0012 SEARCH 1:1000 UNSEEN
IMAP> A0013 SEARCH 1001:2000 UNSEEN
IMAP> A0014 SEARCH 2001:3000 UNSEEN

The problems with this approach are:

- In the most common case, there are very few unseen mails. This method of 
searching is inefficient in those cases.

- If the mailbox size is huge (say having 500k mails), fetchmail gets 
caught in an unproductive cycle.
==========================================================================

-- 
Sunil Shetye.

Re: [fetchmail-devel] [PATCH] imap: introduce imapsearchmax parameter

[Matthias A. <mat...@gm...>]

Am 23.04.2011 01:28, schrieb Andrea Righi:
> To retrieve new messages from a IMAP folder we use the IMAP SEARCH
> command, but we limit the messages processed by the query to 1000.
> This limit is hard-coded and it cannot be changed at runtime.
> 
> This value is not the best option in some cases, especially for large
> folders, because we need to submit a query each 1000 messages.
> 
> The following patch introduces a new option (imapsearchmax) that can be
> used to change this limit at runtime.

I wonder how much of that would survive the planned UID use in IMAP. :)

Re: [fetchmail-devel] imap: introduce imapsearchmax parameter

[Andrea R. <rig...@gm...>]

On Wed, Apr 27, 2011 at 10:41:04AM +0530, Sunil Shetye wrote:
> > > Could you please send your imap server CAPABILITY
> > string?
> > 
> > * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID
> > XLIST CHILDREN
> > * X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE
> 
> ...
> 
> > > 1. Parse the response to the SELECT command:
> 
> Please send from the IMAP logs the output of the SELECT command. You will find something like:
> 
> IMAP> A0001 SELECT "INBOX"
> ....
> IMAP< A0001 OK SELECT completed

- when there's no new mail:

A0001 SELECT "INBOX"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]
* OK [UIDVALIDITY 2]
* 518298 EXISTS
* 0 RECENT
* OK [UIDNEXT 652167]
A0001 OK [READ-WRITE] INBOX selected. (Success)

- when there's at least one new mail:

A0001 SELECT "INBOX"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]
* OK [UIDVALIDITY 2]
* 518299 EXISTS
* 0 RECENT
* OK [UIDNEXT 652168]
A0001 OK [READ-WRITE] INBOX selected. (Success)

> 
> The whole section is required. Please copy them for two cases:
> 
> - When there is atleast one new mail in the folder
> 
> - When there are no new mails in the folder
> 
> Are you also using IDLE? That could require more IMAP logs from your end around the IDLE command.

No, I don't.

> 
> > > 2. Send a STATUS command
> 
> The STATUS command might be a bit expensive as it will scan the folder again. There could also be an issue of mismatch between the open folder and the current status of that folder.

OK.

> 
> > > 3. Send an extended SEARCH command
> 
> This is not supported by your server as it is not advertised in the CAPABILITY string. Implementing this will not help you.

OK.

Thanks,
-Andrea

Re: [fetchmail-devel] timeout not working for TLS

[Matthias A. <mat...@gm...>]

Am 28.04.2011 17:06, schrieb Thomas Jarosch:
> On Thursday, 28. April 2011 14:42:21 Matthias Andree wrote:
>> $ sudo socat -dddD - tcp4-listen:110,reuseaddr
>> 1. paste +OK\n as greeting
>> 2. paste +OK\nSTLS\n.\n as response to STLS, then wait
> 
> Ok, I found the difference. If I wait at the same point as you do,
> the timeout is triggered. Please at step 3.:
> 
> 3. paste "+OK do it\n" and then wait

Got it, I can reproduce the problem.

> Some SSL garbage will appear in socat but that's fine.
> The strace output looks like this:
> 
> write(3, "STLS\r\n", 6)                 = 6 <0.000078>
> setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={90, 0}}, NULL) = 0 <0.000014>
> recv(3, "+OK do it\n", 512, MSG_PEEK)   = 10 <3.055521>
> read(3, "+OK do it\n", 10)              = 10 <0.000015>
> setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0 <0.000012>

ouch.

> ...
> 
> -> The timeout get's disabled again. Any idea why?

I've analyzed it.  The trace in the Git master branch is:

(gdb) bt
#0  set_timeout (timeleft=0) at ../driver.c:100
#1  0x00000000004111a6 in gen_recv (sock=5, buf=0x7fffffff76d0 "+OK go\n",
    size=513) at ../transact.c:1575
#2  0x00000000004208c1 in pop3_ok (sock=0, argbuf=0x7fffffff7930 "STLS")
    at ../pop3.c:116
#3  0x0000000000410e56 in gen_transact (sock=5, fmt=0x42a0a2 "STLS")
    at ../transact.c:1632
#4  0x0000000000420ee4 in pop3_getauth (sock=5, ctl=0x4516d0,
    greeting=0x7fffffff9c30 "") at ../pop3.c:451

IOW, I had overlooked that gen_recv resets the timeout. We extract the
STLS response, which resets the timeout, thus the subsequent SSLOpen
isn't under some timeout.

Sorry for the incomplete fix in 6.3.18, and thanks for the report.  Now
I need to make sure I catch all similar bugs before releasing the next
version, and I need to update and re-issue the corresponding CVE (or a
new one, need to check with the gurus) and security announcement.

The fix needs a thorough analysis of the code.  Note that I've already
queued patches to ditch SSLv2 support, I need to reconsider that, or
making that an option so that distributors can go ahead and update their
6.3.ancient version to 6.3.20 without major incompatibilities.

I'll get back to this.

Thanks again -- I do appreciate bug reports that are easy to reproduce :-)

Best regards
Matthias

Re: [fetchmail-devel] timeout not working for TLS

[Thomas J. <tho...@in...>]

On Thursday, 28. April 2011 14:42:21 Matthias Andree wrote:
> $ sudo socat -dddD - tcp4-listen:110,reuseaddr
> 1. paste +OK\n as greeting
> 2. paste +OK\nSTLS\n.\n as response to STLS, then wait

Ok, I found the difference. If I wait at the same point as you do,
the timeout is triggered. Please at step 3.:

3. paste "+OK do it\n" and then wait

Some SSL garbage will appear in socat but that's fine.
The strace output looks like this:

write(3, "STLS\r\n", 6)                 = 6 <0.000078>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={90, 0}}, NULL) = 0 <0.000014>
recv(3, "+OK do it\n", 512, MSG_PEEK)   = 10 <3.055521>
read(3, "+OK do it\n", 10)              = 10 <0.000015>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0 <0.000012>
...

-> The timeout get's disabled again. Any idea why?

Cheers,
Thomas

Re: [fetchmail-devel] timeout not working for TLS

[Matthias A. <mat...@gm...>]

BTW, timeouts also work for me on openSUSE 11.4 i386, with openSSL
1.0.0c and glibc 2.11.3, and fetchmail 6.3.20-pre1.

Re: [fetchmail-devel] timeout not working for TLS

[Matthias A. <mat...@gm...>]

Am 28.04.2011 13:50, schrieb Thomas Jarosch:
> On Thursday, 28. April 2011 13:18:59 Matthias Andree wrote:
>> Looks like GMX.
> 
> Yes, 1&1 ;)
> 
>> what OS and version (distribution) does this happen on?  Do the
>> -debuginfo packages match the actual RPMs?  Could you try building
>> 6.3.19 from source and use that to somehow reproduce the hang?
> 
> It's our custom distribution which is based on Fedora.
> 
> I recovered the correct -debuginfo packages from the time I built the binary 
> RPMs. If the debuginfo packages don't fit (f.e. glibc header changed),
> gdb will skip those debug symbols with an CRC mismatch error.
> 
>> I wonder if the SSL stuff somehow masks the timeout alarm we're using,
>> that could possibly explain things, but I don't have an idea how to
>> figure that out yet.
> 
> It just came to my mind: We can simulate the situation with socat.
> And I was able to trigger the problem. Try this:
> 
> - Start a "fake POP3 server" with socat: socat - tcp4-listen:110
> 
> - fetchmail connects to this server
> 
> - Paste the welcome greeting:
> +OK POP fake server ready H mimap4
> 
> - fetchmail will reply with: "CAPA"
> 
> - Paste this:
> +OK Capability list follows
> TOP
> USER
> UIDL
> STLS
> SASL PLAIN
> IMPLEMENTATION trinity
> .
> 
> - fetchmail will reply: "STLS"
> 
> - Paste this:
> +OK Begin TLS negotiation
> 
> 
> Then do nothing in socat until the timeout should be triggered.

Good idea.

Timeouts work for me in 6.3.20-pre1, and I haven't changed the timeout
handling since 6.3.18 AFAIR.  The default is 300, I've changed it to 10
only for testing.

Is there anything that masks or traps SIGALRM in your setup?

I'm currently testing on Ubuntu 10.10 amd64 with OpenSSL 0.9.8o and
libc6 which is embedded GNU libc 2.12.1 (both likely with Ubuntu patches).

In one console:

$ sudo socat -dddD - tcp4-listen:110,reuseaddr
1. paste +OK\n as greeting
2. paste +OK\nSTLS\n.\n as response to STLS, then wait

In another console (I hope Thunderbird doesn't trash everything here):

$ LC_ALL=C strace -T ./fetchmail -p pop3 -Nd0 localhost --nosyslog -s
--timeout 10
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3 <0.000023>
connect(3, {sa_family=AF_INET, sin_port=htons(110),
sin_addr=inet_addr("127.0.0.1")}, 16) = 0 <0.000028>
getsockname(3, {sa_family=AF_INET, sin_port=htons(38674),
sin_addr=inet_addr("127.0.0.1")}, [16]) = 0 <0.000025>
close(3)                                = 0 <0.000027>
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3 <0.000026>
connect(3, {sa_family=AF_INET, sin_port=htons(110),
sin_addr=inet_addr("127.0.0.1")}, 16) = 0 <0.000101>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={10, 0}}, NULL) = 0
<0.000016>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
<0.000013>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={10, 0}}, NULL) = 0
<0.000014>
recvfrom(3, "+OK\n", 512, MSG_PEEK, NULL, NULL) = 4 <3.263103>
read(3, "+OK\n", 4)                     = 4 <0.000031>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
<0.000021>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={10, 0}}, NULL) = 0
<0.000020>
write(3, "CAPA\r\n", 6)                 = 6 <0.000090>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={10, 0}}, NULL) = 0
<0.000020>
recvfrom(3, "+OK\n", 512, MSG_PEEK, NULL, NULL) = 4 <1.739998>
read(3, "+OK\n", 4)                     = 4 <0.000030>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
<0.000020>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={10, 0}}, NULL) = 0
<0.000020>
recvfrom(3, "STLS\n", 63, MSG_PEEK, NULL, NULL) = 5 <0.959806>
read(3, "STLS\n", 5)                    = 5 <0.000026>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
<0.000019>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={10, 0}}, NULL) = 0
<0.000020>
recvfrom(3, ".\n", 63, MSG_PEEK, NULL, NULL) = 2 <0.220514>
read(3, ".\n", 2)                       = 2 <0.000030>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
<0.000021>
write(3, "STLS\r\n", 6)                 = 6 <0.000078>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={10, 0}}, NULL) = 0
<0.000031>
recvfrom(3, 0x7fff216f2770, 512, 2, 0, 0) = ? ERESTARTSYS (To be
restarted) <9.999914>
--- SIGALRM (Alarm clock) @ 0 (0) ---
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 <0.000023>
rt_sigprocmask(SIG_UNBLOCK, ~[RTMIN RT_1], NULL, 8) = 0 <0.000020>
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
<0.000021>
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7ff3a6332000 <0.000024>
write(1, "fetchmail: timeout after 10 seco"..., 66fetchmail: timeout
after 10 seconds waiting for server localhost.
) = 66 <0.000031>
rt_sigaction(SIGALRM, {0x40dce0, [], SA_RESTORER, 0x7ff3a4a75c20},
{0x40e0e0, [], SA_RESTORER, 0x7ff3a4a75c20}, 8) = 0 <0.000019>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 0
<0.000020>
close(3)                                = 0 <0.000097>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
<0.000029>
rt_sigaction(SIGALRM, {0x40e0e0, [], SA_RESTORER, 0x7ff3a4a75c20},
{0x40dce0, [], SA_RESTORER, 0x7ff3a4a75c20}, 8) = 0 <0.000019>
write(2, "fetchmail: ", 11fetchmail: )             = 11 <0.000024>
write(2, "socket error while fetching from"..., 51socket error while
fetching from mandree@localhost
) = 51 <0.000023>
setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
<0.000019>
rt_sigaction(SIGALRM, {0x4074f0, [], SA_RESTORER, 0x7ff3a4a75c20},
{0x40e0e0, [], SA_RESTORER, 0x7ff3a4a75c20}, 8) = 0 <0.000018>
write(1, "fetchmail: Query status=2 (SOCKE"..., 35fetchmail: Query
status=2 (SOCKET)
) = 35 <0.000023>

Re: [fetchmail-devel] timeout not working for TLS

[Thomas J. <tho...@in...>]

On Thursday, 28. April 2011 13:18:59 Matthias Andree wrote:
> Looks like GMX.

Yes, 1&1 ;)

> what OS and version (distribution) does this happen on?  Do the
> -debuginfo packages match the actual RPMs?  Could you try building
> 6.3.19 from source and use that to somehow reproduce the hang?

It's our custom distribution which is based on Fedora.

I recovered the correct -debuginfo packages from the time I built the binary 
RPMs. If the debuginfo packages don't fit (f.e. glibc header changed),
gdb will skip those debug symbols with an CRC mismatch error.

> I wonder if the SSL stuff somehow masks the timeout alarm we're using,
> that could possibly explain things, but I don't have an idea how to
> figure that out yet.

It just came to my mind: We can simulate the situation with socat.
And I was able to trigger the problem. Try this:

- Start a "fake POP3 server" with socat: socat - tcp4-listen:110

- fetchmail connects to this server

- Paste the welcome greeting:
+OK POP fake server ready H mimap4

- fetchmail will reply with: "CAPA"

- Paste this:
+OK Capability list follows
TOP
USER
UIDL
STLS
SASL PLAIN
IMPLEMENTATION trinity
.

- fetchmail will reply: "STLS"

- Paste this:
+OK Begin TLS negotiation


Then do nothing in socat until the timeout should be triggered.

Hope that helps,
Thomas

Re: [fetchmail-devel] timeout not working for TLS

[Matthias A. <mat...@gm...>]

Am 28.04.2011 09:38, schrieb Thomas Jarosch:
> Hello Matthias,
> 
> On Thursday, 5. August 2010 00:18:30 Matthias Andree wrote:
>> The attached patch sets the timeout for the getauth() stage (which
>> entails STARTTLS-like negotiation). Please try that too and see if you
>> get timeout reports while fetchmail tries to negotiate TLS. It should.
>>
>> Thanks for the report and offered help in debugging. Would be good if you
>> could report back in a month or so :-)
> 
> I just had another case of this issue on a different box,
> this time running fetchmail 6.3.18.
> 
> Here's all the info I was able to collect:
> 
> Log from fetchmail:
> fetchmail: POP3< +OK Capability list follows
> fetchmail: POP3< TOP
> fetchmail: POP3< USER
> fetchmail: POP3< UIDL
> fetchmail: POP3< STLS
> fetchmail: POP3< SASL PLAIN
> fetchmail: POP3< IMPLEMENTATION trinity
> fetchmail: POP3< .
> fetchmail: POP3> STLS
> fetchmail: POP3< +OK Begin TLS negotiation

Looks like GMX.

> fetchmail was stuck at that line since 21.04.2011.
> 
> [root@intranator log]# strace -p 1505
> Process 1505 attached - interrupt to quit
> read(3,
> 
> (gdb) bt
> #0  0x00c17424 in __kernel_vsyscall ()
> #1  0x00867ef3 in __read_nocancel () at ../sysdeps/unix/syscall-
> template.S:82
> #2  0x0033771e in sock_read () from /usr/lib/libcrypto.so.8
> #3  0x00000003 in ?? ()
> #4  0x09e7cafe in ?? ()
> #5  0x000006bf in ?? ()
> #6  0x89f4e239 in ?? ()
> #7  0x6ce90be1 in ?? ()
> #8  0xa8a2f11c in ?? ()
> #9  0x60917bce in ?? ()
> #10 0x003c8c28 in ?? () from /usr/lib/libcrypto.so.8
> #11 0x09e762f8 in ?? ()
> #12 0x00000000 in ?? ()
> 
> 
> Even though I installed the -debuginfo packages,
> I wasn't able to get a meaningful backtrace.
> 
> So the stall happens before the authentication state?

Hi Thomas,

what OS and version (distribution) does this happen on?  Do the
-debuginfo packages match the actual RPMs?  Could you try building
6.3.19 from source and use that to somehow reproduce the hang?

I wonder if the SSL stuff somehow masks the timeout alarm we're using,
that could possibly explain things, but I don't have an idea how to
figure that out yet.

Sorry - a backtrace could possibly help a bit because I could try
reading the OpenSSL code paths in the OpenSSL sources.

Best regards
Matthias