fetchmail-devel — Development related discussion

[fetchmail-devel] fetchmail 6.3.17-pre1 snapshot and request for help

[Matthias A. <mat...@gm...>]

Greetings,

#1 I am asking interested fetchmail users to test a preview release of
fetchmail. Change details are below.

#2 Also, if you can offer access to test servers that I can send a short
test mail to and then log into to retrieve that test message -
particularly Exchange 2007 is desired, but others besides Cyrus IMAP and
Dovecot are also welcome - please let me know.

#3 Finally, fetchmail needs translators for the program strings. Some
languages (such as those shown below) are in quite good shape, but
others are lacking a bit.

Translation information at
<http://translationproject.org/domain/fetchmail.html>

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

DOWNLOAD this beta software from:
<http://home.pages.de/~mandree/fetchmail/>

The repository can be browsed at and cloned from:
<http://gitorious.org/fetchmail/fetchmail>

Git (the software used to keep the fetchmail source code version
controlled) information is at: <http://git-scm.com/>

CHANGES since the previous formal release of fetchmail listed below.
Unless otherwise noted, the changes were made by Matthias Andree:

# SECURITY FIX
* CVE-2010-1167: Fetchmail before release 6.3.17 did not properly sanitize
  external input (mail headers and UID). When a multi-character locale (such as
  UTF-8) was in use, this could cause memory exhaustion and thus a denial of
  service, because fetchmail's report.c functions assumed that non-success of
  [v]snprintf was due to insufficient buffer size allocation. It would then
  repeatedly reallocate a larger buffer and fail formatting again.
  See fetchmail-SA-2010-02.txt.

# FEATURES
* Fetchmail now supports a --sslcertfile <file> option to specify a "CA bundle"
  file (a file that contains trusted CA certificates). Since these bundled CA
  files do not require c_rehash to be run, they are easier to use and immune to
  OpenSSL library updates that affect the hash function.
* Fetchmail now supports a FETCHMAIL_INCLUDE_DEFAULT_X509_CA_CERTS
  environment variable to force loading the default SSL CA certificate
  locations.

# REGRESSION FIX
* Fix string handling in rcfile scanner, which caused fetchmail to misparse a
  run control file in certain circumstances.  Fixes BerliOS bug #14257.
  Patch by Michael Banack.  This fixes a regression introduced before 6.3.0.

# BUG FIXES
* Plug memory leak when using a "defaults" entry in the run control file.
* Do not print SSL certificate mismatches unless verbose or --sslcertck is
  enabled.
* Do not lose "set invisible" in fetchmailconf. (Michael Barnack)

# CHANGES
* Usability: SSL certificate chains are fully printed in -v -v mode, and there
  are now helpful pointers to --sslcertpath and c_rehash for "unable to get
  local issuer certificate" and self-signed certificates -- these usually hint
  to missing root signing CAs in the certs directory.
* Several fixes for compiler (GCC, Intel C++, CLang) and autotools warnings
* Memory allocation failures will now cause abnormal program abort (SIGABRT),
  not exit with unspecified code.

# DOCUMENTATION
* Fix table of global option to read "set softbounce" where there used to be a
  2nd copy of "set spambounce".  Patch by Michael Banack, BerliOS Bug #17067.
* In the --sslcertpath description, mention that OpenSSL upgrade (and a 0.9.X
  to 1.0.0 upgrade in particular) may require running c_rehash.

# KNOWN BUGS AND WORKAROUNDS:
  (this section floats upwards through the NEWS file so it stays with the
  current release information - however, it was stuck with 6.3.8 for a while)
* fetchmail does not handle messages without Message-ID header well
  (See sourceforge.net bug #780933)
* BSMTP is mostly untested and errors can cause corrupt output.
* Sun Workshop 6 (SPARC) is known to miscompile the configuration file lexer in
  64-bit mode.  Either compile 32-bit code or use GCC to compile 64-bit
  fetchmail.  Note that fetchmail doesn't take advantage of 64-bit code,
  so compiling 32-bit SPARC code should not cause any difficulties.
* fetchmail does not track pending deletes over crashes
* the command line interface is sometimes a bit stubborn, for instance,
  fetchmail -s doesn't work with a daemon running

-- 
Matthias Andree

Re: [fetchmail-devel] save_str problems.

[Matthias A. <mat...@gm...>]

Rainer Weikusat wrote on 2010-04-26:

> This looks very suspiciously like my first attempt at making a 'quick'
> improvement to this code. It will yield about half of the benefit of
> the patch I sent (for 'our' application), because this method cannot
> be used for the fastuidl path in pop3_is_old (as you wrote, I profiled
> this before making the more complicated change). It is also (sorry for
> being so blunt) not very well done. Because of the
>
>     /* do it nonrecursively so the list is in the right order */
>     for (end = idl; *end; end = &(*end)->next)
>         continue;
>
> in save_str_quick, the value passed to the routine should be the
> address of the pointer which needs to be changed when appending to the
> list, eg, assuming the 'savep' change:

Yeah, we can save that last single iteration from end to &end->next and go  
all the way, and we can use a proper initializer. Would make only a minor  
difference though as long as we're still doing linear searches.

> a better implementation would be
>
> svp = &ctl->oldsaved;
>
> [...]
>
> last_nr = try_nr;
> /* save it */
> sv = save_str(svp, id, UID_UNSEEN);
> sv->val.status.num = try_nr;
> svp = &sv->next;
> ---------------------------

Thanks.

> which has the nice bonus property that it doesn't have the conditional
> which goes one way during the first iteration and the other way during
> all that follow inside the loop (the idea isn't mine, I originally
> learnt about it because of some years-old USENET posting of a guy
> whose name I've unfortunately forgotten). I am also rather concerned
> about use of bandwidth than speed. Presently, I am dealing with 76
> POP3 accounts and about a meg of stored UIDs I'd need to download
> every five minutes in order to determine that presently, nothing needs
> to be downloaded (this refers to fastuidl in general, assuming I
> understood the principle correctly without really analyzing the code).

Makes sense, although I'm wondering if it really makes that much of a  
difference for fastuidl.
fastuidl appears to be opportunistically harvesting message numbers, I  
wonder if that's of any use.
I think Sunil wrote that fastUIDL code, I'm Cc:ing him.

> So, thank you very much for your effort, but I'll stick with the
> version in the private fork I am anyway maintaining because of the
> additional features (like 'object-oriented/ vtabled sinks') whose
> usefulness would be very limited outside this particular
> application and/or whose implementation is rather 'commercial' (eg
> #ifdefing away the concurrency control code) than aesthetically/
> technically pleasing.

Not sure I get your point.  I've always wanted to abstract the sink code  
the way the fetch protocols are abstracted, and that was also one thing  
I'd tried to squeeze from the 2008 Google Summer of Code that was supposed  
to provide MAPI (but apparently isn't fit for integration, and apparently  
stalled).

> The savedend-change was just something I
> considered to be more generally useful, hence I 'backported' it to
> 6.3.16 from my HEAD and sent it to the list.

Much appreciated.

> BTW, I had a look at the UIDL patch and whoever wrote that should
> probably spend some time reading RFC4549 (Synchronization Operations
> for Disconnected IMAP4 Clients) which is what I am going to add to the
> imap-code because using the \Seen flag as 'message is old' indicator
> [reportedly] interferes with the gmail web interface.

I'd appreciate if such code could be made public. The server-side \Seen  
tracking is something I have wanted fetchmail to get rid of for long. The  
long-winded way with "smtp_someop() { if (I'm surprisingly using mda)  
mda_someop() else if (using bsmtp) else if (using lmtp) }" is garbage,  
inconcise, inefficient -- and it appears from your description you've  
fixed just that already.

> NB: Nothing of this text is meant as an insult even if it may sound
> like one.

No offense taken.  I can usually tell the difference between criticing my  
work objectively, subjectively, and ad-hominem attacks. :)

-- 
Matthias Andree

Re: [fetchmail-devel] save_str problems.

[Rainer W. <rwe...@ms...>]

Matthias Andree <mat...@gm...> writes:
> http://gitorious.org/fetchmail/fetchmail/commits/for-rainer/
>
> You can download this as tarball, but you still need to follow the
> build-from-Git instructions (basically, run autoreconf -isv first).
>
> It contains a lightweight version of your patch (see the two latest
> commits). The fastuidl fix is less efficient, but also less needed since
> the amount of UIDs added is ld N there, so we get O(n log n) complexity
> for parsing the UIDL responses. The UIDL fix is O(n) for the actual
> insertion, but remains O(n^2) for "message seen yet?" detection.

This looks very suspiciously like my first attempt at making a 'quick'
improvement to this code. It will yield about half of the benefit of
the patch I sent (for 'our' application), because this method cannot
be used for the fastuidl path in pop3_is_old (as you wrote, I profiled
this before making the more complicated change). It is also (sorry for
being so blunt) not very well done. Because of the

    /* do it nonrecursively so the list is in the right order */
    for (end = idl; *end; end = &(*end)->next)
        continue;

in save_str_quick, the value passed to the routine should be the
address of the pointer which needs to be changed when appending to the
list, eg, assuming the 'savep' change:

---------
int ok;
unsigned int first_nr, last_nr, try_nr;
char id [IDLEN+1];
struct idlist *savep = NULL; /** pointer to cache save_str result, speeds up saves */
first_nr = 0;
last_nr = count + 1;

[...]

last_nr = try_nr;
/* save it */
savep = save_str(savep ? &savep : &ctl->oldsaved, id, UID_UNSEEN);
savep->val.status.num = try_nr;
---------

a better implementation would be

--------------------------
int ok;
unsigned int first_nr, last_nr, try_nr;
char id [IDLEN+1];
struct idlist **svp, *sv;
first_nr = 0;
last_nr = count + 1;
svp = &ctl->oldsaved;

[...]

last_nr = try_nr;
/* save it */
sv = save_str(svp, id, UID_UNSEEN);
sv->val.status.num = try_nr;
svp = &sv->next;
---------------------------

which has the nice bonus property that it doesn't have the conditional
which goes one way during the first iteration and the other way during
all that follow inside the loop (the idea isn't mine, I originally
learnt about it because of some years-old USENET posting of a guy
whose name I've unfortunately forgotten). I am also rather concerned
about use of bandwidth than speed. Presently, I am dealing with 76
POP3 accounts and about a meg of stored UIDs I'd need to download
every five minutes in order to determine that presently, nothing needs
to be downloaded (this refers to fastuidl in general, assuming I
understood the principle correctly without really analyzing the code).

So, thank you very much for your effort, but I'll stick with the
version in the private fork I am anyway maintaining because of the
additional features (like 'object-oriented/ vtabled sinks') whose
usefulness would be very limited outside this particular
application and/or whose implementation is rather 'commercial' (eg
#ifdefing away the concurrency control code) than aesthetically/
technically pleasing. The savedend-change was just something I
considered to be more generally useful, hence I 'backported' it to
6.3.16 from my HEAD and sent it to the list.

BTW, I had a look at the UIDL patch and whoever wrote that should
probably spend some time reading RFC4549 (Synchronization Operations
for Disconnected IMAP4 Clients) which is what I am going to add to the
imap-code because using the \Seen flag as 'message is old' indicator
[reportedly] interferes with the gmail web interface.

NB: Nothing of this text is meant as an insult even if it may sound
like one. I am a computer person and not a person person and my
knowledge regarding 'how to deal with other human beings' is still
very limited, not the least because 'other human beings' are usually
drunk, male persons desiring to hit me because I have (again) done
something wrongly I neither understood nor recognized.

Re: [fetchmail-devel] save_str problems.

[Matthias A. <mat...@gm...>]

Please check out

http://gitorious.org/fetchmail/fetchmail/commits/for-rainer/

You can download this as tarball, but you still need to follow the
build-from-Git instructions (basically, run autoreconf -isv first).

It contains a lightweight version of your patch (see the two latest
commits). The fastuidl fix is less efficient, but also less needed since
the amount of UIDs added is ld N there, so we get O(n log n) complexity
for parsing the UIDL responses. The UIDL fix is O(n) for the actual
insertion, but remains O(n^2) for "message seen yet?" detection.

I wonder if I should kill fastuidl in fetchmail 6.4. It has quite a few
quirks and is only useful on low-bandwidth low-latency links. DSL, GSM,
GPRS and thereabouts are high-latency.

Example: On DSL a line w/ 6 MBit/s and 40 ms round trips, the regular
UIDL code is still faster for 10000 UIDs.

The tree above also contains a few cleanups (idlist functions were split
out from uid.c to a new idlist.c file).

CPU is a different issue; to fix this for good, unless I'm mistaken, we
need to split the message flags from the UID linked list.  The former
goes in an array (which needs to store only the mark byte per message),
the latter into a btree (UID -> number).

I may need to kill the slow UIDL emulation code (header-/Message-ID
based) along the way, if so, the speedup will hardly be fit for 6.3.X,
but rather for 6.4.X.

HTH
Matthias

Re: [fetchmail-devel] save_str problems.

[Matthias A. <mat...@gm...>]

Rainer Weikusat wrote on 2010-04-23:

> "Matthias Andree" <mat...@gm...> writes:
>> Rainer Weikusat wrote on 2010-04-21:
>>> The company I work for uses fetchmail to download mails from various
>>> POP- and IMAP-servers in order to offer an anti-spam/ anti-virus
>>> scanning service for smartphone owners. Below is an oprofile excerpt
>>> from the 'download server' regarding fetchmail:
>
> [...]
>
>> You've bumped into a design flaw of fetchmail's  UID handling.
>
> I assume it is rather an instance of 'Learn Lisp. It will make you a
> better programmer for life' :->.

Uh, don't tell me. I've known that when my Debian installation fit onto  
the Conner CP30344 disk in my i486SX25.
Tell that to Eric. And tell him LAST was an abomination and  
defunct-upon-invention. ;)

No, don't tell him. I guess he'd do it in Python today, use a dictionary,  
and leave optimization up to the Python interpreter, and that might be  
reasonable, unless we'd have to have SSL idiosyncrasies under control.  
getmail (Python based) used to not complain about expired certificates,  
and all Charles had to say was "sorry, not my fault".

> [...]
>
>> If we were being blunt and wanted a quick fix, we'd kill the list and
>> use  a vector (array) and realloc(), a flag "vector is sorted", and
>> qsort and  bsearch. That would bring things to O(n log n)
>>
>> If we were to be decent, we'd use the right structure for our purpose,
>> and  that's a hash table (in C++/STL i'd use a hash_set where
>> available and a  set where unavailable). We might want to keep things
>> in a database on  disk, rather than load/write the table for every
>> run.
>>
>> Your patch effort is much appreicated, but however I have serious
>> doubts  about taking it. save_str and id_find are used together and
>> basically  O(n^2) complexity. Even if your patch shaves save_str down
>> to O(n),  id_find usage pattern itself remains O(n^2).
>
> struct idlist * is quite pervasive in the code. It is also used for
> many 'short lists' (eg, ctl->localnames) where a linked list is at
> least good enough. Keeping track of the last item on the oldsaved and
> newsaved lists was just a reasonably non-invasive, fairly quick fix for
> the 'worst offender' case (and the IMHO most braindead ...) I could
> smuggle past my boss[*] as 'immediately useful and not that much
> effort'.

Yes. Only this is one of the makeshift solutions that is not bound to last  
long. GMX split my INBOX when it hit 50,000 messages. Oops. I do NOT want  
to poll that with fetchmail <= 6.3.16. :)

Seriously, how about this approach:

1. create uid_save and uid_find wrappers around save_str and id_find and  
provide a typedef

2. change POP3 and possibly IMAP (see below) to use that => decouple from  
the implementation

3. replace uid_save/uid_find implementations by something decent?


This way all the short lists can continue to use the save_str/id_find  
cruft for the nonce, and the bulk can use a O(n log n) implementation.

> 	[*] I am actually supposed to add UID-support the IMAP
> 	protocol driver and this case of 'walk until your shoes fall
> 	off' just angered me ...

Good to know. We have a patch in the berlios tracker [1], which is halfway  
there, but you need to add UIDVALIDITY support [2], and probably a few  
touch-ups because the patch ancestor and current fetchmail code have  
diverged.
Oh, and of course we can add your company name to the NEWS file, too.  
Sponsored by... written by... <hint, hint> :)

Note I'm planning to change the license to Affero GPL v3 in the not too  
distant future.

[1]  
<http://developer.berlios.de/patch/?func=detailpatch&patch_id=740&group_id=1824>
     also attached

[2] <http://www.rfc-editor.org/rfc/rfc3501.txt>
     2.3.1.1.        Unique Identifier (UID) Message Attribute

Just in case BerliOS drops off the net, find the patch attached (may not  
be available on the list).
Note again: not ready for use!

HTH

-- 
Matthias Andree

Re: [fetchmail-devel] save_str problems.

[Matthias A. <mat...@gm...>]

Rainer Weikusat wrote on 2010-04-21:

> Hello.
>
> The company I work for uses fetchmail to download mails from various
> POP- and IMAP-servers in order to offer an anti-spam/ anti-virus
> scanning service for smartphone owners. Below is an oprofile excerpt
> from the 'download server' regarding fetchmail:

[whoops, I meant to store a draft when I figured it wasn't a 2 minute  
reply, but figured that the message was sent out rather than stored as  
draft, sorry for that]

Hi Rainer,

thanks for the analysis.  You've bumped into a (known) design flaw of  
fetchmail's UID handling.  It has been on the TODO list for ~2 years, but  
since it's not a functional bug, the bug fixes always got priority.

Basically the whole uid.c outfit is using the wrong data structures for  
its purpose. The whole stuff - I guess you figured as much - is meant to  
figure if a certain UID was seen, i. e. is in a set or not, and read/write  
that set from/to disk. We don't need order, only uniqueness, but we do  
need quick access, and the size is dynamic.

uid.c currently uses a linear list of n elements, and iterates over it  
roughly 1.5n times, i. e. we get O(n^2) complexity (without constant  
factors).

Your patch, albeit it helps a bit in that it shaves save_str down to O(n),  
does not fix the fundamental complexity problem: the id_find() stuff and  
with it the whole UID setup, still remains O(n^2) the way we're using it,  
only reduced by a constant factor of, guessing optimistially, 2-3.

I have some ideas for a proper solution:

(a) blunt: instead of the list, use an array with realloc, qsort and  
bsearch.  Ugly, but probably portable to anything that can compile  
fetchmail today.

(b) halfway: depend on <search.h>, use tsearch(3).  Not the ultimate in  
performance, but easy to use and down to O(n log n) complexity, which  
helps quite a bit already.  (hcreate(3) has a fixed size and supports only  
one table per process, which are showstoppers.)  Deemed safe for fetchmail  
6.3.

Even better options:

(c) Use C++ and optionally Boost with unordered_set. Fallback options are  
hash_set and set. Recent 6.3 versions are already intelligible as C and  
C++, so it's a gradual move with one initial bump that pulls in the  
dependencies and switches to C++.

(d) use a disk-based hashed or btree database.  I don't know the optimal  
library though, I've been through this in bogofilter.  sqlite3 pulls in  
SQL overhead (it's not that bad, but still) we don't need, but otherwise  
quite robust. Berkeley DB has decent documentation, STL API and is bullet  
proof -- but there are two nukes that kill it: (1) users messing with the  
files and removing log files, (2) distributions upgrading libdb and  
application without telling the user. It's also a nightmare to  
autodetect.  TokyoCabinet might be a candidate, but its API is pretty raw  
on the bits and I trust its future less than all other databases.


I am very inclined to go for (c) or (d) in fetchmail 6.4 and I think I'm  
determined to go for C++ as that allows much conciser code.

I am loathe to change 6.3 uid.c at all, as basically it's beyond repair  
and needs to be redone from scratch - which is not adequate for a "stable"  
version.

6.4 should split .fetchids to per-account files (we used to have patches  
but never deployed them in baseline code), which greatly simplifies uid  
list handling. See patches at <http://home.pages.de/~mandree/fetchmail/>.

> I am aware that the 'split append' (save_str + explicit adjustment of
> savedend-pointer) isn't exactly beautiful but it is a reasonably
> simple change with a significant positive impact for POP3 mailboxes
> containing lots of ('kept on server') messages.

If a constant factor of - best case guess - 3 is "significant", then yes -  
but it still doesn't scale and your gain is lost again if you increase the  
number of messages by 40 - 80 %.

The moment I saw your patch I immediately thought of AmigaOS (Kickstart)  
exec.library "struct List" and "struct Node" (AROS equivalent at [1])  
which would solve this problem in a beautiful way -- albeit without fixing  
the fundamental performance problem.

I hope you're not too disappointed that I'm loathe to take your patch and  
that you can understand my reasoning.

Best regards
Matthias

[1]  
<http://aros.sourceforge.net/documentation/developers/app-dev/exec-library.php#list-manipulating-functions>

-- 
Matthias Andree

Re: [fetchmail-devel] save_str problems.

[Rainer W. <rwe...@ms...>]

"Matthias Andree" <mat...@gm...> writes:
> Rainer Weikusat wrote on 2010-04-21:
>> The company I work for uses fetchmail to download mails from various
>> POP- and IMAP-servers in order to offer an anti-spam/ anti-virus
>> scanning service for smartphone owners. Below is an oprofile excerpt
>> from the 'download server' regarding fetchmail:

[...]

> You've bumped into a design flaw of fetchmail's  UID handling.

I assume it is rather an instance of 'Learn Lisp. It will make you a
better programmer for life' :->.

[...]

> If we were being blunt and wanted a quick fix, we'd kill the list and
> use  a vector (array) and realloc(), a flag "vector is sorted", and
> qsort and  bsearch. That would bring things to O(n log n)
>
> If we were to be decent, we'd use the right structure for our purpose,
> and  that's a hash table (in C++/STL i'd use a hash_set where
> available and a  set where unavailable). We might want to keep things
> in a database on  disk, rather than load/write the table for every
> run.
>
> Your patch effort is much appreicated, but however I have serious
> doubts  about taking it. save_str and id_find are used together and
> basically  O(n^2) complexity. Even if your patch shaves save_str down
> to O(n),  id_find usage pattern itself remains O(n^2).

struct idlist * is quite pervasive in the code. It is also used for
many 'short lists' (eg, ctl->localnames) where a linked list is at
least good enough. Keeping track of the last item on the oldsaved and
newsaved lists was just a reasonably non-invasive, fairly quick fix for
the 'worst offender' case (and the IMHO most braindead ...) I could
smuggle past my boss[*] as 'immediately useful and not that much
effort'.

	[*] I am actually supposed to add UID-support the IMAP
	protocol driver and this case of 'walk until your shoes fall
	off' just angered me ...
        

Re: [fetchmail-devel] save_str problems.

[Matthias A. <mat...@gm...>]

Rainer Weikusat wrote on 2010-04-21:

> Hello.
>
> The company I work for uses fetchmail to download mails from various
> POP- and IMAP-servers in order to offer an anti-spam/ anti-virus
> scanning service for smartphone owners. Below is an oprofile excerpt
> from the 'download server' regarding fetchmail:

Hi Rainer,

thanks for the analysis.  You've bumped into a design flaw of fetchmail's  
UID handling. It has been on my TODO list for long (and using fetchmail  
with just 500 kept messages was painful), but it was a non-functional  
improvement, and the functional improvements (bug fixes) always "jumped  
the queue" :)

Basically the whole uid.c outfit is using the wrong data structures for  
its purpose. It uses a linear list of n elements, and iterates over it  
1.5n times, i. e. we get O(n^2) complexity (without constants).

The whole stuff - I guess you figured as much - is meant to figure if a  
certain UID is in a set or not, and read/write that set to disk. We don't  
need order, only uniqueness.

If we were being blunt and wanted a quick fix, we'd kill the list and use  
a vector (array) and realloc(), a flag "vector is sorted", and qsort and  
bsearch. That would bring things to O(n log n)

If we were to be decent, we'd use the right structure for our purpose, and  
that's a hash table (in C++/STL i'd use a hash_set where available and a  
set where unavailable). We might want to keep things in a database on  
disk, rather than load/write the table for every run.

Your patch effort is much appreicated, but however I have serious doubts  
about taking it. save_str and id_find are used together and basically  
O(n^2) complexity. Even if your patch shaves save_str down to O(n),  
id_find usage pattern itself remains O(n^2).

>
> ,----
> | CPU: Core 2, speed 2393.92 MHz (estimated)
> | Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) [...]
> | samples  %        image name               symbol name
> | 25110    51.5035  fetchmail                save_str
> | 11399    23.3806  fetchmail                id_find
> | 5851     12.0011  fetchmail                str_in_list
> | 912       1.8706  fetchmail                yylex
> | 558       1.1445  fetchmail                initialize_saved_lists
> | 493       1.0112  fetchmail                do_session
> `----
>
> According to this, the program spent most of its time (51.5%) with
> executing the save_str-routine (uid.c). This routine appends a new
> item to a linked list of structures by traversing the list in order to
> find the last node and then adding the new one. It is [mostly] called
> in a couple of places in pop3.c to update the 'newsaved' and
> 'oldsaved' lists of message UIDs (UIDL). The query structure defined
> in fetchmail.h actually even contains a member which was supposed to
> track the end of the oldsaved list (oldsavedend) but that is only
> initialized (in initialize_saved_lists/ uid.c) and never really used
> for anything. The patch included below changes the code to actually
> track the end of the oldsaved list and introduces a newsavedend struct
> query member in order to do the same for the newsaved list. With this
> change, the save_str-routines now only accounts for 0.4% of the CPU
> time used by the program:
>
> ,----
> | CPU: Core 2, speed 2393.92 MHz (estimated)
> | Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) [...]
> | samples  %        image name               symbol name
> | 19146    48.1588  fetchmail                id_find
> | 9416     23.6845  fetchmail                str_in_list
> | 1452      3.6523  fetchmail                yylex
> | 860       2.1632  fetchmail                initialize_saved_lists
> | 810       2.0374  fetchmail                do_session
> |
> | [...]
> |
> | 167       0.4201  fetchmail                save_str
> | 157       0.3949  fetchmail                parsecmdline
> | 145       0.3647  fetchmail                readheaders
> `----
>
> I am aware that the 'split append' (save_str + explicit adjustment of
> savedend-pointer) isn't exactly beautiful but it is a reasonably
> simple change with a significant positive impact for POP3 mailboxes
> containing lots of ('kept on server') messages.
>
> ----------
> --- fetchmail/fetchmail.h	16 Apr 2010 14:45:15 -0000	1.1.1.3
> +++ fetchmail/fetchmail.h	21 Apr 2010 16:31:34 -0000	1.1.1.3.2.1
> @@ -378,7 +378,7 @@
>      unsigned int uid;		/* UID of user to deliver to */
>      struct idlist *skipped;	/* messages skipped on the mail server */
>      struct idlist *oldsaved, *newsaved;
> -    struct idlist **oldsavedend;
> +    struct idlist **oldsavedend, **newsavedend;
>      char lastdigest[DIGESTLEN];	/* last MD5 hash seen on this  
> connection */
>      char *folder;		/* folder currently being polled */
> --- fetchmail/pop3.c	16 Apr 2010 17:27:18 -0000	1.1.1.3
> +++ fetchmail/pop3.c	21 Apr 2010 16:31:34 -0000	1.1.1.3.2.1
> @@ -881,8 +881,10 @@
>  	    last_nr = try_nr;
> 	    /* save it */
> -	    newl = save_str(&ctl->oldsaved, id, UID_UNSEEN);
> +	    newl = save_str(ctl->oldsavedend, id, UID_UNSEEN);
>  	    newl->val.status.num = try_nr;
> +
> +	    ctl->oldsavedend = &newl->next;
>  	}
>      }
>      if (outlevel >= O_DEBUG && last_nr <= count)
> @@ -970,15 +972,18 @@
>      /* the first try_id messages are known -> copy them to the newsaved  
> list */
>      for( num = first_nr; num < list_len; num++ )
>      {
> -	struct idlist	*newl = save_str(&ctl->newsaved,
> +	struct idlist	*newl = save_str(ctl->newsavedend,
>  				str_from_nr_list(&ctl->oldsaved, num),
>  				UID_UNSEEN);
>  	newl->val.status.num = num - first_nr + 1;
> +
> +	ctl->newsavedend = &newl->next;
>      }
>     if( nolinear ) {
>  	free_str_list(&ctl->oldsaved);
>  	ctl->oldsaved = 0;
> +	ctl->oldsavedend = &ctl->oldsaved;
>  	last = try_id;
>      }
> @@ -998,6 +1003,7 @@
>      (void)folder;
>      /* Ensure that the new list is properly empty */
>      ctl->newsaved = (struct idlist *)NULL;
> +    ctl->newsavedend = &ctl->newsaved;
> #ifdef MBOX
>      /* Alain Knaff suggests this, but it's not RFC standard */
> @@ -1089,9 +1095,11 @@
>  		    {
>  			struct idlist	*old, *newl;
> -			newl = save_str(&ctl->newsaved, id, UID_UNSEEN);
> +			newl = save_str(ctl->newsavedend, id, UID_UNSEEN);
>  			newl->val.status.num = unum;
> +			ctl->newsavedend = &newl->next;
> +
>  			if ((old = str_in_list(&ctl->oldsaved, id, FALSE)))
>  			{
>  			    flag mark = old->val.status.mark;
> @@ -1123,7 +1131,8 @@
>  			     * swap the lists (say, due to socket error),
>  			     * the same mail will not be downloaded again.
>  			     */
> -			    old = save_str(&ctl->oldsaved, id, UID_UNSEEN);
> +			    old = save_str(ctl->oldsavedend, id, UID_UNSEEN);
> +			    ctl->oldsavedend = &old->next;
>  			}
>  			/* save the number */
>  			old->val.status.num = unum;
> @@ -1224,8 +1233,10 @@
>  	}
> 	/* save it */
> -	newl = save_str(&ctl->oldsaved, id, UID_UNSEEN);
> +	newl = save_str(ctl->oldsavedend, id, UID_UNSEEN);
>  	newl->val.status.num = num;
> +
> +	ctl->oldsavedend = &newl->next;
>  	return(FALSE);
>      }
>      else
> --- fetchmail/transact.c	16 Apr 2010 14:45:18 -0000	1.1.1.2
> +++ fetchmail/transact.c	21 Apr 2010 16:31:34 -0000	1.1.1.2.2.1
> @@ -882,8 +882,10 @@
>   		sscanf(line+12, "%s", id);
>   	        if (!str_find( &ctl->newsaved, num))
>  		{
> - 		    struct idlist *newl = save_str(&ctl->newsaved,id,UID_SEEN);
> + 		    struct idlist *newl = save_str(ctl->newsavedend,id,UID_SEEN);
>  		    newl->val.status.num = num;
> +
> +		    ctl->newsavedend = &newl->next;
>  		}
>   	    }
>   	}
> --- fetchmail/uid.c	17 Aug 2009 10:47:46 -0000	1.1.1.1
> +++ fetchmail/uid.c	21 Apr 2010 16:31:34 -0000	1.1.1.1.2.1
> @@ -119,6 +119,7 @@
>  	ctl->oldsaved = (struct idlist *)NULL;
>  	ctl->newsaved = (struct idlist *)NULL;
>  	ctl->oldsavedend = &ctl->oldsaved;
> +	ctl->newsavedend = &ctl->newsaved;
>      }
>     errno = 0;
> @@ -151,6 +152,7 @@
>  	char saveddelim1;
>  	char *delimp2;
>  	char saveddelim2 = '\0';	/* pacify -Wall */
> +	struct idlist *old;
> 	while (fgets(buf, POPBUFSIZE, tmpfp) != (char *)NULL)
>  	{
> @@ -215,7 +217,8 @@
>  		for (ctl = hostlist; ctl; ctl = ctl->next) {
>  		    if (strcasecmp(host, ctl->server.queryname) == 0
>  			    && strcasecmp(user, ctl->remotename) == 0) {
> -			save_str(&ctl->oldsaved, id, UID_SEEN);
> +			old = save_str(ctl->oldsavedend, id, UID_SEEN);
> +			ctl->oldsavedend = &old->next;
>  			break;
>  		    }
>  		}
> @@ -547,7 +550,9 @@
>  	if (outlevel >= O_DEBUG)
>  	    report(stdout, GT_("swapping UID lists\n"));
>  	ctl->oldsaved = ctl->newsaved;
> +	ctl->oldsavedend = ctl->newsavedend;
>  	ctl->newsaved = (struct idlist *) NULL;
> +	ctl->newsavedend = &ctl->newsaved;
>  	free_str_list(&temp);
>      }
>      /* in fast uidl, there is no need to swap lists: the old state of
> @@ -581,6 +586,7 @@
>  	    report(stdout, GT_("discarding new UID list\n"));
>  	free_str_list(&ctl->newsaved);
>  	ctl->newsaved = (struct idlist *) NULL;
> +	ctl->newsavedend = &ctl->newsaved;
>      }
>  }
> _______________________________________________
> fetchmail-devel mailing list
> fet...@li...
> https://lists.berlios.de/mailman/listinfo/fetchmail-devel


-- 
Matthias Andree

[fetchmail-devel] save_str problems.

[Rainer W. <rwe...@ms...>]

Hello.

The company I work for uses fetchmail to download mails from various
POP- and IMAP-servers in order to offer an anti-spam/ anti-virus
scanning service for smartphone owners. Below is an oprofile excerpt
from the 'download server' regarding fetchmail:

,----
| CPU: Core 2, speed 2393.92 MHz (estimated)
| Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) [...]
| samples  %        image name               symbol name
| 25110    51.5035  fetchmail                save_str
| 11399    23.3806  fetchmail                id_find
| 5851     12.0011  fetchmail                str_in_list
| 912       1.8706  fetchmail                yylex
| 558       1.1445  fetchmail                initialize_saved_lists
| 493       1.0112  fetchmail                do_session
`----

According to this, the program spent most of its time (51.5%) with
executing the save_str-routine (uid.c). This routine appends a new
item to a linked list of structures by traversing the list in order to
find the last node and then adding the new one. It is [mostly] called
in a couple of places in pop3.c to update the 'newsaved' and
'oldsaved' lists of message UIDs (UIDL). The query structure defined
in fetchmail.h actually even contains a member which was supposed to
track the end of the oldsaved list (oldsavedend) but that is only
initialized (in initialize_saved_lists/ uid.c) and never really used
for anything. The patch included below changes the code to actually
track the end of the oldsaved list and introduces a newsavedend struct
query member in order to do the same for the newsaved list. With this
change, the save_str-routines now only accounts for 0.4% of the CPU
time used by the program:

,----
| CPU: Core 2, speed 2393.92 MHz (estimated)
| Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) [...]
| samples  %        image name               symbol name
| 19146    48.1588  fetchmail                id_find
| 9416     23.6845  fetchmail                str_in_list
| 1452      3.6523  fetchmail                yylex
| 860       2.1632  fetchmail                initialize_saved_lists
| 810       2.0374  fetchmail                do_session
| 
| [...]
| 
| 167       0.4201  fetchmail                save_str
| 157       0.3949  fetchmail                parsecmdline
| 145       0.3647  fetchmail                readheaders
`----

I am aware that the 'split append' (save_str + explicit adjustment of
savedend-pointer) isn't exactly beautiful but it is a reasonably
simple change with a significant positive impact for POP3 mailboxes
containing lots of ('kept on server') messages.

----------
--- fetchmail/fetchmail.h	16 Apr 2010 14:45:15 -0000	1.1.1.3
+++ fetchmail/fetchmail.h	21 Apr 2010 16:31:34 -0000	1.1.1.3.2.1
@@ -378,7 +378,7 @@
     unsigned int uid;		/* UID of user to deliver to */
     struct idlist *skipped;	/* messages skipped on the mail server */
     struct idlist *oldsaved, *newsaved;
-    struct idlist **oldsavedend;
+    struct idlist **oldsavedend, **newsavedend;
     char lastdigest[DIGESTLEN];	/* last MD5 hash seen on this connection */
     char *folder;		/* folder currently being polled */
 
--- fetchmail/pop3.c	16 Apr 2010 17:27:18 -0000	1.1.1.3
+++ fetchmail/pop3.c	21 Apr 2010 16:31:34 -0000	1.1.1.3.2.1
@@ -881,8 +881,10 @@
 	    last_nr = try_nr;
 
 	    /* save it */
-	    newl = save_str(&ctl->oldsaved, id, UID_UNSEEN);
+	    newl = save_str(ctl->oldsavedend, id, UID_UNSEEN);
 	    newl->val.status.num = try_nr;
+
+	    ctl->oldsavedend = &newl->next;
 	}
     }
     if (outlevel >= O_DEBUG && last_nr <= count)
@@ -970,15 +972,18 @@
     /* the first try_id messages are known -> copy them to the newsaved list */
     for( num = first_nr; num < list_len; num++ )
     {
-	struct idlist	*newl = save_str(&ctl->newsaved, 
+	struct idlist	*newl = save_str(ctl->newsavedend, 
 				str_from_nr_list(&ctl->oldsaved, num),
 				UID_UNSEEN);
 	newl->val.status.num = num - first_nr + 1;
+
+	ctl->newsavedend = &newl->next;
     }
 
     if( nolinear ) {
 	free_str_list(&ctl->oldsaved);
 	ctl->oldsaved = 0;
+	ctl->oldsavedend = &ctl->oldsaved;
 	last = try_id;
     }
 
@@ -998,6 +1003,7 @@
     (void)folder;
     /* Ensure that the new list is properly empty */
     ctl->newsaved = (struct idlist *)NULL;
+    ctl->newsavedend = &ctl->newsaved;
 
 #ifdef MBOX
     /* Alain Knaff suggests this, but it's not RFC standard */
@@ -1089,9 +1095,11 @@
 		    {
 			struct idlist	*old, *newl;
 
-			newl = save_str(&ctl->newsaved, id, UID_UNSEEN);
+			newl = save_str(ctl->newsavedend, id, UID_UNSEEN);
 			newl->val.status.num = unum;
 
+			ctl->newsavedend = &newl->next;
+
 			if ((old = str_in_list(&ctl->oldsaved, id, FALSE)))
 			{
 			    flag mark = old->val.status.mark;
@@ -1123,7 +1131,8 @@
 			     * swap the lists (say, due to socket error),
 			     * the same mail will not be downloaded again.
 			     */
-			    old = save_str(&ctl->oldsaved, id, UID_UNSEEN);
+			    old = save_str(ctl->oldsavedend, id, UID_UNSEEN);
+			    ctl->oldsavedend = &old->next;
 			}
 			/* save the number */
 			old->val.status.num = unum;
@@ -1224,8 +1233,10 @@
 	}
 
 	/* save it */
-	newl = save_str(&ctl->oldsaved, id, UID_UNSEEN);
+	newl = save_str(ctl->oldsavedend, id, UID_UNSEEN);
 	newl->val.status.num = num;
+
+	ctl->oldsavedend = &newl->next;
 	return(FALSE);
     }
     else
--- fetchmail/transact.c	16 Apr 2010 14:45:18 -0000	1.1.1.2
+++ fetchmail/transact.c	21 Apr 2010 16:31:34 -0000	1.1.1.2.2.1
@@ -882,8 +882,10 @@
  		sscanf(line+12, "%s", id);
  	        if (!str_find( &ctl->newsaved, num))
 		{
- 		    struct idlist *newl = save_str(&ctl->newsaved,id,UID_SEEN);
+ 		    struct idlist *newl = save_str(ctl->newsavedend,id,UID_SEEN);
 		    newl->val.status.num = num;
+
+		    ctl->newsavedend = &newl->next;
 		}
  	    }
  	}
--- fetchmail/uid.c	17 Aug 2009 10:47:46 -0000	1.1.1.1
+++ fetchmail/uid.c	21 Apr 2010 16:31:34 -0000	1.1.1.1.2.1
@@ -119,6 +119,7 @@
 	ctl->oldsaved = (struct idlist *)NULL;
 	ctl->newsaved = (struct idlist *)NULL;
 	ctl->oldsavedend = &ctl->oldsaved;
+	ctl->newsavedend = &ctl->newsaved;
     }
 
     errno = 0;
@@ -151,6 +152,7 @@
 	char saveddelim1;
 	char *delimp2;
 	char saveddelim2 = '\0';	/* pacify -Wall */
+	struct idlist *old;
 
 	while (fgets(buf, POPBUFSIZE, tmpfp) != (char *)NULL)
 	{
@@ -215,7 +217,8 @@
 		for (ctl = hostlist; ctl; ctl = ctl->next) {
 		    if (strcasecmp(host, ctl->server.queryname) == 0
 			    && strcasecmp(user, ctl->remotename) == 0) {
-			save_str(&ctl->oldsaved, id, UID_SEEN);
+			old = save_str(ctl->oldsavedend, id, UID_SEEN);
+			ctl->oldsavedend = &old->next;
 			break;
 		    }
 		}
@@ -547,7 +550,9 @@
 	if (outlevel >= O_DEBUG)
 	    report(stdout, GT_("swapping UID lists\n"));
 	ctl->oldsaved = ctl->newsaved;
+	ctl->oldsavedend = ctl->newsavedend;
 	ctl->newsaved = (struct idlist *) NULL;
+	ctl->newsavedend = &ctl->newsaved;
 	free_str_list(&temp);
     }
     /* in fast uidl, there is no need to swap lists: the old state of
@@ -581,6 +586,7 @@
 	    report(stdout, GT_("discarding new UID list\n"));
 	free_str_list(&ctl->newsaved);
 	ctl->newsaved = (struct idlist *) NULL;
+	ctl->newsavedend = &ctl->newsaved;
     }
 }
 

Re: [fetchmail-devel] "set invisible" bug in fetchmailconf.py

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 22.04.2010 21:40, schrieb Michael Banack:
> The "Enable Invisible?" option in fetchmailconf.py is not saving in the
> configuration file.  It loads it correctly, but when you save it the
> option is dropped.
> 
> There's a 2-line patch that fixes it in my tree on gitorious.
> 
> a7f59a3e23b54aebd143b4be852a8364ed0a503f
> 
> Fixed set invisible bug in fetchmailconf.py

Thanks, I've cherrypicked this patch.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (FreeBSD)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkvQtpIACgkQvmGDOQUufZXSTgCgy19KnU5T1Ygm2J5GTL1GZ9fV
Q9kAoKuNQL3nqE/Og0UKM83IyxLut0Bo
=YxaX
-----END PGP SIGNATURE-----

[fetchmail-devel] "set invisible" bug in fetchmailconf.py

[Michael B. <bo...@gm...>]

The "Enable Invisible?" option in fetchmailconf.py is not saving in the
configuration file.  It loads it correctly, but when you save it the
option is dropped.

There's a 2-line patch that fixes it in my tree on gitorious.

a7f59a3e23b54aebd143b4be852a8364ed0a503f

Fixed set invisible bug in fetchmailconf.py



--Michael Banack

[fetchmail-devel] New Chinese (simplified) PO file for 'fetchmail' (version 6.3.16)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Chinese (simplified) team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/zh_CN.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] [Bug #17073] ssl certificate verification broken in fetchmail 6.3.16

[<ad...@be...>]

Bug #17073, was updated on 2010-Apr-12 16:15
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Closed
Resolution: Invalid
Bug Group: None
Priority: 5
Submitted by: darose
Assigned to : none
Summary: ssl certificate verification broken in fetchmail 6.3.16 

Details: As I indicated in a bug at the Arch Linux distro site (http://bugs.archlinux.org/task/19043) upgrading to FM 6.3.16 broke my mail fetch. Downgrading back to 6.3.14 fixes it.

[darose@darsys12 ~]$ /usr/bin/fetchmail -v
fetchmail: 6.3.16 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:41:11 AM EDT: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: darose.net key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: Server certificate verification error: unable to get local issuer certificate
139713097279144:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1056:
fetchmail: SSL connection failed.
fetchmail: socket error while fetching from da...@da...@darose.net
fetchmail: 6.3.16 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:41:12 AM EDT: poll completed
fetchmail: Query status=2 (SOCKET)
fetchmail: normal termination, status 2


[darose@darsys12 ~]$ /usr/bin/fetchmail -v
fetchmail: 6.3.14 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:40:43 AM EDT: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: darose.net key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: POP3< +OK Hello there.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK Here's what I can do:
fetchmail: POP3< STLS
fetchmail: POP3< TOP
fetchmail: POP3< USER
fetchmail: POP3< LOGIN-DELAY 10
fetchmail: POP3< PIPELINING
fetchmail: POP3< UIDL
fetchmail: POP3< IMPLEMENTATION Courier Mail Server
fetchmail: POP3< .
fetchmail: POP3> USER da...@da...
fetchmail: POP3< +OK Password required.
fetchmail: POP3> PASS *
fetchmail: POP3< +OK logged in.
fetchmail: POP3> STAT
fetchmail: POP3< +OK 0 0
fetchmail: No mail for da...@da... at darose.net
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Bye-bye.
fetchmail: 6.3.14 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:40:44 AM EDT: poll completed
fetchmail: normal termination, status 1


My fetchmailrc is as follows:

set postmaster "root"
set bouncemail
set no spambounce
set properties ""

poll darose.net via <my dreamhost mail server>.mail.dreamhost.com
with proto POP3
user 'da...@da...' there with password '<my password>' is 'darose' here options fetchall ssl sslcertck sslcertpath /home/darose/certs


And the certs directory contains dreamhost's "ndn" (New Dream Network) ssl cert:

[darose@darsys12 .fetchmail]$ ls -l /home/darose/certs
total 16
lrwxrwxrwx 1 darose users 10 Aug 1 2008 17a3f64c.0 -> ndn.ca.pem
lrwxrwxrwx 1 darose users 19 Aug 1 2008 2bbe502b.0 -> mail.darose.net.pem
-rw-r--r-- 1 darose users 1578 Aug 1 2008 mail.darose.net.pem
-rw-r--r-- 1 darose users 2151 Aug 1 2008 ndn.ca.crt
-rw-r--r-- 1 darose users 1546 Aug 1 2008 ndn.ca.der
-rw-r--r-- 1 darose users 2151 Aug 1 2008 ndn.ca.pem

Any ideas? Thanks!

Follow-Ups:

Date: 2010-Apr-12 21:49
By: m-a

Comment:
Submitter confirmed that running "c_rehash home/darose/certs" fixed the issue. <http://bugs.archlinux.org/task/19043#comment60352>
-------------------------------------------------------

Date: 2010-Apr-12 18:54
By: m-a

Comment:
JFTR, debugging this uncovered several critical c_rehash bugs in OpenSSL 1.0.0 (and possibly earlier versions). I've filed a patch with OpenSSL's Request Tracker, but haven't received a request ID yet. The OpenSSL patch has also been filed to the Archlinux "Flyspray" bug tracker URL posted by David (darose@).

-------------------------------------------------------

Date: 2010-Apr-12 18:04
By: m-a

Comment:
No I haven't closed the bug prematurely. fetchmail isn't responsible for how OpenSSL thinks on presented certificates, or how you hash your --sslcertpath.

I have established (in further testing) that the bug is in fact outside fetchmail.

Particularly, OpenSSL 0.9.8 is hashing 17a3f64c.0 -> ndn.ca.pem when OpenSSL 1.0.0 is hashing 05e36882.0 -> ndn.ca.pem.

c_rehash is peculiar, especially if you've got multiple openssl versions installed in parallel. Be sure to run OpenSSL 1.0.0's c_rehash and to have 1.0.0's openssl first in the path - and note that c_rehash will remove the symlinks that the other version's c_rehash had installed. Best to cp -al your certs directory to a new one if you need both OpenSSL versions installed.

I was testing with 0.9.8k and 1.0.0 (both on openSUSE) and 0.9.8n (Cygwin).
-------------------------------------------------------

Date: 2010-Apr-12 17:45
By: darose

Comment:
Egads!  I think you've closed this bug a bit hastily.  We really have not yet established that fetchmail is not the source of the bug here - particularly given the fact that switching from 6.3.14 to 6.3.16 lets me recreate the problem at will.

In answer to your questions:

> Does fetchmail parse your rcfile properly (try fetchmail -Vv)?

Yes.

> What type of file system (ext3, ext4, nfs, afs, cifs, coda, smbfs...) is your /home/darose... on?

XFS

> What OpenSSL version are you using?

openssl 1.0.0
Which version did you use to test with below?

> Are libssl or libcrypto linked statically or dynamically?

I have no idea.  (How would I find this out?)

> Is LD_PRELOAD in effect?

Again, I have no idea.  (Again, how would I find this out?)
-------------------------------------------------------

Date: 2010-Apr-12 17:10
By: m-a

Comment:
Please do not make up hostnames for SSL-related reports, otherwise they may be impossible to debug.  This time, I could guess your hostname with a bit of Googling, and it works for me on openSUSE 11.2 and Cygwin 1.7. 

As pointers:
Does fetchmail parse your rcfile properly (try fetchmail -Vv)?
What type of file system (ext3, ext4, nfs, afs, cifs, coda, smbfs...) is your /home/darose... on? 
What OpenSSL version are you using?  
Are libssl or libcrypto linked statically or dynamically? 
Is LD_PRELOAD in effect?

Here's the trace:

<pre>
$ wget -nv https://dreamhost.com/ca/ndn.ca.crt
2010-04-12 16:58:50 URL:https://dreamhost.com/ca/ndn.ca.crt [2151/2151] -> "ndn.ca.crt" [1]
$ mv ndn.ca.crt ndn.ca.pem
$ c_rehash .
Doing .
ndn.ca.pem => 17a3f64c.0
$ LC_ALL=C build/fetchmail -v --auth external --user nobody --sslcertpath /tmp --sslcertck --ssl -p pop3 a1.balanced.homie.mail.dreamhost.com
fetchmail: 6.3.16 querying a1.balanced.homie.mail.dreamhost.com (protocol POP3) at Mon Apr 12 16:56:56 2010: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: a1.balanced.homie.mail.dreamhost.com key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: POP3< +OK Hello there.
</pre>

Looks ok, we get past the SSL negotiation successfully.
The relevant diff between 6.3.14 and 6.3.16 is in socket.c adds OpenSSL_add_all_algorithms(), but I fail to see how it could possibly trigger an issue here. If it did, then it would be OpenSSL's fault anyways, not fetchmail's.

relevant differences 6.3.14 -> 6.3.16 in socket.c:

<pre>
@@ -819,16 +814,16 @@
-int SSLOpen(int sock, char *mycert, char *mykey, char *myproto, int certck, cha
r *certpath,
+int SSLOpen(int sock, char *mycert, char *mykey, const char *myproto, int certc
k, char *certpath,
     char *fingerprint, char *servercname, char *label, char **remotename)
 {
         struct stat randstat;
         int i;

        SSL_load_error_strings();
-       SSLeay_add_ssl_algorithms(); /* synonym for SSL_library_init() */
-
-#ifdef SSL_ENABLE
+       SSL_library_init();
+       OpenSSL_add_all_algorithms(); /* see Debian Bug#576430 and manpage */
+
         if (stat("/dev/random", &randstat)  &&
             stat("/dev/urandom", &randstat)) {
           /* Neither /dev/random nor /dev/urandom are present, so add
</pre>
-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=17073&group_id=1824

[fetchmail-devel] [Bug #17073] ssl certificate verification broken in fetchmail 6.3.16

[<ad...@be...>]

Bug #17073, was updated on 2010-Apr-12 16:15
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Closed
Resolution: Invalid
Bug Group: None
Priority: 5
Submitted by: darose
Assigned to : none
Summary: ssl certificate verification broken in fetchmail 6.3.16 

Details: As I indicated in a bug at the Arch Linux distro site (http://bugs.archlinux.org/task/19043) upgrading to FM 6.3.16 broke my mail fetch. Downgrading back to 6.3.14 fixes it.

[darose@darsys12 ~]$ /usr/bin/fetchmail -v
fetchmail: 6.3.16 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:41:11 AM EDT: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: darose.net key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: Server certificate verification error: unable to get local issuer certificate
139713097279144:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1056:
fetchmail: SSL connection failed.
fetchmail: socket error while fetching from da...@da...@darose.net
fetchmail: 6.3.16 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:41:12 AM EDT: poll completed
fetchmail: Query status=2 (SOCKET)
fetchmail: normal termination, status 2


[darose@darsys12 ~]$ /usr/bin/fetchmail -v
fetchmail: 6.3.14 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:40:43 AM EDT: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: darose.net key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: POP3< +OK Hello there.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK Here's what I can do:
fetchmail: POP3< STLS
fetchmail: POP3< TOP
fetchmail: POP3< USER
fetchmail: POP3< LOGIN-DELAY 10
fetchmail: POP3< PIPELINING
fetchmail: POP3< UIDL
fetchmail: POP3< IMPLEMENTATION Courier Mail Server
fetchmail: POP3< .
fetchmail: POP3> USER da...@da...
fetchmail: POP3< +OK Password required.
fetchmail: POP3> PASS *
fetchmail: POP3< +OK logged in.
fetchmail: POP3> STAT
fetchmail: POP3< +OK 0 0
fetchmail: No mail for da...@da... at darose.net
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Bye-bye.
fetchmail: 6.3.14 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:40:44 AM EDT: poll completed
fetchmail: normal termination, status 1


My fetchmailrc is as follows:

set postmaster "root"
set bouncemail
set no spambounce
set properties ""

poll darose.net via <my dreamhost mail server>.mail.dreamhost.com
with proto POP3
user 'da...@da...' there with password '<my password>' is 'darose' here options fetchall ssl sslcertck sslcertpath /home/darose/certs


And the certs directory contains dreamhost's "ndn" (New Dream Network) ssl cert:

[darose@darsys12 .fetchmail]$ ls -l /home/darose/certs
total 16
lrwxrwxrwx 1 darose users 10 Aug 1 2008 17a3f64c.0 -> ndn.ca.pem
lrwxrwxrwx 1 darose users 19 Aug 1 2008 2bbe502b.0 -> mail.darose.net.pem
-rw-r--r-- 1 darose users 1578 Aug 1 2008 mail.darose.net.pem
-rw-r--r-- 1 darose users 2151 Aug 1 2008 ndn.ca.crt
-rw-r--r-- 1 darose users 1546 Aug 1 2008 ndn.ca.der
-rw-r--r-- 1 darose users 2151 Aug 1 2008 ndn.ca.pem

Any ideas? Thanks!

Follow-Ups:

Date: 2010-Apr-12 18:54
By: m-a

Comment:
JFTR, debugging this uncovered several critical c_rehash bugs in OpenSSL 1.0.0 (and possibly earlier versions). I've filed a patch with OpenSSL's Request Tracker, but haven't received a request ID yet. The OpenSSL patch has also been filed to the Archlinux "Flyspray" bug tracker URL posted by David (darose@).

-------------------------------------------------------

Date: 2010-Apr-12 18:04
By: m-a

Comment:
No I haven't closed the bug prematurely. fetchmail isn't responsible for how OpenSSL thinks on presented certificates, or how you hash your --sslcertpath.

I have established (in further testing) that the bug is in fact outside fetchmail.

Particularly, OpenSSL 0.9.8 is hashing 17a3f64c.0 -> ndn.ca.pem when OpenSSL 1.0.0 is hashing 05e36882.0 -> ndn.ca.pem.

c_rehash is peculiar, especially if you've got multiple openssl versions installed in parallel. Be sure to run OpenSSL 1.0.0's c_rehash and to have 1.0.0's openssl first in the path - and note that c_rehash will remove the symlinks that the other version's c_rehash had installed. Best to cp -al your certs directory to a new one if you need both OpenSSL versions installed.

I was testing with 0.9.8k and 1.0.0 (both on openSUSE) and 0.9.8n (Cygwin).
-------------------------------------------------------

Date: 2010-Apr-12 17:45
By: darose

Comment:
Egads!  I think you've closed this bug a bit hastily.  We really have not yet established that fetchmail is not the source of the bug here - particularly given the fact that switching from 6.3.14 to 6.3.16 lets me recreate the problem at will.

In answer to your questions:

> Does fetchmail parse your rcfile properly (try fetchmail -Vv)?

Yes.

> What type of file system (ext3, ext4, nfs, afs, cifs, coda, smbfs...) is your /home/darose... on?

XFS

> What OpenSSL version are you using?

openssl 1.0.0
Which version did you use to test with below?

> Are libssl or libcrypto linked statically or dynamically?

I have no idea.  (How would I find this out?)

> Is LD_PRELOAD in effect?

Again, I have no idea.  (Again, how would I find this out?)
-------------------------------------------------------

Date: 2010-Apr-12 17:10
By: m-a

Comment:
Please do not make up hostnames for SSL-related reports, otherwise they may be impossible to debug.  This time, I could guess your hostname with a bit of Googling, and it works for me on openSUSE 11.2 and Cygwin 1.7. 

As pointers:
Does fetchmail parse your rcfile properly (try fetchmail -Vv)?
What type of file system (ext3, ext4, nfs, afs, cifs, coda, smbfs...) is your /home/darose... on? 
What OpenSSL version are you using?  
Are libssl or libcrypto linked statically or dynamically? 
Is LD_PRELOAD in effect?

Here's the trace:

<pre>
$ wget -nv https://dreamhost.com/ca/ndn.ca.crt
2010-04-12 16:58:50 URL:https://dreamhost.com/ca/ndn.ca.crt [2151/2151] -> "ndn.ca.crt" [1]
$ mv ndn.ca.crt ndn.ca.pem
$ c_rehash .
Doing .
ndn.ca.pem => 17a3f64c.0
$ LC_ALL=C build/fetchmail -v --auth external --user nobody --sslcertpath /tmp --sslcertck --ssl -p pop3 a1.balanced.homie.mail.dreamhost.com
fetchmail: 6.3.16 querying a1.balanced.homie.mail.dreamhost.com (protocol POP3) at Mon Apr 12 16:56:56 2010: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: a1.balanced.homie.mail.dreamhost.com key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: POP3< +OK Hello there.
</pre>

Looks ok, we get past the SSL negotiation successfully.
The relevant diff between 6.3.14 and 6.3.16 is in socket.c adds OpenSSL_add_all_algorithms(), but I fail to see how it could possibly trigger an issue here. If it did, then it would be OpenSSL's fault anyways, not fetchmail's.

relevant differences 6.3.14 -> 6.3.16 in socket.c:

<pre>
@@ -819,16 +814,16 @@
-int SSLOpen(int sock, char *mycert, char *mykey, char *myproto, int certck, cha
r *certpath,
+int SSLOpen(int sock, char *mycert, char *mykey, const char *myproto, int certc
k, char *certpath,
     char *fingerprint, char *servercname, char *label, char **remotename)
 {
         struct stat randstat;
         int i;

        SSL_load_error_strings();
-       SSLeay_add_ssl_algorithms(); /* synonym for SSL_library_init() */
-
-#ifdef SSL_ENABLE
+       SSL_library_init();
+       OpenSSL_add_all_algorithms(); /* see Debian Bug#576430 and manpage */
+
         if (stat("/dev/random", &randstat)  &&
             stat("/dev/urandom", &randstat)) {
           /* Neither /dev/random nor /dev/urandom are present, so add
</pre>
-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=17073&group_id=1824

[fetchmail-devel] [Bug #17073] ssl certificate verification broken in fetchmail 6.3.16

[<ad...@be...>]

Bug #17073, was updated on 2010-Apr-12 16:15
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Closed
Resolution: Invalid
Bug Group: None
Priority: 5
Submitted by: darose
Assigned to : none
Summary: ssl certificate verification broken in fetchmail 6.3.16 

Details: As I indicated in a bug at the Arch Linux distro site (http://bugs.archlinux.org/task/19043) upgrading to FM 6.3.16 broke my mail fetch. Downgrading back to 6.3.14 fixes it.

[darose@darsys12 ~]$ /usr/bin/fetchmail -v
fetchmail: 6.3.16 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:41:11 AM EDT: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: darose.net key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: Server certificate verification error: unable to get local issuer certificate
139713097279144:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1056:
fetchmail: SSL connection failed.
fetchmail: socket error while fetching from da...@da...@darose.net
fetchmail: 6.3.16 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:41:12 AM EDT: poll completed
fetchmail: Query status=2 (SOCKET)
fetchmail: normal termination, status 2


[darose@darsys12 ~]$ /usr/bin/fetchmail -v
fetchmail: 6.3.14 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:40:43 AM EDT: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: darose.net key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: POP3< +OK Hello there.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK Here's what I can do:
fetchmail: POP3< STLS
fetchmail: POP3< TOP
fetchmail: POP3< USER
fetchmail: POP3< LOGIN-DELAY 10
fetchmail: POP3< PIPELINING
fetchmail: POP3< UIDL
fetchmail: POP3< IMPLEMENTATION Courier Mail Server
fetchmail: POP3< .
fetchmail: POP3> USER da...@da...
fetchmail: POP3< +OK Password required.
fetchmail: POP3> PASS *
fetchmail: POP3< +OK logged in.
fetchmail: POP3> STAT
fetchmail: POP3< +OK 0 0
fetchmail: No mail for da...@da... at darose.net
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Bye-bye.
fetchmail: 6.3.14 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:40:44 AM EDT: poll completed
fetchmail: normal termination, status 1


My fetchmailrc is as follows:

set postmaster "root"
set bouncemail
set no spambounce
set properties ""

poll darose.net via <my dreamhost mail server>.mail.dreamhost.com
with proto POP3
user 'da...@da...' there with password '<my password>' is 'darose' here options fetchall ssl sslcertck sslcertpath /home/darose/certs


And the certs directory contains dreamhost's "ndn" (New Dream Network) ssl cert:

[darose@darsys12 .fetchmail]$ ls -l /home/darose/certs
total 16
lrwxrwxrwx 1 darose users 10 Aug 1 2008 17a3f64c.0 -> ndn.ca.pem
lrwxrwxrwx 1 darose users 19 Aug 1 2008 2bbe502b.0 -> mail.darose.net.pem
-rw-r--r-- 1 darose users 1578 Aug 1 2008 mail.darose.net.pem
-rw-r--r-- 1 darose users 2151 Aug 1 2008 ndn.ca.crt
-rw-r--r-- 1 darose users 1546 Aug 1 2008 ndn.ca.der
-rw-r--r-- 1 darose users 2151 Aug 1 2008 ndn.ca.pem

Any ideas? Thanks!

Follow-Ups:

Date: 2010-Apr-12 18:04
By: m-a

Comment:
No I haven't closed the bug prematurely. fetchmail isn't responsible for how OpenSSL thinks on presented certificates, or how you hash your --sslcertpath.

I have established (in further testing) that the bug is in fact outside fetchmail.

Particularly, OpenSSL 0.9.8 is hashing 17a3f64c.0 -> ndn.ca.pem when OpenSSL 1.0.0 is hashing 05e36882.0 -> ndn.ca.pem.

c_rehash is peculiar, especially if you've got multiple openssl versions installed in parallel. Be sure to run OpenSSL 1.0.0's c_rehash and to have 1.0.0's openssl first in the path - and note that c_rehash will remove the symlinks that the other version's c_rehash had installed. Best to cp -al your certs directory to a new one if you need both OpenSSL versions installed.

I was testing with 0.9.8k and 1.0.0 (both on openSUSE) and 0.9.8n (Cygwin).
-------------------------------------------------------

Date: 2010-Apr-12 17:45
By: darose

Comment:
Egads!  I think you've closed this bug a bit hastily.  We really have not yet established that fetchmail is not the source of the bug here - particularly given the fact that switching from 6.3.14 to 6.3.16 lets me recreate the problem at will.

In answer to your questions:

> Does fetchmail parse your rcfile properly (try fetchmail -Vv)?

Yes.

> What type of file system (ext3, ext4, nfs, afs, cifs, coda, smbfs...) is your /home/darose... on?

XFS

> What OpenSSL version are you using?

openssl 1.0.0
Which version did you use to test with below?

> Are libssl or libcrypto linked statically or dynamically?

I have no idea.  (How would I find this out?)

> Is LD_PRELOAD in effect?

Again, I have no idea.  (Again, how would I find this out?)
-------------------------------------------------------

Date: 2010-Apr-12 17:10
By: m-a

Comment:
Please do not make up hostnames for SSL-related reports, otherwise they may be impossible to debug.  This time, I could guess your hostname with a bit of Googling, and it works for me on openSUSE 11.2 and Cygwin 1.7. 

As pointers:
Does fetchmail parse your rcfile properly (try fetchmail -Vv)?
What type of file system (ext3, ext4, nfs, afs, cifs, coda, smbfs...) is your /home/darose... on? 
What OpenSSL version are you using?  
Are libssl or libcrypto linked statically or dynamically? 
Is LD_PRELOAD in effect?

Here's the trace:

<pre>
$ wget -nv https://dreamhost.com/ca/ndn.ca.crt
2010-04-12 16:58:50 URL:https://dreamhost.com/ca/ndn.ca.crt [2151/2151] -> "ndn.ca.crt" [1]
$ mv ndn.ca.crt ndn.ca.pem
$ c_rehash .
Doing .
ndn.ca.pem => 17a3f64c.0
$ LC_ALL=C build/fetchmail -v --auth external --user nobody --sslcertpath /tmp --sslcertck --ssl -p pop3 a1.balanced.homie.mail.dreamhost.com
fetchmail: 6.3.16 querying a1.balanced.homie.mail.dreamhost.com (protocol POP3) at Mon Apr 12 16:56:56 2010: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: a1.balanced.homie.mail.dreamhost.com key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: POP3< +OK Hello there.
</pre>

Looks ok, we get past the SSL negotiation successfully.
The relevant diff between 6.3.14 and 6.3.16 is in socket.c adds OpenSSL_add_all_algorithms(), but I fail to see how it could possibly trigger an issue here. If it did, then it would be OpenSSL's fault anyways, not fetchmail's.

relevant differences 6.3.14 -> 6.3.16 in socket.c:

<pre>
@@ -819,16 +814,16 @@
-int SSLOpen(int sock, char *mycert, char *mykey, char *myproto, int certck, cha
r *certpath,
+int SSLOpen(int sock, char *mycert, char *mykey, const char *myproto, int certc
k, char *certpath,
     char *fingerprint, char *servercname, char *label, char **remotename)
 {
         struct stat randstat;
         int i;

        SSL_load_error_strings();
-       SSLeay_add_ssl_algorithms(); /* synonym for SSL_library_init() */
-
-#ifdef SSL_ENABLE
+       SSL_library_init();
+       OpenSSL_add_all_algorithms(); /* see Debian Bug#576430 and manpage */
+
         if (stat("/dev/random", &randstat)  &&
             stat("/dev/urandom", &randstat)) {
           /* Neither /dev/random nor /dev/urandom are present, so add
</pre>
-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=17073&group_id=1824

[fetchmail-devel] [Bug #17073] ssl certificate verification broken in fetchmail 6.3.16

[<ad...@be...>]

Bug #17073, was updated on 2010-Apr-12 10:15
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Closed
Resolution: Works For Me
Bug Group: None
Priority: 5
Submitted by: darose
Assigned to : none
Summary: ssl certificate verification broken in fetchmail 6.3.16 

Details: As I indicated in a bug at the Arch Linux distro site (http://bugs.archlinux.org/task/19043) upgrading to FM 6.3.16 broke my mail fetch. Downgrading back to 6.3.14 fixes it.

[darose@darsys12 ~]$ /usr/bin/fetchmail -v
fetchmail: 6.3.16 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:41:11 AM EDT: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: darose.net key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: Server certificate verification error: unable to get local issuer certificate
139713097279144:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1056:
fetchmail: SSL connection failed.
fetchmail: socket error while fetching from da...@da...@darose.net
fetchmail: 6.3.16 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:41:12 AM EDT: poll completed
fetchmail: Query status=2 (SOCKET)
fetchmail: normal termination, status 2


[darose@darsys12 ~]$ /usr/bin/fetchmail -v
fetchmail: 6.3.14 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:40:43 AM EDT: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: darose.net key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: POP3< +OK Hello there.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK Here's what I can do:
fetchmail: POP3< STLS
fetchmail: POP3< TOP
fetchmail: POP3< USER
fetchmail: POP3< LOGIN-DELAY 10
fetchmail: POP3< PIPELINING
fetchmail: POP3< UIDL
fetchmail: POP3< IMPLEMENTATION Courier Mail Server
fetchmail: POP3< .
fetchmail: POP3> USER da...@da...
fetchmail: POP3< +OK Password required.
fetchmail: POP3> PASS *
fetchmail: POP3< +OK logged in.
fetchmail: POP3> STAT
fetchmail: POP3< +OK 0 0
fetchmail: No mail for da...@da... at darose.net
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Bye-bye.
fetchmail: 6.3.14 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:40:44 AM EDT: poll completed
fetchmail: normal termination, status 1


My fetchmailrc is as follows:

set postmaster "root"
set bouncemail
set no spambounce
set properties ""

poll darose.net via <my dreamhost mail server>.mail.dreamhost.com
with proto POP3
user 'da...@da...' there with password '<my password>' is 'darose' here options fetchall ssl sslcertck sslcertpath /home/darose/certs


And the certs directory contains dreamhost's "ndn" (New Dream Network) ssl cert:

[darose@darsys12 .fetchmail]$ ls -l /home/darose/certs
total 16
lrwxrwxrwx 1 darose users 10 Aug 1 2008 17a3f64c.0 -> ndn.ca.pem
lrwxrwxrwx 1 darose users 19 Aug 1 2008 2bbe502b.0 -> mail.darose.net.pem
-rw-r--r-- 1 darose users 1578 Aug 1 2008 mail.darose.net.pem
-rw-r--r-- 1 darose users 2151 Aug 1 2008 ndn.ca.crt
-rw-r--r-- 1 darose users 1546 Aug 1 2008 ndn.ca.der
-rw-r--r-- 1 darose users 2151 Aug 1 2008 ndn.ca.pem

Any ideas? Thanks!

Follow-Ups:

Date: 2010-Apr-12 11:45
By: darose

Comment:
Egads!  I think you've closed this bug a bit hastily.  We really have not yet established that fetchmail is not the source of the bug here - particularly given the fact that switching from 6.3.14 to 6.3.16 lets me recreate the problem at will.

In answer to your questions:

> Does fetchmail parse your rcfile properly (try fetchmail -Vv)?

Yes.

> What type of file system (ext3, ext4, nfs, afs, cifs, coda, smbfs...) is your /home/darose... on?

XFS

> What OpenSSL version are you using?

openssl 1.0.0
Which version did you use to test with below?

> Are libssl or libcrypto linked statically or dynamically?

I have no idea.  (How would I find this out?)

> Is LD_PRELOAD in effect?

Again, I have no idea.  (Again, how would I find this out?)
-------------------------------------------------------

Date: 2010-Apr-12 11:10
By: m-a

Comment:
Please do not make up hostnames for SSL-related reports, otherwise they may be impossible to debug.  This time, I could guess your hostname with a bit of Googling, and it works for me on openSUSE 11.2 and Cygwin 1.7. 

As pointers:
Does fetchmail parse your rcfile properly (try fetchmail -Vv)?
What type of file system (ext3, ext4, nfs, afs, cifs, coda, smbfs...) is your /home/darose... on? 
What OpenSSL version are you using?  
Are libssl or libcrypto linked statically or dynamically? 
Is LD_PRELOAD in effect?

Here's the trace:

<pre>
$ wget -nv https://dreamhost.com/ca/ndn.ca.crt
2010-04-12 16:58:50 URL:https://dreamhost.com/ca/ndn.ca.crt [2151/2151] -> "ndn.ca.crt" [1]
$ mv ndn.ca.crt ndn.ca.pem
$ c_rehash .
Doing .
ndn.ca.pem => 17a3f64c.0
$ LC_ALL=C build/fetchmail -v --auth external --user nobody --sslcertpath /tmp --sslcertck --ssl -p pop3 a1.balanced.homie.mail.dreamhost.com
fetchmail: 6.3.16 querying a1.balanced.homie.mail.dreamhost.com (protocol POP3) at Mon Apr 12 16:56:56 2010: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: a1.balanced.homie.mail.dreamhost.com key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: POP3< +OK Hello there.
</pre>

Looks ok, we get past the SSL negotiation successfully.
The relevant diff between 6.3.14 and 6.3.16 is in socket.c adds OpenSSL_add_all_algorithms(), but I fail to see how it could possibly trigger an issue here. If it did, then it would be OpenSSL's fault anyways, not fetchmail's.

relevant differences 6.3.14 -> 6.3.16 in socket.c:

<pre>
@@ -819,16 +814,16 @@
-int SSLOpen(int sock, char *mycert, char *mykey, char *myproto, int certck, cha
r *certpath,
+int SSLOpen(int sock, char *mycert, char *mykey, const char *myproto, int certc
k, char *certpath,
     char *fingerprint, char *servercname, char *label, char **remotename)
 {
         struct stat randstat;
         int i;

        SSL_load_error_strings();
-       SSLeay_add_ssl_algorithms(); /* synonym for SSL_library_init() */
-
-#ifdef SSL_ENABLE
+       SSL_library_init();
+       OpenSSL_add_all_algorithms(); /* see Debian Bug#576430 and manpage */
+
         if (stat("/dev/random", &randstat)  &&
             stat("/dev/urandom", &randstat)) {
           /* Neither /dev/random nor /dev/urandom are present, so add
</pre>
-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=17073&group_id=1824

[fetchmail-devel] [Bug #17073] ssl certificate verification broken in fetchmail 6.3.16

[<ad...@be...>]

Bug #17073, was updated on 2010-Apr-12 16:15
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Closed
Resolution: Works For Me
Bug Group: None
Priority: 5
Submitted by: darose
Assigned to : none
Summary: ssl certificate verification broken in fetchmail 6.3.16 

Details: As I indicated in a bug at the Arch Linux distro site (http://bugs.archlinux.org/task/19043) upgrading to FM 6.3.16 broke my mail fetch. Downgrading back to 6.3.14 fixes it.

[darose@darsys12 ~]$ /usr/bin/fetchmail -v
fetchmail: 6.3.16 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:41:11 AM EDT: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: darose.net key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: Server certificate verification error: unable to get local issuer certificate
139713097279144:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1056:
fetchmail: SSL connection failed.
fetchmail: socket error while fetching from da...@da...@darose.net
fetchmail: 6.3.16 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:41:12 AM EDT: poll completed
fetchmail: Query status=2 (SOCKET)
fetchmail: normal termination, status 2


[darose@darsys12 ~]$ /usr/bin/fetchmail -v
fetchmail: 6.3.14 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:40:43 AM EDT: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: darose.net key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: POP3< +OK Hello there.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK Here's what I can do:
fetchmail: POP3< STLS
fetchmail: POP3< TOP
fetchmail: POP3< USER
fetchmail: POP3< LOGIN-DELAY 10
fetchmail: POP3< PIPELINING
fetchmail: POP3< UIDL
fetchmail: POP3< IMPLEMENTATION Courier Mail Server
fetchmail: POP3< .
fetchmail: POP3> USER da...@da...
fetchmail: POP3< +OK Password required.
fetchmail: POP3> PASS *
fetchmail: POP3< +OK logged in.
fetchmail: POP3> STAT
fetchmail: POP3< +OK 0 0
fetchmail: No mail for da...@da... at darose.net
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Bye-bye.
fetchmail: 6.3.14 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:40:44 AM EDT: poll completed
fetchmail: normal termination, status 1


My fetchmailrc is as follows:

set postmaster "root"
set bouncemail
set no spambounce
set properties ""

poll darose.net via <my dreamhost mail server>.mail.dreamhost.com
with proto POP3
user 'da...@da...' there with password '<my password>' is 'darose' here options fetchall ssl sslcertck sslcertpath /home/darose/certs


And the certs directory contains dreamhost's "ndn" (New Dream Network) ssl cert:

[darose@darsys12 .fetchmail]$ ls -l /home/darose/certs
total 16
lrwxrwxrwx 1 darose users 10 Aug 1 2008 17a3f64c.0 -> ndn.ca.pem
lrwxrwxrwx 1 darose users 19 Aug 1 2008 2bbe502b.0 -> mail.darose.net.pem
-rw-r--r-- 1 darose users 1578 Aug 1 2008 mail.darose.net.pem
-rw-r--r-- 1 darose users 2151 Aug 1 2008 ndn.ca.crt
-rw-r--r-- 1 darose users 1546 Aug 1 2008 ndn.ca.der
-rw-r--r-- 1 darose users 2151 Aug 1 2008 ndn.ca.pem

Any ideas? Thanks!

Follow-Ups:

Date: 2010-Apr-12 17:10
By: m-a

Comment:
Please do not make up hostnames for SSL-related reports, otherwise they may be impossible to debug.  This time, I could guess your hostname with a bit of Googling, and it works for me on openSUSE 11.2 and Cygwin 1.7. 

As pointers:
Does fetchmail parse your rcfile properly (try fetchmail -Vv)?
What type of file system (ext3, ext4, nfs, afs, cifs, coda, smbfs...) is your /home/darose... on? 
What OpenSSL version are you using?  
Are libssl or libcrypto linked statically or dynamically? 
Is LD_PRELOAD in effect?

Here's the trace:

<pre>
$ wget -nv https://dreamhost.com/ca/ndn.ca.crt
2010-04-12 16:58:50 URL:https://dreamhost.com/ca/ndn.ca.crt [2151/2151] -> "ndn.ca.crt" [1]
$ mv ndn.ca.crt ndn.ca.pem
$ c_rehash .
Doing .
ndn.ca.pem => 17a3f64c.0
$ LC_ALL=C build/fetchmail -v --auth external --user nobody --sslcertpath /tmp --sslcertck --ssl -p pop3 a1.balanced.homie.mail.dreamhost.com
fetchmail: 6.3.16 querying a1.balanced.homie.mail.dreamhost.com (protocol POP3) at Mon Apr 12 16:56:56 2010: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: a1.balanced.homie.mail.dreamhost.com key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: POP3< +OK Hello there.
</pre>

Looks ok, we get past the SSL negotiation successfully.
The relevant diff between 6.3.14 and 6.3.16 is in socket.c adds OpenSSL_add_all_algorithms(), but I fail to see how it could possibly trigger an issue here. If it did, then it would be OpenSSL's fault anyways, not fetchmail's.

relevant differences 6.3.14 -> 6.3.16 in socket.c:

<pre>
@@ -819,16 +814,16 @@
-int SSLOpen(int sock, char *mycert, char *mykey, char *myproto, int certck, cha
r *certpath,
+int SSLOpen(int sock, char *mycert, char *mykey, const char *myproto, int certc
k, char *certpath,
     char *fingerprint, char *servercname, char *label, char **remotename)
 {
         struct stat randstat;
         int i;

        SSL_load_error_strings();
-       SSLeay_add_ssl_algorithms(); /* synonym for SSL_library_init() */
-
-#ifdef SSL_ENABLE
+       SSL_library_init();
+       OpenSSL_add_all_algorithms(); /* see Debian Bug#576430 and manpage */
+
         if (stat("/dev/random", &randstat)  &&
             stat("/dev/urandom", &randstat)) {
           /* Neither /dev/random nor /dev/urandom are present, so add
</pre>
-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=17073&group_id=1824

[fetchmail-devel] [Bug #17073] ssl certificate verification broken in fetchmail 6.3.16

[<ad...@be...>]

Bug #17073, was updated on 2010-Apr-12 10:15
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Open
Resolution: None
Bug Group: None
Priority: 5
Submitted by: darose
Assigned to : none
Summary: ssl certificate verification broken in fetchmail 6.3.16 

Details: As I indicated in a bug at the Arch Linux distro site (http://bugs.archlinux.org/task/19043) upgrading to FM 6.3.16 broke my mail fetch. Downgrading back to 6.3.14 fixes it.

[darose@darsys12 ~]$ /usr/bin/fetchmail -v
fetchmail: 6.3.16 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:41:11 AM EDT: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: darose.net key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: Server certificate verification error: unable to get local issuer certificate
139713097279144:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1056:
fetchmail: SSL connection failed.
fetchmail: socket error while fetching from da...@da...@darose.net
fetchmail: 6.3.16 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:41:12 AM EDT: poll completed
fetchmail: Query status=2 (SOCKET)
fetchmail: normal termination, status 2


[darose@darsys12 ~]$ /usr/bin/fetchmail -v
fetchmail: 6.3.14 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:40:43 AM EDT: poll started
Trying to connect to 208.97.132.208/995...connected.
fetchmail: Issuer Organization: New Dream Network, LLC
fetchmail: Issuer CommonName: New Dream Network Certificate Authority
fetchmail: Server CommonName: *.mail.dreamhost.com
fetchmail: darose.net key fingerprint: 17:F7:F2:FF:4A:9D:C3:D3:2B:8A:E9:12:47:C4:A4:28
fetchmail: POP3< +OK Hello there.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK Here's what I can do:
fetchmail: POP3< STLS
fetchmail: POP3< TOP
fetchmail: POP3< USER
fetchmail: POP3< LOGIN-DELAY 10
fetchmail: POP3< PIPELINING
fetchmail: POP3< UIDL
fetchmail: POP3< IMPLEMENTATION Courier Mail Server
fetchmail: POP3< .
fetchmail: POP3> USER da...@da...
fetchmail: POP3< +OK Password required.
fetchmail: POP3> PASS *
fetchmail: POP3< +OK logged in.
fetchmail: POP3> STAT
fetchmail: POP3< +OK 0 0
fetchmail: No mail for da...@da... at darose.net
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Bye-bye.
fetchmail: 6.3.14 querying darose.net (protocol POP3) at Sun 11 Apr 2010 10:40:44 AM EDT: poll completed
fetchmail: normal termination, status 1


My fetchmailrc is as follows:

set postmaster "root"
set bouncemail
set no spambounce
set properties ""

poll darose.net via <my dreamhost mail server>.mail.dreamhost.com
with proto POP3
user 'da...@da...' there with password '<my password>' is 'darose' here options fetchall ssl sslcertck sslcertpath /home/darose/certs


And the certs directory contains dreamhost's "ndn" (New Dream Network) ssl cert:

[darose@darsys12 .fetchmail]$ ls -l /home/darose/certs
total 16
lrwxrwxrwx 1 darose users 10 Aug 1 2008 17a3f64c.0 -> ndn.ca.pem
lrwxrwxrwx 1 darose users 19 Aug 1 2008 2bbe502b.0 -> mail.darose.net.pem
-rw-r--r-- 1 darose users 1578 Aug 1 2008 mail.darose.net.pem
-rw-r--r-- 1 darose users 2151 Aug 1 2008 ndn.ca.crt
-rw-r--r-- 1 darose users 1546 Aug 1 2008 ndn.ca.der
-rw-r--r-- 1 darose users 2151 Aug 1 2008 ndn.ca.pem

Any ideas? Thanks!

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=17073&group_id=1824

[fetchmail-devel] New Slovak PO file for 'fetchmail' (version 6.3.16)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Slovak team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/sk.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] [Bug #14257] Incorrect parsing of fetchmailrc

[<ad...@be...>]

Bug #14257, was updated on 2008-Jul-23 10:07
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Closed
Resolution: Fixed
Bug Group: None
Priority: 5
Submitted by: juanvaqueroponc
Assigned to : m-a
Summary: Incorrect parsing of fetchmailrc

Details: The username in question is being parsed as 'sername "aaaaaaaaaaaaa@aaaaaaaaaaaaa'

It is quite difficult to trigger this bug.
If I change some lines the bug does not trigger.
I have got to the attached file that triggers the bug, but
if I remove the last line the same bug seems not to trigger.


Follow-Ups:

Date: 2010-Apr-10 12:13
By: bob5972

Comment:
I agree that using xstrdup would probably be better.  The parser just copies the string when it gets it anyway.  I just didn't want to change the contract between the lexer and the parser if I wasn't sure what it was and who would free it.  I've got some time though, so I'll go ahead and check it out if you want.
-------------------------------------------------------

Date: 2010-Apr-10 08:32
By: m-a

Comment:
Turns out that this bug was a regression I'd introduced in March 2005 when trying to fix memory leaks - but I fixed them the wrong way.

I've committed michael's change as http://gitorious.org/fetchmail/fetchmail/commit/1c6fb40e79a1344c35dcc69b292f9b6913cd4e95 - the fix is to appear in a (not-so-distant) future fetchmail 6.3.17 release.

Could I ask you a favour and propose that you either mail patches to the fetchmail-devel@ mailing list or upload them using BerliOS's patch tracker? Much easier to merge them for me that way.

I can also merge from a private branch of yours in Gitorious if you like.
-------------------------------------------------------

Date: 2010-Apr-10 07:47
By: m-a

Comment:
Michael, that's just awesome - thanks for the pointer to the bug!

The bug is that the .sval union member must contain a copy from the string. The lexer may have to read ahead, and this can then cause yytext() to change. It's somewhat documented, so I think it's not masking a deeper issue.

I wonder if transferring the string in an xstrdup() like manner would be better here. I guess I'm signed up to validating all sval uses. :-)

-------------------------------------------------------

Date: 2010-Apr-09 18:02
By: bob5972

Comment:
I don't know if this is the right way to fix this, but it works.  The contents of yytext are changing between when the lexer returns yylval.sval and when the parser reads it.  I don't know why that would be happening, so this might be masking a deeper issue.  It may need to be tested further, but it does fix the problem.

This is extremely finicky though.  A clearer test case that triggers it is here http://www.ucf.ics.uci.edu/~bob5972/fetchmailrc.bug (It's basically his test case with the names labeled something other than aaaa).


>From 46a52eac0c1705339642bbe5c277e6bac11235ad Mon Sep 17 00:00:00 2001
From: Michael Banack <bo...@gm...>
Date: Fri, 9 Apr 2010 17:56:05 -0700
Subject: [PATCH] Changed the lexer to copy string values out of yytext.

Fixes bug #14257.
---
 rcfile_l.l |   54 +++++++++++++++++++++++++++++++++++++++++-------------
 1 files changed, 41 insertions(+), 13 deletions(-)

diff --git a/rcfile_l.l b/rcfile_l.l
index 8dcd8af..384f8df 100644
--- a/rcfile_l.l
+++ b/rcfile_l.l
@@ -34,18 +34,36 @@ int prc_lineno = 1;
 %%
 
 \"[^\"]*\"	{
-			yytext[yyleng-1] = '\0';
-			escapes(yytext+1, yytext);
-			yyleng = strlen(yytext);
-			yylval.sval = yytext;
+			static char *in = NULL;
+			static size_t ins = 0;
+			
+			if ((size_t)yyleng + 1 > ins) {
+			    ins = yyleng + 1;
+			    in = (char *)xrealloc(in, ins);
+			}
+			memcpy(in, yytext, yyleng);
+			
+			in[yyleng-1] = '\0';
+			escapes(in+1, in);
+			yyleng = strlen(in);
+			yylval.sval = in;
                         SETSTATE(0);
 			return STRING;
 		}
 \'[^\']*\'	{
-			yytext[yyleng-1] = '\0';
-			escapes(yytext+1, yytext);
-			yyleng = strlen(yytext);
-			yylval.sval = yytext;
+			static char *in = NULL;
+			static size_t ins = 0;
+			
+			if ((size_t)yyleng + 1 > ins) {
+			    ins = yyleng + 1;
+			    in = (char *)xrealloc(in, ins);
+			}
+			memcpy(in, yytext, yyleng);
+			
+			in[yyleng-1] = '\0';
+			escapes(in+1, in);
+			yyleng = strlen(in);
+			yylval.sval = in;
                         SETSTATE(0);
 			return STRING;
 		}
@@ -53,8 +71,8 @@ int prc_lineno = 1;
 "*"		{ BEGIN(0); return WILDCARD; }
 
 <NAME>[^=;:, \t\r\n]+	{
-			static char *in;
-			static size_t ins;
+			static char *in = NULL;
+			static size_t ins = 0;
 
 			if ((size_t)yyleng + 1 > ins) {
 			    ins = yyleng + 1;
@@ -222,9 +240,19 @@ options		{/* EMPTY */}
 -?[0-9]+	{ yylval.number = atoi(yytext); return NUMBER; }
 
 [^=;:, \t\r\n]+	{
-			escapes(yytext, yytext);
-			yyleng = strlen(yytext);
-			yylval.sval = yytext;
+			static char *in = NULL;
+			static size_t ins = 0;
+
+			if ((size_t)yyleng + 1 > ins) {
+			    ins = yyleng + 1;
+			    in = (char *)xrealloc(in, ins);
+			}
+			memcpy(in, yytext, yyleng);
+			in[yyleng] = '\0';
+			
+			escapes(in, in);
+			yyleng = strlen(in);
+			yylval.sval = in;
 			return STRING;
 		}
 
-- 
1.7.0.4


-------------------------------------------------------

Date: 2009-Apr-23 06:09
By: m-a

Comment:
Is there any improvement when compiling things with a recent version of flex?

It's just stabbing into the dark, but perhaps it helps.
-------------------------------------------------------

Date: 2008-Jul-23 10:11
By: juanvaqueroponc

Comment:
begin-base64 600 fetchmailrc.gz
H4sIAAAAAAACA+2dzW7aQBDH73mKFagoOezuoeohVin0UqmHqnkDtE2IiuQE
hEl767MXDLb3Y5xgs3ic5E+kgI1hfp6dr501cDe/N0/pJhMPd0YMVuvl7YNZ
pELeiw/fBuJ+vrn9bdL0YrVMU2Hsm1gt1xtxff1JbF+0Wd4uU3Hz8+ajeMrm
60fzMBcD490GYmWy7O9yfVc9NxBZlopsvhGLxz+LbPErnYvhTlBSSrsY+sJf
kNhIjtoKCE+OFPb9x9ebtsLCk3j2jdu/64vaoW+BQEoarad2Cop9GseIoYz4
NNsN7Sm32BME1YqrlbY3YiNG5Y6p817ek3ucw7bK/0v7yMMBJtq5NFKehdCx
5EoNjrYUA4kiGOztzjVzsBMGbViC7U1Gkkiy2+k/sAVV4+Iqqg83Nt5SzO5f
onVAq6lDPBW7xyQ8IUkV1ndZvfjK0TULDpfpM0ZFLlVznCiDXHWGuNHixOsq
lnDndsdleMhVt9B5ZEq8uopk37HOiPBRnIbsnF0Rip0RJRCzRTCaJKs3SLuW
twcpeMJSlLS3OlcWm67iDVQz189vCUeh5VdMecFke7DREQahgTb2xnmWqW99
eKjcQsUzeh7DayrascGO1U5k6FhKaO8Bu4d6N2sYRa+jGignoLJ9s3TN4OmJ
MePt3Wi/Z+y1wmIGmaYKrp+VdRvkC8XK87CcYHc9xIk+UKe5ZZRcFE8nRUNC
VUGDlS+Ghk5q0SSW3Kmp6+MQh7EWyHzCldUu5ig9Q/uO4mQtABR3WhB+ASL4
UpM1FspN8tpP/Jq5nVb1ZP0KijdSK6uAk8FiluIe2pgJ7TiIYdhHV3Q6U92n
2/PnrO5CmiD6Xt4sy3owctOm87IyNSbMpXLP/JqToJqCacHc1aQL5Ggtm9aT
1bhZnLOQ63vnkLVuDvxCk20JhoUvN791nWi9qojUkKUgtmgmw6jh4OXcQzfG
iX6EXv4coOkm3KEDl3feeOpwFaoq5JRMM906e+tDHiWmwbUzLeniBkWdZzLW
aGj+YF3RVNY6O7SK+dZn/W6RoCtlxumt33/XB0NwW4GWjv/5R8r9rPQcFt/y
jBRl4LoycR3UGNM+NBuO9VTZk8qoTBkyjHrq4IO5C+ZrrqNy9XXsbuZ7VPcL
8tbFAMwjXp/XJC+hIlfoLHb2CRmRxibGWMuE9S8pVhNfPrAHU5ECqIgJMxNz
EJrktMQPQ9N4QfOkxmM5ZgzrjNY1B4b8uMbxgjvo+rW4zGzqhX37qhL+pS4C
EAhAAAIQ6MAVTsSgneIK1QQ6AQIQgACE5xGG/YjXVWMR44IcBgQgAOFdIPQs
TiJzwEiBgMoKwwKEN40wnhSrX9JMtn/VxxFn1qL2ZCw+O/sJ7i8YOoQ2DAsQ
gAAXBcJRCFVWDVCQTuGrGBYgOAiIF4gXGBYgAAEIQEDGeHv9x32HcWzdOc1H
lD2vxIJQBAEBCIgZr1onwvB+1TFMFwhwYOgECEA4Z4JzklyxHw6EuIZhAQIQ
4KJAgHFiWIAABCAAAQhAAMI7/YYCt3mC7ysAAhCAAAQgAAGtCAwLhgUIQAAC
EJAxMCxAAAIQgAAEIODDOrATIAABLgrDaI1Q/MZisfgoSjAeJvoXrg+U5I+T
7+61dXDn37lkSq0J+hdCmfysl1DWUrdxGG2g/0PjdzVZoAAA
====

-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=14257&group_id=1824

[fetchmail-devel] [Bug #14257] Incorrect parsing of fetchmailrc

[<ad...@be...>]

Bug #14257, was updated on 2008-Jul-23 19:07
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Closed
Resolution: Fixed
Bug Group: None
Priority: 5
Submitted by: juanvaqueroponc
Assigned to : m-a
Summary: Incorrect parsing of fetchmailrc

Details: The username in question is being parsed as 'sername "aaaaaaaaaaaaa@aaaaaaaaaaaaa'

It is quite difficult to trigger this bug.
If I change some lines the bug does not trigger.
I have got to the attached file that triggers the bug, but
if I remove the last line the same bug seems not to trigger.


Follow-Ups:

Date: 2010-Apr-10 17:32
By: m-a

Comment:
Turns out that this bug was a regression I'd introduced in March 2005 when trying to fix memory leaks - but I fixed them the wrong way.

I've committed michael's change as http://gitorious.org/fetchmail/fetchmail/commit/1c6fb40e79a1344c35dcc69b292f9b6913cd4e95 - the fix is to appear in a (not-so-distant) future fetchmail 6.3.17 release.

Could I ask you a favour and propose that you either mail patches to the fetchmail-devel@ mailing list or upload them using BerliOS's patch tracker? Much easier to merge them for me that way.

I can also merge from a private branch of yours in Gitorious if you like.
-------------------------------------------------------

Date: 2010-Apr-10 16:47
By: m-a

Comment:
Michael, that's just awesome - thanks for the pointer to the bug!

The bug is that the .sval union member must contain a copy from the string. The lexer may have to read ahead, and this can then cause yytext() to change. It's somewhat documented, so I think it's not masking a deeper issue.

I wonder if transferring the string in an xstrdup() like manner would be better here. I guess I'm signed up to validating all sval uses. :-)

-------------------------------------------------------

Date: 2010-Apr-10 03:02
By: bob5972

Comment:
I don't know if this is the right way to fix this, but it works.  The contents of yytext are changing between when the lexer returns yylval.sval and when the parser reads it.  I don't know why that would be happening, so this might be masking a deeper issue.  It may need to be tested further, but it does fix the problem.

This is extremely finicky though.  A clearer test case that triggers it is here http://www.ucf.ics.uci.edu/~bob5972/fetchmailrc.bug (It's basically his test case with the names labeled something other than aaaa).


>From 46a52eac0c1705339642bbe5c277e6bac11235ad Mon Sep 17 00:00:00 2001
From: Michael Banack <bo...@gm...>
Date: Fri, 9 Apr 2010 17:56:05 -0700
Subject: [PATCH] Changed the lexer to copy string values out of yytext.

Fixes bug #14257.
---
 rcfile_l.l |   54 +++++++++++++++++++++++++++++++++++++++++-------------
 1 files changed, 41 insertions(+), 13 deletions(-)

diff --git a/rcfile_l.l b/rcfile_l.l
index 8dcd8af..384f8df 100644
--- a/rcfile_l.l
+++ b/rcfile_l.l
@@ -34,18 +34,36 @@ int prc_lineno = 1;
 %%
 
 \"[^\"]*\"	{
-			yytext[yyleng-1] = '\0';
-			escapes(yytext+1, yytext);
-			yyleng = strlen(yytext);
-			yylval.sval = yytext;
+			static char *in = NULL;
+			static size_t ins = 0;
+			
+			if ((size_t)yyleng + 1 > ins) {
+			    ins = yyleng + 1;
+			    in = (char *)xrealloc(in, ins);
+			}
+			memcpy(in, yytext, yyleng);
+			
+			in[yyleng-1] = '\0';
+			escapes(in+1, in);
+			yyleng = strlen(in);
+			yylval.sval = in;
                         SETSTATE(0);
 			return STRING;
 		}
 \'[^\']*\'	{
-			yytext[yyleng-1] = '\0';
-			escapes(yytext+1, yytext);
-			yyleng = strlen(yytext);
-			yylval.sval = yytext;
+			static char *in = NULL;
+			static size_t ins = 0;
+			
+			if ((size_t)yyleng + 1 > ins) {
+			    ins = yyleng + 1;
+			    in = (char *)xrealloc(in, ins);
+			}
+			memcpy(in, yytext, yyleng);
+			
+			in[yyleng-1] = '\0';
+			escapes(in+1, in);
+			yyleng = strlen(in);
+			yylval.sval = in;
                         SETSTATE(0);
 			return STRING;
 		}
@@ -53,8 +71,8 @@ int prc_lineno = 1;
 "*"		{ BEGIN(0); return WILDCARD; }
 
 <NAME>[^=;:, \t\r\n]+	{
-			static char *in;
-			static size_t ins;
+			static char *in = NULL;
+			static size_t ins = 0;
 
 			if ((size_t)yyleng + 1 > ins) {
 			    ins = yyleng + 1;
@@ -222,9 +240,19 @@ options		{/* EMPTY */}
 -?[0-9]+	{ yylval.number = atoi(yytext); return NUMBER; }
 
 [^=;:, \t\r\n]+	{
-			escapes(yytext, yytext);
-			yyleng = strlen(yytext);
-			yylval.sval = yytext;
+			static char *in = NULL;
+			static size_t ins = 0;
+
+			if ((size_t)yyleng + 1 > ins) {
+			    ins = yyleng + 1;
+			    in = (char *)xrealloc(in, ins);
+			}
+			memcpy(in, yytext, yyleng);
+			in[yyleng] = '\0';
+			
+			escapes(in, in);
+			yyleng = strlen(in);
+			yylval.sval = in;
 			return STRING;
 		}
 
-- 
1.7.0.4


-------------------------------------------------------

Date: 2009-Apr-23 15:09
By: m-a

Comment:
Is there any improvement when compiling things with a recent version of flex?

It's just stabbing into the dark, but perhaps it helps.
-------------------------------------------------------

Date: 2008-Jul-23 19:11
By: juanvaqueroponc

Comment:
begin-base64 600 fetchmailrc.gz
H4sIAAAAAAACA+2dzW7aQBDH73mKFagoOezuoeohVin0UqmHqnkDtE2IiuQE
hEl767MXDLb3Y5xgs3ic5E+kgI1hfp6dr501cDe/N0/pJhMPd0YMVuvl7YNZ
pELeiw/fBuJ+vrn9bdL0YrVMU2Hsm1gt1xtxff1JbF+0Wd4uU3Hz8+ajeMrm
60fzMBcD490GYmWy7O9yfVc9NxBZlopsvhGLxz+LbPErnYvhTlBSSrsY+sJf
kNhIjtoKCE+OFPb9x9ebtsLCk3j2jdu/64vaoW+BQEoarad2Cop9GseIoYz4
NNsN7Sm32BME1YqrlbY3YiNG5Y6p817ek3ucw7bK/0v7yMMBJtq5NFKehdCx
5EoNjrYUA4kiGOztzjVzsBMGbViC7U1Gkkiy2+k/sAVV4+Iqqg83Nt5SzO5f
onVAq6lDPBW7xyQ8IUkV1ndZvfjK0TULDpfpM0ZFLlVznCiDXHWGuNHixOsq
lnDndsdleMhVt9B5ZEq8uopk37HOiPBRnIbsnF0Rip0RJRCzRTCaJKs3SLuW
twcpeMJSlLS3OlcWm67iDVQz189vCUeh5VdMecFke7DREQahgTb2xnmWqW99
eKjcQsUzeh7DayrascGO1U5k6FhKaO8Bu4d6N2sYRa+jGignoLJ9s3TN4OmJ
MePt3Wi/Z+y1wmIGmaYKrp+VdRvkC8XK87CcYHc9xIk+UKe5ZZRcFE8nRUNC
VUGDlS+Ghk5q0SSW3Kmp6+MQh7EWyHzCldUu5ig9Q/uO4mQtABR3WhB+ASL4
UpM1FspN8tpP/Jq5nVb1ZP0KijdSK6uAk8FiluIe2pgJ7TiIYdhHV3Q6U92n
2/PnrO5CmiD6Xt4sy3owctOm87IyNSbMpXLP/JqToJqCacHc1aQL5Ggtm9aT
1bhZnLOQ63vnkLVuDvxCk20JhoUvN791nWi9qojUkKUgtmgmw6jh4OXcQzfG
iX6EXv4coOkm3KEDl3feeOpwFaoq5JRMM906e+tDHiWmwbUzLeniBkWdZzLW
aGj+YF3RVNY6O7SK+dZn/W6RoCtlxumt33/XB0NwW4GWjv/5R8r9rPQcFt/y
jBRl4LoycR3UGNM+NBuO9VTZk8qoTBkyjHrq4IO5C+ZrrqNy9XXsbuZ7VPcL
8tbFAMwjXp/XJC+hIlfoLHb2CRmRxibGWMuE9S8pVhNfPrAHU5ECqIgJMxNz
EJrktMQPQ9N4QfOkxmM5ZgzrjNY1B4b8uMbxgjvo+rW4zGzqhX37qhL+pS4C
EAhAAAIQ6MAVTsSgneIK1QQ6AQIQgACE5xGG/YjXVWMR44IcBgQgAOFdIPQs
TiJzwEiBgMoKwwKEN40wnhSrX9JMtn/VxxFn1qL2ZCw+O/sJ7i8YOoQ2DAsQ
gAAXBcJRCFVWDVCQTuGrGBYgOAiIF4gXGBYgAAEIQEDGeHv9x32HcWzdOc1H
lD2vxIJQBAEBCIgZr1onwvB+1TFMFwhwYOgECEA4Z4JzklyxHw6EuIZhAQIQ
4KJAgHFiWIAABCAAAQhAAMI7/YYCt3mC7ysAAhCAAAQgAAGtCAwLhgUIQAAC
EJAxMCxAAAIQgAAEIODDOrATIAABLgrDaI1Q/MZisfgoSjAeJvoXrg+U5I+T
7+61dXDn37lkSq0J+hdCmfysl1DWUrdxGG2g/0PjdzVZoAAA
====

-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=14257&group_id=1824

[fetchmail-devel] [Bug #14257] Incorrect parsing of fetchmailrc

[<ad...@be...>]

Bug #14257, was updated on 2008-Jul-23 19:07
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Open
Resolution: None
Bug Group: None
Priority: 4
Submitted by: juanvaqueroponc
Assigned to : none
Summary: Incorrect parsing of fetchmailrc

Details: The username in question is being parsed as 'sername "aaaaaaaaaaaaa@aaaaaaaaaaaaa'

It is quite difficult to trigger this bug.
If I change some lines the bug does not trigger.
I have got to the attached file that triggers the bug, but
if I remove the last line the same bug seems not to trigger.


Follow-Ups:

Date: 2010-Apr-10 16:47
By: m-a

Comment:
Michael, that's just awesome - thanks for the pointer to the bug!

The bug is that the .sval union member must contain a copy from the string. The lexer may have to read ahead, and this can then cause yytext() to change. It's somewhat documented, so I think it's not masking a deeper issue.

I wonder if transferring the string in an xstrdup() like manner would be better here. I guess I'm signed up to validating all sval uses. :-)

-------------------------------------------------------

Date: 2010-Apr-10 03:02
By: bob5972

Comment:
I don't know if this is the right way to fix this, but it works.  The contents of yytext are changing between when the lexer returns yylval.sval and when the parser reads it.  I don't know why that would be happening, so this might be masking a deeper issue.  It may need to be tested further, but it does fix the problem.

This is extremely finicky though.  A clearer test case that triggers it is here http://www.ucf.ics.uci.edu/~bob5972/fetchmailrc.bug (It's basically his test case with the names labeled something other than aaaa).


>From 46a52eac0c1705339642bbe5c277e6bac11235ad Mon Sep 17 00:00:00 2001
From: Michael Banack <bo...@gm...>
Date: Fri, 9 Apr 2010 17:56:05 -0700
Subject: [PATCH] Changed the lexer to copy string values out of yytext.

Fixes bug #14257.
---
 rcfile_l.l |   54 +++++++++++++++++++++++++++++++++++++++++-------------
 1 files changed, 41 insertions(+), 13 deletions(-)

diff --git a/rcfile_l.l b/rcfile_l.l
index 8dcd8af..384f8df 100644
--- a/rcfile_l.l
+++ b/rcfile_l.l
@@ -34,18 +34,36 @@ int prc_lineno = 1;
 %%
 
 \"[^\"]*\"	{
-			yytext[yyleng-1] = '\0';
-			escapes(yytext+1, yytext);
-			yyleng = strlen(yytext);
-			yylval.sval = yytext;
+			static char *in = NULL;
+			static size_t ins = 0;
+			
+			if ((size_t)yyleng + 1 > ins) {
+			    ins = yyleng + 1;
+			    in = (char *)xrealloc(in, ins);
+			}
+			memcpy(in, yytext, yyleng);
+			
+			in[yyleng-1] = '\0';
+			escapes(in+1, in);
+			yyleng = strlen(in);
+			yylval.sval = in;
                         SETSTATE(0);
 			return STRING;
 		}
 \'[^\']*\'	{
-			yytext[yyleng-1] = '\0';
-			escapes(yytext+1, yytext);
-			yyleng = strlen(yytext);
-			yylval.sval = yytext;
+			static char *in = NULL;
+			static size_t ins = 0;
+			
+			if ((size_t)yyleng + 1 > ins) {
+			    ins = yyleng + 1;
+			    in = (char *)xrealloc(in, ins);
+			}
+			memcpy(in, yytext, yyleng);
+			
+			in[yyleng-1] = '\0';
+			escapes(in+1, in);
+			yyleng = strlen(in);
+			yylval.sval = in;
                         SETSTATE(0);
 			return STRING;
 		}
@@ -53,8 +71,8 @@ int prc_lineno = 1;
 "*"		{ BEGIN(0); return WILDCARD; }
 
 <NAME>[^=;:, \t\r\n]+	{
-			static char *in;
-			static size_t ins;
+			static char *in = NULL;
+			static size_t ins = 0;
 
 			if ((size_t)yyleng + 1 > ins) {
 			    ins = yyleng + 1;
@@ -222,9 +240,19 @@ options		{/* EMPTY */}
 -?[0-9]+	{ yylval.number = atoi(yytext); return NUMBER; }
 
 [^=;:, \t\r\n]+	{
-			escapes(yytext, yytext);
-			yyleng = strlen(yytext);
-			yylval.sval = yytext;
+			static char *in = NULL;
+			static size_t ins = 0;
+
+			if ((size_t)yyleng + 1 > ins) {
+			    ins = yyleng + 1;
+			    in = (char *)xrealloc(in, ins);
+			}
+			memcpy(in, yytext, yyleng);
+			in[yyleng] = '\0';
+			
+			escapes(in, in);
+			yyleng = strlen(in);
+			yylval.sval = in;
 			return STRING;
 		}
 
-- 
1.7.0.4


-------------------------------------------------------

Date: 2009-Apr-23 15:09
By: m-a

Comment:
Is there any improvement when compiling things with a recent version of flex?

It's just stabbing into the dark, but perhaps it helps.
-------------------------------------------------------

Date: 2008-Jul-23 19:11
By: juanvaqueroponc

Comment:
begin-base64 600 fetchmailrc.gz
H4sIAAAAAAACA+2dzW7aQBDH73mKFagoOezuoeohVin0UqmHqnkDtE2IiuQE
hEl767MXDLb3Y5xgs3ic5E+kgI1hfp6dr501cDe/N0/pJhMPd0YMVuvl7YNZ
pELeiw/fBuJ+vrn9bdL0YrVMU2Hsm1gt1xtxff1JbF+0Wd4uU3Hz8+ajeMrm
60fzMBcD490GYmWy7O9yfVc9NxBZlopsvhGLxz+LbPErnYvhTlBSSrsY+sJf
kNhIjtoKCE+OFPb9x9ebtsLCk3j2jdu/64vaoW+BQEoarad2Cop9GseIoYz4
NNsN7Sm32BME1YqrlbY3YiNG5Y6p817ek3ucw7bK/0v7yMMBJtq5NFKehdCx
5EoNjrYUA4kiGOztzjVzsBMGbViC7U1Gkkiy2+k/sAVV4+Iqqg83Nt5SzO5f
onVAq6lDPBW7xyQ8IUkV1ndZvfjK0TULDpfpM0ZFLlVznCiDXHWGuNHixOsq
lnDndsdleMhVt9B5ZEq8uopk37HOiPBRnIbsnF0Rip0RJRCzRTCaJKs3SLuW
twcpeMJSlLS3OlcWm67iDVQz189vCUeh5VdMecFke7DREQahgTb2xnmWqW99
eKjcQsUzeh7DayrascGO1U5k6FhKaO8Bu4d6N2sYRa+jGignoLJ9s3TN4OmJ
MePt3Wi/Z+y1wmIGmaYKrp+VdRvkC8XK87CcYHc9xIk+UKe5ZZRcFE8nRUNC
VUGDlS+Ghk5q0SSW3Kmp6+MQh7EWyHzCldUu5ig9Q/uO4mQtABR3WhB+ASL4
UpM1FspN8tpP/Jq5nVb1ZP0KijdSK6uAk8FiluIe2pgJ7TiIYdhHV3Q6U92n
2/PnrO5CmiD6Xt4sy3owctOm87IyNSbMpXLP/JqToJqCacHc1aQL5Ggtm9aT
1bhZnLOQ63vnkLVuDvxCk20JhoUvN791nWi9qojUkKUgtmgmw6jh4OXcQzfG
iX6EXv4coOkm3KEDl3feeOpwFaoq5JRMM906e+tDHiWmwbUzLeniBkWdZzLW
aGj+YF3RVNY6O7SK+dZn/W6RoCtlxumt33/XB0NwW4GWjv/5R8r9rPQcFt/y
jBRl4LoycR3UGNM+NBuO9VTZk8qoTBkyjHrq4IO5C+ZrrqNy9XXsbuZ7VPcL
8tbFAMwjXp/XJC+hIlfoLHb2CRmRxibGWMuE9S8pVhNfPrAHU5ECqIgJMxNz
EJrktMQPQ9N4QfOkxmM5ZgzrjNY1B4b8uMbxgjvo+rW4zGzqhX37qhL+pS4C
EAhAAAIQ6MAVTsSgneIK1QQ6AQIQgACE5xGG/YjXVWMR44IcBgQgAOFdIPQs
TiJzwEiBgMoKwwKEN40wnhSrX9JMtn/VxxFn1qL2ZCw+O/sJ7i8YOoQ2DAsQ
gAAXBcJRCFVWDVCQTuGrGBYgOAiIF4gXGBYgAAEIQEDGeHv9x32HcWzdOc1H
lD2vxIJQBAEBCIgZr1onwvB+1TFMFwhwYOgECEA4Z4JzklyxHw6EuIZhAQIQ
4KJAgHFiWIAABCAAAQhAAMI7/YYCt3mC7ysAAhCAAAQgAAGtCAwLhgUIQAAC
EJAxMCxAAAIQgAAEIODDOrATIAABLgrDaI1Q/MZisfgoSjAeJvoXrg+U5I+T
7+61dXDn37lkSq0J+hdCmfysl1DWUrdxGG2g/0PjdzVZoAAA
====

-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=14257&group_id=1824

[fetchmail-devel] [Bug #14257] Incorrect parsing of fetchmailrc

[<ad...@be...>]

Bug #14257, was updated on 2008-Jul-23 10:07
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Open
Resolution: None
Bug Group: None
Priority: 4
Submitted by: juanvaqueroponc
Assigned to : none
Summary: Incorrect parsing of fetchmailrc

Details: The username in question is being parsed as 'sername "aaaaaaaaaaaaa@aaaaaaaaaaaaa'

It is quite difficult to trigger this bug.
If I change some lines the bug does not trigger.
I have got to the attached file that triggers the bug, but
if I remove the last line the same bug seems not to trigger.


Follow-Ups:

Date: 2010-Apr-09 18:02
By: bob5972

Comment:
I don't know if this is the right way to fix this, but it works.  The contents of yytext are changing between when the lexer returns yylval.sval and when the parser reads it.  I don't know why that would be happening, so this might be masking a deeper issue.  It may need to be tested further, but it does fix the problem.

This is extremely finicky though.  A clearer test case that triggers it is here http://www.ucf.ics.uci.edu/~bob5972/fetchmailrc.bug (It's basically his test case with the names labeled something other than aaaa).


>From 46a52eac0c1705339642bbe5c277e6bac11235ad Mon Sep 17 00:00:00 2001
From: Michael Banack <bo...@gm...>
Date: Fri, 9 Apr 2010 17:56:05 -0700
Subject: [PATCH] Changed the lexer to copy string values out of yytext.

Fixes bug #14257.
---
 rcfile_l.l |   54 +++++++++++++++++++++++++++++++++++++++++-------------
 1 files changed, 41 insertions(+), 13 deletions(-)

diff --git a/rcfile_l.l b/rcfile_l.l
index 8dcd8af..384f8df 100644
--- a/rcfile_l.l
+++ b/rcfile_l.l
@@ -34,18 +34,36 @@ int prc_lineno = 1;
 %%
 
 \"[^\"]*\"	{
-			yytext[yyleng-1] = '\0';
-			escapes(yytext+1, yytext);
-			yyleng = strlen(yytext);
-			yylval.sval = yytext;
+			static char *in = NULL;
+			static size_t ins = 0;
+			
+			if ((size_t)yyleng + 1 > ins) {
+			    ins = yyleng + 1;
+			    in = (char *)xrealloc(in, ins);
+			}
+			memcpy(in, yytext, yyleng);
+			
+			in[yyleng-1] = '\0';
+			escapes(in+1, in);
+			yyleng = strlen(in);
+			yylval.sval = in;
                         SETSTATE(0);
 			return STRING;
 		}
 \'[^\']*\'	{
-			yytext[yyleng-1] = '\0';
-			escapes(yytext+1, yytext);
-			yyleng = strlen(yytext);
-			yylval.sval = yytext;
+			static char *in = NULL;
+			static size_t ins = 0;
+			
+			if ((size_t)yyleng + 1 > ins) {
+			    ins = yyleng + 1;
+			    in = (char *)xrealloc(in, ins);
+			}
+			memcpy(in, yytext, yyleng);
+			
+			in[yyleng-1] = '\0';
+			escapes(in+1, in);
+			yyleng = strlen(in);
+			yylval.sval = in;
                         SETSTATE(0);
 			return STRING;
 		}
@@ -53,8 +71,8 @@ int prc_lineno = 1;
 "*"		{ BEGIN(0); return WILDCARD; }
 
 <NAME>[^=;:, \t\r\n]+	{
-			static char *in;
-			static size_t ins;
+			static char *in = NULL;
+			static size_t ins = 0;
 
 			if ((size_t)yyleng + 1 > ins) {
 			    ins = yyleng + 1;
@@ -222,9 +240,19 @@ options		{/* EMPTY */}
 -?[0-9]+	{ yylval.number = atoi(yytext); return NUMBER; }
 
 [^=;:, \t\r\n]+	{
-			escapes(yytext, yytext);
-			yyleng = strlen(yytext);
-			yylval.sval = yytext;
+			static char *in = NULL;
+			static size_t ins = 0;
+
+			if ((size_t)yyleng + 1 > ins) {
+			    ins = yyleng + 1;
+			    in = (char *)xrealloc(in, ins);
+			}
+			memcpy(in, yytext, yyleng);
+			in[yyleng] = '\0';
+			
+			escapes(in, in);
+			yyleng = strlen(in);
+			yylval.sval = in;
 			return STRING;
 		}
 
-- 
1.7.0.4


-------------------------------------------------------

Date: 2009-Apr-23 06:09
By: m-a

Comment:
Is there any improvement when compiling things with a recent version of flex?

It's just stabbing into the dark, but perhaps it helps.
-------------------------------------------------------

Date: 2008-Jul-23 10:11
By: juanvaqueroponc

Comment:
begin-base64 600 fetchmailrc.gz
H4sIAAAAAAACA+2dzW7aQBDH73mKFagoOezuoeohVin0UqmHqnkDtE2IiuQE
hEl767MXDLb3Y5xgs3ic5E+kgI1hfp6dr501cDe/N0/pJhMPd0YMVuvl7YNZ
pELeiw/fBuJ+vrn9bdL0YrVMU2Hsm1gt1xtxff1JbF+0Wd4uU3Hz8+ajeMrm
60fzMBcD490GYmWy7O9yfVc9NxBZlopsvhGLxz+LbPErnYvhTlBSSrsY+sJf
kNhIjtoKCE+OFPb9x9ebtsLCk3j2jdu/64vaoW+BQEoarad2Cop9GseIoYz4
NNsN7Sm32BME1YqrlbY3YiNG5Y6p817ek3ucw7bK/0v7yMMBJtq5NFKehdCx
5EoNjrYUA4kiGOztzjVzsBMGbViC7U1Gkkiy2+k/sAVV4+Iqqg83Nt5SzO5f
onVAq6lDPBW7xyQ8IUkV1ndZvfjK0TULDpfpM0ZFLlVznCiDXHWGuNHixOsq
lnDndsdleMhVt9B5ZEq8uopk37HOiPBRnIbsnF0Rip0RJRCzRTCaJKs3SLuW
twcpeMJSlLS3OlcWm67iDVQz189vCUeh5VdMecFke7DREQahgTb2xnmWqW99
eKjcQsUzeh7DayrascGO1U5k6FhKaO8Bu4d6N2sYRa+jGignoLJ9s3TN4OmJ
MePt3Wi/Z+y1wmIGmaYKrp+VdRvkC8XK87CcYHc9xIk+UKe5ZZRcFE8nRUNC
VUGDlS+Ghk5q0SSW3Kmp6+MQh7EWyHzCldUu5ig9Q/uO4mQtABR3WhB+ASL4
UpM1FspN8tpP/Jq5nVb1ZP0KijdSK6uAk8FiluIe2pgJ7TiIYdhHV3Q6U92n
2/PnrO5CmiD6Xt4sy3owctOm87IyNSbMpXLP/JqToJqCacHc1aQL5Ggtm9aT
1bhZnLOQ63vnkLVuDvxCk20JhoUvN791nWi9qojUkKUgtmgmw6jh4OXcQzfG
iX6EXv4coOkm3KEDl3feeOpwFaoq5JRMM906e+tDHiWmwbUzLeniBkWdZzLW
aGj+YF3RVNY6O7SK+dZn/W6RoCtlxumt33/XB0NwW4GWjv/5R8r9rPQcFt/y
jBRl4LoycR3UGNM+NBuO9VTZk8qoTBkyjHrq4IO5C+ZrrqNy9XXsbuZ7VPcL
8tbFAMwjXp/XJC+hIlfoLHb2CRmRxibGWMuE9S8pVhNfPrAHU5ECqIgJMxNz
EJrktMQPQ9N4QfOkxmM5ZgzrjNY1B4b8uMbxgjvo+rW4zGzqhX37qhL+pS4C
EAhAAAIQ6MAVTsSgneIK1QQ6AQIQgACE5xGG/YjXVWMR44IcBgQgAOFdIPQs
TiJzwEiBgMoKwwKEN40wnhSrX9JMtn/VxxFn1qL2ZCw+O/sJ7i8YOoQ2DAsQ
gAAXBcJRCFVWDVCQTuGrGBYgOAiIF4gXGBYgAAEIQEDGeHv9x32HcWzdOc1H
lD2vxIJQBAEBCIgZr1onwvB+1TFMFwhwYOgECEA4Z4JzklyxHw6EuIZhAQIQ
4KJAgHFiWIAABCAAAQhAAMI7/YYCt3mC7ysAAhCAAAQgAAGtCAwLhgUIQAAC
EJAxMCxAAAIQgAAEIODDOrATIAABLgrDaI1Q/MZisfgoSjAeJvoXrg+U5I+T
7+61dXDn37lkSq0J+hdCmfysl1DWUrdxGG2g/0PjdzVZoAAA
====

-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=14257&group_id=1824

[fetchmail-devel] [Bug #16134] Incorrect library check in configure

[<ad...@be...>]

Bug #16134, was updated on 2009-Aug-13 17:00
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Closed
Resolution: Fixed
Bug Group: None
Priority: 3
Submitted by: voidmage
Assigned to : m-a
Summary: Incorrect library check in configure

Details: This is a followup of very old Gentoo bugs: 231400 and 185652.
It's a -Wl,--as-needed problem.
Most of it is a Gentoo only problem, except for two things:
- MD5_Init is in libcrypto, not libssl, so the check in configure.ac
is incorrect (well, OK, it's not documented that well,
which functions are in libssl and which in libcrypto)
- why it did work with that check removed -
is that check not really needed ?

Right now (since 6.3.9 at least), only thing I needed to do
to make things work, were changing "ssl" to "crypto" in
that AC_CHECK_LIB check.

In one of Flameeyes' blog posts it said that somewhere after
binutils 2.19, they've changed the behavior, so perhaps
even current state will work, but it's not that way now.

Follow-Ups:

Date: 2010-Apr-09 09:53
By: m-a

Comment:
Sorry for the late reply; this got fixed in fetchmail 6.3.12.

Thanks a million for forwarding Gentoo bugs to the upstream, and please keep doing that if new bugs come in.
-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=16134&group_id=1824

[fetchmail-devel] [Bug #16172] Fetchmail eats mail when unable to connect to SMTP

[<ad...@be...>]

Bug #16172, was updated on 2009-Aug-24 10:38
Here is a current snapshot of the bug.

Project: fetchmail
Category: None
Status: Open
Resolution: Works For Me
Bug Group: None
Priority: 7
Submitted by: bennovdbogaard
Assigned to : m-a
Summary: Fetchmail eats mail when unable to connect to SMTP

Details: Hello everyone,

I use fetchmail in various production environments and I discovered some unwanted behaviour.

After fetchmail fetches mail from a pop mail server it removes messages after they are retrieved. But if the SMTP to deliver these messages to isn't available, it drops the messages and gives up instantly.

Could this behaviour be altered by first checking if the SMTP server is reachable and then decide to fetch the messages?

Yours,
Benno van den Bogaard <be...@ho...>

Follow-Ups:

Date: 2010-Apr-09 09:51
By: m-a

Comment:
Benno,

can I _please_ get a verbose trace showing the message loss? Thank you.
-------------------------------------------------------

Date: 2009-Aug-24 14:05
By: m-a

Comment:
Please prove through complete logs that fetchmail 6.3.10 or 6.3.11 loses mail and to show it's not a server fault (some servers are known to delete mail after retrieval even if the client hasn't requested deletion; also, some may make deletes effective before QUIT).

See http://www.fetchmail.info/fetchmail-FAQ.html#G3 for a list of pieces of information that we need to debug the issue.

Fetchmail has been designed to NOT delete messages from the server before it has successfully delivered them.

-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=16172&group_id=1824