fetchmail-devel — Development related discussion

Re: [fetchmail-devel] BerliOS/mailing list companion web site X.509 certificates (was: Multiple sslfingerprint options and digests)

[Matthias A. <mat...@gm...>]

Am 11.08.2009, 23:44 Uhr, schrieb grarpamp <gra...@gm...>:

> Note3: The lists.berlios.de cert is expired, and MD5, though that
> doesn't matter :) The Berlios_CA cert may be self-signed.

Berlios's certificate management is a mystery to me (to spare other  
rants), and I can't seem to get a hold of an admin who could provide me  
with the CA's root certificate or to be bothered about expired or  
non-matching certficiates.

If anyone has a direct link to prod them to proper certificate management  
practices, that would be quite welcome...

-- 
Matthias Andree

[fetchmail-devel] [Bug #11576] fetchmail does not report SSL negotiation errors properly

[<ad...@be...>]

Bug #11576, was updated on 2007-Jul-17 20:34
Here is a current snapshot of the bug.

Project: Community Fetchmail
Category: None
Status: Closed
Resolution: Works For Me
Bug Group: None
Priority: 5
Submitted by: psusi
Assigned to : none
Summary: fetchmail does not report SSL negotiation errors properly

Details: Matthias Andree asked me to file this bug report after our discussion on the fetchmail-users list in the thread titled "Invalid SSL certificate".  


Follow-Ups:

Date: 2009-Aug-19 11:15
By: m-a

Comment:
I believe this is fixed in fetchmail 6.3.11.
-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=11576&group_id=1824

[fetchmail-devel] [Bug #16000] fetchmail should support multiple sslfingerprint's

[<ad...@be...>]

Bug #16000, was updated on 2009-Jul-16 02:40
Here is a current snapshot of the bug.

Project: Community Fetchmail
Category: None
Status: Open
Resolution: None
Bug Group: None
Priority: 5
Submitted by: jw9
Assigned to : none
Summary: fetchmail should support multiple sslfingerprint's

Details: Some mail servers, such as pop3.live.com, have multiple hosts available at a single IP address (currently pop3.live.com appears to have two hosts behind that one IP address).

Other mail servers, such as imap.gmail.com, resolve to rotating IP addresses.

It should be possible to specify multiple 'sslfingerprint' entries so that any one of those fingerprints can match.


Follow-Ups:

Date: 2009-Aug-19 11:13
By: m-a

Comment:
reopening after grarpamp's comments.

On the suggestion, based on the rationale below, I might indeed rather correct the documentation and indeed support certificate fingerprint lists. I do however not subscribe to the assertion that the fingerprint were necessary to detect MITM attacks - the certificates should be sufficient as long as you're happy with the signing CA's verification policies (including the hash used).

I currently second the view that verifying both the MD5 and SHA1 of a certificate has been considered reasonably safe recently, i. e. I'm not aware of efficient attacks that collide both MD5 and SHA1 at the same time.

I wonder if the hash grouping can be configured in a simpler way, for instance --sslfingerprint {MD5}DE:AD:BE:...:EF&{SHA1}F0:0D:04:13:...:37 or similar. I'll need to think about this a bit more when it comes to the real implementation. It may be awkward to edit on your netbook screen or smartphone, but it's likely simpler to implement.

Given this is more a feature request than a real bug, it will probably not make a 6.3.X release though.
-------------------------------------------------------

Date: 2009-Aug-11 23:55
By: grarpamp

Comment:
Adding content for those who use this interface... 

Hi :) Folks appear to be misunderstanding things
about the underlying crypto and why fingerprints are a necessary
safety check.

If one simply relies on checking the cert chain and blindly trusts
that model, as most of the world is conditioned to believe it is
sound, you are open to attack. Namely, of the sort in the posted
url, it is a good read, as are the referenced prior art. Both MD5
and SHA1 are theory broken and MD5 demonstrably so, as noted in the
literatre. So the hash part is of concern, at least in this case.

A CA takes a MD5 over certain fields in the cert and signs that MD5
digest, not the entire cert contents. CA's previously used MD5 and
failed to change their cert practices to SHA1 until a real world
break, oops. Anyways... So I don't have to swipe their private key.
All I have to do to forge a server cert is copy the signature from
any candidate real cert into my cert and manufacture the rest of
my cert contents so that they: a) work, b) make the fields have the
same digest as the real one. Which, as we all know by now is possible,
given time or new art, due to the whole hash collision thing.

However... if I copy a known good cert via trusted channels, and
take both the MD5 and SHA1 fingerprints of the DER form of that
cert, it's much closer to cryptographically impossible to manufacture
a cert that would match evarything. I could just as easily make
formal contact with the cert holder and get the prints that way or
receive them as part of a high security programme. Banks do this.

So there are two different fingerprints covering different areas.

Checking fingerprints over the entire DER cert will foil this attack
because it detects the contents that I manipulated to match the
field digest that was later signed. That is why openssl, browsers,
ssh, etc provide fingerprint calculating and verification functions...
totally separate from signature calc/verif capabilities.

So two types of checks should always be performed for maximum
security:

 A) cert chain trust = sig(hash(contents)) -> pki
  This is done with openssl verify -verbose
  aka: sslcertck
 B) DER fingerprint = hash(sig(hash(contents))
  This is done with openssl x509 -in <PEMcert> -noout -fingerprint -<sha1|md5>
   or
  openssl x509 -in <PEMcert> -outform DER | <sha1|md5>
  aka: sslfingerprint... with the enhancements in the prior note

Enhancing fetchmail to doubly check B with both MD5 and SHa1 shouldn't
be much extra coding or user effort. As an aside, it's fairly common
to find places using/publishing both hashes for various purposes.

Removing the check in B places all the beans on the hash function
in A, which is not good. Private keys are not even involved in this.
Defense in depth will save your butt :)

Give the user tools, let them decide how many they wish to deploy
and manage. I would adapt the docs to say that one should use both
A and B, or neither, with some short reasons why. And continue to
provide example openssl usage as is there now, to allow them to
accomplish it.

Note1: Not checking the fingerprints on self-signed certs... such
as are commonly found in internal corporate nets, edu's, org's, and
the many end user, developer, hacker and other low budget/tier
sites... is asking for trouble. Not publishing your prints, signed
by OpenPGP, is bad too.

Note2: The random serial fix just helps eliminate easier forms of
the hash collision attack, see the paper. It doesn't necessarily
prevent other forms or completely new math attacks, and definitely
doesn't prevent brute force.

Note3: The lists.berlios.de cert is expired, and MD5, though that
doesn't matter :) The Berlios_CA cert may be self-signed.

I'm out of time at the moment to
help out with correcting what is already in here, perhaps later.

Thanks again for keeping up with all the maintenance to fetchmail :)
-------------------------------------------------------

Date: 2009-Aug-11 23:50
By: none

Comment:
Adding content for those who use this interface...

Hi all :)

Fetchmail needs the ability to specify multiple 'sslfingerprint'
options and digests per 'poll' stanza.
  
I noticed this when connecting to gmail. I think it's a load sharing
thing providers use. Google also has their own intermediate CA now
so more cert deployments could be expected.
  
At the moment, if you specify multiple sslfingerprint rc lines or
command line options, only the last one specified is checked. Any
prior sslfingerprint options are silently ignored. And the only
digest available is md5.
  
This is obviously frustrating users mail collection attempts :(
  
  
To reproduce the problem, run this and examine the certs and hashes.
The gmail certs happen to verify back to the root CA's. The
44a8...f1a9 cert occurs more frequently, the 9273...9f33 cert is
rarer. I've attached them both and the CA's involved.
  
If you are not seeing multiple certs, you may need to 'travel' and
debug with different source ip addresses... such as provided with
a worldwide proxy network like Tor.
  
i=10 ; while : ; do
 LD_PRELOAD=/tmp/tor/libtsocks.so.1 TSOCKS_CONF_FILE=/tmp/tor/tsocks.conf \
  openssl s_client -connect pop.gmail.com:pop3s < /dev/null > tcert$i 2>&1
 echo -n "$i: "
 openssl x509 -noout -fingerprint -md5 -in tcert$i
 i=`expr $i + 1`
done
  
  
Also related... from the BRANCH_6-3 NEWS file:
  
* --sslfingerprint may be removed from a future fetchmail version,
 because it's just too easily abused to create a false sense of
 security.
  
PLEASE don't remove this feature. I like it a lot and use it
everywhere possible, with --ssl and --sslcertck at the same time
as well, from trusted sources beforehand of course. Knowing when
the cert changes out from under you is very useful, even if only
for debugging or system verification. Especially if you connect
from untrusted locations or paths. Or you want to know about and
resolve them. It's also handy if the machine a user is connecting
from has a poor, untrusted or nonexistent collection of CA certs.
In that case, just the digest can be imported and used. Note that
openssl does not ship with a CA cert collection... admins usually
have to get it from mozilla... if they value PKI enough to think
of and bother with it.

There are blurbs in the man page that even talk about self-signed
certs being accepted with 'sslcertck'.


Overall, it would be good to see this feature enhanced to do both
md5 and sha1 to make cert collision attacks much more improbable
going forward... at least until the NIST 'SHA-3' hash competition
is done and that trickles down to the CA's and apps years from now.
Usage in the case of a single cert, where both must match, might
be like...

poll
 sslfingerprint-md5 <fp_md5>
 sslfingerprint-sha1 <fp_sha1>

In the case where multiple certs may be presented, group the hashes
as a set. Both hashes in the set must match. First check set cert1,
then set cert2, then set cert[n] until one set, or none, match the
presented cert...

poll
 sslfingerprint-cert1-md5 <fp_md5>
 sslfingerprint-cert1-sha1 <fp_sha1>
 sslfingerprint-cert2-md5 <fp_md5>
 sslfingerprint-cert2-sha1 <fp_sha1>
 sslfingerprint-cert[n]-md5 <fp_md5>
 sslfingerprint-cert[n]-sha1 <fp_sha1>

The '--' command line version of this should also do the right
thing.

Fetchmail should also print both the md5 and sha1 fingerprints from
the received certificate in the debug output.

Note that openssl supports mdc2, md2, md5, sha1. I understand that
some CA's also used, or may still use, the non-sha1 digests. Though
I guess sha1 is the only one used by sane CA's for new certs these
days. I don't know how many old ones are still in use on hosts. md5
and sha1 are enough until SHA-3.

And since the past always seems to apply to some future scenario
with new twists, this is surely not the last of the breed to be
proven out: http://www.win.tue.nl/hashclash/rogue-ca/

Extra checks are always good thing to have available.
Openssl probably has simple library routines to do this.

Thanks guys, and for fetchmail too, been using it for years :)
-------------------------------------------------------

Date: 2009-Jul-16 20:20
By: m-a

Comment:
Compatibility. Basically 6.3.X should be as compatible with 6.2.0 as reasonably possible - and it turned out that 6.3.X has lived for quite a while now... Compatibility was needed at the time to pave the upgrade path for 6.2.X users, as 6.3.0 had around one hundred bug fixes over 6.2.5 - and those were more important than cleanups.
-------------------------------------------------------

Date: 2009-Jul-16 17:44
By: jw9

Comment:
Good.

You should remove sslfingerprint asap.

Cannot understand why the feature was implemented in the first place.  And why such a broken "feature" has endured for so long.

Have a nice day.

-------------------------------------------------------

Date: 2009-Jul-16 15:19
By: m-a

Comment:
[Now the bug tracker is messing things up, my comment appeared twice in spite of being submitted once around 14:35, not again at 15:03. Sorry for that, but it has happened outside my control.]

I don't doubt your technical and intellectual skills to download the server's certificate or its fingerprint. BUT then your security hinges on the useless assumption that your connection is not under attack in the very moment that you're downloading the cert/fingerprint. You can't know if you're under attack unless you can verify the FP through a secure channel: You're using the same channel you'd later use for the actual mail-fetching connection; so you'd happily download the attacker's certificate or fingerprint.

Fetching the fingerprint through a DIFFERENT or at least a known-secure channel is crucial here.

I will not abolish the option for 6.3.X versions for compatibility reasons. If I'll drop it later is left for later decision, as the fingerprint is the easier alternative compared to downloading the *server* certificate and stuffing it into <prefix>/ssl/certs/. The canonical way is still to install the root-signing certificate though.

I don't see much use in listing hundreds of ssl fingerprints and keeping that list up to date when instead you can just save the certificate of the signing CA. It's less of a hassle. And if you can't trust, for instance, CyberTrust, then don't install their certificate, but Microsoft's certificate instead - providing you have a way of verifying that...

I'll see to marking the sslfingerprint feature candidate for removal for 6.4.X or newer fetchmail releases.
-------------------------------------------------------

Date: 2009-Jul-16 15:03
By: m-a

Comment:
Two things:

1a. sslfingerprint is unnecessary in "--sslcertck --sslproto tls1" and in "--sslcertck --sslproto ssl3 --ssl" configurations, and shouldn't be used. It doesn't add security unless your ISP provides you with the finger prints through a separate channel (such as your walking up to the site and fetching a sheet with finger prints).

The assumption is that the MITM (=attacker) cannot forge the signature of the certification authorities you trust. He may forge the server's certificate, but he'll hardly be able to forge the root signing certificate from Equifax, Verisign, or whoever. sslcertck is sufficient unless you follow stupid guides that suggest capturing and saving the server certificate (while that works to shut up complaints about certificates, it doesn't add protection against MITM attacks: you might already download the counterfeit certificate that you save as trusted, if you're already under attack at the time).

1b. sslfingerprint, as you observed, isn't useful with certain load balancing setups, when servers have individual certificates that are signed by a common CA.

2. for complaints about the newest-first ordering of the comments, please file a support request against BerliOS itself, not against individual projects.
-------------------------------------------------------

Date: 2009-Jul-16 15:02
By: jw9

Comment:
I can determine the proper fingerprint from any other server on any other network that I have access to.  I don't have to ask any "ISP" to provide fingerprints through another channel because I am quite able to do that myself.

So, as you state, it does add extra security.  Especially in today's world where most people have access to more than one "ISP" or network (eg your workplace if you have one, and most people do!).

In which case you should allow a set of sslfingerprints to be specified.

Otherwise, to be consistent with your argument (that sslfingerprint is unnecessary), you should completely remove the sslfingerprint option.

Fix it or lose it.

-------------------------------------------------------

Date: 2009-Jul-16 14:35
By: m-a

Comment:
Two things:

1a. sslfingerprint is unnecessary in "--sslcertck --sslproto tls1" and in "--sslcertck --sslproto ssl3 --ssl" configurations, and shouldn't be used. It doesn't add security unless your ISP provides you with the finger prints through a separate channel (such as your walking up to the site and fetching a sheet with finger prints).

The assumption is that the MITM (=attacker) cannot forge the signature of the certification authorities you trust. He may forge the server's certificate, but he'll hardly be able to forge the root signing certificate from Equifax, Verisign, or whoever. sslcertck is sufficient unless you follow stupid guides that suggest capturing and saving the server certificate (while that works to shut up complaints about certificates, it doesn't add protection against MITM attacks: you might already download the counterfeit certificate that you save as trusted, if you're already under attack at the time).

1b. sslfingerprint, as you observed, isn't useful with certain load balancing setups, when servers have individual certificates that are signed by a common CA.

2. for complaints about the newest-first ordering of the comments, please file a support request against BerliOS itself, not against individual projects.
-------------------------------------------------------

Date: 2009-Jul-16 13:09
By: jw9

Comment:
It would appear that additional comments appear in a push-down stack order.  This makes it very hard to follow the natural order of things.  This is very strange.

Here in the western world we are used to reading left-to-right and top-to-bottom.  We certainly don't expect to have to read from bottom-to-top.

I suppose I can turn my monitor upside down and then they might follow in a proper order.  But wait, that would make the characters appear upside-down!

In any case, if one is to be able to check the sslfingerprint of a server to protect against man-in-the-middle attacks, and if the fingerprint does actually change randomly (and not because of a man-in-the-middle attack), then we need to be able to specify a set of fingerprints and only require that at least one of them matches.

-------------------------------------------------------

Date: 2009-Jul-16 13:03
By: jw9

Comment:
Why do comments appear in such a strange place (after my original submission)?  Shouldn't additional comments follow in a logical order after other people's followup comments?  Very strange!

-------------------------------------------------------

Date: 2009-Jul-16 13:00
By: jw9

Comment:
II think you misunderstand.  The sslfingerprint is used to verify the identity of the server and to protect against man-in-the-middle attacks.  The configuration also has proper certificate and "ssslcertck" and "sslcertpath" settings.

To protect again man-in-the-middle attacks one needs to be able to ensure that the fingerprint of the server hasn't changed.

-------------------------------------------------------

Date: 2009-Jul-16 12:33
By: m-a

Comment:
I don't think fetchmail should do that.

sslfingerprints are just a convenience shortcut, and closely resemble stuffing the server's certificate into the directory of trusted certification authority certificates, usually /usr/ssl/certs, /etc/ssl/certs (and running c_rehash afterwards), or similar.

The canonical way is to install the GTE CyberTrust Global Root (for Microsoft/live.com) or the Equifax Secure Certificate Authority (for Google Mail) and use certificate-based authentication.

See also:
http://gagravarr.org/writing/openssl-certs/others.shtml
-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=16000&group_id=1824

[fetchmail-devel] regression bug fix for fetchmail 6.3.11 security release

[Matthias A. <mat...@gm...>]

Greetings,

fetchmail 6.3.11 was recently released to address an SSL/TLS certificate
verification weakness that allowed man-in-the-middle (MITM) attacks to
go unnoticed with specially-crafted certificates.

Unfortunately, the fix has had a minor flaw that causes program aborts on
SSL connection on some systems, depending on the compiler used, and on the
libc version and configuration, short, on the computer systems.

As a workaround, fetchmail -v (i. e. verbose mode) should work.

Jürgen Edner has reported the issue.
Thomas Heinz has independently reporter, and also analyzed and fixed the
issue, see http://bugs.gentoo.org/show_bug.cgi?id=280760 for details. I
approve of and am including his patch below and also as attachment for your
convenience. Please apply on top of 6.3.11 and rebuild your fetchmail
version if 6.3.11 causes problems similar to the one reported in
http://article.gmane.org/gmane.mail.fetchmail.user/8976 .

Unfortunately, this fix propagation was also delayed by a one-week
vacation of the active fetchmail maintainer.

I'm sorry for the inconvenience that 6.3.11 has caused.

I'll see to a soonish 6.3.12 release (which also updates a few
translations that 6.3.11 had to leave pending as the fix was somewhat
urgent) for those who are uncomfortable with patching.

Best regards
Matthias

--- socket.c.org	2009-08-08 16:01:49.000000000 +0200
+++ socket.c	2009-08-08 16:03:17.000000000 +0200
@@ -628,9 +628,10 @@
 				report(stdout, GT_("Unknown Issuer CommonName\n"));
 		}
 		if ((i = X509_NAME_get_text_by_NID(subj, NID_commonName, buf, sizeof(buf))) != -1) {
-			if (outlevel >= O_VERBOSE)
+			if (outlevel >= O_VERBOSE) {
 				report(stdout, GT_("Server CommonName: %s\n"), (tt = sdump(buf, i)));
-			xfree(tt);
+				xfree(tt);
+			}
 			if ((size_t)i >= sizeof(buf) - 1) {
 				/* Possible truncation. In this case, this is a DNS name, so this
 				 * is really bad. We do not tolerate this even in the non-strict case. */

[fetchmail-devel] [Bug #16134] Incorrect library check in configure

[<ad...@be...>]

Bug #16134, was updated on 2009-Aug-13 07:00
Here is a current snapshot of the bug.

Project: Community Fetchmail
Category: None
Status: Open
Resolution: None
Bug Group: None
Priority: 5
Submitted by: voidmage
Assigned to : none
Summary: Incorrect library check in configure

Details: This is a followup of very old Gentoo bugs: 231400 and 185652.
It's a -Wl,--as-needed problem.
Most of it is a Gentoo only problem, except for two things:
- MD5_Init is in libcrypto, not libssl, so the check in configure.ac
is incorrect (well, OK, it's not documented that well,
which functions are in libssl and which in libcrypto)
- why it did work with that check removed -
is that check not really needed ?

Right now (since 6.3.9 at least), only thing I needed to do
to make things work, were changing "ssl" to "crypto" in
that AC_CHECK_LIB check.

In one of Flameeyes' blog posts it said that somewhere after
binutils 2.19, they've changed the behavior, so perhaps
even current state will work, but it's not that way now.

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=16134&group_id=1824

[fetchmail-devel] [Bug #16000] fetchmail should support multiple sslfingerprint's

[<ad...@be...>]

Bug #16000, was updated on 2009-Jul-15 20:40
Here is a current snapshot of the bug.

Project: Community Fetchmail
Category: None
Status: Closed
Resolution: Invalid
Bug Group: None
Priority: 5
Submitted by: jw9
Assigned to : none
Summary: fetchmail should support multiple sslfingerprint's

Details: Some mail servers, such as pop3.live.com, have multiple hosts available at a single IP address (currently pop3.live.com appears to have two hosts behind that one IP address).

Other mail servers, such as imap.gmail.com, resolve to rotating IP addresses.

It should be possible to specify multiple 'sslfingerprint' entries so that any one of those fingerprints can match.


Follow-Ups:

Date: 2009-Aug-11 17:55
By: grarpamp

Comment:
Adding content for those who use this interface... 

Hi :) Folks appear to be misunderstanding things
about the underlying crypto and why fingerprints are a necessary
safety check.

If one simply relies on checking the cert chain and blindly trusts
that model, as most of the world is conditioned to believe it is
sound, you are open to attack. Namely, of the sort in the posted
url, it is a good read, as are the referenced prior art. Both MD5
and SHA1 are theory broken and MD5 demonstrably so, as noted in the
literatre. So the hash part is of concern, at least in this case.

A CA takes a MD5 over certain fields in the cert and signs that MD5
digest, not the entire cert contents. CA's previously used MD5 and
failed to change their cert practices to SHA1 until a real world
break, oops. Anyways... So I don't have to swipe their private key.
All I have to do to forge a server cert is copy the signature from
any candidate real cert into my cert and manufacture the rest of
my cert contents so that they: a) work, b) make the fields have the
same digest as the real one. Which, as we all know by now is possible,
given time or new art, due to the whole hash collision thing.

However... if I copy a known good cert via trusted channels, and
take both the MD5 and SHA1 fingerprints of the DER form of that
cert, it's much closer to cryptographically impossible to manufacture
a cert that would match evarything. I could just as easily make
formal contact with the cert holder and get the prints that way or
receive them as part of a high security programme. Banks do this.

So there are two different fingerprints covering different areas.

Checking fingerprints over the entire DER cert will foil this attack
because it detects the contents that I manipulated to match the
field digest that was later signed. That is why openssl, browsers,
ssh, etc provide fingerprint calculating and verification functions...
totally separate from signature calc/verif capabilities.

So two types of checks should always be performed for maximum
security:

 A) cert chain trust = sig(hash(contents)) -> pki
  This is done with openssl verify -verbose
  aka: sslcertck
 B) DER fingerprint = hash(sig(hash(contents))
  This is done with openssl x509 -in <PEMcert> -noout -fingerprint -<sha1|md5>
   or
  openssl x509 -in <PEMcert> -outform DER | <sha1|md5>
  aka: sslfingerprint... with the enhancements in the prior note

Enhancing fetchmail to doubly check B with both MD5 and SHa1 shouldn't
be much extra coding or user effort. As an aside, it's fairly common
to find places using/publishing both hashes for various purposes.

Removing the check in B places all the beans on the hash function
in A, which is not good. Private keys are not even involved in this.
Defense in depth will save your butt :)

Give the user tools, let them decide how many they wish to deploy
and manage. I would adapt the docs to say that one should use both
A and B, or neither, with some short reasons why. And continue to
provide example openssl usage as is there now, to allow them to
accomplish it.

Note1: Not checking the fingerprints on self-signed certs... such
as are commonly found in internal corporate nets, edu's, org's, and
the many end user, developer, hacker and other low budget/tier
sites... is asking for trouble. Not publishing your prints, signed
by OpenPGP, is bad too.

Note2: The random serial fix just helps eliminate easier forms of
the hash collision attack, see the paper. It doesn't necessarily
prevent other forms or completely new math attacks, and definitely
doesn't prevent brute force.

Note3: The lists.berlios.de cert is expired, and MD5, though that
doesn't matter :) The Berlios_CA cert may be self-signed.

I'm out of time at the moment to
help out with correcting what is already in here, perhaps later.

Thanks again for keeping up with all the maintenance to fetchmail :)
-------------------------------------------------------

Date: 2009-Aug-11 17:50
By: none

Comment:
Adding content for those who use this interface...

Hi all :)

Fetchmail needs the ability to specify multiple 'sslfingerprint'
options and digests per 'poll' stanza.
  
I noticed this when connecting to gmail. I think it's a load sharing
thing providers use. Google also has their own intermediate CA now
so more cert deployments could be expected.
  
At the moment, if you specify multiple sslfingerprint rc lines or
command line options, only the last one specified is checked. Any
prior sslfingerprint options are silently ignored. And the only
digest available is md5.
  
This is obviously frustrating users mail collection attempts :(
  
  
To reproduce the problem, run this and examine the certs and hashes.
The gmail certs happen to verify back to the root CA's. The
44a8...f1a9 cert occurs more frequently, the 9273...9f33 cert is
rarer. I've attached them both and the CA's involved.
  
If you are not seeing multiple certs, you may need to 'travel' and
debug with different source ip addresses... such as provided with
a worldwide proxy network like Tor.
  
i=10 ; while : ; do
 LD_PRELOAD=/tmp/tor/libtsocks.so.1 TSOCKS_CONF_FILE=/tmp/tor/tsocks.conf \
  openssl s_client -connect pop.gmail.com:pop3s < /dev/null > tcert$i 2>&1
 echo -n "$i: "
 openssl x509 -noout -fingerprint -md5 -in tcert$i
 i=`expr $i + 1`
done
  
  
Also related... from the BRANCH_6-3 NEWS file:
  
* --sslfingerprint may be removed from a future fetchmail version,
 because it's just too easily abused to create a false sense of
 security.
  
PLEASE don't remove this feature. I like it a lot and use it
everywhere possible, with --ssl and --sslcertck at the same time
as well, from trusted sources beforehand of course. Knowing when
the cert changes out from under you is very useful, even if only
for debugging or system verification. Especially if you connect
from untrusted locations or paths. Or you want to know about and
resolve them. It's also handy if the machine a user is connecting
from has a poor, untrusted or nonexistent collection of CA certs.
In that case, just the digest can be imported and used. Note that
openssl does not ship with a CA cert collection... admins usually
have to get it from mozilla... if they value PKI enough to think
of and bother with it.

There are blurbs in the man page that even talk about self-signed
certs being accepted with 'sslcertck'.


Overall, it would be good to see this feature enhanced to do both
md5 and sha1 to make cert collision attacks much more improbable
going forward... at least until the NIST 'SHA-3' hash competition
is done and that trickles down to the CA's and apps years from now.
Usage in the case of a single cert, where both must match, might
be like...

poll
 sslfingerprint-md5 <fp_md5>
 sslfingerprint-sha1 <fp_sha1>

In the case where multiple certs may be presented, group the hashes
as a set. Both hashes in the set must match. First check set cert1,
then set cert2, then set cert[n] until one set, or none, match the
presented cert...

poll
 sslfingerprint-cert1-md5 <fp_md5>
 sslfingerprint-cert1-sha1 <fp_sha1>
 sslfingerprint-cert2-md5 <fp_md5>
 sslfingerprint-cert2-sha1 <fp_sha1>
 sslfingerprint-cert[n]-md5 <fp_md5>
 sslfingerprint-cert[n]-sha1 <fp_sha1>

The '--' command line version of this should also do the right
thing.

Fetchmail should also print both the md5 and sha1 fingerprints from
the received certificate in the debug output.

Note that openssl supports mdc2, md2, md5, sha1. I understand that
some CA's also used, or may still use, the non-sha1 digests. Though
I guess sha1 is the only one used by sane CA's for new certs these
days. I don't know how many old ones are still in use on hosts. md5
and sha1 are enough until SHA-3.

And since the past always seems to apply to some future scenario
with new twists, this is surely not the last of the breed to be
proven out: http://www.win.tue.nl/hashclash/rogue-ca/

Extra checks are always good thing to have available.
Openssl probably has simple library routines to do this.

Thanks guys, and for fetchmail too, been using it for years :)
-------------------------------------------------------

Date: 2009-Jul-16 14:20
By: m-a

Comment:
Compatibility. Basically 6.3.X should be as compatible with 6.2.0 as reasonably possible - and it turned out that 6.3.X has lived for quite a while now... Compatibility was needed at the time to pave the upgrade path for 6.2.X users, as 6.3.0 had around one hundred bug fixes over 6.2.5 - and those were more important than cleanups.
-------------------------------------------------------

Date: 2009-Jul-16 11:44
By: jw9

Comment:
Good.

You should remove sslfingerprint asap.

Cannot understand why the feature was implemented in the first place.  And why such a broken "feature" has endured for so long.

Have a nice day.

-------------------------------------------------------

Date: 2009-Jul-16 09:19
By: m-a

Comment:
[Now the bug tracker is messing things up, my comment appeared twice in spite of being submitted once around 14:35, not again at 15:03. Sorry for that, but it has happened outside my control.]

I don't doubt your technical and intellectual skills to download the server's certificate or its fingerprint. BUT then your security hinges on the useless assumption that your connection is not under attack in the very moment that you're downloading the cert/fingerprint. You can't know if you're under attack unless you can verify the FP through a secure channel: You're using the same channel you'd later use for the actual mail-fetching connection; so you'd happily download the attacker's certificate or fingerprint.

Fetching the fingerprint through a DIFFERENT or at least a known-secure channel is crucial here.

I will not abolish the option for 6.3.X versions for compatibility reasons. If I'll drop it later is left for later decision, as the fingerprint is the easier alternative compared to downloading the *server* certificate and stuffing it into <prefix>/ssl/certs/. The canonical way is still to install the root-signing certificate though.

I don't see much use in listing hundreds of ssl fingerprints and keeping that list up to date when instead you can just save the certificate of the signing CA. It's less of a hassle. And if you can't trust, for instance, CyberTrust, then don't install their certificate, but Microsoft's certificate instead - providing you have a way of verifying that...

I'll see to marking the sslfingerprint feature candidate for removal for 6.4.X or newer fetchmail releases.
-------------------------------------------------------

Date: 2009-Jul-16 09:03
By: m-a

Comment:
Two things:

1a. sslfingerprint is unnecessary in "--sslcertck --sslproto tls1" and in "--sslcertck --sslproto ssl3 --ssl" configurations, and shouldn't be used. It doesn't add security unless your ISP provides you with the finger prints through a separate channel (such as your walking up to the site and fetching a sheet with finger prints).

The assumption is that the MITM (=attacker) cannot forge the signature of the certification authorities you trust. He may forge the server's certificate, but he'll hardly be able to forge the root signing certificate from Equifax, Verisign, or whoever. sslcertck is sufficient unless you follow stupid guides that suggest capturing and saving the server certificate (while that works to shut up complaints about certificates, it doesn't add protection against MITM attacks: you might already download the counterfeit certificate that you save as trusted, if you're already under attack at the time).

1b. sslfingerprint, as you observed, isn't useful with certain load balancing setups, when servers have individual certificates that are signed by a common CA.

2. for complaints about the newest-first ordering of the comments, please file a support request against BerliOS itself, not against individual projects.
-------------------------------------------------------

Date: 2009-Jul-16 09:02
By: jw9

Comment:
I can determine the proper fingerprint from any other server on any other network that I have access to.  I don't have to ask any "ISP" to provide fingerprints through another channel because I am quite able to do that myself.

So, as you state, it does add extra security.  Especially in today's world where most people have access to more than one "ISP" or network (eg your workplace if you have one, and most people do!).

In which case you should allow a set of sslfingerprints to be specified.

Otherwise, to be consistent with your argument (that sslfingerprint is unnecessary), you should completely remove the sslfingerprint option.

Fix it or lose it.

-------------------------------------------------------

Date: 2009-Jul-16 08:35
By: m-a

Comment:
Two things:

1a. sslfingerprint is unnecessary in "--sslcertck --sslproto tls1" and in "--sslcertck --sslproto ssl3 --ssl" configurations, and shouldn't be used. It doesn't add security unless your ISP provides you with the finger prints through a separate channel (such as your walking up to the site and fetching a sheet with finger prints).

The assumption is that the MITM (=attacker) cannot forge the signature of the certification authorities you trust. He may forge the server's certificate, but he'll hardly be able to forge the root signing certificate from Equifax, Verisign, or whoever. sslcertck is sufficient unless you follow stupid guides that suggest capturing and saving the server certificate (while that works to shut up complaints about certificates, it doesn't add protection against MITM attacks: you might already download the counterfeit certificate that you save as trusted, if you're already under attack at the time).

1b. sslfingerprint, as you observed, isn't useful with certain load balancing setups, when servers have individual certificates that are signed by a common CA.

2. for complaints about the newest-first ordering of the comments, please file a support request against BerliOS itself, not against individual projects.
-------------------------------------------------------

Date: 2009-Jul-16 07:09
By: jw9

Comment:
It would appear that additional comments appear in a push-down stack order.  This makes it very hard to follow the natural order of things.  This is very strange.

Here in the western world we are used to reading left-to-right and top-to-bottom.  We certainly don't expect to have to read from bottom-to-top.

I suppose I can turn my monitor upside down and then they might follow in a proper order.  But wait, that would make the characters appear upside-down!

In any case, if one is to be able to check the sslfingerprint of a server to protect against man-in-the-middle attacks, and if the fingerprint does actually change randomly (and not because of a man-in-the-middle attack), then we need to be able to specify a set of fingerprints and only require that at least one of them matches.

-------------------------------------------------------

Date: 2009-Jul-16 07:03
By: jw9

Comment:
Why do comments appear in such a strange place (after my original submission)?  Shouldn't additional comments follow in a logical order after other people's followup comments?  Very strange!

-------------------------------------------------------

Date: 2009-Jul-16 07:00
By: jw9

Comment:
II think you misunderstand.  The sslfingerprint is used to verify the identity of the server and to protect against man-in-the-middle attacks.  The configuration also has proper certificate and "ssslcertck" and "sslcertpath" settings.

To protect again man-in-the-middle attacks one needs to be able to ensure that the fingerprint of the server hasn't changed.

-------------------------------------------------------

Date: 2009-Jul-16 06:33
By: m-a

Comment:
I don't think fetchmail should do that.

sslfingerprints are just a convenience shortcut, and closely resemble stuffing the server's certificate into the directory of trusted certification authority certificates, usually /usr/ssl/certs, /etc/ssl/certs (and running c_rehash afterwards), or similar.

The canonical way is to install the GTE CyberTrust Global Root (for Microsoft/live.com) or the Equifax Secure Certificate Authority (for Google Mail) and use certificate-based authentication.

See also:
http://gagravarr.org/writing/openssl-certs/others.shtml
-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=16000&group_id=1824

[fetchmail-devel] [Bug #16000] fetchmail should support multiple sslfingerprint's

[<ad...@be...>]

Bug #16000, was updated on 2009-Jul-16 02:40
Here is a current snapshot of the bug.

Project: Community Fetchmail
Category: None
Status: Closed
Resolution: Invalid
Bug Group: None
Priority: 5
Submitted by: jw9
Assigned to : none
Summary: fetchmail should support multiple sslfingerprint's

Details: Some mail servers, such as pop3.live.com, have multiple hosts available at a single IP address (currently pop3.live.com appears to have two hosts behind that one IP address).

Other mail servers, such as imap.gmail.com, resolve to rotating IP addresses.

It should be possible to specify multiple 'sslfingerprint' entries so that any one of those fingerprints can match.


Follow-Ups:

Date: 2009-Aug-11 23:50
By: none

Comment:
Adding content for those who use this interface...

Hi all :)

Fetchmail needs the ability to specify multiple 'sslfingerprint'
options and digests per 'poll' stanza.
  
I noticed this when connecting to gmail. I think it's a load sharing
thing providers use. Google also has their own intermediate CA now
so more cert deployments could be expected.
  
At the moment, if you specify multiple sslfingerprint rc lines or
command line options, only the last one specified is checked. Any
prior sslfingerprint options are silently ignored. And the only
digest available is md5.
  
This is obviously frustrating users mail collection attempts :(
  
  
To reproduce the problem, run this and examine the certs and hashes.
The gmail certs happen to verify back to the root CA's. The
44a8...f1a9 cert occurs more frequently, the 9273...9f33 cert is
rarer. I've attached them both and the CA's involved.
  
If you are not seeing multiple certs, you may need to 'travel' and
debug with different source ip addresses... such as provided with
a worldwide proxy network like Tor.
  
i=10 ; while : ; do
 LD_PRELOAD=/tmp/tor/libtsocks.so.1 TSOCKS_CONF_FILE=/tmp/tor/tsocks.conf \
  openssl s_client -connect pop.gmail.com:pop3s < /dev/null > tcert$i 2>&1
 echo -n "$i: "
 openssl x509 -noout -fingerprint -md5 -in tcert$i
 i=`expr $i + 1`
done
  
  
Also related... from the BRANCH_6-3 NEWS file:
  
* --sslfingerprint may be removed from a future fetchmail version,
 because it's just too easily abused to create a false sense of
 security.
  
PLEASE don't remove this feature. I like it a lot and use it
everywhere possible, with --ssl and --sslcertck at the same time
as well, from trusted sources beforehand of course. Knowing when
the cert changes out from under you is very useful, even if only
for debugging or system verification. Especially if you connect
from untrusted locations or paths. Or you want to know about and
resolve them. It's also handy if the machine a user is connecting
from has a poor, untrusted or nonexistent collection of CA certs.
In that case, just the digest can be imported and used. Note that
openssl does not ship with a CA cert collection... admins usually
have to get it from mozilla... if they value PKI enough to think
of and bother with it.

There are blurbs in the man page that even talk about self-signed
certs being accepted with 'sslcertck'.


Overall, it would be good to see this feature enhanced to do both
md5 and sha1 to make cert collision attacks much more improbable
going forward... at least until the NIST 'SHA-3' hash competition
is done and that trickles down to the CA's and apps years from now.
Usage in the case of a single cert, where both must match, might
be like...

poll
 sslfingerprint-md5 <fp_md5>
 sslfingerprint-sha1 <fp_sha1>

In the case where multiple certs may be presented, group the hashes
as a set. Both hashes in the set must match. First check set cert1,
then set cert2, then set cert[n] until one set, or none, match the
presented cert...

poll
 sslfingerprint-cert1-md5 <fp_md5>
 sslfingerprint-cert1-sha1 <fp_sha1>
 sslfingerprint-cert2-md5 <fp_md5>
 sslfingerprint-cert2-sha1 <fp_sha1>
 sslfingerprint-cert[n]-md5 <fp_md5>
 sslfingerprint-cert[n]-sha1 <fp_sha1>

The '--' command line version of this should also do the right
thing.

Fetchmail should also print both the md5 and sha1 fingerprints from
the received certificate in the debug output.

Note that openssl supports mdc2, md2, md5, sha1. I understand that
some CA's also used, or may still use, the non-sha1 digests. Though
I guess sha1 is the only one used by sane CA's for new certs these
days. I don't know how many old ones are still in use on hosts. md5
and sha1 are enough until SHA-3.

And since the past always seems to apply to some future scenario
with new twists, this is surely not the last of the breed to be
proven out: http://www.win.tue.nl/hashclash/rogue-ca/

Extra checks are always good thing to have available.
Openssl probably has simple library routines to do this.

Thanks guys, and for fetchmail too, been using it for years :)
-------------------------------------------------------

Date: 2009-Jul-16 20:20
By: m-a

Comment:
Compatibility. Basically 6.3.X should be as compatible with 6.2.0 as reasonably possible - and it turned out that 6.3.X has lived for quite a while now... Compatibility was needed at the time to pave the upgrade path for 6.2.X users, as 6.3.0 had around one hundred bug fixes over 6.2.5 - and those were more important than cleanups.
-------------------------------------------------------

Date: 2009-Jul-16 17:44
By: jw9

Comment:
Good.

You should remove sslfingerprint asap.

Cannot understand why the feature was implemented in the first place.  And why such a broken "feature" has endured for so long.

Have a nice day.

-------------------------------------------------------

Date: 2009-Jul-16 15:19
By: m-a

Comment:
[Now the bug tracker is messing things up, my comment appeared twice in spite of being submitted once around 14:35, not again at 15:03. Sorry for that, but it has happened outside my control.]

I don't doubt your technical and intellectual skills to download the server's certificate or its fingerprint. BUT then your security hinges on the useless assumption that your connection is not under attack in the very moment that you're downloading the cert/fingerprint. You can't know if you're under attack unless you can verify the FP through a secure channel: You're using the same channel you'd later use for the actual mail-fetching connection; so you'd happily download the attacker's certificate or fingerprint.

Fetching the fingerprint through a DIFFERENT or at least a known-secure channel is crucial here.

I will not abolish the option for 6.3.X versions for compatibility reasons. If I'll drop it later is left for later decision, as the fingerprint is the easier alternative compared to downloading the *server* certificate and stuffing it into <prefix>/ssl/certs/. The canonical way is still to install the root-signing certificate though.

I don't see much use in listing hundreds of ssl fingerprints and keeping that list up to date when instead you can just save the certificate of the signing CA. It's less of a hassle. And if you can't trust, for instance, CyberTrust, then don't install their certificate, but Microsoft's certificate instead - providing you have a way of verifying that...

I'll see to marking the sslfingerprint feature candidate for removal for 6.4.X or newer fetchmail releases.
-------------------------------------------------------

Date: 2009-Jul-16 15:03
By: m-a

Comment:
Two things:

1a. sslfingerprint is unnecessary in "--sslcertck --sslproto tls1" and in "--sslcertck --sslproto ssl3 --ssl" configurations, and shouldn't be used. It doesn't add security unless your ISP provides you with the finger prints through a separate channel (such as your walking up to the site and fetching a sheet with finger prints).

The assumption is that the MITM (=attacker) cannot forge the signature of the certification authorities you trust. He may forge the server's certificate, but he'll hardly be able to forge the root signing certificate from Equifax, Verisign, or whoever. sslcertck is sufficient unless you follow stupid guides that suggest capturing and saving the server certificate (while that works to shut up complaints about certificates, it doesn't add protection against MITM attacks: you might already download the counterfeit certificate that you save as trusted, if you're already under attack at the time).

1b. sslfingerprint, as you observed, isn't useful with certain load balancing setups, when servers have individual certificates that are signed by a common CA.

2. for complaints about the newest-first ordering of the comments, please file a support request against BerliOS itself, not against individual projects.
-------------------------------------------------------

Date: 2009-Jul-16 15:02
By: jw9

Comment:
I can determine the proper fingerprint from any other server on any other network that I have access to.  I don't have to ask any "ISP" to provide fingerprints through another channel because I am quite able to do that myself.

So, as you state, it does add extra security.  Especially in today's world where most people have access to more than one "ISP" or network (eg your workplace if you have one, and most people do!).

In which case you should allow a set of sslfingerprints to be specified.

Otherwise, to be consistent with your argument (that sslfingerprint is unnecessary), you should completely remove the sslfingerprint option.

Fix it or lose it.

-------------------------------------------------------

Date: 2009-Jul-16 14:35
By: m-a

Comment:
Two things:

1a. sslfingerprint is unnecessary in "--sslcertck --sslproto tls1" and in "--sslcertck --sslproto ssl3 --ssl" configurations, and shouldn't be used. It doesn't add security unless your ISP provides you with the finger prints through a separate channel (such as your walking up to the site and fetching a sheet with finger prints).

The assumption is that the MITM (=attacker) cannot forge the signature of the certification authorities you trust. He may forge the server's certificate, but he'll hardly be able to forge the root signing certificate from Equifax, Verisign, or whoever. sslcertck is sufficient unless you follow stupid guides that suggest capturing and saving the server certificate (while that works to shut up complaints about certificates, it doesn't add protection against MITM attacks: you might already download the counterfeit certificate that you save as trusted, if you're already under attack at the time).

1b. sslfingerprint, as you observed, isn't useful with certain load balancing setups, when servers have individual certificates that are signed by a common CA.

2. for complaints about the newest-first ordering of the comments, please file a support request against BerliOS itself, not against individual projects.
-------------------------------------------------------

Date: 2009-Jul-16 13:09
By: jw9

Comment:
It would appear that additional comments appear in a push-down stack order.  This makes it very hard to follow the natural order of things.  This is very strange.

Here in the western world we are used to reading left-to-right and top-to-bottom.  We certainly don't expect to have to read from bottom-to-top.

I suppose I can turn my monitor upside down and then they might follow in a proper order.  But wait, that would make the characters appear upside-down!

In any case, if one is to be able to check the sslfingerprint of a server to protect against man-in-the-middle attacks, and if the fingerprint does actually change randomly (and not because of a man-in-the-middle attack), then we need to be able to specify a set of fingerprints and only require that at least one of them matches.

-------------------------------------------------------

Date: 2009-Jul-16 13:03
By: jw9

Comment:
Why do comments appear in such a strange place (after my original submission)?  Shouldn't additional comments follow in a logical order after other people's followup comments?  Very strange!

-------------------------------------------------------

Date: 2009-Jul-16 13:00
By: jw9

Comment:
II think you misunderstand.  The sslfingerprint is used to verify the identity of the server and to protect against man-in-the-middle attacks.  The configuration also has proper certificate and "ssslcertck" and "sslcertpath" settings.

To protect again man-in-the-middle attacks one needs to be able to ensure that the fingerprint of the server hasn't changed.

-------------------------------------------------------

Date: 2009-Jul-16 12:33
By: m-a

Comment:
I don't think fetchmail should do that.

sslfingerprints are just a convenience shortcut, and closely resemble stuffing the server's certificate into the directory of trusted certification authority certificates, usually /usr/ssl/certs, /etc/ssl/certs (and running c_rehash afterwards), or similar.

The canonical way is to install the GTE CyberTrust Global Root (for Microsoft/live.com) or the Equifax Secure Certificate Authority (for Google Mail) and use certificate-based authentication.

See also:
http://gagravarr.org/writing/openssl-certs/others.shtml
-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=16000&group_id=1824

Re: [fetchmail-devel] [fetchmail-users] Multiple sslfingerprint options and digests

[grarpamp <gra...@gm...>]

Hi :) Folks in that ticket appear to be misunderstanding things
about the underlying crypto and why fingerprints are a necessary
safety check.

If one simply relies on checking the cert chain and blindly trusts
that model, as most of the world is conditioned to believe it is
sound, you are open to attack. Namely, of the sort in the posted
url, it is a good read, as are the referenced prior art. Both MD5
and SHA1 are theory broken and MD5 demonstrably so, as noted in the
literatre. So the hash part is of concern, at least in this case.

A CA takes a MD5 over certain fields in the cert and signs that MD5
digest, not the entire cert contents. CA's previously used MD5 and
failed to change their cert practices to SHA1 until a real world
break, oops. Anyways... So I don't have to swipe their private key.
All I have to do to forge a server cert is copy the signature from
any candidate real cert into my cert and manufacture the rest of
my cert contents so that they: a) work, b) make the fields have the
same digest as the real one. Which, as we all know by now is possible,
given time or new art, due to the whole hash collision thing.

However... if I copy a known good cert via trusted channels, and
take both the MD5 and SHA1 fingerprints of the DER form of that
cert, it's much closer to cryptographically impossible to manufacture
a cert that would match evarything. I could just as easily make
formal contact with the cert holder and get the prints that way or
receive them as part of a high security programme. Banks do this.

So there are two different fingerprints covering different areas.

Checking fingerprints over the entire DER cert will foil this attack
because it detects the contents that I manipulated to match the
field digest that was later signed. That is why openssl, browsers,
ssh, etc provide fingerprint calculating and verification functions...
totally separate from signature calc/verif capabilities.

So two types of checks should always be performed for maximum
security:

 A) cert chain trust = sig(hash(contents)) -> pki
  This is done with openssl verify -verbose
  aka: sslcertck
 B) DER fingerprint = hash(sig(hash(contents))
  This is done with openssl x509 -in <PEMcert> -noout -fingerprint -<sha1|md5>
   or
  openssl x509 -in <PEMcert> -outform DER | <sha1|md5>
  aka: sslfingerprint... with the enhancements in the prior note

Enhancing fetchmail to doubly check B with both MD5 and SHa1 shouldn't
be much extra coding or user effort. As an aside, it's fairly common
to find places using/publishing both hashes for various purposes.

Removing the check in B places all the beans on the hash function
in A, which is not good. Private keys are not even involved in this.
Defense in depth will save your butt :)

Give the user tools, let them decide how many they wish to deploy
and manage. I would adapt the docs to say that one should use both
A and B, or neither, with some short reasons why. And continue to
provide example openssl usage as is there now, to allow them to
accomplish it.

Note1: Not checking the fingerprints on self-signed certs... such
as are commonly found in internal corporate nets, edu's, org's, and
the many end user, developer, hacker and other low budget/tier
sites... is asking for trouble. Not publishing your prints, signed
by OpenPGP, is bad too.

Note2: The random serial fix just helps eliminate easier forms of
the hash collision attack, see the paper. It doesn't necessarily
prevent other forms or completely new math attacks, and definitely
doesn't prevent brute force.

Note3: The lists.berlios.de cert is expired, and MD5, though that
doesn't matter :) The Berlios_CA cert may be self-signed.

Hey, thanks for the pointer to the bug thing! I'll try to append
to the seemingly closed ticket. I'm out of time at the moment to
help out with correcting what is already in there, perhaps later.

And again for keeping up with all the maintenance to fetchmail :)

[fetchmail-devel] Multiple sslfingerprint options and digests

[grarpamp <gra...@gm...>]

Hi all :)

Fetchmail needs the ability to specify multiple 'sslfingerprint'
options and digests per 'poll' stanza.

I noticed this when connecting to gmail. I think it's a load sharing
thing providers use. Google also has their own intermediate CA now
so more cert deployments could be expected.

At the moment, if you specify multiple sslfingerprint rc lines or
command line options, only the last one specified is checked. Any
prior sslfingerprint options are silently ignored. And the only
digest available is md5.

This is obviously frustrating users mail collection attempts :(


To reproduce the problem, run this and examine the certs and hashes.
The gmail certs happen to verify back to the root CA's. The
44a8...f1a9 cert occurs more frequently, the 9273...9f33 cert is
rarer. I've attached them both and the CA's involved.

If you are not seeing multiple certs, you may need to 'travel' and
debug with different source ip addresses... such as provided with
a worldwide proxy network like Tor.

i=10 ; while : ; do
 LD_PRELOAD=/tmp/tor/libtsocks.so.1 TSOCKS_CONF_FILE=/tmp/tor/tsocks.conf \
  openssl s_client -connect pop.gmail.com:pop3s < /dev/null > tcert$i 2>&1
 echo -n "$i: "
 openssl x509 -noout -fingerprint -md5 -in tcert$i
 i=`expr $i + 1`
done


Also related... from the BRANCH_6-3 NEWS file:

* --sslfingerprint may be removed from a future fetchmail version,
 because it's just too easily abused to create a false sense of
 security.

PLEASE don't remove this feature. I like it a lot and use it
everywhere possible, with --ssl and --sslcertck at the same time
as well, from trusted sources beforehand of course. Knowing when
the cert changes out from under you is very useful, even if only
for debugging or system verification. Especially if you connect
from untrusted locations or paths. Or you want to know about and
resolve them. It's also handy if the machine a user is connecting
from has a poor, untrusted or nonexistent collection of CA certs.
In that case, just the digest can be imported and used. Note that
openssl does not ship with a CA cert collection... admins usually
have to get it from mozilla... if they value PKI enough to think
of and bother with it.

There are blurbs in the man page that even talk about self-signed
certs being accepted with 'sslcertck'.


Overall, it would be good to see this feature enhanced to do both
md5 and sha1 to make cert collision attacks much more improbable
going forward... at least until the NIST 'SHA-3' hash competition
is done and that trickles down to the CA's and apps years from now.
Usage in the case of a single cert, where both must match, might
be like...

poll
 sslfingerprint-md5 <fp_md5>
 sslfingerprint-sha1 <fp_sha1>

In the case where multiple certs may be presented, group the hashes
as a set. Both hashes in the set must match. First check set cert1,
then set cert2, then set cert[n] until one set, or none, match the
presented cert...

poll
 sslfingerprint-cert1-md5 <fp_md5>
 sslfingerprint-cert1-sha1 <fp_sha1>
 sslfingerprint-cert2-md5 <fp_md5>
 sslfingerprint-cert2-sha1 <fp_sha1>
 sslfingerprint-cert[n]-md5 <fp_md5>
 sslfingerprint-cert[n]-sha1 <fp_sha1>

The '--' command line version of this should also do the right
thing.

Fetchmail should also print both the md5 and sha1 fingerprints from
the received certificate in the debug output.

Note that openssl supports mdc2, md2, md5, sha1. I understand that
some CA's also used, or may still use, the non-sha1 digests. Though
I guess sha1 is the only one used by sane CA's for new certs these
days. I don't know how many old ones are still in use on hosts. md5
and sha1 are enough until SHA-3.

And since the past always seems to apply to some future scenario
with new twists, this is surely not the last of the breed to be
proven out: http://www.win.tue.nl/hashclash/rogue-ca/

Extra checks are always good thing to have available.
Openssl probably has simple library routines to do this.

Thanks guys, and for fetchmail too, been using it for years :)


Hashes for people googling this:
MD5 Fingerprint=92:73:17:4C:34:4B:68:F7:B2:17:71:42:0D:7F:9F:33
SHA1 Fingerprint=30:36:A5:96:52:B6:64:18:12:E0:2C:98:82:1B:A4:C2:81:4F:BB:B5
34ceaf75
MD5 Fingerprint=44:A8:E9:2C:FB:A9:7E:6D:F9:DB:F3:62:B2:9E:F1:A9
SHA1 Fingerprint=51:21:45:CE:CE:99:19:87:7D:CE:3F:52:C0:31:0F:7E:FB:B4:6A:6F
7f549ca4
MD5 Fingerprint=67:CB:9D:C0:13:24:8A:82:9B:B2:17:1E:D1:1B:EC:D4
SHA1 Fingerprint=D2:32:09:AD:23:D3:14:23:21:74:E4:0D:7F:9D:62:13:97:86:63:3A
594f1775
MD5 Fingerprint=4E:5F:E4:78:5F:B5:68:37:DC:A1:C3:26:F9:F6:73:0B
SHA1 Fingerprint=F8:DF:AB:A0:BF:37:48:24:5B:4E:46:E7:6E:89:82:13:1F:ED:D7:46
d9b7a851

[fetchmail-devel] New Spanish PO file for 'fetchmail' (version 6.3.11)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Spanish team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/es.po

(This file, 'fetchmail-6.3.11.es.po', has just now been sent to you in
a separate email.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Polish PO file for 'fetchmail' (version 6.3.11)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Polish team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/pl.po

(This file, 'fetchmail-6.3.11.pl.po', has just now been sent to you in
a separate email.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Catalan PO file for 'fetchmail' (version 6.3.11)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Catalan team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/ca.po

(This file, 'fetchmail-6.3.11.ca.po', has just now been sent to you in
a separate email.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Japanese PO file for 'fetchmail' (version 6.3.11)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Japanese team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/ja.po

(This file, 'fetchmail-6.3.11.ja.po', has just now been sent to you in
a separate email.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Czech PO file for 'fetchmail' (version 6.3.11)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Czech team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/cs.po

(This file, 'fetchmail-6.3.11.cs.po', has just now been sent to you in
a separate email.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New template for 'fetchmail' made available

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.  (If you have
any questions, send them to <coo...@tr...>.)

A new POT file for textual domain 'fetchmail' has been made available
to the language teams for translation.  It is archived as:

    http://translationproject.org/POT-files/fetchmail-6.3.11.pot

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

Below is the URL which has been provided to the translators of your
package.  Please inform the translation coordinator, at the address
at the bottom, if this information is not current:

    http://www.dt.e-technik.uni-dortmund.de/~ma/fetchmail/fetchmail-6.3.11.tar.bz2

Translated PO files will later be automatically e-mailed to you.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] fetchmail 6.3.11 security fix release available (STABLE)

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings,

I am announcing the release of fetchmail 6.3.11.  This new stable
version of fetchmail fixes a security bug, CVE-2009-2666, and two other
issues, see below for details.

The software is available from:
<http://developer.berlios.de/project/showfiles.php?group_id=1824>

The fetchmail home pages are:
<http://www.fetchmail.info/>  or  <http://fetchmail.berlios.de/>

These are the relevant changes in 6.3.11 since 6.3.10;
unless otherwise noted, changes to this release were made by Matthias Andree:

# SECURITY BUGFIXES
* CVE-2009-2666: SSL NUL prefix impersonation attack through NULs in a
  part of a X.509 certificate's CommonName and subjectAltName fields. These
  fields use opaque strings with a separate length field, so that the NUL
  character isn't a special character inside the certificate. Fetchmail, being
  written in the C language, used to treat these strings as C strings
  nonetheless, so that the domain comparison would end at the first embedded NUL
  character, rather than at the real end of the string.
    Fetchmail will now abort certificate verification as failed if NULs are
  encountered inside either of these fields regardless of their position, and
  drop the connection even if --sslcertck is not used, because NUL is not a
  valid character in legitimate DNS names.
  See fetchmail-SA-2009-01.txt for details, including a minimal patch.

# BUGFIXES
* Remove the spurious message "message delimiter found while scanning headers".
  RFC-5322 syntax states that the delimiter is part of the body, and the body is
  optional.
* Convert all non-printable characters in certificate Subject/Issuer
  Common Name or Subject Alternative Name fields to ANSI-C hex escapes (\xnn,
  where nn are hex digits).

# TRANSLATION UPDATES AND ADDITIONS (ordered by language name):
* [zh_CN] Chinese/Simplified (Ji ZhengYu)
* [es]    Spanish/Castilian (Francisco Molinero)

# ADVANCE WARNING OF FEATURES TO BE REMOVED OR CHANGED IN FUTURE VERSIONS
(There are no plans to remove features from a 6.3.X release, but they may be
removed from a 6.4.0 or newer release.)
* The MX and host alias DNS lookups that fetchmail performs in multidrop mode
  are based on assumptions that are rarely met in practice, somewhat defective,
  deprecated and may be removed from a future fetchmail version.
  They have never supported IPv6 (including IPv6-mapped IPv4).
  Non-DNS based alias keywords such as "aka" will remain in fetchmail.
* The monitor and interface options may be removed from a future fetchmail
  version as they are not reasonably portable across operating systems.
* POP2 is obsolete, support will be removed from a future fetchmail version.
* RPOP is obsolete, support will be removed from a future fetchmail release.
* --sslcertck will become a default setting in a future fetchmail version.
* --sslfingerprint may be removed from a future fetchmail version, because it's
  just too easily abused to create a false sense of security.
* The multidrop To/Cc guessing code along with the fragile duplicate suppressor
  is deprecated and may be removed from a future release.
* The "envelope Received" option may be removed from a future release, because
  the Received header was never meant to be machine-readable, the format varies
  widely, and various other differences in behavior make parsing Received an
  unreliable undertaking. The envelope option as such will remain though, in
  order to support Delivered-To, X-Envelope-To, X-Original-To and similar.
  See also <http://home.pages.de/~mandree/mail/multidrop>.
* The --enable-fallback (fall back to MDA if MTA unavailable) will be removed
  from a future fetchmail release, because it makes fetchmail's behavior
  inconsistent and confusing.
* The "protocol auto" default inside fetchmail may be removed from a future
  fetchmail release. Explicit configuration of the protocol is recommended.
* Kerberos IV support may be removed from a future fetchmail release.
* SIGHUP wakeup support may be removed from a future fetchmail release and
  cause fetchmail to terminate - it was broken for many years.
* Support for operating systems that are not sufficiently POSIX compliant may be
  removed or operation on such systems may be suboptimal for future releases.
  This means that fetchmail may only continue to work on C99 and POSIX 2001
  based systems.
* The maintainer may migrate fetchmail to C++ with STL or C#, and impose further
  requirements (dependencies), such as Boost or other class libraries.
* The softbounce option default will change to "false" in the next release.

- -- 
Matthias Andree
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkp6HawACgkQvmGDOQUufZVGJwCfUVdPy62qU0zffx8I4RPFGlNg
KosAoMaqkDewu0x/+QSpCLL5yz5eg8fU
=lTMq
-----END PGP SIGNATURE-----

Re: [fetchmail-devel] Tracking pending POP3 deletes to solve some problems

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Thu, Jul 23, 2009:
> > I mean, please merge my UID parser code patch in fetchmail, if it is
> > still suitable.
> 
> >> > the UID parser also had an option to mark bad mails. This would be
> >> > used in such cases where there is a repeated delivery failure on the
> >> > same mail. Once a certain bad count is reached, fetchmail will stop
> >> > attempting to download the mail.
> >>
> >> I don't think fetchmail has such a feature in the baseline code. The
> >> internal uid data structure is:
> >
> >> From my UID parser code patch:
> 
> Is  
> http://funknet.net/fetchmail/patches/2003-11-27-6.2.5-shetye-splituid.diff  
> the patch you're talking of?

Yes. Is that patch so old? Obviously, that patch is not going to apply
cleanly.

I thought that when you mentioned about changing the .fetchids format
as discussed in 2006, you were referring to this patch. Sorry for
digressing.

-- 
Sunil Shetye.

Re: [fetchmail-devel] Tracking pending POP3 deletes to solve some problems

[Matthias A. <mat...@gm...>]

Am 23.07.2009, 12:36 Uhr, schrieb Sunil Shetye  
<sh...@bo...>:

> I mean, please merge my UID parser code patch in fetchmail, if it is
> still suitable.

>> > the UID parser also had an option to mark bad mails. This would be
>> > used in such cases where there is a repeated delivery failure on the
>> > same mail. Once a certain bad count is reached, fetchmail will stop
>> > attempting to download the mail.
>>
>> I don't think fetchmail has such a feature in the baseline code. The
>> internal uid data structure is:
>
>> From my UID parser code patch:

Is  
http://funknet.net/fetchmail/patches/2003-11-27-6.2.5-shetye-splituid.diff  
the patch you're talking of?

> struct uidlist
> {
>     union
>     {
>        unsigned char *id;
>        long int nid;                   /* IMAP ids are integers! */
>     };
>     int                num;
>     int                mark;           /* UID-index information */
> #define UID_UNSEEN     0               /* hasn't been seen */
> #define UID_SEEN       1               /* seen, but not deleted */
> #define UID_DELETED    2               /* this message has been deleted  
> */
> #define UID_EXPUNGED   3               /* this message has been expunged  
> */
> #define UID_OVERSIZED  4               /* this message is oversized */
> #define UID_SKIPPED    5               /* this message has been skipped  
> */
> #define UID_OLD_DELETED        6       /* this message was deleted but  
> not expunged! */
> #define UID_ERROR      99              /* processing error */
>     time_t     dltime;                 /* time of delivery */
>     int                errcount;       /* count of errors while  
> downloading this mail */
>     int                size;           /* size of the mail */
>     struct uidlist *next;
> };

-- 
Matthias Andree

Re: [fetchmail-devel] Tracking pending POP3 deletes to solve some problems

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Thu, Jul 23, 2009:
> > ====================================================================
> > Case 1: After fetchmail has sent the initial MAIL FROM: to the SMTP
> > server/invoked the MDA with %F substitution:
> >
> > I am assuming here that the SMTP server/MDA is doing a DNS
> > lookup/basic spam testing on the sender address.
> >
> > So, in the SMTP server case, fetchmail is waiting for a response from
> > the server after sending the initial MAIL FROM: while the SMTP server
> > is doing the DNS lookup/basic spam testing.
> >
> > In the MDA case, fetchmail goes ahead with popen() and starts writing
> > the body to the MDA. As the MDA is still doing the DNS lookup/basic
> > spam testing and has not yet started reading the body, the pipe buffer
> > will get full and fetchmail will get blocked on an fwrite() later.
> >
> > While fetchmail is blocked, the POP3 mailserver is waiting for
> > fetchmail to read the entire mail body.
> >
> > Note that the write() timeout for the POP3 mailserver may be shorter
> > than the read() timeout. This means that the POP3 mailserver is more
> > likely to timeout faster if it finds that the body is not getting
> > drained at all in a reasonable amount of time.
> > ====================================================================
> > Case 2: After fetchmail has sent the entire mail to the SMTP server/MDA:
> >
> > Here, the remote mailserver is waiting for a command from fetchmail,
> > while fetchmail is waiting for a response from the SMTP server/exit
> > code from the MDA.
> >
> > As mentioned above, the read() timeout may be longer for the POP3
> > mailserver and so it may not mind waiting for the next command.
> > ====================================================================
> 
> I'm not sure if I want to design fetchmail around guesses if read() or  
> write() timeouts on the server are set differently.

What I mean to say is that Case 1 above must be occurring more
frequently than reported. Hence, it should be tackled first. Of
course, I could be wrong and so Case 2 should be tackled first.

> >> a. queue downloaded messages before handing them off for delivery. This
> >> avoids timeouts that originate in the SMTP/LMTP server or MDA that
> >> fetchmail forwards to.
> >
> > This should work. Of course, fetchmail will have to work entirely with
> > UIDs as it will have to reconnect later and mark delivered mails for
> > deletion.
> 
> Yup. That's what I want to do anyways in the next after-6.3.N releases (if  
> I'll call them 6.4 or 6.5 or 7.0, I'll decide later).
> 
> So we'd have to polish the existing UID patches for IMAP to support  
> UIDVALIDITY (not a major issue, once you detect UIDVALIDITY changes, you  
> discard all stored UIDs) - OTOH if fetchmail deals cleanly with the  
> existing \Seen and \Deleted flags, we don't even need that for IMAP. I  
> need to check the IMAP4r1 transaction model though.

Ok.

> >> Fixing 2 is sort of a requisite for solving 1 in way a or c - we need to
> >> track more state. This does entail changing the .fetchids format as
> >> discussed in 2006, but the UID parser appeared very tolerant even at  
> >> that
> >> time, so that an extension would be possible and backwards compatible. I
> >> would feel more comfortable checking that again, but I think I checked
> >> thoroughly in 2006 already. Even if we must change the .fetchids
> >> format/layout, I'm open to it.
> >
> > Well, changing the .fetchids format is anyway a must. If you can
> > incorporate the UID parser, it will be great. If I remember correctly,
> 
> I'm not sure what you mean by "incorporate" here.

I mean, please merge my UID parser code patch in fetchmail, if it is
still suitable.

> > the UID parser also had an option to mark bad mails. This would be
> > used in such cases where there is a repeated delivery failure on the
> > same mail. Once a certain bad count is reached, fetchmail will stop
> > attempting to download the mail.
> 
> I don't think fetchmail has such a feature in the baseline code. The  
> internal uid data structure is:

From my UID parser code patch:

struct uidlist
{
    union
    {
       unsigned char *id;
       long int nid;                   /* IMAP ids are integers! */
    };
    int                num;
    int                mark;           /* UID-index information */
#define UID_UNSEEN     0               /* hasn't been seen */
#define UID_SEEN       1               /* seen, but not deleted */
#define UID_DELETED    2               /* this message has been deleted */
#define UID_EXPUNGED   3               /* this message has been expunged */
#define UID_OVERSIZED  4               /* this message is oversized */
#define UID_SKIPPED    5               /* this message has been skipped */
#define UID_OLD_DELETED        6       /* this message was deleted but not expunged! */
#define UID_ERROR      99              /* processing error */
    time_t     dltime;                 /* time of delivery */
    int                errcount;       /* count of errors while downloading this mail */
    int                size;           /* size of the mail */
    struct uidlist *next;
};

> I'm considering a general solution that doesn't require such an analysis,  
> but solves all of the issues at the same time.
> 
> WRT tracking the DELE/QUIT races in POP3, I am wondering about the  
> handling of the QUIT. Can we see a difference between "server hasn't  
> received QUIT" and "we haven't seen the answer"? In other words, will the  
> server's TCP stack hand the QUIT command to the server application  
> software even if TCP couldn't send the ACK? I think it will, because the  
> ACK itself needn't be ACKed and the server often won't care if we don't  
> see the +OK after QUIT...
> 
> The other option is top track UIDs with "to be deleted" and "deleted, QUIT  
> +OK pending" and "deleted, QUIT acknowledged":
> 
> - "QUIT acknowledged" is easy, we don't save that state per UID, but just  
> drop the corresponding UID as the server will do the same.
> 
> - "to be deleted" means we're positively sure that the transaction was  
> rolled back (because we haven't sent the QUIT command) - we need a  
> workaround server option though, because some servers can be configured to  
> spoil the protocol dangerously and commit DELEtes on loss of connection  
> unless there's a RSET. We can assume the server won't reassign the UID  
> until the next cycle (*)
> 
> - "deleted, QUIT +OK pending" is for your borderline case, we've sent the  
> QUIT to the TCP/IP stack but haven't seen the +OK response. If we see more  
> than half of the UIDs marked QUIT +OK pending in the next cycle, we'll  
> mark them "to be deleted", if it's less than half, we'll forget them and  
> re-fetch. The other option is to hash a subset of whitespace-normalized  
> message headers (Received, Message-ID, perhaps others, making sure to  
> avoid X-Status or other mutable headers) to accompany the UID. We could  
> hash headers as they pass by in forwarding and only re-fetch them in your  
> "we send QUIT but don't see +OK" case if we don't trust the UID. I wonder  
> if we should do that.

Well, the 'UID_OLD_DELETED' mark mentioned above was meant to address these
cases.

> WRT getting stuck on one message, we could record message UIDs and mark  
> them as "fetch attempted before", perhaps with a counter. We'd set this  
> state if we ever send a TOP or RETR for a new message and keep this state  
> for the next poll cycle. This would be less than "seen". On the next poll  
> cycle, we'll fetch new messages before those marked "fetch attempted  
> before". This would allow new mail to be fetched even if we get stuck on  
> particular messages through server, fetchmail, or MTA/MDA bugs. If we add  
> a counter, we can mark a message "broken" if the counter exceeds a  
> threshold and give up on it without deleting it, and request manual  
> intervention from the postmaster (in multidrop) or addressee (in  
> singledrop).

The 'errcount' field mentioned above was meant for this purpose.

-- 
Sunil Shetye.

Re: [fetchmail-devel] Tracking pending POP3 deletes to solve some problems

[Matthias A. <mat...@gm...>]

Am 22.07.2009, 18:20 Uhr, schrieb Sunil Shetye  
<sh...@bo...>:


> There are actually two possibilities where the delay could be
> occurring.
>
> ====================================================================
> Case 1: After fetchmail has sent the initial MAIL FROM: to the SMTP
> server/invoked the MDA with %F substitution:
>
> I am assuming here that the SMTP server/MDA is doing a DNS
> lookup/basic spam testing on the sender address.
>
> So, in the SMTP server case, fetchmail is waiting for a response from
> the server after sending the initial MAIL FROM: while the SMTP server
> is doing the DNS lookup/basic spam testing.
>
> In the MDA case, fetchmail goes ahead with popen() and starts writing
> the body to the MDA. As the MDA is still doing the DNS lookup/basic
> spam testing and has not yet started reading the body, the pipe buffer
> will get full and fetchmail will get blocked on an fwrite() later.
>
> While fetchmail is blocked, the POP3 mailserver is waiting for
> fetchmail to read the entire mail body.
>
> Note that the write() timeout for the POP3 mailserver may be shorter
> than the read() timeout. This means that the POP3 mailserver is more
> likely to timeout faster if it finds that the body is not getting
> drained at all in a reasonable amount of time.
> ====================================================================
> Case 2: After fetchmail has sent the entire mail to the SMTP server/MDA:
>
> Here, the remote mailserver is waiting for a command from fetchmail,
> while fetchmail is waiting for a response from the SMTP server/exit
> code from the MDA.
>
> As mentioned above, the read() timeout may be longer for the POP3
> mailserver and so it may not mind waiting for the next command.
> ====================================================================

I'm not sure if I want to design fetchmail around guesses if read() or  
write() timeouts on the server are set differently.

>> Generally, I see several approaches to 1:
>>
>> a. queue downloaded messages before handing them off for delivery. This
>> avoids timeouts that originate in the SMTP/LMTP server or MDA that
>> fetchmail forwards to.
>
> This should work. Of course, fetchmail will have to work entirely with
> UIDs as it will have to reconnect later and mark delivered mails for
> deletion.

Yup. That's what I want to do anyways in the next after-6.3.N releases (if  
I'll call them 6.4 or 6.5 or 7.0, I'll decide later).

So we'd have to polish the existing UID patches for IMAP to support  
UIDVALIDITY (not a major issue, once you detect UIDVALIDITY changes, you  
discard all stored UIDs) - OTOH if fetchmail deals cleanly with the  
existing \Seen and \Deleted flags, we don't even need that for IMAP. I  
need to check the IMAP4r1 transaction model though.

>> b. Alternatively, we could try making fetchmail multithreaded and  
>> keeping the POP3 server happy by spamming it with NOOP. I'm not sure  
>> how good this works, how many POP3 servers implement NOOP, how many  
>> NOOP in sequence
>> they tolerate. Given fetchmail's design, it's very intrusive and amounts
>> to a rewrite of several major parts. It would have other benefits, but
>> it's a major effort.
>
> This will not work in Case 1. There, the POP3 mailserver is obviously
> in no mood for NOOPs and may even treat it as a protocol error if it
> gets a command even though the complete body has not been sent.

True. So this approach is out for yet another reason.

>> c. Alternatively, we could try to reconnect after loss of connection -
>> however, we may lose prior DELE commands when we don't send QUIT, so
>> again, we need to bundle DELE requests at the end or for a separate
>> transaction.  Given that many sites (including hotmail, where Tony had  
>> his
>> problem) limit the number of logins per unit of time, often to once per  
>> 15
>> minutes, we can't preventively send QUIT so as not to lock ourselves  
>> out.
>> Anyways, the solution means we would do 2.
>
> In Case 1 above, the POP3 mailserver may have timed out before sending
> the entire body. If this is happening repeatedly, fetchmail will
> always fail on the same mail.

Also solved by queueing (a.)

>> Fixing 2 is sort of a requisite for solving 1 in way a or c - we need to
>> track more state. This does entail changing the .fetchids format as
>> discussed in 2006, but the UID parser appeared very tolerant even at  
>> that
>> time, so that an extension would be possible and backwards compatible. I
>> would feel more comfortable checking that again, but I think I checked
>> thoroughly in 2006 already. Even if we must change the .fetchids
>> format/layout, I'm open to it.
>
> Well, changing the .fetchids format is anyway a must. If you can
> incorporate the UID parser, it will be great. If I remember correctly,

I'm not sure what you mean by "incorporate" here.

> the UID parser also had an option to mark bad mails. This would be
> used in such cases where there is a repeated delivery failure on the
> same mail. Once a certain bad count is reached, fetchmail will stop
> attempting to download the mail.

I don't think fetchmail has such a feature in the baseline code. The  
internal uid data structure is:

struct idlist
{
     char *id;
     union
     {
         struct
         {
             int         num;
             flag        mark;           /* UID-index information */
#define UID_UNSEEN      0               /* hasn't been seen */
#define UID_SEEN        1               /* seen, but not deleted */
#define UID_DELETED     2               /* this message has been marked  
deleted */
#define UID_EXPUNGED    3               /* this message has been expunged  
*/
         }
         status;
         char *id2;
     } val;
     struct idlist *next;
};


You may be referring to fetchmail marking /servers/ "wedged" if it sees  
too many timeouts on a particular server (this only works in daemon mode).

>> Functionally, we'd probably need to bundle DELEs into a bulk operation  
>> of
>> "DELE n1 DELE n2 DELE n3 ... DELE nm QUIT" so that we have a reasonable
>> chance that the server isn't going away from boredom between the first
>> DELE and the QUIT, and we have more chances to avoid UID reassignment  
>> and
>> "delete wrong message" issues that happen in the race Sunil described,  
>> i.
>> e. if the network dies if the server executes QUIT but fetchmail doesn't
>> see the +OK response.
>
> This should be possible.
>
> I have not gone through the cases you have mentioned yet, but it would
> be better to categorize them as Case 1 or Case 2 (or both!) first
> before deciding the course of action. For SMTP server, it will be
> simple as the time between the SMTP transactions will give a clear
> indication in syslog. For MDA, this will probably require an strace
> output.

I'm considering a general solution that doesn't require such an analysis,  
but solves all of the issues at the same time.

WRT tracking the DELE/QUIT races in POP3, I am wondering about the  
handling of the QUIT. Can we see a difference between "server hasn't  
received QUIT" and "we haven't seen the answer"? In other words, will the  
server's TCP stack hand the QUIT command to the server application  
software even if TCP couldn't send the ACK? I think it will, because the  
ACK itself needn't be ACKed and the server often won't care if we don't  
see the +OK after QUIT...

The other option is top track UIDs with "to be deleted" and "deleted, QUIT  
+OK pending" and "deleted, QUIT acknowledged":

- "QUIT acknowledged" is easy, we don't save that state per UID, but just  
drop the corresponding UID as the server will do the same.

- "to be deleted" means we're positively sure that the transaction was  
rolled back (because we haven't sent the QUIT command) - we need a  
workaround server option though, because some servers can be configured to  
spoil the protocol dangerously and commit DELEtes on loss of connection  
unless there's a RSET. We can assume the server won't reassign the UID  
until the next cycle (*)

- "deleted, QUIT +OK pending" is for your borderline case, we've sent the  
QUIT to the TCP/IP stack but haven't seen the +OK response. If we see more  
than half of the UIDs marked QUIT +OK pending in the next cycle, we'll  
mark them "to be deleted", if it's less than half, we'll forget them and  
re-fetch. The other option is to hash a subset of whitespace-normalized  
message headers (Received, Message-ID, perhaps others, making sure to  
avoid X-Status or other mutable headers) to accompany the UID. We could  
hash headers as they pass by in forwarding and only re-fetch them in your  
"we send QUIT but don't see +OK" case if we don't trust the UID. I wonder  
if we should do that.

(*) there is another borderline case, that is: UID reassignment if ANOTHER  
client (other than fetchmail) deletes messages. I think we need to rule  
that out through documentation and tell users that only *one particular*  
client must ever delete messages from a POP3 server. If that's THunderbird  
or something, the user will have to run fetchmail in "keep" mode,  
otherwise, if he runs fetchmail without "keep", the other POP3 client must  
be configured to leave ALL messages on the server.


WRT getting stuck on one message, we could record message UIDs and mark  
them as "fetch attempted before", perhaps with a counter. We'd set this  
state if we ever send a TOP or RETR for a new message and keep this state  
for the next poll cycle. This would be less than "seen". On the next poll  
cycle, we'll fetch new messages before those marked "fetch attempted  
before". This would allow new mail to be fetched even if we get stuck on  
particular messages through server, fetchmail, or MTA/MDA bugs. If we add  
a counter, we can mark a message "broken" if the counter exceeds a  
threshold and give up on it without deleting it, and request manual  
intervention from the postmaster (in multidrop) or addressee (in  
singledrop).

Seems like I should draw the full finite state machine (FSM)...

Best regards

-- 
Matthias Andree

Re: [fetchmail-devel] Tracking pending POP3 deletes to solve some problems

[Sunil S. <sh...@bo...>]

Dear Matthias,

Quoting from Matthias Andree's mail on Wed, Jul 22, 2009:
> There seems to be a general issue with fetchmail around fetching large  
> messages, spam filtering, timeouts and thereabouts.
> 
> 
> Some three years ago I made a foray into tracking pending DELEtes that we  
> could no longer issue or that were reverted by POP3 servers through socket  
> errors (often timeouts or network failures, not too clear to me at the  
> time), for reference, see
> 
>   - https://lists.berlios.de/pipermail/fetchmail-users/2006-May/000409.html
> 
> and followups. At the time, we discussed several concerns around this  
> issue and I eventually shelved a solution for lack of ideas and time.

...

> It seems the problem has two faces:
> 
> 1. Timeout on the server we're fetching from, because the SMTP server or  
> MDA takes too long.

There are actually two possibilities where the delay could be
occurring.

====================================================================
Case 1: After fetchmail has sent the initial MAIL FROM: to the SMTP
server/invoked the MDA with %F substitution:

I am assuming here that the SMTP server/MDA is doing a DNS
lookup/basic spam testing on the sender address. 

So, in the SMTP server case, fetchmail is waiting for a response from
the server after sending the initial MAIL FROM: while the SMTP server
is doing the DNS lookup/basic spam testing.

In the MDA case, fetchmail goes ahead with popen() and starts writing
the body to the MDA. As the MDA is still doing the DNS lookup/basic
spam testing and has not yet started reading the body, the pipe buffer
will get full and fetchmail will get blocked on an fwrite() later.

While fetchmail is blocked, the POP3 mailserver is waiting for
fetchmail to read the entire mail body.

Note that the write() timeout for the POP3 mailserver may be shorter
than the read() timeout. This means that the POP3 mailserver is more
likely to timeout faster if it finds that the body is not getting
drained at all in a reasonable amount of time.
====================================================================
Case 2: After fetchmail has sent the entire mail to the SMTP server/MDA:

Here, the remote mailserver is waiting for a command from fetchmail,
while fetchmail is waiting for a response from the SMTP server/exit
code from the MDA.

As mentioned above, the read() timeout may be longer for the POP3
mailserver and so it may not mind waiting for the next command.
====================================================================

Of course, a combination of the above two cases is also possible.

> Generally, I see several approaches to 1:
> 
> a. queue downloaded messages before handing them off for delivery. This  
> avoids timeouts that originate in the SMTP/LMTP server or MDA that  
> fetchmail forwards to.

This should work. Of course, fetchmail will have to work entirely with
UIDs as it will have to reconnect later and mark delivered mails for
deletion.

> b. Alternatively, we could try making fetchmail multithreaded and keeping  
> the POP3 server happy by spamming it with NOOP. I'm not sure how good this  
> works, how many POP3 servers implement NOOP, how many NOOP in sequence  
> they tolerate. Given fetchmail's design, it's very intrusive and amounts  
> to a rewrite of several major parts. It would have other benefits, but  
> it's a major effort.

This will not work in Case 1. There, the POP3 mailserver is obviously
in no mood for NOOPs and may even treat it as a protocol error if it
gets a command even though the complete body has not been sent.

> c. Alternatively, we could try to reconnect after loss of connection -  
> however, we may lose prior DELE commands when we don't send QUIT, so  
> again, we need to bundle DELE requests at the end or for a separate  
> transaction.  Given that many sites (including hotmail, where Tony had his  
> problem) limit the number of logins per unit of time, often to once per 15  
> minutes, we can't preventively send QUIT so as not to lock ourselves out.  
> Anyways, the solution means we would do 2.

In Case 1 above, the POP3 mailserver may have timed out before sending
the entire body. If this is happening repeatedly, fetchmail will
always fail on the same mail.

> Fixing 2 is sort of a requisite for solving 1 in way a or c - we need to  
> track more state. This does entail changing the .fetchids format as  
> discussed in 2006, but the UID parser appeared very tolerant even at that  
> time, so that an extension would be possible and backwards compatible. I  
> would feel more comfortable checking that again, but I think I checked  
> thoroughly in 2006 already. Even if we must change the .fetchids  
> format/layout, I'm open to it.

Well, changing the .fetchids format is anyway a must. If you can
incorporate the UID parser, it will be great. If I remember correctly,
the UID parser also had an option to mark bad mails. This would be
used in such cases where there is a repeated delivery failure on the
same mail. Once a certain bad count is reached, fetchmail will stop
attempting to download the mail.

> Functionally, we'd probably need to bundle DELEs into a bulk operation of  
> "DELE n1 DELE n2 DELE n3 ... DELE nm QUIT" so that we have a reasonable  
> chance that the server isn't going away from boredom between the first  
> DELE and the QUIT, and we have more chances to avoid UID reassignment and  
> "delete wrong message" issues that happen in the race Sunil described, i.  
> e. if the network dies if the server executes QUIT but fetchmail doesn't  
> see the +OK response.

This should be possible.

I have not gone through the cases you have mentioned yet, but it would
be better to categorize them as Case 1 or Case 2 (or both!) first
before deciding the course of action. For SMTP server, it will be
simple as the time between the SMTP transactions will give a clear
indication in syslog. For MDA, this will probably require an strace
output.

-- 
Sunil Shetye.

[fetchmail-devel] Tracking pending POP3 deletes to solve some problems

[Matthias A. <mat...@gm...>]

Dear Sunil, dear list readers,

There seems to be a general issue with fetchmail around fetching large  
messages, spam filtering, timeouts and thereabouts.


Some three years ago I made a foray into tracking pending DELEtes that we  
could no longer issue or that were reverted by POP3 servers through socket  
errors (often timeouts or network failures, not too clear to me at the  
time), for reference, see

  - https://lists.berlios.de/pipermail/fetchmail-users/2006-May/000409.html

and followups. At the time, we discussed several concerns around this  
issue and I eventually shelved a solution for lack of ideas and time.

Originally reported by Frédéric Marchal around that time (2006 or 2005, I  
didn't check).

However, I have had at least two plausible reports (perhaps more that I  
don't currently recall or connect) since. In these reports, the local  
delivery took so long that the server disconnected fetchmail before it  
could even send the DELE, or fetchmail disconnected itself if the SMTP  
server took too long to actually accept a message.

One newer issue was filed to the Berlios bug tracker as #10972 by Viktor  
Binzberger in 2007.

The other was reported to the fetchmail-users@ list last Friday  
(2009-07-17) by Tony Morehen; I have received a full log off-list.

While Viktor's problem is worked around in fetchmail 6.3.10 through  
extending the SMTP client timeouts to the minimum RFC 5321 (SMTP,  
obsoleting RFC 2821 and RFC 821) recommended periods, this isn't going to  
help if the POP3 server drops the connection as early as in Tony's case -  
plus he's using an MDA anyways.


It seems the problem has two faces:

1. Timeout on the server we're fetching from, because the SMTP server or  
MDA takes too long.

2. Tracking what messages were successfully fetched and delivered, and  
which subset of these were flushed.


Generally, I see several approaches to 1:

a. queue downloaded messages before handing them off for delivery. This  
avoids timeouts that originate in the SMTP/LMTP server or MDA that  
fetchmail forwards to.

b. Alternatively, we could try making fetchmail multithreaded and keeping  
the POP3 server happy by spamming it with NOOP. I'm not sure how good this  
works, how many POP3 servers implement NOOP, how many NOOP in sequence  
they tolerate. Given fetchmail's design, it's very intrusive and amounts  
to a rewrite of several major parts. It would have other benefits, but  
it's a major effort.

c. Alternatively, we could try to reconnect after loss of connection -  
however, we may lose prior DELE commands when we don't send QUIT, so  
again, we need to bundle DELE requests at the end or for a separate  
transaction.  Given that many sites (including hotmail, where Tony had his  
problem) limit the number of logins per unit of time, often to once per 15  
minutes, we can't preventively send QUIT so as not to lock ourselves out.  
Anyways, the solution means we would do 2.


Another workaround is making sure that your MTA or MDA accepts messages  
quickly (a few seconds at most, as common in Postfix unless set to  
non-default pre-queue content inspection) and uses the spam filter  
off-line after accepting the message. I understand that it's not always  
the best solution, but since the POP3 or IMAP server has accepted the  
message already, we seriously can only either forward or drop it anyways -  
bouncing is dangerous, since the envelope sender might be forged; I think  
messages can only sensibly be rejected before being accepted by your  
respective ISP through SMTP.

We could add a trivial queueing agent separately and link it through the  
MDA option for early experiments (sort of nullmailer for inbound  
messages), but


Fixing 2 is sort of a requisite for solving 1 in way a or c - we need to  
track more state. This does entail changing the .fetchids format as  
discussed in 2006, but the UID parser appeared very tolerant even at that  
time, so that an extension would be possible and backwards compatible. I  
would feel more comfortable checking that again, but I think I checked  
thoroughly in 2006 already. Even if we must change the .fetchids  
format/layout, I'm open to it.

Functionally, we'd probably need to bundle DELEs into a bulk operation of  
"DELE n1 DELE n2 DELE n3 ... DELE nm QUIT" so that we have a reasonable  
chance that the server isn't going away from boredom between the first  
DELE and the QUIT, and we have more chances to avoid UID reassignment and  
"delete wrong message" issues that happen in the race Sunil described, i.  
e. if the network dies if the server executes QUIT but fetchmail doesn't  
see the +OK response.


Anybody care to share their thoughts?  I'm looking forward to them.

Best regards and thanks for your time.

-- 
Matthias Andree

Re: [fetchmail-devel] NO messages

[Rob M. <rob...@gm...>]

On Sun, Jul 19, 2009 at 00:11, ml41782<ml...@ya...> wrote:
>
> I'm setup to fetch messages from my ISP to my linux box. Ubuntu 9.04
> I'm not seeing any messages on my box. So I started logging.

> fetchmail: SMTP connect to localhost failed
> fetchmail: SMTP transaction error while fetching from
> mlu...@va...@mail.vab1.com and delivering to SMTP host localhost
> fetchmail: Query status=10 (SMTP)

It appears that your SMTP server is either not running or
misconfigured.  Check it's configuration.

If that doesn't resolve your problem, please read section G3 of the
Fetchmail FAQ.

-- 
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

[fetchmail-devel] NO messages

[ml41782 <ml...@ya...>]

I'm setup to fetch messages from my ISP to my linux box. Ubuntu 9.04
I'm not seeing any messages on my box. So I started logging. 




fetchmail: reading message mlu...@va...@mail.emailsrvr.com:1 of 42 (1534 octets) (log message incomplete)fetchmail: SMTP connect to localhost failed
fetchmail: SMTP transaction error while fetching from mlu...@va...@mail.vab1.com and delivering to SMTP host localhost
fetchmail: Query status=10 (SMTP)
fetchmail: sleeping at Sat 18 Jul 2009 06:51:04 PM EDT for 240 seconds
fetchmail: terminated with signal 15
fetchmail: starting fetchmail 6.3.9-rc2 daemon 
fetchmail: 42 messages for mlu...@va... at secure.emailsrvr.com (66848 octets).
fetchmail: reading message mlu...@va...@secure.emailsrvr.com:1 of 42 (1534 octets) (log message incomplete)fetchmail: terminated with signal 15
fetchmail: starting fetchmail 6.3.9-rc2 daemon 
fetchmail: 42 messages for mlu...@va... at secure.emailsrvr.com (66848 octets).


here is my .fetchmailrc file

otg1017@Webserver:~$ cat .fetchmailrc   permissions are set to 600
set logfile /home/otg1017/tmp/fetchmail.log
set daemon 240
poll "secure.emailsrvr.com"
protocol pop3
user 'mlu...@va...' there with password 'ABCXYZ123' is 'otg1017' here
keep


Keep is set because if I dont then my messages are gone. 


      

[fetchmail-devel] [Bug #16000] fetchmail should support multiple sslfingerprint's

[<ad...@be...>]

Bug #16000, was updated on 2009-Jul-16 02:40
Here is a current snapshot of the bug.

Project: Community Fetchmail
Category: None
Status: Closed
Resolution: Invalid
Bug Group: None
Priority: 5
Submitted by: jw9
Assigned to : none
Summary: fetchmail should support multiple sslfingerprint's

Details: Some mail servers, such as pop3.live.com, have multiple hosts available at a single IP address (currently pop3.live.com appears to have two hosts behind that one IP address).

Other mail servers, such as imap.gmail.com, resolve to rotating IP addresses.

It should be possible to specify multiple 'sslfingerprint' entries so that any one of those fingerprints can match.


Follow-Ups:

Date: 2009-Jul-16 20:20
By: m-a

Comment:
Compatibility. Basically 6.3.X should be as compatible with 6.2.0 as reasonably possible - and it turned out that 6.3.X has lived for quite a while now... Compatibility was needed at the time to pave the upgrade path for 6.2.X users, as 6.3.0 had around one hundred bug fixes over 6.2.5 - and those were more important than cleanups.
-------------------------------------------------------

Date: 2009-Jul-16 17:44
By: jw9

Comment:
Good.

You should remove sslfingerprint asap.

Cannot understand why the feature was implemented in the first place.  And why such a broken "feature" has endured for so long.

Have a nice day.

-------------------------------------------------------

Date: 2009-Jul-16 15:19
By: m-a

Comment:
[Now the bug tracker is messing things up, my comment appeared twice in spite of being submitted once around 14:35, not again at 15:03. Sorry for that, but it has happened outside my control.]

I don't doubt your technical and intellectual skills to download the server's certificate or its fingerprint. BUT then your security hinges on the useless assumption that your connection is not under attack in the very moment that you're downloading the cert/fingerprint. You can't know if you're under attack unless you can verify the FP through a secure channel: You're using the same channel you'd later use for the actual mail-fetching connection; so you'd happily download the attacker's certificate or fingerprint.

Fetching the fingerprint through a DIFFERENT or at least a known-secure channel is crucial here.

I will not abolish the option for 6.3.X versions for compatibility reasons. If I'll drop it later is left for later decision, as the fingerprint is the easier alternative compared to downloading the *server* certificate and stuffing it into <prefix>/ssl/certs/. The canonical way is still to install the root-signing certificate though.

I don't see much use in listing hundreds of ssl fingerprints and keeping that list up to date when instead you can just save the certificate of the signing CA. It's less of a hassle. And if you can't trust, for instance, CyberTrust, then don't install their certificate, but Microsoft's certificate instead - providing you have a way of verifying that...

I'll see to marking the sslfingerprint feature candidate for removal for 6.4.X or newer fetchmail releases.
-------------------------------------------------------

Date: 2009-Jul-16 15:03
By: m-a

Comment:
Two things:

1a. sslfingerprint is unnecessary in "--sslcertck --sslproto tls1" and in "--sslcertck --sslproto ssl3 --ssl" configurations, and shouldn't be used. It doesn't add security unless your ISP provides you with the finger prints through a separate channel (such as your walking up to the site and fetching a sheet with finger prints).

The assumption is that the MITM (=attacker) cannot forge the signature of the certification authorities you trust. He may forge the server's certificate, but he'll hardly be able to forge the root signing certificate from Equifax, Verisign, or whoever. sslcertck is sufficient unless you follow stupid guides that suggest capturing and saving the server certificate (while that works to shut up complaints about certificates, it doesn't add protection against MITM attacks: you might already download the counterfeit certificate that you save as trusted, if you're already under attack at the time).

1b. sslfingerprint, as you observed, isn't useful with certain load balancing setups, when servers have individual certificates that are signed by a common CA.

2. for complaints about the newest-first ordering of the comments, please file a support request against BerliOS itself, not against individual projects.
-------------------------------------------------------

Date: 2009-Jul-16 15:02
By: jw9

Comment:
I can determine the proper fingerprint from any other server on any other network that I have access to.  I don't have to ask any "ISP" to provide fingerprints through another channel because I am quite able to do that myself.

So, as you state, it does add extra security.  Especially in today's world where most people have access to more than one "ISP" or network (eg your workplace if you have one, and most people do!).

In which case you should allow a set of sslfingerprints to be specified.

Otherwise, to be consistent with your argument (that sslfingerprint is unnecessary), you should completely remove the sslfingerprint option.

Fix it or lose it.

-------------------------------------------------------

Date: 2009-Jul-16 14:35
By: m-a

Comment:
Two things:

1a. sslfingerprint is unnecessary in "--sslcertck --sslproto tls1" and in "--sslcertck --sslproto ssl3 --ssl" configurations, and shouldn't be used. It doesn't add security unless your ISP provides you with the finger prints through a separate channel (such as your walking up to the site and fetching a sheet with finger prints).

The assumption is that the MITM (=attacker) cannot forge the signature of the certification authorities you trust. He may forge the server's certificate, but he'll hardly be able to forge the root signing certificate from Equifax, Verisign, or whoever. sslcertck is sufficient unless you follow stupid guides that suggest capturing and saving the server certificate (while that works to shut up complaints about certificates, it doesn't add protection against MITM attacks: you might already download the counterfeit certificate that you save as trusted, if you're already under attack at the time).

1b. sslfingerprint, as you observed, isn't useful with certain load balancing setups, when servers have individual certificates that are signed by a common CA.

2. for complaints about the newest-first ordering of the comments, please file a support request against BerliOS itself, not against individual projects.
-------------------------------------------------------

Date: 2009-Jul-16 13:09
By: jw9

Comment:
It would appear that additional comments appear in a push-down stack order.  This makes it very hard to follow the natural order of things.  This is very strange.

Here in the western world we are used to reading left-to-right and top-to-bottom.  We certainly don't expect to have to read from bottom-to-top.

I suppose I can turn my monitor upside down and then they might follow in a proper order.  But wait, that would make the characters appear upside-down!

In any case, if one is to be able to check the sslfingerprint of a server to protect against man-in-the-middle attacks, and if the fingerprint does actually change randomly (and not because of a man-in-the-middle attack), then we need to be able to specify a set of fingerprints and only require that at least one of them matches.

-------------------------------------------------------

Date: 2009-Jul-16 13:03
By: jw9

Comment:
Why do comments appear in such a strange place (after my original submission)?  Shouldn't additional comments follow in a logical order after other people's followup comments?  Very strange!

-------------------------------------------------------

Date: 2009-Jul-16 13:00
By: jw9

Comment:
II think you misunderstand.  The sslfingerprint is used to verify the identity of the server and to protect against man-in-the-middle attacks.  The configuration also has proper certificate and "ssslcertck" and "sslcertpath" settings.

To protect again man-in-the-middle attacks one needs to be able to ensure that the fingerprint of the server hasn't changed.

-------------------------------------------------------

Date: 2009-Jul-16 12:33
By: m-a

Comment:
I don't think fetchmail should do that.

sslfingerprints are just a convenience shortcut, and closely resemble stuffing the server's certificate into the directory of trusted certification authority certificates, usually /usr/ssl/certs, /etc/ssl/certs (and running c_rehash afterwards), or similar.

The canonical way is to install the GTE CyberTrust Global Root (for Microsoft/live.com) or the Equifax Secure Certificate Authority (for Google Mail) and use certificate-based authentication.

See also:
http://gagravarr.org/writing/openssl-certs/others.shtml
-------------------------------------------------------

For detailed info, follow this link:
http://developer.berlios.de/bugs/?func=detailbug&bug_id=16000&group_id=1824