fetchmail-users — User questions and discussion

Re: [Fetchmail-users] fetchmail vs divecot server setup for pop3 and imap

[Gene H. <ghe...@sh...>]

On Friday 06 March 2020 15:18:37 Matthias Andree wrote:

> Am 06.03.20 um 19:48 schrieb Gene Heskett:
> > fetchmail: POP3> DELE 9
> > fetchmail: POP3< +OK Marked to be deleted.
> > fetchmail: POP3> QUIT
> > fetchmail: POP3< +OK Logging out, messages deleted.
> > fetchmail: 6.4.2 querying pop.shentel.net (protocol POP3) at Fri 06
> > Mar 2020 01:43:17 PM EST: poll completed
> > fetchmail: New UID list from pop.shentel.net:
> >  <empty>
> > fetchmail: not swapping UID lists, no UIDs seen this query
> > fetchmail: sleeping at Fri 06 Mar 2020 01:43:17 PM EST for 120
> > seconds
> >
> > A blast of 9 messages, but despite claiming deleted, they are not,
> > and there's 247 to nuke right now.
> >
> > Your turn & thank you.
>
> You're welcome. Actually it's your ISP's turn now, because if their
> Dovecot claims the messages deleted and the same messages remain,
> something needs fixing on their end.
>
And their response has always been that they have clients using both 
interfaces who would be upset to discover that mail fetched by a 
mailfetcher and then deleted was missing when they next came in with a 
#$^(&%* browser. I drop html encoded email in my spam collector 
directory. usually unread because its generally spam.
>
>
> _______________________________________________
> Fetchmail-users mailing list
> Fet...@li...
> https://lists.sourceforge.net/lists/listinfo/fetchmail-users


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] fetchmail vs divecot server setup for pop3 and imap

[Matthias A. <mat...@gm...>]

Am 06.03.20 um 19:48 schrieb Gene Heskett:
>
> fetchmail: POP3> DELE 9
> fetchmail: POP3< +OK Marked to be deleted.
> fetchmail: POP3> QUIT
> fetchmail: POP3< +OK Logging out, messages deleted.
> fetchmail: 6.4.2 querying pop.shentel.net (protocol POP3) at Fri 06 Mar 
> 2020 01:43:17 PM EST: poll completed
> fetchmail: New UID list from pop.shentel.net:
>  <empty>
> fetchmail: not swapping UID lists, no UIDs seen this query
> fetchmail: sleeping at Fri 06 Mar 2020 01:43:17 PM EST for 120 seconds
>
> A blast of 9 messages, but despite claiming deleted, they are not, and 
> there's 247 to nuke right now.
>
> Your turn & thank you. 
You're welcome. Actually it's your ISP's turn now, because if their
Dovecot claims the messages deleted and the same messages remain,
something needs fixing on their end.

Re: [Fetchmail-users] fetchmail vs divecot server setup for pop3 and imap

[Gene H. <ghe...@sh...>]

On Friday 06 March 2020 10:54:26 Matthias Andree wrote:

> Am 06.03.20 um 15:32 schrieb Gene Heskett:
> > What sort of a debugging command would expose that to my perusal?
>
> just make sure you don't use -s or --silent and add one -v or
> --verbose.
>
> Then check the syslog or logfile if output wasn't to console (Note the
> logfile must exist before running fetchmail, else logging is lost).
>
> >> Because on well-behaved
> >> POP3 servers, DELE are only deletion requests that are queued up
> >> and kept pending unapplied until the client sends a "QUIT" command,
> >> when the DELE will actually be committed.
> >>
> >> Also, does your mail store have proper permissions for Dovecot to
> >> actually write, if you are sharing it with Webmail?
> >
> > My mail store is the /home/gene/Mail/etc database tde's version of
> > kmail-1.9 maintains.  So I guess I am not understanding what you are
> > seeking.
>
> If you see a QUIT/+OK conversation at the end of the verbose log, then
> that "cannot DELE" is a question to ask your ISP.

You will have to tell me where to put that --verbose. An attempt to add 
it to my launcher script is an exit causeing syntax error.

here is the script that starts it for his uptime as it also used by my 
sa-learn script to restart it after being killed for the duration of the 
sa-learn execution, done daily.

#!/bin/sh
killall fetchmail
sleep 2
/usr/local/bin/fetchmail -d120 -v -v --fetchmailrc /home/gene/.fetchmailrc

is the script that restarts it, located in /home/gene/bin since I cannot 
seem to start it in /etc/init.d, so I have to start it as me after 
logging in.

I got well p.o.ed  trying to keep logs in /var/logs, logrotate kept 
changeing perms=no permission, so both fetchmail.log and procmail.log 
have been moved to /home/gene/log, no perms problems there.

Strangely enough the double -v -v now works, it did not work with 6.3.26, 
ever. But since I succeeded in launching the new version, there's bee no 
new mail. The -v -v is 100% verbose and I will need to remove it 
shortly. But now I am waiting for a mail no more kmail just beeped at 
me.
trace logged:
fetchmail:  flushed
fetchmail: POP3> DELE 8
fetchmail: POP3< +OK Marked to be deleted.
fetchmail: POP3> LIST 9
fetchmail: POP3< +OK 9 7036
fetchmail: POP3> RETR 9
fetchmail: POP3< +OK 7036 octets
fetchmail: reading message ghe...@sh...@pop.shentel.net:9 of 9 
(7036 octets)About to rewrite Return-Path: 
<lin...@vg...>...
...rewritten version is Return-Path: 
<lin...@vg...>.
fetchmail: About to rewrite From:   Steven Rostedt 
<ro...@go...>...
...rewritten version is From:   Steven Rostedt <ro...@go...>.
fetchmail: About to rewrite To:     lin...@vg...,...
...rewritten version is To:     lin...@vg...,.
fetchmail: About to rewrite Cc:     Thomas Gleixner 
<tg...@li...>,...
...rewritten version is Cc:     Thomas Gleixner <tg...@li...>,.
fetchmail: About to rewrite Sender: 
lin...@vg......
...rewritten version is Sender: lin...@vg....
fetchmail: about to deliver with: /usr/bin/procmail -d gene
fetchmail:  flushed
fetchmail: POP3> DELE 9
fetchmail: POP3< +OK Marked to be deleted.
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Logging out, messages deleted.
fetchmail: 6.4.2 querying pop.shentel.net (protocol POP3) at Fri 06 Mar 
2020 01:43:17 PM EST: poll completed
fetchmail: New UID list from pop.shentel.net:
 <empty>
fetchmail: not swapping UID lists, no UIDs seen this query
fetchmail: sleeping at Fri 06 Mar 2020 01:43:17 PM EST for 120 seconds

A blast of 9 messages, but despite claiming deleted, they are not, and 
there's 247 to nuke right now.

Your turn & thank you. 

>
> _______________________________________________
> Fetchmail-users mailing list
> Fet...@li...
> https://lists.sourceforge.net/lists/listinfo/fetchmail-users


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] UTF-8 support?

[Matthias A. <mat...@gm...>]

Am 06.03.20 um 15:48 schrieb Sergey Organov:
>> Yup, and client support doesn't seem abundant. Python's poplib appears
>> to support it in recent versions.
>>
>> So, bottom line: I'll put this on the TODO for 6.5, but I will probably
>> not implement downgrading or conversion code, fetchmail will need to
>> assume the destination SMTP/LMTP server or MDA is 8bit clean
>> (8BITMIME/SMTPUTF8).
>>
>> This will take a while.
>> https://gitlab.com/fetchmail/fetchmail/-/issues/14
> Thanks a lot for taking care of this, Matthias!
>
> Right now getting 3-4 such surrogate mails a day is not a big deal, but
> it might start to affect more people in the future, as these new servers
> start to spread.
>
The question is how much open UTF-8 is really sent - this would not be
overly compatible and will still often need downconversion to 7-bit mail...

Re: [Fetchmail-users] fetchmail vs divecot server setup for pop3 and imap

[Matthias A. <mat...@gm...>]

Am 06.03.20 um 15:32 schrieb Gene Heskett:
> What sort of a debugging command would expose that to my perusal? 

just make sure you don't use -s or --silent and add one -v or --verbose.

Then check the syslog or logfile if output wasn't to console (Note the
logfile must exist before running fetchmail, else logging is lost).

>> Because on well-behaved 
>> POP3 servers, DELE are only deletion requests that are queued up and
>> kept pending unapplied until the client sends a "QUIT" command, when
>> the DELE will actually be committed.
>>
>> Also, does your mail store have proper permissions for Dovecot to
>> actually write, if you are sharing it with Webmail?
> My mail store is the /home/gene/Mail/etc database tde's version of 
> kmail-1.9 maintains.  So I guess I am not understanding what you are 
> seeking.
If you see a QUIT/+OK conversation at the end of the verbose log, then
that "cannot DELE" is a question to ask your ISP.

Re: [Fetchmail-users] UTF-8 support?

[Sergey O. <sor...@gm...>]

Matthias Andree <mat...@gm...> writes:

> Am 06.03.20 um 14:19 schrieb Sergey Organov:

[...]

>> Anyway, there is no other mail client by fetchmail on my side that
>> connects to the server, and my googling about the problem revealed that
>> some recent mail servers start to issue such warning e-mail indeed when
>> they can't detect proper UTF-8 support when server needs to present a
>> message with UTF-8 encoded headers to client.
> Yup, and client support doesn't seem abundant. Python's poplib appears
> to support it in recent versions.
>
> So, bottom line: I'll put this on the TODO for 6.5, but I will probably
> not implement downgrading or conversion code, fetchmail will need to
> assume the destination SMTP/LMTP server or MDA is 8bit clean
> (8BITMIME/SMTPUTF8).
>
> This will take a while.
> https://gitlab.com/fetchmail/fetchmail/-/issues/14

Thanks a lot for taking care of this, Matthias!

Right now getting 3-4 such surrogate mails a day is not a big deal, but
it might start to affect more people in the future, as these new servers
start to spread.

Thanks again and have a nice weekend!

-- Sergey

Re: [Fetchmail-users] fetchmail vs divecot server setup for pop3 and imap

[Gene H. <ghe...@sh...>]

On Friday 06 March 2020 09:01:04 Matthias Andree wrote:

> Am 06.03.20 um 14:28 schrieb Gene Heskett:
> > greetings all;
> >
> > And dovecot ignores fetchmails dele, make me login with a browser to
> > delete old mail I've already fetched. Since traffic is several
> > hundred a day, thats rather a nuisance.
>
> Gene, what distro is your Dovecot on?

Not my dovecot, my ISP's.

> Normally I consider it 
> well-behaved. Can you confirm that your session completes through the
> "QUIT" command and does not abort prematurely?

What sort of a debugging command would expose that to my perusal?

> Because on well-behaved 
> POP3 servers, DELE are only deletion requests that are queued up and
> kept pending unapplied until the client sends a "QUIT" command, when
> the DELE will actually be committed.
>
> Also, does your mail store have proper permissions for Dovecot to
> actually write, if you are sharing it with Webmail?

My mail store is the /home/gene/Mail/etc database tde's version of 
kmail-1.9 maintains.  So I guess I am not understanding what you are 
seeking.

Thanks.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] UTF-8 support?

[Matthias A. <mat...@gm...>]

Am 06.03.20 um 14:19 schrieb Sergey Organov:
> Thanks, that's what I've expected.
>
> To be honest, I don't even understand what actual problem those 2 RFCs
> are to solve. I mean I don't understand why server can't just send what
> it has "as is".

Traditionally, from "way back then"; meaning short after the first human
landing on the moon, we used parity bits in US-ASCII systems and nobody
thought of umlauts, Cyrillic or whatnot. The early transmission
protocols were 7-bit protocols, no 8th bit permitted, hence crutches
such as the UTF-7 stuff in IMAP, MIME content transfer encodings
(base64, quoted-printable) for headers and whatnot.

Some transmission protocols actively prohibit sending 8 bit data, and
have been extended that clients and servers can negotiate 8-bit clean
transmissions.

The UTF8 RFC set strives to pull the plug from that "you must only send
7-bit data" requirement, but requires negotiation between client and server.

> Do you mean: https://tools.ietf.org/html/rfc6858 ?
> Dunno, but to me it looks like server rather does not create surrogate
> messages.
You just showed us one, and I'll leave this snippet in.
>>> Please use an E-mail reader that supports POP3 with UTF-8
>>> (see https://tools.ietf.org/html/rfc6856.html).
>>>
>>> This can also happen when the sender's E-mail program does not
>>> correctly format the sent message.
>> What is your mail server type and address?
> It's corporate server mail.javad.com and I dunno its type. Probably I
> can figure through our IT team if needed.
No need, it identifies on the greeting line as Courier-IMAP. It appears
that Courier-IMAP 5.0.0 introduced UTF8 support in the way you're
currently seeing, quoting its ChangeLog:
> 5.0.0
>
> 2018-07-21  Sam Varshavchik  <mr...@co...>
>
>         * pop3dserver.c: update Courier-IMAP to support UTF8 POP3. Update
>         version of the courierpop3dsizelist cache file.
>
> 2018-07-17  Sam Varshavchik  <mr...@co...>
>
>         * courier-imap, sqwebmail: update Courier-IMAP to support UTF8
> IMAP.
>         Convert maildir folders to use UTF-8 for folder names. Add
>         --checkutf8 and --convutf8 options to maildirmake to convert
>         pre-UTF8 maildirs to UTF8 maildirs, a mandatory upgrade procedure.
>
> ...

> If I get a patch, I think I can test it against the server. I guess
> it should be something rather simple like sending additional
> command claiming UTF-8 support? 

Nah, there's a bit more to it because at least for IMAP I've seen this
incurs protocol changes from UTF-7 to UTF-8 after sending it, and I
haven't fully read the RFCs yet to know what's on the table to do with
them, so we'll at least need to:

* implement CAPABILITY detection

* implement sending UTF8 willingness commands where applicable

* assess where protocol is changed in a way that affects fetchmail's
command generators and response parsers

* assess where fetchmail documentation needs to be changed

* TBD


And it's not just that we can assume UTF8 a given. It's been around for
years, and thank you for pointing me to the RFCs, but I queried all my
servers that I regularly poll and none talks UTF8 today except my
Courier IMAP 5.0.8 test installation.

So nothing's simple...

>
> I didn't try to include attachment and I'd rather not, as it might be
> confidential.
Oh, of course. I only now understand that the mail server was wrapping
the original message with this surrogate banner.
> Why? Suppose the server got proper e-mail with UTF-8 encoded headers.
> Now fetchmail tries to fetch the e-mail, and server in turn tries to
> ensure that it will send content that will be understood by the
> requester. As fetchmail doesn't support relevant RFC indeed, server
> decides support is absent and issues warning message.
Actually it's more of a wrapper message, on second reading of your
original post.
> Anyway, there is no other mail client by fetchmail on my side that
> connects to the server, and my googling about the problem revealed that
> some recent mail servers start to issue such warning e-mail indeed when
> they can't detect proper UTF-8 support when server needs to present a
> message with UTF-8 encoded headers to client.
Yup, and client support doesn't seem abundant. Python's poplib appears
to support it in recent versions.

So, bottom line: I'll put this on the TODO for 6.5, but I will probably
not implement downgrading or conversion code, fetchmail will need to
assume the destination SMTP/LMTP server or MDA is 8bit clean
(8BITMIME/SMTPUTF8).

This will take a while. https://gitlab.com/fetchmail/fetchmail/-/issues/14

Re: [Fetchmail-users] fetchmail vs divecot server setup for pop3 and imap

[Matthias A. <mat...@gm...>]

Am 06.03.20 um 14:28 schrieb Gene Heskett:
> greetings all;
>
> And dovecot ignores fetchmails dele, make me login with a browser to
> delete old mail I've already fetched. Since traffic is several hundred a
> day, thats rather a nuisance.

Gene, what distro is your Dovecot on? Normally I consider it
well-behaved. Can you confirm that your session completes through the
"QUIT" command and does not abort prematurely? Because on well-behaved
POP3 servers, DELE are only deletion requests that are queued up and
kept pending unapplied until the client sends a "QUIT" command, when the
DELE will actually be committed.

Also, does your mail store have proper permissions for Dovecot to
actually write, if you are sharing it with Webmail?

[Fetchmail-users] fetchmail vs divecot server setup for pop3 and imap

[Gene H. <ghe...@sh...>]

greetings all;

And dovecot ignores fetchmails dele, make me login with a browser to 
delete old mail I've already fetched. Since traffic is several hundred a 
day, thats rather a nuisance.

Is there an option I can switch on to make it act like a proper pop3 
client in the face of dovecot's ignoreing the dele?

Thanks, 6.4.2 just installed, 6.3.26 was sent a SIGHUP and other than 
logging it, has gone on its merry way, now reporting its 6.4.2.  Thats 
how a locally built install is supposed to work, thank you.

6.3.26 has served long and well here.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] UTF-8 support?

[Sergey O. <sor...@gm...>]

Matthias Andree <mat...@gm...> writes:

> Am 06.03.20 um 12:01 schrieb Sergey Organov:
>> Hello,
>>
>> Does fetchmail support UTF-8 for IMAP and POP, as described in:
>>
>> https://tools.ietf.org/html/rfc6855
>>
>> and
>>
>> https://tools.ietf.org/html/rfc6856
>>
>> ?
>
> Fetchmail is 8-bit clean, but no release in existence today sends UTF8
> commands.

Thanks, that's what I've expected.

To be honest, I don't even understand what actual problem those 2 RFCs
are to solve. I mean I don't understand why server can't just send what
it has "as is".

>
>> I get my mail using fetchmail  and recently started to receive
>> notification e-mails of the form:
>>
>> Subject: Cannot display Unicode content
>>
>> This E-mail message was determined to be Unicode-formatted
>> but your E-mail reader does not support Unicode E-mail.
> E-Mail UTF-8 is a different thing that protocol commands and responses
> being UTF-8. But it might be that your server is creating surrogate
> messages.

Do you mean: https://tools.ietf.org/html/rfc6858 ?

Dunno, but to me it looks like server rather does not create surrogate
messages.

>> Please use an E-mail reader that supports POP3 with UTF-8
>> (see https://tools.ietf.org/html/rfc6856.html).
>>
>> This can also happen when the sender's E-mail program does not
>> correctly format the sent message.
>
> What is your mail server type and address?

It's corporate server mail.javad.com and I dunno its type. Probably I
can figure through our IT team if needed.

> It might be that the server detects "client not sending UTF-8 but
> non-ASCII stuff in content" and decides "so send this warning e-mail
> instead of real content".
>
> I don't have a UTF8 server to test against apparently... so this would
> make debugging impossible.

If I get a patch, I think I can test it against the server. I guess
it should be something rather simple like sending additional
command claiming UTF-8 support? 

>
>> The original message is included as a separate attachment
>> so that it can be downloaded manually.
>>

> The list strips attachments, please send it to me again off-list,

I didn't try to include attachment and I'd rather not, as it might be
confidential.

> but I'd guess that this is actually a failure of the mail client or
> the sender's mail client.

Why? Suppose the server got proper e-mail with UTF-8 encoded headers.
Now fetchmail tries to fetch the e-mail, and server in turn tries to
ensure that it will send content that will be understood by the
requester. As fetchmail doesn't support relevant RFC indeed, server
decides support is absent and issues warning message.

Anyway, there is no other mail client by fetchmail on my side that
connects to the server, and my googling about the problem revealed that
some recent mail servers start to issue such warning e-mail indeed when
they can't detect proper UTF-8 support when server needs to present a
message with UTF-8 encoded headers to client.

-- Sergey

Re: [Fetchmail-users] UTF-8 support?

[Matthias A. <mat...@gm...>]

Am 06.03.20 um 12:01 schrieb Sergey Organov:
> Hello,
>
> Does fetchmail support UTF-8 for IMAP and POP, as described in:
>
> https://tools.ietf.org/html/rfc6855
>
> and
>
> https://tools.ietf.org/html/rfc6856
>
> ?

Fetchmail is 8-bit clean, but no release in existence today sends UTF8
commands.

> I get my mail using fetchmail  and recently started to receive
> notification e-mails of the form:
>
> Subject: Cannot display Unicode content
>
> This E-mail message was determined to be Unicode-formatted
> but your E-mail reader does not support Unicode E-mail.
E-Mail UTF-8 is a different thing that protocol commands and responses
being UTF-8. But it might be that your server is creating surrogate
messages.
> Please use an E-mail reader that supports POP3 with UTF-8
> (see https://tools.ietf.org/html/rfc6856.html).
>
> This can also happen when the sender's E-mail program does not
> correctly format the sent message.

What is your mail server type and address? It might be that the server
detects "client not sending UTF-8 but non-ASCII stuff in content" and
decides "so send this warning e-mail instead of real content".

I don't have a UTF8 server to test against apparently... so this would
make debugging impossible.

> The original message is included as a separate attachment
> so that it can be downloaded manually.
>
The list strips attachments, please send it to me again off-list, but
I'd guess that this is actually a failure of the mail client or the
sender's mail client.

Re: [Fetchmail-users] UTF-8 support?

[Sergey O. <sor...@gm...>]

"Carlos E. R." <rob...@te...> writes:

> On 06/03/2020 12.01, Sergey Organov wrote:
>> Hello,
>>
>> Does fetchmail support UTF-8 for IMAP and POP, as described in:
>>
>> https://tools.ietf.org/html/rfc6855
>>
>> and
>>
>> https://tools.ietf.org/html/rfc6856
>>
>> ?
>>
>> I get my mail using fetchmail  and recently started to receive
>> notification e-mails of the form:
>>
>> Subject: Cannot display Unicode content
>>
>> This E-mail message was determined to be Unicode-formatted
>> but your E-mail reader does not support Unicode E-mail.
>
> I don't think fetchmail (or any mail downloader) has any effect. The
> content is not altered at all, it may come in any charset.

Yes, but it looks like fetchmail doesn't tell the server that this is
the case, and I don't find anything about it in the fetchmail manual.

> It is your mail client software which has to interpret it.

fetchmail is the mail client here, the tool that connect to the POP
server. POP sever has no idea what kind of client fetchmail is, and if
fetchmail fails to claim UTF-8 support properly, POP server issues the
warning I get.

That's how I see the situation.

-- 
Sergey

Re: [Fetchmail-users] UTF-8 support?

[Carlos E. R. <rob...@te...>]

On 06/03/2020 12.01, Sergey Organov wrote:
> Hello,
> 
> Does fetchmail support UTF-8 for IMAP and POP, as described in:
> 
> https://tools.ietf.org/html/rfc6855
> 
> and
> 
> https://tools.ietf.org/html/rfc6856
> 
> ?
> 
> I get my mail using fetchmail  and recently started to receive
> notification e-mails of the form:
> 
> Subject: Cannot display Unicode content
> 
> This E-mail message was determined to be Unicode-formatted
> but your E-mail reader does not support Unicode E-mail.

I don't think fetchmail (or any mail downloader) has any effect. The 
content is not altered at all, it may come in any charset. It is your 
mail client software which has to interpret it.

-- 
Cheers / Saludos,

		Carlos E. R.
		(from 15.1 x86_64 at Telcontar)

[Fetchmail-users] UTF-8 support?

[Sergey O. <sor...@gm...>]

Hello,

Does fetchmail support UTF-8 for IMAP and POP, as described in:

https://tools.ietf.org/html/rfc6855

and

https://tools.ietf.org/html/rfc6856

?

I get my mail using fetchmail  and recently started to receive
notification e-mails of the form:

Subject: Cannot display Unicode content

This E-mail message was determined to be Unicode-formatted
but your E-mail reader does not support Unicode E-mail.

Please use an E-mail reader that supports POP3 with UTF-8
(see https://tools.ietf.org/html/rfc6856.html).

This can also happen when the sender's E-mail program does not
correctly format the sent message.

The original message is included as a separate attachment
so that it can be downloaded manually.

-- 
Sergey

[Fetchmail-users] ANNOUNCE: fetchmail-6.4.2 has been released today, 2020-02-14 (fetchmailconf overhaul, bugfixes)

[Matthias A. <mat...@gm...>]

Greetings,

I have released fetchmail-6.4.2, it is available from sourceforge.net, see:
<https://sourceforge.net/projects/fetchmail/files/branch_6.4/>

It is a RECOMMENDED update for all users, it fixes one SSL bug, updates
the manual page and Chinese translation,
and brings fetchmailconf into the 2020's, by making it compatible with Python 3
(but it requires the future package, see below).

The distribution is now in lzip (*) format, but xz format remains available.
(*) https://www.nongnu.org/lzip/

Older releases are herewith officially unsupported.

Where to get it - Deep link to tarball:
<https://sourceforge.net/projects/fetchmail/files/branch_6.4/fetchmail-6.4.2.tar.lz/download>
How to verify:
GnuPG detached signature:
<https://sourceforge.net/projects/fetchmail/files/branch_6.4/fetchmail-6.4.2.tar.lz.asc/download>

RIPEMD160(fetchmail-6.4.2.tar.lz)= 87b8b51bb8c781ce59e817a4189cd76783ac313b
SHA1(fetchmail-6.4.2.tar.lz)= 5e3b1cffcf80634d4a9ac2b16d10c4260285fb6b
SHA256(fetchmail-6.4.2.tar.lz)= 971361dca9bea227154c715ee88098b8ffc66fd4dbdb0dbbf1c77ace59e1461a
SHA512(fetchmail-6.4.2.tar.lz)= f8f7466f1891ffd0a72a042e4b942056acf1c0f153f44cb096e5e0aa5847400257e2880a78eaa682db94074205d9a311449e9743ed6cd254a8732d3afe3ef9a0

RIPEMD160(fetchmail-6.4.2.tar.xz)= 16faf2d06aa568f14f334b6c616d6ac1a34983d2
SHA1(fetchmail-6.4.2.tar.xz)= cdd6f0f4c8b99be3d3950763c53472c4ad764312
SHA256(fetchmail-6.4.2.tar.xz)= e21f6b3326f29fdb0c4786b5602aa4b9e668805424d0708eb42be6395c1ca630
SHA512(fetchmail-6.4.2.tar.xz)= 8ec62a5df81b9b8c5e5479d35a10aded22aca74f671cded339dc7ae1c78d8a8638dfe4f04be35334184b5b27f3fb857402dc5b587cca8ede3f5b9b268b37edc1

Status:

I plan to only do bug fixes to fetchmail-6.4.x so that upgrading
to newer 6.4.x releases should be acceptable under many "stable branch"
maintenance policies. More intrusive changes will happen to versions
that have higher numbers such as 6.5.y or 7.n.m (with m, n, y being numbers).


These are the release notes from NEWS:

--------------------------------------------------------------------------------
fetchmail-6.4.2 (released 2020-02-14, 27473 LoC):

## BREAKING CHANGES:
* fetchmailconf now supports Python 3 and currently requires the "future" 
  package, see https://pypi.org/project/future/.
* fetchmailconf: The minimum supported version is now Python 2.7.13, but it is 
  recommended to use at least 2.7.16 (due to its massive SSL updates).
  Older Python versions may check SSL certificates not strictly enough,
  which may cause fetchmail to complain later, if the certificate verify fails.
* fetchmailconf now autoprobes SSL-wrapped connections (ports 993 and 995 for 
  IMAP and POP3) as well and by preference.
* fetchmailconf now defaults newly created users to "ssl" if either of the
  existing users sets ssl, or if the server has freshly been probed and
  found supporting ssl.
  There is a caveat: adding a user to an existing server without probing it
  again may skip adding ssl. (This does not prevent STARTTLS.)

## BUG FIXES:
* Fix three bugs in fetchmail.man (one unterminated string to .IP macro, one
  line that ran into a .PP macro, .TH date format), and remove one .br request 
  from inside the table, which is unsupported by FreeBSD 12's mandoc(1) 
  formatter.  FreeBSD Bug#241032, reported by Helge Oldach.
* Further man page fixes and additions by Chris Mayo and Gregor Zattler.
* When evaluating the need for STARTTLS in non-default configurations (SSL 
  certificate validation turned off), fetchmail would only consider --sslproto 
  tls1 as requiring STARTTLS, now all non-empty protocol versions do.
* fetchmailconf now properly writes "no sslcertck" if sslcertck is disabled.
* fetchmailconf now catches and reports OS errors (including DNS errors) when 
  autoprobing.  Reported as Gitlab issue #12 by Sergey Alirzaev.
* fetchmailconf received a host of other bugfixes, see the Git commit log.

## CHANGES:
* Make t.smoke more robust and use temporary directory as FETCHMAILHOME, to make
  sure that the home directory resolves for the user running the test suite
  even if the environment isn't perfect. Reported by Konstantin Belousov,
  analysed by Corey Halpin, FreeBSD Bug#240914.

## UPDATED TRANSLATION - THANKS TO:
* zh_CN: Boyuan Yang [Chinese (simplified)]

--------------------------------------------------------------------------------

Happy fetching,
Matthias

Re: [Fetchmail-users] error with preconnect

[grarpamp <gra...@gm...>]

On 2/10/20, Joe Acquisto-j4 <jo...@j4...> wrote:
> /etc/fetchmailrc contains:

Consider trying the one statement per line multiline format.
This avoids line wrapping past screen, avoids filler words,
makes quotes pairs easier to check, each line and option
can be commented, indent denotes pre/post and all the options
as valid only inside a poll, easy to look at / validate / edit,
and more etc...

poll mail.somehost.com
 preconnect "/bin/date >> /var/log/fetchmail.log"
 proto imap
 ssl
 # sslcertck
 sslfingerprint "blah"
 timeout 120
 username 'me...@ga...'
 password 'B1nksSt1nks'
 folder 'Inbox','Junk'
 fetchlimit 10
 # envelope Delivered-to
 to 'me...@co...'
 # smtphost  192.168.ccc.ddd
 # keep
 # check


When forwarding 'to' one singledrop,
multidrop envelope seems quite moot.

> Trying to connect to ::1/25...connection failed.
> fetchmail: connection to localhost:smtp [::1/25] failed: Connection
> refused.

SMTP is not running on localhost IPv6 ::1/25.
Run it, tell it where it is, fix dns, firewall, IPv4/6 stack, etc

> This worked when fetchmail and the local mail server were not on the same
> box.
>
> "postconnect" works, but places the time stamp after the connection.
> Preconnect prior to poll produces a syntax error on startup.
>
> A minor issue, but curious as to what I am mis understanding about
> preconnect.

Re: [Fetchmail-users] error with preconnect

[Matthias A. <mat...@gm...>]

Am 10.02.20 um 18:05 schrieb Joe Acquisto-j4:
> Currently at latest release candidate, but this error happened with earlier versions a well.
>
> /etc/fetchmailrc contains:
>
> poll mail.somehost.com with proto IMAP timeout 120 envelope Delivered-to
>     user 'me...@ga...' there with password 'B1nksSt1nks' is 'me...@co...' here folder 'Inbox','Junk' fetchlimit 10 ssl sslfingerprint "blah"
> #  smtphost  192.168.ccc.ddd
> preconnect "/bin/date >> /var/log/fetchmail.log"
>
> This is what log shows:

That log is not complete.

Also, when making up logs, use example.com and example.net - see
https://tools.ietf.org/html/rfc2606

somehost.com *most likely does not belong to you.*

> fetchmail: IMAP< A0006 OK Fetch completed (0.001 + 0.000 secs).
> fetchmail: IMAP> A0007 FETCH 1 RFC822.HEADER
> fetchmail: IMAP< * 1 FETCH (RFC822.HEADER {4003}
> fetchmail: reading message me...@ga...@mail.somehost.com:1 of 1 (4003 header octets)Trying to connect to ::1/25...connection failed.
> fetchmail: connection to localhost:smtp [::1/25] failed: Connection refused.
>
>
> This worked when fetchmail and the local mail server were not on the same box.
>
> "postconnect" works, but places the time stamp after the connection.   Preconnect prior to poll produces a syntax error on startup.
>
> A minor issue, but curious as to what I am mis understanding about preconnect.
fetchmail reports all pre-/postconnect issues as PS_SYNTAX no matter
what. Check whatever is in the log before that point, such as:
> fetchmail: 6.4.2-rc3 querying mail.example.com (protocol IMAP) at Mo
> 10 Feb 2020 21:10:17 CET: poll started
> sh: /var/log/fetchmail.log: Permission denied
> fetchmail: pre-connection command failed with status 1

[Fetchmail-users] error with preconnect

[Joe Acquisto-j. <jo...@j4...>]

Currently at latest release candidate, but this error happened with earlier versions a well.

/etc/fetchmailrc contains:

poll mail.somehost.com with proto IMAP timeout 120 envelope Delivered-to
    user 'me...@ga...' there with password 'B1nksSt1nks' is 'me...@co...' here folder 'Inbox','Junk' fetchlimit 10 ssl sslfingerprint "blah"
#  smtphost  192.168.ccc.ddd
preconnect "/bin/date >> /var/log/fetchmail.log"

This is what log shows:

fetchmail: IMAP< A0006 OK Fetch completed (0.001 + 0.000 secs).
fetchmail: IMAP> A0007 FETCH 1 RFC822.HEADER
fetchmail: IMAP< * 1 FETCH (RFC822.HEADER {4003}
fetchmail: reading message me...@ga...@mail.somehost.com:1 of 1 (4003 header octets)Trying to connect to ::1/25...connection failed.
fetchmail: connection to localhost:smtp [::1/25] failed: Connection refused.


This worked when fetchmail and the local mail server were not on the same box.

"postconnect" works, but places the time stamp after the connection.   Preconnect prior to poll produces a syntax error on startup.

A minor issue, but curious as to what I am mis understanding about preconnect.

joe a.

[Fetchmail-users] The 6.4.2-rc3 release candidate #3 of fetchmail is available [fetchmailconf improvements]

[Matthias A. <mat...@gm...>]

[Re-sent announcement with fixed Subject: line.]

The 6.4.2-rc3 release of fetchmail is now available at the usual locations,
including <http://sourceforge.net/projects/fetchmail>.

The source archive is available at:
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.2-rc3.tar.lz>
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.2-rc3.tar.xz>

Please test the revised fetchmailconf, it now also needs Python's "future"
package, supports Python 3, and learned how to autoprobe via SSL and/or IPv6.

These are the relevant changes from 6.4.2-rc2 to -rc3.
fetchmailconf.py is now at version 1.63.

NEWS: reword/reformat a bit.
fetchmailconf.py: hostname qualification fixup
fetchmailconf.py: Show hostname in configuration selector.
NEWS: mention fetchmailconf's improved error handling for OSErrors in get_greetline()
fetchmailconf: Catch errors from get_greetline()
fetchmailconf: Add missing line separator in RunWindow.
fetchmailconf: delete server entries properly.

[Fetchmail-users] The 6.4.2-rc2 release candidate #2 of fetchmail is available [fetchmailconf improvements]

[Matthias A. <mat...@gm...>]

The 6.4.2-rc3 release of fetchmail is now available at the usual locations,
including <http://sourceforge.net/projects/fetchmail>.

The source archive is available at:
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.2-rc3.tar.lz>
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.2-rc3.tar.xz>

Please test the revised fetchmailconf, it now also needs Python's "future"
package, supports Python 3, and learned how to autoprobe via SSL and/or IPv6.

These are the relevant changes from 6.4.2-rc2 to -rc3.
fetchmailconf.py is now at version 1.63.

NEWS: reword/reformat a bit.
fetchmailconf.py: hostname qualification fixup
fetchmailconf.py: Show hostname in configuration selector.
NEWS: mention fetchmailconf's improved error handling for OSErrors in get_greetline()
fetchmailconf: Catch errors from get_greetline()
fetchmailconf: Add missing line separator in RunWindow.
fetchmailconf: delete server entries properly.

Re: [Fetchmail-users] TLS fingerprints, and option --pinnedpublickey <sha256//b64hash>

[grarpamp <gra...@gm...>]

It's info for people to understand various background on why
--pinnedpublickey is showing up now in softwares that speak TLS.
If users like to pool together and implement, and have available to
use the feature as desired or not, in whatever upcoming release,
great for them. If not, then they won't have it yet is all.
Simple, no problem, no need to shoot when even ...

Here are more background link reasons, and sample code, and related...

https://owasp.org/www-community/controls/Certificate_and_Public_Key_Pinning
https://owasp.org/www-project-cheat-sheets/cheatsheets/Pinning_Cheat_Sheet

https://www.netcraft.com/internet-data-mining/ssl-survey/
https://www.ssllabs.com/ssl-pulse/
https://arstechnica.com/gadgets/2018/10/browser-vendors-unite-to-end-support-for-20-year-old-tls-1-0/
https://www.bleepingcomputer.com/news/security/ietf-approves-tls-13-as-internet-standard/
https://en.wikipedia.org/wiki/Transport_Layer_Security
https://tools.ietf.org/html/rfc8446


> minor releases

pinnedpublickey would be new, not change, so doesn't apply.

Who nobody said anything about when TLS.
Though given deprecation of SSL since decade,
the 6.4 -> 7.x would seem be good for s/SSL/TLS/ig
forward so as not to perpetuate SSL in overall thoughts
when TLS is the majority standard now and ssl dead by then.

Regardless of option names, posters are often writing mail things
as "SSL", this implies maybe some missing knowledge that
it is actually TLS nowdays in use since decade, even in use
by fetchmail in maybe their particular case, so no problem
to help not perpetuate that, or to help accuracy. See above links.

> Aunt Tillie

Aunt Tillie has a support team, it's called Niece Julie,
else she'd just be called Lady Tillie. And if Lady Tillie is
here using unix fetchmail, good for her, and confident
she can search and read links like other amazing users.

> And what prevents a state

Yes various actors are the reasons a variety of options
and tools are made to combine together against
various adversaries, including pinnedpublickey, PGP,
tor, etc. It is about having many options in depth, not
one single one waiting to fail.

People could and do even pinnedpublickey with i2p/onion/etc
services to detect potential attack before moving data,
even on web like facebook onion, and mail systems too.

> I see no difference to state orders forcing the mail service to reveal
> unencrypted information.
> plaintext

Some jurisdictions and attacks and spies and vantage points treat or
can hit the wire different from the service. Everything on the wire needs
wrapped in TLS, and TLS has some various use cases for various options.
What content or crypto travels inside is of no relavance to fetchmail binary.

> Mozilla (who are normally the source for the trust lists)

Microsoft originates its own, so do a few others, they all
overlap and intershare.

> I think they have removed CAs that were involved in scandals

Removal of obvious scandals logically does not and cannot
certify the remaining CA's as being pure now, or in the future.

> Oh they do serve a purpose, and that's permitting delegating server
> authentication.

Intermediate org level certs issued by roots seem quite rare,
and LetsEnc haven't yet [?] issue any. Roots signing
whatever server certs are handed to them is different from
delegating intermediates to orgs.

> Those that mess with certificates, will, too.

Such attacks against big gmail size frontends affecting
everyone loudly would be stupid expose of attacker.
Real adversaries can also target individual users at their network
edges where such little noise is hardly heard from them.

> 5/1/384/512/3 hashes

CA's are mostly using sha256 now, people have 256
in their toolsets, pinnedpublickey matches that nicely,
and 256 isn't broken yet so can suffice alone without
composition till then, unlike 5/1 which are broken.
Sha-3 is in hedge pool.

> ...alleviating the need to keep huge CRLs.

Many implementations still don't bother checking CRLs,
they just trust whatever looks like a CA. And checking
CRLs is a massive metadata privacy and monetization
leak that some users might not want to do.

> Which is its very purpose.

Fingerprints can be held over whatever desired cert
parameters to serve different purposes. If a user
really cares what some CA says, they can pin the sig.
If they don't care what CA says, can pin just the pubkey linked.
Or check both. The existance of two print methods
proves more than one purpose.

> historically you'd find tons of instructions

Backward compat is fine, but not to the point
of inhibiting good movement into the future.

> validate the fingerprints *in practice.

Plenty of old blogs would say to contact the mail
provider and ask. As before, now many of them
are publishing attestations in public places for users to
various degrees of trust assignability. Better than nothing.


> IMPLICITLY follows an INSECURE!!! trust-on-first-use setting

Post mentioned variety of means to verify assess a security position.
Contrary, most users use of TLS is actually completely unverified,
and based entirely on "trusting" non-fiduciary third parties and other
entities they have no reason or agreement to trust. By doing some
verification and pinning many parties and attacks are removed.
Even TOFU via simple vpn/tor observatory check can be better
at defeating some threats than untrusted CA "trust".
Fine if users don't want to pin, most don't, but they should have
and know some things on they could and why.

> lobby all the mail service providers to actually upgrade their TLS stuff
> to be v1.3 capable.

Many of the big providers, and the privacy oriented ones,
already have 1.3 at the head of their ciphersuite preference list.
Now that 1.3 is standard, the rest are adding it and dropping
older than 1.2 in the process, its linked above.

> the transmission channel often only has opportunistic TLS.

That's not fetchmail business :) Msmtp and others can pin too.

> out of perspective for the mail transmission chain between
> the sender and the fetchmail user as a whole system.

Defense in depth.

> add SHA256 and stronger hashes for fingerprinting

SHA256 is fine alone. The adding was to give users
options to not use broken md5 / sha1 pins. Any composition
was probably meant as easy to do logic hedge. Years
ago most public attestations used md5 / sha1, at then
either sha 256 or 3 were thus non broken options.
Attestation seems to be slowly moving to sha256 now
(since the certs hash internally already did too), but that
doesn't affect which strong one is used for pinning, other than
to be easy cut paste compare compatible to today attestations.


Happy fetchmailing :)

Re: [Fetchmail-users] TLS fingerprints, and option --pinnedpublickey <sha256//b64hash>

[Matthias A. <mat...@gm...>]

Am 06.02.20 um 03:07 schrieb grarpamp:
> On 2/5/20, Matthias Andree <mat...@gm...> wrote:
>> Am 05.02.20 um 20:06 schrieb Joe Acquisto-j4:
>>> chasing a gmail issue.   The  ssl fingerprint seems to change, seemingly
>>> with what IP it happens to connect.
>> SSL fingerprints are mostly non-workable for those of the big sites that
>> use per-host certificates, and fetchmail won't let you list dozens, let
>> a lone use a secure hash (it uses MD5).
>>
>> Be sure to install the Mozilla root certificates (most distributions
>> have packages such as ca-certificates or nss_root_ca) that integrate
>> with the default OpenSSL configuration such that the certificates can be
>> verified automatically, do not use --sslfingerprint, do not use
>> --nosslcertck (but do use --sslcertck on 6.3.x) and move on.
> Everyone should know it's called TLS now, stop calling SSL or using SSL.

grarpamp,

I need to say that very bluntly, posts like these are becoming annoying.

We don't change established option names and break everyone's
configuration in minor releases.

> Mail providers tend to be reasonably well known and well run entities,
> and all the new privacy startups. Many of them PGP sign their own
> TLS fingerprints, post them OOB to twitter, keybase, onions, etc.
> Google and some others use intermediate certs that can be loaded.
> The web is becoming self authenticating, easily validated, no need
> for silly CA's anymore.

So you're going to teach Aunt Tillie how to do that for each and every
service she uses, and also support her via this list? Unless you're
committed to getting things practical, none of this is going to happen.
And this includes telling them to watch all the typical suspect places
for changes, and watch canaries if the service posts one, and what not.

And what prevents a state order ...

> CA's have always provided exactly zero benefit to anyone (other than those
> who invested stock in the CA fee and spy scam), in deference to well known
> TLS fingerprints. Further, the CA's all many 100's of them, are big source of
> MITM threat... bogus court orders, govt force, coersion, rogue moles,
> exploit, etc...

...from telling them to tee out the decrypted traffic behind the
certificate, or just the mail, and at the same time forcing them to STFU
and not tell anyone.

I see no difference to state orders forcing the mail service to reveal
unencrypted information. If you'll want really secure communication,
you're not going to achieve that with e-mail via SMTP/IMAP/POP3, but
you'll need to set up quite a different system.

> anyone using the bundles should be afraid... look at all those completely
> untrustable CA's. Do not start with a file full of exploits, start with an
> empty cacert file and work up to the minimum you need to live.

This is implying that Mozilla (who are normally the source for the trust
lists) were shipping trust recommendations for "completely untrustable"
CAs. I think they have removed CAs that were involved in scandals such
as issuing wildcard trusts or other betrayals.

> Root CA's, even the Lets Encrypt, just don't serve much purpose for
> services users use... they are only profiteers and nag preventers,
> not MITM preventers, not server iron authentication providers, not
> compromise detectors, not auditors, etc.
Oh they do serve a purpose, and that's permitting delegating server
authentication.
> Good service providers will never use a compromised TLS private key,
> and by extension they will never seek CA renewal over that public key.
> Any service that is stupid about this principle will get publicly bashed.
Those that mess with certificates, will, too.
> Fetchmail allows both TLS fingerprint and CA to match,
> or either one if one of them is used solo.
> Both MD5 and SHA1 are completely broken.
Which is, in current fetchmail versions 6.4.x, is rather an argument to
not use the MD5 fingerprint but instead use stronger certification.
> Lets Encrypt forces, (and the holders of intermediate CA's like
> Google do,) very frequent renew their expired signatures over the
> same old non-compromised TLS public key,
...alleviating the need to keep huge CRLs.
> Fetchmail's TLS fingerprint check fails upon every new CA sig.

Which is its very purpose. Nobody claims X.509 certificate
fingerprinting were particularly practical, and historically you'd find
tons of instructions on The Net that would describe how to manually
implement something like TOFU or manual certificate pinning, and I don't
ever recall those documents telling people how to validate the
fingerprints *in practice.

And cURL's option and instructions "PUBLIC KEY EXTRACTION" here

> https://curl.haxx.se/libcurl/c/CURLOPT_PINNEDPUBLICKEY.html
> --pinnedpubkey <hashes|file>

IMPLICITLY follows an INSECURE!!! trust-on-first-use setting so is
actually just as useful or useless as --sslfingerprint (assuming I were
to implement SHA256/384/512 and other hashes), I don't see how that's
more secure.

The priority with the 6.4 release was getting TLS 1.1/1.2/1.3 support
out, and disentangling "talk-TLS-right-away" vs. STARTTLS schemes from
the protocol version. Also the recommendations (use STARTTLS to preserve
the IANA port vs. preferring talk-TLS-directly services) have changed
over time.

Adding more features and testing those (for the ancient system that 6.4
is to support) would have deferred that release even longer, which
wasn't an option.

>> I suppose Google's non-SSL port (which would have had to use STARTTLS or
>> possibly risked broken clients volunteering their password over
>> unencrypted links) is firewalled and blackholes inbound traffic.
> Users should always choose TLS wherever possible instead of
> any sort of plaintext. And pin TLS to 1.3 or latest possible.

In the past we haven't cared too much if TLS was negotiated over the
separate port directly or in-band after STARTTLS, but you might want to
lobby all the mail service providers to actually upgrade their TLS stuff
to be v1.3 capable.

None of this fixes any problem inherent with e-mail being transmitted
and/or stored in plain text somewhere, somehow, even if the fetching
channel is secured, the transmission channel often only has
opportunistic TLS.

If you need really secure communications, e-mail certainly isn't going
to provide that, look elsewhere, and I have yet to see a proper argument
why and where fetchmail alone were the weak link of the chain and could
fix that. You're arguing about attaining perfection and perfect security
on the final link between MSP and user, but that seems out of
perspective for the mail transmission chain between the sender and the
fetchmail user as a whole system.


The only take-home message from your lengthy post for me was a plea to
add SHA256 and stronger hashes for fingerprinting and/or comparing the
entire public key. Which in itself wasn't a new request, it only
couldn't make 6.4 - that release branch had to be cut somewhere.

[Fetchmail-users] TLS fingerprints, and option --pinnedpublickey <sha256//b64hash>

[grarpamp <gra...@gm...>]

On 2/5/20, Matthias Andree <mat...@gm...> wrote:
> Am 05.02.20 um 20:06 schrieb Joe Acquisto-j4:
>> chasing a gmail issue.   The  ssl fingerprint seems to change, seemingly
>> with what IP it happens to connect.
>
> SSL fingerprints are mostly non-workable for those of the big sites that
> use per-host certificates, and fetchmail won't let you list dozens, let
> a lone use a secure hash (it uses MD5).
>
> Be sure to install the Mozilla root certificates (most distributions
> have packages such as ca-certificates or nss_root_ca) that integrate
> with the default OpenSSL configuration such that the certificates can be
> verified automatically, do not use --sslfingerprint, do not use
> --nosslcertck (but do use --sslcertck on 6.3.x) and move on.

Everyone should know it's called TLS now, stop calling SSL or using SSL.

Gmail's fingerprints seem to roll globally all at once,
then remain stable till the next, roughly speaking.

Mail providers tend to be reasonably well known and well run entities,
and all the new privacy startups. Many of them PGP sign their own
TLS fingerprints, post them OOB to twitter, keybase, onions, etc.
Google and some others use intermediate certs that can be loaded.
The web is becoming self authenticating, easily validated, no need
for silly CA's anymore.

CA's have always provided exactly zero benefit to anyone (other than those
who invested stock in the CA fee and spy scam), in deference to well known
TLS fingerprints. Further, the CA's all many 100's of them, are big source of
MITM threat... bogus court orders, govt force, coersion, rogue moles,
exploit, etc...
anyone using the bundles should be afraid... look at all those completely
untrustable CA's. Do not start with a file full of exploits, start with an
empty cacert file and work up to the minimum you need to live.

Root CA's, even the Lets Encrypt, just don't serve much purpose for
services users use... they are only profiteers and nag preventers,
not MITM preventers, not server iron authentication providers, not
compromise detectors, not auditors, etc.

Good service providers will never use a compromised TLS private key,
and by extension they will never seek CA renewal over that public key.
Any service that is stupid about this principle will get publicly bashed.
Everyone knows to regenerate TLS private keys after compromise.
CA expiry is only barely marginally useful as a compromise detection
eyeball during cert swap process.

Fetchmail allows both TLS fingerprint and CA to match,
or either one if one of them is used solo.
Both MD5 and SHA1 are completely broken.

However, due in part to todays more self authenticating world,
users have big maintenance pain with any use of TLS fingerprint
for the non compromised case.

Lets Encrypt forces, (and the holders of intermediate CA's like
Google do,) very frequent renew their expired signatures over the
same old non-compromised TLS public key,
much more frequently than old CA signing periods did.
These pubkey lifetimes tend to be longer than even those periods,
arguably supported by todays ephemeral session keys.

Fetchmail's TLS fingerprint check fails upon every new CA sig.
This makes more needless work for users to update
fingerprints, even though nothing was compromised.

Keep todays TLS fingerprint option over the entire
CA signed DER form.

Reduce users work (and dependency on pointless CAs),
by adding support for fingerprint of public key itself.
If can even be used in conjunction with sslcertck and sslfingerprint.

Curl, wget, and others can use this pubkey method, here it is...


https://curl.haxx.se/libcurl/c/CURLOPT_PINNEDPUBLICKEY.html
--pinnedpubkey <hashes|file>
Tells curl to use the specified public key file (or hashes) to
verify the peer. This can be a path to a file which contains a
single public key in PEM or DER format, or any number of base64
encoded sha256 hashes preceded by 'sha256//' and separated by ';'.
When negotiating a TLS or SSL connection, the server sends a
certificate indicating its identity. A public key is extracted from
this certificate and if it does not exactly match the public key(s)
provided to this option, curl will abort the connection before sending
or receiving any data.


> I suppose Google's non-SSL port (which would have had to use STARTTLS or
> possibly risked broken clients volunteering their password over
> unencrypted links) is firewalled and blackholes inbound traffic.

In various standards and server talks, STARTTLS seems to finally be
trending to be deprecated, with preference now to mandatory native TLS.

Users should always choose TLS wherever possible instead of
any sort of plaintext. And pin TLS to 1.3 or latest possible.

Re: [Fetchmail-users] gmail timeout using IMAP

[Matthias A. <mat...@gm...>]

Am 05.02.20 um 20:06 schrieb Joe Acquisto-j4:
> Well . . . oops?  Don't know how I  missed the port issue.  At some point the 
> "ssl" option was deleted from the user line.   
>
> That resolved the connection failure.      Working after a few fetch cycles and
> chasing a gmail issue.   The  ssl fingerprint seems to change, seemingly with what
> IP it happens to connect.

SSL fingerprints are mostly non-workable for those of the big sites that
use per-host certificates, and fetchmail won't let you list dozens, let
a lone use a secure hash (it uses MD5).

Be sure to install the Mozilla root certificates (most distributions
have packages such as ca-certificates or nss_root_ca) that integrate
with the default OpenSSL configuration such that the certificates can be
verified automatically, do not use --sslfingerprint, do not use
--nosslcertck (but do use --sslcertck on 6.3.x) and move on.

> While I do find almost immediate response when connecting (human eyeball time)

I suppose Google's non-SSL port (which would have had to use STARTTLS or
possibly risked broken clients volunteering their password over
unencrypted links) is firewalled and blackholes inbound traffic.

And yes, Google isn't exactly following the IMAP RFCs (standards) to the
letter, or even the traditional model of an IMAP server.