fetchmail-devel

Re: [fetchmail-devel] [fetchmail-users] master: TODO.txt

[Matthias A. <mat...@gm...>]

Am 24.04.2013 10:37, schrieb grarpamp:
> Noted some things
> 
> - blacklist DigiNotar/Comodo/T<C3><BC>rktrust hacks/certs, possibly
> with Chrome's serial# list?
> 
> I would not hardcode this but instead place fingerprints in multiple
> global/per_host 'fpdeny' config options. In part because testing
> infrastructure with these certs is valuable. And at least that way,
> even if they're lazy and only use sslcertck, if some emergency
> arises they can add a negative print there affecting global/per_host.
> Additionally, point the user to where they can find and then build
> their own updated cert store free from all such junk. As well as
> point them to some doc about the importance of fingerprint checking.

Up front, thanks a bunch for the feedback.  We should move to
fetchmail-devel though...

I am wondering - especially about switching SSL library, too, because
OpenSSL requires you to jump quite through a few hoops for even standard
stuff, like CRLs and OCSP.

> https://mxr.mozilla.org/mozilla-central/source/security/nss/lib/ckfw/builtins/certdata.txt?raw=1
>   ftp://ftp.mozilla.org/pub/security/nss/releases/
> https://github.com/agl/extract-nss-root-certs.git

> I'll try to remember to add this to the 'cert' ticket when I find
> it again.

<http://svnweb.freebsd.org/ports/head/security/ca_root_nss/files/MAca-bundle.pl.in?revision=312617&view=markup>

> - CRYPTO: remove sslfingerprint? too easily abused (see NEWS)
> 
> I trust this is by now just an old entry. If not, please don't :)

This line is to be dropped from TODO.txt -- I've seen too many
certification "authorities" that did not deserve this name, and I seem
to have someone willing to tell users in the fetchmail lobby how to make
_good_ use of this feature.

The missing link is that you hardly ever get the certificate
fingerprints on the "how to configure Outlook, blahmail, whatever for
fetching mail from us" on the ISP help pages, or even better, by snail
mail when they send you account data.

> #### config flexibility
> Consider making 'poll [thing]'s thing just a label string. And
> breaking apart the config into types:  'polls' with 'poll [label]',
> 'hosts' with 'host [label]', and 'accounts' with 'account [label]'.
> Put whatever you want in a label ... 'blah' 'jo...@sc...',
> 'foo.com', '1.2.3.4' ... but treat it as a label. 

Basically we're already quite close, we'd only have to make sure that
adding a poll argument on the command line permits specifying an
account. See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=705291>,
for instance.

Your notion of

poll thing   host foo.example.org   account blah

is spelt

poll thing   via foo.example.org    user blah

today :-)

We lack group selection capability, though, as detailed in the Debian
BTS report linked above.

> Collect all poll types
> into a set for daemon or oneshot runs. Account types are
> always singly available for commandline oneshot's. 'netaddr'
> may be fqdn, or ip (dns down/insecure scenario). Error out
> if a label dependency does not exist. Order of types in file
> does not matter, indented bits are type content. Includes
> work.

That would be missing.

> 
> ./fetch -p  # polls
> ./fetch -a play2
> ./fetch -a mywork -h work1_pop  # temporary swap :)
> ./fetch -a play1  # bombs on foo1
> 
> It's really really hard to get fetchmail to do anything like this
> today. Unless you forget it and frontend it with your own config
> and call engine. These sort of configs are powerful.

And require particular attention in tracing output (and possibly imply
"verbose" mode) so that we can give users a hint what they are doing and
what fetchmail is making of it, so that we can later properly dissect
bug reports.

Well, the other frequently asked feature is "poll multiple hosts at the
same time" (especially on high-speed-but-high-latency-links, such as
DSL, satellite), which needs to go hand in hand here.

The actual multithreading-the-input is not too hard to come by, the
question is how to limit the output side, and to assess if that is
necessary.

If you think ISP-grade "POP collector option" (disregarding it violates
most site's security policy to share credentials with another entity),
you may want to configure "poll 50 accounts at the same time but only
use 20 outbound SMTP threads".  This requires caching mail, either in
RAM, or on disk.

I have updated the master's TODO.txt a tiny bit (near the end).

Re: [fetchmail-devel] [fetchmail-users] master: TODO.txt

[grarpamp <gra...@gm...>]

> Up front, thanks a bunch for the feedback.  We should move to

BTW, 6.3.26 happens to work here. Now if I could just get --password
before I end up stealing the framework for it from some other option :)

> I am wondering - especially about switching SSL library, too, because
> OpenSSL requires you to jump quite through a few hoops for even standard
> stuff, like CRLs and OCSP.

Don't know much there, not really a programmer that way. Haven't
got around to trying the GNUTLS binary cert tools yet either.
https://en.wikipedia.org/wiki/Comparison_of_TLS_Implementations

> to have someone willing to tell users in the fetchmail lobby how to make

I'm hoping to search out some overall cert security docs online for
this someday. They'd be far better and easier to point to.

> The missing link is that you hardly ever get the certificate
> fingerprints on the "how to configure Outlook, blahmail, whatever for
> fetching mail from us" on the ISP help pages, or even better, by snail
> mail when they send you account data.

I do see some that show GUI pics and so forth, though they're
usually related to accepting a self-signed or intermediate cert.
There are often some short blog style posts if you search out some
problematic FP you're dealing with. I'll be testing pinning with Firefox
then Thunderbird sometime this year. Maybe the world will come
up with a better cert scheme in a few years.

Re: [fetchmail-devel] [fetchmail-users] master: TODO.txt

[grarpamp <gra...@gm...>]

>> #### config flexibility
>> Consider making 'poll [thing]'s thing just a label string. And
>> breaking apart the config into types:  'polls' with 'poll [label]',
>> 'hosts' with 'host [label]', and 'accounts' with 'account [label]'.
>> Put whatever you want in a label ... 'blah' 'jo...@sc...',
>> 'foo.com', '1.2.3.4' ... but treat it as a label.
>
> Basically we're already quite close, we'd only have to make sure that
> adding a poll argument on the command line permits specifying an
> account. See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=705291>,
> for instance.
>
> Your notion of
> poll thing   host foo.example.org   account blah
> is spelt
> poll thing   via foo.example.org    user blah
> today :-)

Well sortof, except that in the absence of a 'via', the poll
'thing' is treated as actual user data, not merely a part of,
or a private label for, the FM config building engine. It
almost seems multiplexed and needs to explicitly not be.

I have one of mine constructed as, with n incremented
poll sections, but can't remember why...
poll user1@fqdn1
 via fqdn
 username user1@fqdn1

Additionally, who wants to literally poll a host as in 'poll host'?
You generally in mind want to poll an account, or some accounts,
and then figure out where they are.
Though there's probably value in some shorthand submechanism
to in fact poll everything that configures out to be residing at the
same host. But I think giving accounts their independance from
hosts and their configuration would come first.

This was/is all a sleepy draft... I think I meant the following but would
need to think on it all more...

# polls
poll play
 account play1
 account play2
poll kids
 ...

./fetch  # all polls, or maybe as ./fetch -p, whatever the default
./fetch -p play,kids  # some listed polls, perhaps even '-p !work'.

> Collect all poll types into a set for daemon or oneshot runs.

I don't think I meant this as a parallel thing, but if no specific
poll specified,
collate them all from the config for serial processing, whether daemon
or cmdline oneshot (-N).

> We lack group selection capability, though, as detailed in the Debian
> BTS report linked above.

Reading just the report, I thought it was a logic issue. Then
the response tied in with this.

> bug reports.

Routers often have a mode that emits the config as interpreted, but
with private info such as passwords hashed out, for posting, storing,
etc.


### parallel stuff

> And require particular attention in tracing output (and possibly imply

Possibly refer to some threading might be early bits below.

> Well, the other frequently asked feature is "poll multiple hosts at the
> same time" (especially on high-speed-but-high-latency-links, such as
> DSL, satellite), which needs to go hand in hand here.

Possibly refer to some maildir yes bits below.

> The actual multithreading-the-input is not too hard to come by, the

I'm not sure for fetchmail to be threaded/forked in general right away.
Though I did throw it in there with the different 'interval [time]' hints.

A gotcha with multiples... if a user has them, they may wish them to
appear to be reasonably separate to an observer. If they keep polling
the same accounts at the same time and not via different nets or shared
proxies, they're linkable. Though if the config was a bit more flexible the
equivalent of cron plus a random jitter driving ./fetch -a accountX
would work instead of threading out the 'interval [period + jitter]' polls.

Fetchmail > server... surely an account must be serial locked
to prevent dupes inbound and various server state issues.

> question is how to limit the output side, and to assess if that is
> necessary.

FM > MDA > maildir, maildir can handle it just fine. I often have 10
or more fetches dumping into a maildir at once. Except for
old offline archives, there's no mbox here :) Other than ensuring
fetchmail can be externally driven that way with say one FM
config file, and a state issue doc warning against polling the same
account in parallel, parallelizing FM itself seems a lot of work right now.

> If you think ISP-grade "POP collector option"
> you may want to configure "poll 50 accounts at the same time but only
> use 20 outbound SMTP threads".  This requires caching mail, either in
> RAM, or on disk.

You mean loop through all 50 polls with a 20 wide parallel sliding
window... I'd think FM > MDA > maildir would eliminate cache so long
as say maildrop had safe tmp file creation and return codes back to FM.
I admit not having looked into that. And I'd warn against mbox if any of
this is pursued.