Re: [fetchmail-devel] [fetchmail-users] master: TODO.txt

[grarpamp <gra...@gm...>]

>> #### config flexibility
>> Consider making 'poll [thing]'s thing just a label string. And
>> breaking apart the config into types:  'polls' with 'poll [label]',
>> 'hosts' with 'host [label]', and 'accounts' with 'account [label]'.
>> Put whatever you want in a label ... 'blah' 'jo...@sc...',
>> 'foo.com', '1.2.3.4' ... but treat it as a label.
>
> Basically we're already quite close, we'd only have to make sure that
> adding a poll argument on the command line permits specifying an
> account. See <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=705291>,
> for instance.
>
> Your notion of
> poll thing   host foo.example.org   account blah
> is spelt
> poll thing   via foo.example.org    user blah
> today :-)

Well sortof, except that in the absence of a 'via', the poll
'thing' is treated as actual user data, not merely a part of,
or a private label for, the FM config building engine. It
almost seems multiplexed and needs to explicitly not be.

I have one of mine constructed as, with n incremented
poll sections, but can't remember why...
poll user1@fqdn1
 via fqdn
 username user1@fqdn1

Additionally, who wants to literally poll a host as in 'poll host'?
You generally in mind want to poll an account, or some accounts,
and then figure out where they are.
Though there's probably value in some shorthand submechanism
to in fact poll everything that configures out to be residing at the
same host. But I think giving accounts their independance from
hosts and their configuration would come first.

This was/is all a sleepy draft... I think I meant the following but would
need to think on it all more...

# polls
poll play
 account play1
 account play2
poll kids
 ...

./fetch  # all polls, or maybe as ./fetch -p, whatever the default
./fetch -p play,kids  # some listed polls, perhaps even '-p !work'.

> Collect all poll types into a set for daemon or oneshot runs.

I don't think I meant this as a parallel thing, but if no specific
poll specified,
collate them all from the config for serial processing, whether daemon
or cmdline oneshot (-N).

> We lack group selection capability, though, as detailed in the Debian
> BTS report linked above.

Reading just the report, I thought it was a logic issue. Then
the response tied in with this.

> bug reports.

Routers often have a mode that emits the config as interpreted, but
with private info such as passwords hashed out, for posting, storing,
etc.


### parallel stuff

> And require particular attention in tracing output (and possibly imply

Possibly refer to some threading might be early bits below.

> Well, the other frequently asked feature is "poll multiple hosts at the
> same time" (especially on high-speed-but-high-latency-links, such as
> DSL, satellite), which needs to go hand in hand here.

Possibly refer to some maildir yes bits below.

> The actual multithreading-the-input is not too hard to come by, the

I'm not sure for fetchmail to be threaded/forked in general right away.
Though I did throw it in there with the different 'interval [time]' hints.

A gotcha with multiples... if a user has them, they may wish them to
appear to be reasonably separate to an observer. If they keep polling
the same accounts at the same time and not via different nets or shared
proxies, they're linkable. Though if the config was a bit more flexible the
equivalent of cron plus a random jitter driving ./fetch -a accountX
would work instead of threading out the 'interval [period + jitter]' polls.

Fetchmail > server... surely an account must be serial locked
to prevent dupes inbound and various server state issues.

> question is how to limit the output side, and to assess if that is
> necessary.

FM > MDA > maildir, maildir can handle it just fine. I often have 10
or more fetches dumping into a maildir at once. Except for
old offline archives, there's no mbox here :) Other than ensuring
fetchmail can be externally driven that way with say one FM
config file, and a state issue doc warning against polling the same
account in parallel, parallelizing FM itself seems a lot of work right now.

> If you think ISP-grade "POP collector option"
> you may want to configure "poll 50 accounts at the same time but only
> use 20 outbound SMTP threads".  This requires caching mail, either in
> RAM, or on disk.

You mean loop through all 50 polls with a 20 wide parallel sliding
window... I'd think FM > MDA > maildir would eliminate cache so long
as say maildrop had safe tmp file creation and return codes back to FM.
I admit not having looked into that. And I'd warn against mbox if any of
this is pursued.