Re: [fetchmail-devel] idle polling on multiple servers

[Matthias A. <mat...@gm...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 30.06.2013 09:47, schrieb Andy Spencer:
> Hi, I've been looking into adding support for polling more than one
> server using the IMAP IDLE feature. I'm very new to the fetchmail source
> code, so any suggestions or comments would be helpful :)

Andy,

thank you for taking the initiative.

The existing IDLE code is very intrusive (not to say ugly) and fragile,
as you have probably noticed, and I would much rather toss it out rather
than extend it.  That does not mean I want to abolish the feature, but
just the implementation.

> Currently, IDLE support is implemented in the imap_getrange function by
> blocking the getrange call until new messages are available. Afterwards,
> it forces a repoll and repeats the process waiting for new messages. The
> down side of this is that it starves out all the other servers because
> the IMAP do_session never returns (except on errors).

The whole machinery will have to undergo revision in order to support
client-side IMAP "seen" status tracking without need to tamper with the
flags, and in order to support PIPELINING.

> From what I understand, many people have expressed concerns about
> polling multiple servers concurrently and I agree that using threads or
> forks would be a bad idea. However, we should be able to use poll(2) or
> select(2) to accomplish the same thing without the threading issues. As
> it turns out, fetchmail uses select to implement it's interruptible_idle
> function anyway.

These syscalls have a history of not working properly, too, so the
implementation means is a moot point.

The main issue with concurrent polls is that some parts of the code
(notably the SSL socket code as well as the sink) use global variables.

> I've put up a working prototype for polling multiple servers here if
> anyone is interested in taking a look at it. Please treat it with some
> skepticism though, it still needs a lot of work ;)
> 
>     http://pileus.org/git/?p=~andy/fetchmail;h=idle
> 
> 
> The basic idea is that the blocking getrange is removed and the servers
> socket is saved between polls and passed to select() so that fetchmail
> can wake up when new mail arrives.
> 
> 
> I added two new functions to the 'struct method' table that replace the
> existing, internal, imap_idle function:
> 
>   start_idle:  called at the end of a fetch, this puts the connection in
>                IDLE mode so that we can wake up when mail arrives
> 
>   finish_idle: called at the start of the next poll, resets the
> 	       connection to 'normal mode'
> 
> 
> There's also a couple additions to 'struct hostdata' (I don't know if
> this the correct structure to use, but it seems to work).
> 
>   true_idle:   set to TRUE if the server supports IDLE polling,
>                otherwise we may need to fake it somehow
> 
>   idle_socket: the socket saved between polls, the connection setup and
> 	       destruction in do_session is skipped while idling
> 
>   idle_ready:  set to TRUE after calling select() if data is available
>                for reading on the socket

Without having looked at your code yet (ENOTIME):

That data might be a disconnect because we've been idling too long, to
mark the most obvious failure case.  I am not so sure that caching
connections between polls is a workable, and that the ratio of effort
and outcome is right.  NOTIFY might give more benefit for the same effort.

> Whew, I hope that wasn't too long, I'll keep poking around at this
> during the next few weeks and see if I can clean it up a bit. I thought
> it would be good to give the other developers a heads up though.

I have a feeling that any attempts to patch up the current IDLE code
structure is spending efforts in the wrong places.

I think a more radical approach to the overall poll setup and action is
warranted, one that does not put one layer of context (through global
variables, struct query, hostdata, whatever) over the next, which is all
laid out differently, and is likely getting in the way of cached
connections, too.

I think the end will more look like a collection of objects.  There is
no new design specification for fetchmail yet, however.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlHTWPgACgkQvmGDOQUufZWU0QCfUexWl9gtgQ9PmoDkqJJtXTrS
VlwAoPtmMZwqk9tXPG9vVbqA7JsaRtgj
=qQE8
-----END PGP SIGNATURE-----