fetchmail-users

[Fetchmail-users] fetchmail not polling multiple servers when IMAP IDLE used

[Jeremy C. <jd...@ko...>]

Hi, I'm hoping this is a config mistake or brain fart on my part, but
I've spent a lot of time looking at fetchmail(1) and can't find anything
that would explain this behaviour.

I'm using fetchmail 6.3.26 on FreeBSD 9.3-STABLE.

The following is in my .fetchmailrc:

set daemon 120

poll imap.server
	protocol imap
	username user password pass
  	idle
	ssl sslfingerprint "aa:bb..."

poll pop3.server
	protocol pop3
	username user password pass
	ssl sslfingerprint "cc:dd..."

The behaviour I expect from this, based on the fetchmail(1) man page, is
that because I'm using daemon mode, fetchmail should remain running and
keep an active IMAP IDLE connection to imap.server, but then
periodically (every 120 seconds) poll (via POP3) pop3.server.  These are
two completely different servers, just to clarify.

Well, pop3.server never gets polled.  Using "set logfile blah.log" then
looking at that logfile never shows the daemon never attempting to
communicate to pop3.server barring this one scenario:

If I reverse the order of the two sections (i.e. put the pop3 one first)
and start fetchmail, it polls pop3.server once -- and only once -- then
proceeds to do IMAP IDLE to imap.server from that point forward.  It
never re-polls pop3.server.

Things I've tried:

* Setting "interval 60" (or some equivalent) inside of the pop3.server
  polling block -- no difference (and fetchmail --configdump shows that
  the parameter changes from 'interval':0 to 'interval':60)

* Running "fetchmail" by itself, which should "wake up" the daemon and
  attempt polling from defined servers -- it never does, meaning it acts
  as if the pop3.server poll section doesn't exist

* fetchmail --configdump shows both imap.server and pop3.server, and
  both are 'active':TRUE

If I remove use of the IMAP IDLE feature, things behave as I'd expect.
(I see the "sleeping at {date} for N seconds" message in the log, etc.)

It's almost as if use of IMAP IDLE causes the daemon to effectively
ignore any other poll sections.

Thoughts?

-- 
| Jeremy Chadwick                                   jd...@ko... |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

Re: [Fetchmail-users] fetchmail not polling multiple servers when IMAP IDLE used

[Jeremy C. <jd...@ko...>]

On Tue, Nov 18, 2014 at 07:52:24PM -0600, Corey Halpin wrote:
> On 2014-11-18, Jeremy Chadwick wrote:
> > It's almost as if use of IMAP IDLE causes the daemon to effectively
> > ignore any other poll sections.
> 
> From the fetchmail manpage, in "The Run Control File" section, under
> "Singledrop vs Multidrop options":
> 
>   The 'idle' option is intended to be used with IMAP servers
>   supporting the RFC2177 IDLE command extension, but does not strictly
>   require it.  If it is enabled, and fetchmail detects that IDLE is
>   supported, an IDLE will be issued at the end of each poll.  This
>   will tell the IMAP server to hold the connection open and notify the
>   client when new mail is available.  If IDLE is not supported,
>   fetchmail will simulate it by periodically issuing NOOP. If you need
>   to poll a link frequently, IDLE can save bandwidth by eliminating
>   TCP/IP connects and LOGIN/LOGOUT sequences. On the other hand, an
>   IDLE connection will eat almost all of your fetchmail's time,
>   because it will never drop the connection and allow other polls to
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   occur unless the server times out the IDLE.  It also doesn't work
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   with multiple folders; only the first folder will ever be polled.

Thank you Corey!

Yup, I didn't actually manage to read that in the man page.  Bummer,
although that conflicts with the following from section "GENERAL
OPERATION":

  Each  server name that you specify following the options on the
  command line will be queried.  If you don't specify any servers on
  the command line, each 'poll' entry in your ~/.fetchmailrc file
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  will be queried.
  ^^^^^^^^^^^^^^^^

Back to cronjobs, either that or maintaining two separate fetchmail
config files + pid files, etc.  :/

-- 
| Jeremy Chadwick                                   jd...@ko... |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

Re: [Fetchmail-users] fetchmail not polling multiple servers when IMAP IDLE used

[Matthias A. <mat...@gm...>]

Am 19.11.2014 um 06:24 schrieb Jeremy Chadwick:

> Yup, I didn't actually manage to read that in the man page.  Bummer,
> although that conflicts with the following from section "GENERAL
> OPERATION":
> 
>   Each  server name that you specify following the options on the
>   command line will be queried.  If you don't specify any servers on
>   the command line, each 'poll' entry in your ~/.fetchmailrc file
>                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   will be queried.
>   ^^^^^^^^^^^^^^^^


> 
> Back to cronjobs, either that or maintaining two separate fetchmail
> config files + pid files, etc.  :/

Jeremy,

sorry about that.  In my opinion IMAP IDLE is somewhat useless due to
its limited applicability.  Either you bind resources on server and
client end keeping lots of connections open, or it simply won't work.
It may have been conceived as an INBOX monitor but once we start talking
about server-side filters, possibly sieve, that assumption seems
far-fetched...

Thanks for pointing out the inconsistency in the manual page, I have
seen do that, and the next fetchmail releases will error out if --idle
is combined with multiple mailboxes and/or accounts.

Cheers,
Matthias

Re: [Fetchmail-users] fetchmail not polling multiple servers when IMAP IDLE used

[Hendrik H. <hen...@gm...>]

Hi,
using separate config folders and pids works great for me. I added a
script that creates the fetchmail files based on my own config file and
monitor all that with a cron job.
regards,
Hendrik

On 19/11/14 06:24, Jeremy Chadwick wrote:
> On Tue, Nov 18, 2014 at 07:52:24PM -0600, Corey Halpin wrote:
>> On 2014-11-18, Jeremy Chadwick wrote:
>>> It's almost as if use of IMAP IDLE causes the daemon to effectively
>>> ignore any other poll sections.
>> From the fetchmail manpage, in "The Run Control File" section, under
>> "Singledrop vs Multidrop options":
>>
>>   The 'idle' option is intended to be used with IMAP servers
>>   supporting the RFC2177 IDLE command extension, but does not strictly
>>   require it.  If it is enabled, and fetchmail detects that IDLE is
>>   supported, an IDLE will be issued at the end of each poll.  This
>>   will tell the IMAP server to hold the connection open and notify the
>>   client when new mail is available.  If IDLE is not supported,
>>   fetchmail will simulate it by periodically issuing NOOP. If you need
>>   to poll a link frequently, IDLE can save bandwidth by eliminating
>>   TCP/IP connects and LOGIN/LOGOUT sequences. On the other hand, an
>>   IDLE connection will eat almost all of your fetchmail's time,
>>   because it will never drop the connection and allow other polls to
>>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>   occur unless the server times out the IDLE.  It also doesn't work
>>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>   with multiple folders; only the first folder will ever be polled.
> Thank you Corey!
>
> Yup, I didn't actually manage to read that in the man page.  Bummer,
> although that conflicts with the following from section "GENERAL
> OPERATION":
>
>   Each  server name that you specify following the options on the
>   command line will be queried.  If you don't specify any servers on
>   the command line, each 'poll' entry in your ~/.fetchmailrc file
>                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   will be queried.
>   ^^^^^^^^^^^^^^^^
>
> Back to cronjobs, either that or maintaining two separate fetchmail
> config files + pid files, etc.  :/
>

Re: [Fetchmail-users] fetchmail not polling multiple servers when IMAP IDLE used

[Matthias A. <mat...@gm...>]

Am 23.11.2014 um 13:13 schrieb Peter Pentchev:
> On Sat, Nov 22, 2014 at 12:45:28AM +0100, Matthias Andree wrote:
>> Am 19.11.2014 um 06:24 schrieb Jeremy Chadwick:
>>
>>> Yup, I didn't actually manage to read that in the man page.  Bummer,
>>> although that conflicts with the following from section "GENERAL
>>> OPERATION":
>>>
>>>   Each  server name that you specify following the options on the
>>>   command line will be queried.  If you don't specify any servers on
>>>   the command line, each 'poll' entry in your ~/.fetchmailrc file
>>>                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>   will be queried.
>>>   ^^^^^^^^^^^^^^^^
>>
>>
>>>
>>> Back to cronjobs, either that or maintaining two separate fetchmail
>>> config files + pid files, etc.  :/
>>
>> Jeremy,
>>
>> sorry about that.  In my opinion IMAP IDLE is somewhat useless due to
>> its limited applicability.  Either you bind resources on server and
>> client end keeping lots of connections open, or it simply won't work.
>> It may have been conceived as an INBOX monitor but once we start talking
>> about server-side filters, possibly sieve, that assumption seems
>> far-fetched...
>>
>> Thanks for pointing out the inconsistency in the manual page, I have
>> seen do that, and the next fetchmail releases will error out if --idle
>> is combined with multiple mailboxes and/or accounts.
> 
> Hmm, shouldn't it only error out in daemon mode?  I guess it might be
> fine for a single round of checks for new mail.
> 
> G'luck,
> Peter
> 

Hi Peter,

--idle does not look at --daemon, so --idle will make fetchmail sit on
the first polled IMAP mailbox that offers idle support and loop there,
so I presume the refusal should be independent of daemon mode.

Cheers,
Matthias