fetchmail-devel — Development related discussion

Re: [fetchmail-devel] [PATCHes] Misc. cleanups

[Nico G. <ni...@ng...>]

Hi,
* Miloslav Trmac <mi...@re...> [2006-03-12 01:25]:
[...] 
> -    if (retval = krb5_cc_get_principal(context, ccdef, &client)) {
> +    if ((retval = krb5_cc_get_principal(context, ccdef, &client)) != 0) {

[...] 
What is your point here? I can't see a real reason to do 
this because its the same.
Regards Nico
-- 
Nico Golde - JAB: ni...@ja... | GPG: 0x73647CFF
http://www.ngolde.de | http://www.muttng.org | http://grml.org
Forget about that mouse with 3/4/5 buttons -
gimme a keyboard with 103/104/105 keys!

[fetchmail-devel] [PATCHes] Misc. cleanups

[Miloslav T. <mi...@re...>]

Hello,
as usual I have choosen the perfect time for cleanup patches :)

The patches avoid most warnings with gcc -Wall -Wextra:
- fetchmail-signed.patch: Avoid implicit conversions between
  char * and unsigned char *; they are not defined by ISO C and gcc
  currently warns about them.  Most places were changed to use "char *",
  or "void *" where general binary data is handled.
- fetchmail-comparisons.patch: Change variable types or add explicit
  casts to make all casts use types with consistent type signedness,
  for clarity
- fetchmail-clean.patch: Remove or (void) out unused parameters, avoid
  assignments directly in if ().

- fetchmail-lmtp.patch: A real bug detected by the compiler, "," has
  lower priority than "||".

Thanks,
	Mirek

Re: [fetchmail-devel] Re: Incorrect header line and lost mails

[Frederic M. <fre...@wo...>]

Sunil Shetye wrote:
> Quoting from Matthias Andree's mail on Tue, Mar 07, 2006:
>   
>> what should we do WRT fastuidl?  We seem to need sanity checks to disable
>> the fastuidl search for servers such as Fr?d?ric's.
>>     
>
> One option is to change the default to a lower value from the current
> value of 10. The lowest practical value is 2, which will use linear
> search in alternate polls.
>
> A default of 4 should be alright. It will use linear search in the
> first poll, followed by binary search in the next three polls,
> followed by linear search in the next poll, and so on.
>
> Other solutions are also possible. For example, keeping a map of UID
> and message number in memory (with corrections for mails deleted after
> logout). Then, when there is a mismatch in the message number expected
> and the message number received from the remote server for any one
> mail, a linear search should be forced in the next(*) poll. This would
> be equivalent to setting fastuidl to 2 if this mismatch happens in
> every poll.
>
> Note that Frederic has claimed that messages with low UID never get
> downloaded. This is only partially correct. In his case, messages with
> low UID never get downloaded whenever a binary search is performed.
> However, a linear search is performed once in every ten polls
> (starting with the first poll) and these messages which were missed
> out earlier will get downloaded in this poll with linear search.
>
>   
Thank you for the clarification ! I was wondering why none of my user 
had complained about any lost e-mail. We receive a lot of spam but I 
would have been surprised if all the lost e-mails were only spam :-)

A suitable solution would be to have an entry in the FAQ with the 
settings required for a smooth operation with Mercury. If I understand 
it right, the only bad effect is a long delay (up to 10 times the poll 
delay) in the delivery of some e-mails. It isn't that bad.

BTW, I reported this sorting feature and the long lines wrapping problem 
to the Mercury mailing list but I don't expect any answer. It is closed 
source and with only one developer. David Harris seems to be busy fixing 
other more urgent problems in his mail client Pegasus Mail.

Frederic

[fetchmail-devel] Re: Incorrect header line and lost mails

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Tue, Mar 07, 2006:
> what should we do WRT fastuidl?  We seem to need sanity checks to disable
> the fastuidl search for servers such as Fr?d?ric's.

One option is to change the default to a lower value from the current
value of 10. The lowest practical value is 2, which will use linear
search in alternate polls.

A default of 4 should be alright. It will use linear search in the
first poll, followed by binary search in the next three polls,
followed by linear search in the next poll, and so on.

Other solutions are also possible. For example, keeping a map of UID
and message number in memory (with corrections for mails deleted after
logout). Then, when there is a mismatch in the message number expected
and the message number received from the remote server for any one
mail, a linear search should be forced in the next(*) poll. This would
be equivalent to setting fastuidl to 2 if this mismatch happens in
every poll.

Note that Frederic has claimed that messages with low UID never get
downloaded. This is only partially correct. In his case, messages with
low UID never get downloaded whenever a binary search is performed.
However, a linear search is performed once in every ten polls
(starting with the first poll) and these messages which were missed
out earlier will get downloaded in this poll with linear search.

-- 
Sunil Shetye.

(*) There is a reason why a linear search should not be forced in this
poll after detecting the mismatch. The same reason applies as to why
fastuidl cannot be disabled for servers such as that of Frederic.

The fastuidl and the fetchsizelimit options have been developed
keeping in mind a large mailbox with thousands of mails being polled
over a slow dialup line which is prone to frequent disconnection.
fetchmail used to previously download all UIDs and all mail sizes
right at the start of the poll. Thus, the amount of data transferred
even before the start of the download of the first mail used to be
huge. For example, if there are 2000 mails in the folder, the data
transferred would be around 110kb (assuming a UID size of 40b) for
POP3. Assuming that an average mail size is 5kb, this would mean that
data equivalent to 22 mails have been downloaded even before
downloading the first mail.

The fetchsizelimit option delays the downloading of the mail sizes.
The fastuidl option delays the downloading of UIDs when doing binary
search. Thus, the first (new) mail is downloaded pretty fast.

Forcing linear search on this poll after detecting the mismatch will
effectively disable fastuidl if this mismatch is detected on every
poll.

Re: [fetchmail-devel] Re: [fetchmail-users] Incorrect header line and lost mails

[Matthias A. <mat...@gm...>]

Frederic Marchal wrote:

> The only option left is to read the whole UID list each time. Is it
> possible to configure fetchmail to do that ?

Try "--fastuidl 0" on the command line.

Quote from the manual:

  --fastuidl <number>
         (Keyword: fastuidl) Do a binary instead of linear search for the
         first unseen UID. Binary search avoids downloading the  UIDs  of
         all  mails.  This  saves  time (especially in daemon mode) where
         downloading the same set of UIDs in each  poll  is  a  waste  of
         bandwidth.  The  number 'n' indicates how rarely a linear search
         should be done. In daemon mode, linear search is used once  fol-
         lowed  by  binary searches in 'n-1' polls if 'n' is greater than
         1; binary search is always used if 'n' is 1;  linear  search  is
         always  used  if  'n' is 0. In non-daemon mode, binary search is
         used if 'n' is 1; otherwise linear search is used.  This  option
         works with POP3 only.

Sunil,

what should we do WRT fastuidl?  We seem to need sanity checks to disable
the fastuidl search for servers such as Frédéric's.

Thanks,
Matthias

Re: [fetchmail-devel] Re: [fetchmail-users] Incorrect header line and lost mails

[Frederic M. <fre...@wo...>]

Matthias Andree wrote:
> Does fetchmail -vvv reveal any more detail? For instance, it would show
> the UIDL lists, and some editing with cut, sort, a text editor and
> finally the "comm" command might show what's really going on.
>
>   
It wasn't long and here is the result. I reconstructed the three UID 
lists here below from the UID fetchmail fetches.

First run, 5 messages are waiting, the response to the UIDL command is 
as follow. All the UID are listed.

POP3< 1 3QWN3Z3.CNM34671D94
POP3< 2 5KS8NWT.CNM34671DAD
POP3< 3 GWBROT7.CNM34671DE2
POP3< 4 IZ3YMGF.CNM34671D48
POP3< 5 YDL1GPM.CNM34671DAD

Next run, 8 messages are on the server and fetchmail tries to find the 
first unseen UID starting from number 4 and goes back to number 2

POP3< +OK 2 3QWN3Z3.CNM34671D94
POP3< +OK 3 5KS8NWT.CNM34671DAD
POP3< +OK 4 D76QP56.CNM34671F2A
POP3< +OK 5 GWBROT7.CNM34671DE2
POP3< +OK 6 IZ3YMGF.CNM34671D48
POP3< +OK 7 VYMMFQG.CNM34671EFC
POP3< +OK 8 YDL1GPM.CNM34671DAD

Next run, 10 messages, fetchmail seeks the first unseen message from 
number 5

POP3< +OK 5 D76QP56.CNM34671F2A
POP3< +OK 6 GWBROT7.CNM34671DE2
POP3< +OK 7 IZ3YMGF.CNM34671D48
POP3< +OK 8 N09R1OZ.CNM34671FB1
POP3< +OK 9 VYMMFQG.CNM34671EFC
POP3< +OK 10 YDL1GPM.CNM34671DAD

As we can see, the UID numbers are not the same from one run to the 
other. Mercury returns a UID list sorted alphabetically on the UID string !

As a consequence, trying to find the last unseen message from a 
dichotomic search on such a UID list always fails.

It also explains why I thought some messages were downloaded more than 
one time. They weren't but I believed so because the same message 
numbers were downloaded several times. They were simply different UID. 
But on the other side, messages with a low UID (alphabetically) were 
clearly never downloaded.

The only option left is to read the whole UID list each time. Is it 
possible to configure fetchmail to do that ?

Frederic

Re: [fetchmail-devel] Re: [fetchmail-users] Incorrect header line and lost mails

[Frederic M. <fre...@wo...>]

Matthias Andree wrote:
> Frederic Marchal <fre...@wo...> writes:
>
>   
>> Why is it reporting 1353 messages seen, then 1355 and then 1358 without
>> downloading any mail ? I have no answer and I can't figure out why it
>> would be related to my patch. Moreover, and this is related to the same
>> problem, Fetchmail keeps downloading some e-mails two or three times or
>> it delays some e-mails until it eventually download them. It looks like
>> the UIDL management is seriously broken but it may be on my version
>> only. I just have to find out...
>>     
>
> Does fetchmail -vvv reveal any more detail? For instance, it would show
> the UIDL lists, and some editing with cut, sort, a text editor and
> finally the "comm" command might show what's really going on.
>
>   

I haven't tried that on the production server because of the daunting 
amount of output it generates. I'll try it now and see if I can get 
something out of it.

Frederic

Re: [fetchmail-devel] Re: [fetchmail-users] Incorrect header line and lost mails

[Matthias A. <mat...@gm...>]

Frederic Marchal <fre...@wo...> writes:

> Here is a suspicious case from my log with the "keep" option active:
>
> Feb 20 17:36:26 localhost fetchmail[14623]: mise en sommeil à lun 20 fév
> 2006 17:36:26 CET
> Feb 20 17:41:26 localhost fetchmail[14623]: réveillé à lun 20 fév 2006
> 17:41:26 CET
> Feb 20 17:41:29 localhost fetchmail[14623]: 1353 messages (1353 déjà
> vus) pour xxx#yyy.be dans 192.168.100.1 (70802116 octets).

[...]

> Why is it reporting 1353 messages seen, then 1355 and then 1358 without
> downloading any mail ? I have no answer and I can't figure out why it
> would be related to my patch. Moreover, and this is related to the same
> problem, Fetchmail keeps downloading some e-mails two or three times or
> it delays some e-mails until it eventually download them. It looks like
> the UIDL management is seriously broken but it may be on my version
> only. I just have to find out...

Does fetchmail -vvv reveal any more detail? For instance, it would show
the UIDL lists, and some editing with cut, sort, a text editor and
finally the "comm" command might show what's really going on.

-- 
Matthias Andree

[fetchmail-devel] [PATCH] Re: BUG (?) fetchmail 6.3.2 hangs with 'idle' on OpenBSD

[Sunil S. <sh...@bo...>]

Quoting from Casper Gripenberg's mail on Thu, Mar 02, 2006:
> > I have finally analysed the patch by Brendan Lynch and found that it
> > has correctly identified the source of the problem. So, please junk my
> > previous patch and try this one. Again, log to syslog to get the time
> > aspect correct.
> 
> Looks like it's working better now. No delays this time. Cool :)
> Here's the output:
> 
> 19:03:44 : IMAP> A0012 NOOP 
> 19:03:44 : IMAP< A0012 OK NOOP completed 
> 19:03:53 : IMAP< * 1 EXISTS 
> 19:03:53 : IMAP> A0013 NOOP 
> 19:03:53 : IMAP< * 1 RECENT 
> 19:03:53 : IMAP< A0013 OK NOOP completed 
> 19:03:53 : 1 message waiting after re-poll 

Cool. There has been no response from Brendan regarding the new patch.
However, I believe that it should work correctly.

Matthias, could you consider this patch for 6.3.3? This patch fixes
the following issues:

- Clear the tag when waiting for asynchronous response from IMAP
  server. Based on original patch by Brendan Lynch.

- Set the stage and timeout to different values only when waiting for
  asynchronous response. This is either after sending "IDLE" or after
  not sending any command.

- Restore stage and timeout after sending "DONE" in "IDLE" mode while
  waiting for "OK IDLE completed". Also, use default stage and timeout
  when transacting "NOOP".

- After getting "EXISTS" while waiting for asynchronous response, send
  a "NOOP" immediately to get an updated "RECENT" response too.

- Have only one return statement in imap_idle(). Tracing the function
  is now easier.

-- 
Sunil Shetye.

[fetchmail-devel] Re: [fetchmail-users] Re: RETR with UIDL

[Matthias A. <mat...@gm...>]

Jakob Hirsch schrieb am 2006-03-06:

> > I hope we can also add the often-requested "delete after N days" that
> > ESR refused (probably because he was scared of the implications with the
> > long-winding UID code), but I'm not sure if time permits that for 6.4.X.
> 
> Yet another MUA feature. :)
> 
> How are you going to do this? Adding a third column to the fetchids file?

Probably, although this might become a tagged format so it's extensible.
The current UID implementation is very inefficient and needs to be
revised.

> > The user should not need to care about protocol details before he runs
> 
> He should not, but often enough he has to. I prefer to have full control
> instead of having to work around some program's magic. But I understand
> that my view is surely not the same as some usual user's. I hope it's
> possible to satisfy both user groups.

Magic only if nothing is configured explicitly.

> > Server-side tracking is a constant source of troubles with respect to
> > interrupted connections, software faults, aborts, reboots, any other
> > things, and relying on POP3 server state beyond UIDL just doesn't work
> > well, so everything will be switched to client-side tracking.
> 
> Not that I care much about that, but server-vs-client side tracking is
> not only an implementional difference, but also a functional one, so I'd
> be reluctant to rip out SST. Some broken servers should not be a reason
> to abandon a feature.

fetchmail's foremost task is to fetch every message, exactly once. This
cannot be achieved with server-side tracking.

> > (insecure, removed with RFC-1460 in 1993), "LAST" (removed with RFC-1725
> 
> Usually I agree to get rid of old stuff, most of the time they are
> kludges with incosistent behaviour and are replaced by a superior
> mechanism. But that's not true for LAST, which has no 100% (in POP3).
> And it's still supported by modern servers like Dovecot.

LAST is a nonstandard extension. Perhaps I'll keep it in as fallback for
--keep on servers that don't talk UIDL, but IF it keeps in, only as last
resort.

> > talking RFC-1939 (POP3)/1734 (AUTH)/2449 (TLS)/2222 (SASL).
> 
> But things like ntlm/msn authentication and pop3s/imaps are still supported?

Probably yes, modulo the NTLM bugs that got reported.

-- 
Matthias Andree

[fetchmail-devel] Re: [fetchmail-users] Re: RETR with UIDL

[Jakob H. <jh...@pl...>]

Matthias Andree wrote:

> UIDL is a pretty standard MUA (used by major MUAs) mechanism for "keep
> messages on server" setups, and it's very likely I'll make UIDL default
> in future versions.

Sure, even OE supports it. But still, the default operation of all
client I know is "download and delete", so UIDL is not used. The major
use of fetchmail is being some kind of MUA "proxy" (not in technical
sense), so it should (by default) behave like a (default) MUA.

> I hope we can also add the often-requested "delete after N days" that
> ESR refused (probably because he was scared of the implications with the
> long-winding UID code), but I'm not sure if time permits that for 6.4.X.

Yet another MUA feature. :)

How are you going to do this? Adding a third column to the fetchids file?

>> available on the server). The magic fetchmail does is nice and works in
>> most cases, but final control should be left to the user.
> 
> The user should not need to care about protocol details before he runs

He should not, but often enough he has to. I prefer to have full control
instead of having to work around some program's magic. But I understand
that my view is surely not the same as some usual user's. I hope it's
possible to satisfy both user groups.

> Server-side tracking is a constant source of troubles with respect to
> interrupted connections, software faults, aborts, reboots, any other
> things, and relying on POP3 server state beyond UIDL just doesn't work
> well, so everything will be switched to client-side tracking.

Not that I care much about that, but server-vs-client side tracking is
not only an implementional difference, but also a functional one, so I'd
be reluctant to rip out SST. Some broken servers should not be a reason
to abandon a feature.

> (insecure, removed with RFC-1460 in 1993), "LAST" (removed with RFC-1725

Usually I agree to get rid of old stuff, most of the time they are
kludges with incosistent behaviour and are replaced by a superior
mechanism. But that's not true for LAST, which has no 100% (in POP3).
And it's still supported by modern servers like Dovecot.

> talking RFC-1939 (POP3)/1734 (AUTH)/2449 (TLS)/2222 (SASL).

But things like ntlm/msn authentication and pop3s/imaps are still supported?

Re: [fetchmail-devel] Re: [fetchmail-users] Incorrect header line and lost mails

[Frederic M. <fre...@wo...>]

Matthias Andree wrote:
> Greetings,
>
> I think we'll shelve this problem and revisit in 6.4.X. I've thought
> about it again, and there is no good solution. It is unclear whether
> some mailer spoiled the blank line between header and body, and the
> incorrect header line delimits header from body, or whether it's just
> broken folding as observed by you.
>   
> I am very much in favor of Rob MacGregor's suggestion of putting the
> message in a message/rfc822 container, and if user addresses have been
> figured from the header, forward there, else to the fallback postmaster
> (usually the calling user, literally "postmaster" if run as root, or
> whichever is configured in the rcfile), and I'd be more comfortable if
> such major changes can get testing first before being deployed. If it
> works well, it might get backported to 6.3.X depending on how fast those
> evolve.
>
> I hope that's acceptable. If you want messages passed on in spite of
> problematic headers, and let other parts of the mail system sort things
> out, try the attached patch. It's highly experimental, may cause older
> bugs to reappear that were masked (which please report, particularly
> segfaults!), and isn't suitable for application in production or
> distributions.
>   

Thank you Matthias, but I patched it myself as I promised. I should have 
told you about it, but I haven't said a word because I'm experiencing 
some big UIDL troubles with version 6.3.2 (I back ported my fix to that 
version to try it in production) and I don't know if the problem is 
related to my fix or if it is a known problem in 6.3.2 (as the recent 
e-mails on this list tend to say).

Here is a suspicious case from my log with the "keep" option active:

Feb 20 17:36:26 localhost fetchmail[14623]: mise en sommeil à lun 20 fév 
2006 17:36:26 CET
Feb 20 17:41:26 localhost fetchmail[14623]: réveillé à lun 20 fév 2006 
17:41:26 CET
Feb 20 17:41:29 localhost fetchmail[14623]: 1353 messages (1353 déjà 
vus) pour xxx#yyy.be dans 192.168.100.1 (70802116 octets).
Feb 20 17:41:29 localhost fetchmail[14623]: mise en sommeil à lun 20 fév 
2006 17:41:29 CET
Feb 20 17:46:29 localhost fetchmail[14623]: réveillé à lun 20 fév 2006 
17:46:29 CET
Feb 20 17:46:31 localhost fetchmail[14623]: 1355 messages (1355 déjà 
vus) pour xxx#yyy.be dans 192.168.100.1 (70808100 octets).
Feb 20 17:46:31 localhost fetchmail[14623]: mise en sommeil à lun 20 fév 
2006 17:46:31 CET
Feb 20 17:51:31 localhost fetchmail[14623]: réveillé à lun 20 fév 2006 
17:51:31 CET
Feb 20 17:51:34 localhost fetchmail[14623]: 1358 messages (1358 déjà 
vus) pour xxx#yyy.be dans 192.168.100.1 (70825184 octets).
Feb 20 17:51:34 localhost fetchmail[14623]: mise en sommeil à lun 20 fév 
2006 17:51:34 CET

Why is it reporting 1353 messages seen, then 1355 and then 1358 without 
downloading any mail ? I have no answer and I can't figure out why it 
would be related to my patch. Moreover, and this is related to the same 
problem, Fetchmail keeps downloading some e-mails two or three times or 
it delays some e-mails until it eventually download them. It looks like 
the UIDL management is seriously broken but it may be on my version 
only. I just have to find out...

If I remove the "keep" option everything looks good and I can't see any 
mail loss but, on the other hand, I can't compare the mails on the 
server to those actually downloaded.

I also think my fix won't be acceptable to you because I let the message 
pass without any change (I can't encapsulate the offending e-mail in a 
container. It is beyond the time I can spend on this project) and I keep 
processing the e-mail until the first blank line is found even if it is 
in the body (I don't know what would happen if the end of the mail was 
reached without encountering a blank line).

Anyway, if you want to have a look at my patch in its current state, I 
can send it to you.

Frederic

Re: [fetchmail-devel] [BUG] fetchmail hangs during pop3 pull after a mail with a null char

[David G. <da...@dg...>]

Matthias Andree wrote:

>David Greaves <da...@dg...> wrote, in October 2004:
>
>  
>
>>It's been working for years with these occasional hangs that have been
>>fixed by popping the bad messages and manually filing them. I finally
>>had a bad message arrive when I was in a position to debug!
>>
>>Summary : fetchmail hangs during pop3 pull after a mail with a null char.
>>
>>The mail with a null char is pulled OK but then rejected by local
>>Cyrus lmtp and bounced to postmaster via exim4.20
>>The next pop3 pull then fails.
>>I've made an effort to trace and I think the hang occurs due to a
>>double call to SMTP_ok which is empty the second time. I am pretty
>>sure the second call originates at sink.c line 1433.
>>    
>>
>
>David, could you try to reproduce the problem with fetchmail 6.3.2 and
>see if the problem is gone?  The hang might have been related to LMTP
>bugs or rather fetchmail 6.3.1 and older not properly talking SMTP any
>more after having talked LMTP, so chances are the hang no longer occurs
>with 6.3.2.
>
>Thanks,
>  
>
OK
I'm away for 2 weeks so I'll have to check when I return...

David

-- 

Re: [fetchmail-devel] Re: BUG (?) fetchmail 6.3.2 hangs with 'idle' on OpenBSD

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> The credit for this goes to Chris Boyle. Surprising that there is no
> mention in the logs and NEWS.
>
> <http://lists.ccil.org/pipermail/fetchmail-friends/2003-July/007705.html>
> <http://lists.ccil.org/pipermail/fetchmail-friends/2003-July/007713.html>

Thanks for digging this up. Better late than never, I've added Chris to
the NEWS file to credit him. The NEWS file of 6.3.3 will mention him in
two places (in 6.2.4 and 6.3.3).

Chris, since Eric is no longer active with the project, please accept
/my/ apologies for Eric's missing your NOOP-based IMAP4 IDLE emulation
work in the NEWS file and log entries between fetchmail 6.2.2 and 6.2.4.
(I hope the mail address still works.)

-- 
Matthias Andree

Re: [fetchmail-devel] Re: [fetchmail-users] Incorrect header line and lost mails

[Matthias A. <mat...@gm...>]

Greetings,

I think we'll shelve this problem and revisit in 6.4.X. I've thought
about it again, and there is no good solution. It is unclear whether
some mailer spoiled the blank line between header and body, and the
incorrect header line delimits header from body, or whether it's just
broken folding as observed by you.

I am very much in favor of Rob MacGregor's suggestion of putting the
message in a message/rfc822 container, and if user addresses have been
figured from the header, forward there, else to the fallback postmaster
(usually the calling user, literally "postmaster" if run as root, or
whichever is configured in the rcfile), and I'd be more comfortable if
such major changes can get testing first before being deployed. If it
works well, it might get backported to 6.3.X depending on how fast those
evolve.

I hope that's acceptable. If you want messages passed on in spite of
problematic headers, and let other parts of the mail system sort things
out, try the attached patch. It's highly experimental, may cause older
bugs to reappear that were masked (which please report, particularly
segfaults!), and isn't suitable for application in production or
distributions.

Regards,

-- 
Matthias Andree

Re: [fetchmail-devel] [BUG] fetchmail hangs during pop3 pull after a mail with a null char

[Matthias A. <mat...@gm...>]

David Greaves <da...@dg...> wrote, in October 2004:

> It's been working for years with these occasional hangs that have been
> fixed by popping the bad messages and manually filing them. I finally
> had a bad message arrive when I was in a position to debug!
>
> Summary : fetchmail hangs during pop3 pull after a mail with a null char.
>
> The mail with a null char is pulled OK but then rejected by local
> Cyrus lmtp and bounced to postmaster via exim4.20
> The next pop3 pull then fails.
> I've made an effort to trace and I think the hang occurs due to a
> double call to SMTP_ok which is empty the second time. I am pretty
> sure the second call originates at sink.c line 1433.

David, could you try to reproduce the problem with fetchmail 6.3.2 and
see if the problem is gone?  The hang might have been related to LMTP
bugs or rather fetchmail 6.3.1 and older not properly talking SMTP any
more after having talked LMTP, so chances are the hang no longer occurs
with 6.3.2.

Thanks,

-- 
Matthias Andree

[fetchmail-devel] Re: handle imap EXPUNGE better

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Wed, Mar 01, 2006:
> Well, OK. We'll take this as a safety measure, but I maintain that the
> real solution is to properly cache the mailbox state and update it as
> untagged responses come in.

Caching is certainly possible. However, matching the fetchmail counter
to the imap server counter is probably going to be computationally
expensive. Is it worth it? For example, the simple 'number'
calculation used three times in imap.c:

    number -= expunged;

will have to be replaced by a loop to find out how many mails less
than 'number' have been expunged. Suppose, fetchmail starts polling
the mailbox and then another email client deletes mails 3 and 5 just
when fetchmail is reading mail 1. Lets say that fetchmail analyses the
notification correctly after it has deleted and expunged mail 1:

IMAP< * 1 EXPUNGE	    # deletion of mail 1; expunged = 1
IMAP< * 2 EXPUNGE	    # deletion of mail 3; expunged = 2
IMAP< * 2 EXPUNGE	    # deletion of mail 5; expunged = 3

So, expunged is now 3 here. Now, suppose fetchmail tries to download
mail 2. The above calculation will not work as 'number' will be -1.
The above calculation will of course work correctly for mail 6
onwards.

The current simplistic assumption works as all expunged mails are
necessarily less than 'number'.

> I've merged the patch onto branches/BRANCH_6-3 for the nonce, and will
> update the trunk from there after the 6.3.3 release.

Thanks.

-- 
Sunil Shetye.

Re: [fetchmail-devel] Re: BUG (?) fetchmail 6.3.2 hangs with 'idle' on OpenBSD

[Casper G. <cas...@ko...>]

On Thu, Mar 02, 2006 at 06:32:13PM +0530, Sunil Shetye wrote:
> Quoting from Casper Gripenberg's mail on Wed, Mar 01, 2006:
> [...]
> So, the EXISTS notification is actually coming unattached and not as a
> part of response to any command. Interesting.

Yes. That how it seems.

> I have finally analysed the patch by Brendan Lynch and found that it
> has correctly identified the source of the problem. So, please junk my
> previous patch and try this one. Again, log to syslog to get the time
> aspect correct.

Looks like it's working better now. No delays this time. Cool :)
Here's the output:

19:03:16 : IMAP> A0011 NOOP 
19:03:16 : IMAP< A0011 OK NOOP completed 
19:03:44 : IMAP> A0012 NOOP 
19:03:44 : IMAP< A0012 OK NOOP completed 
19:03:52 sendmail[21312]: k22H3qr1021312: from=casper, size=60, class=0, nrcpts=1, msgid=<200603021703.k22H3qr1021312@xxxx>, relay=casper@localhost
19:03:53 sm-mta[556]: k22H3qb5000556: from=<casper@xxxx>, size=392, class=0, nrcpts=1, msgid=<200603021703.k22H3qr1021312@xxxx>, proto=ESMTP, daemon=MTA, relay=casper@localhost [127.0.0.1]
19:03:53 sendmail[21312]: k22H3qr1021312: to=cas...@ko..., ctladdr=casper (1000/1000), delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=30050, relay=localhost.homeip.net. [127.0.0.1], dsn=2.0.0, stat=Sent (k22H3qb5000556 Message accepted for delivery)
19:03:53 : IMAP< * 1 EXISTS 
19:03:53 : IMAP> A0013 NOOP 
19:03:53 : IMAP< * 1 RECENT 
19:03:53 : IMAP< A0013 OK NOOP completed 
19:03:53 : 1 message waiting after re-poll 
19:03:53 : IMAP> A0014 SEARCH UNSEEN NOT DELETED 
19:03:53 : IMAP< * SEARCH 1 
19:03:53 : 1 is unseen 
19:03:53 : IMAP< A0014 OK SEARCH completed 
19:03:53 : 1 is first unseen 

Casper

[fetchmail-devel] Re: BUG (?) fetchmail 6.3.2 hangs with 'idle' on OpenBSD

[Sunil S. <sh...@bo...>]

Quoting from Casper Gripenberg's mail on Wed, Mar 01, 2006:
> Ok I see. On the IMAP server I use the mail delivery is instant. 
> I tried it with direct telnet to the smtp server, and as soon as 
> I send the mail I see it on the IMAP side. There is absolutely 
> no delay.

Interesting.

> Here is the output. I cleaned it up a bit, and it includes my
> mta output too, so you can see when my testmail leaves my machine.

...

> 11:51:38 : IMAP> A0005 NOOP
> 11:51:38 : IMAP< A0005 OK NOOP completed
> 11:51:55 : IMAP< * 1 EXISTS
> 11:51:55 : IMAP> A0006 EXPUNGE
> 11:51:55 : IMAP< * 1 RECENT
> 11:51:55 : IMAP< A0006 OK EXPUNGE completed

So, the EXISTS notification is actually coming unattached and not as a
part of response to any command. Interesting.

> Ok. Right. It was a shoot from the hip fix. I just wanted to avoid the
> delay if possible. But if it's not avoidable in any proper way then
> that is the way it is. I guess people can live with getting their 
> e-mails a few seconds later, it's not THAT critical :) But for me
> the delay in there will still bother me..can't help it, sorry :)
> For anyone else, I guess they will never know. I would have never
> known hadn't I started investigating the problem anyway.

I have finally analysed the patch by Brendan Lynch and found that it
has correctly identified the source of the problem. So, please junk my
previous patch and try this one. Again, log to syslog to get the time
aspect correct.

This patch fixes the imap_idle() code in several ways:

- tag is emptied before waiting for asynchronous notification. This
  comes from the patch by Brendan Lynch.

- stage is currently being set to STAGE_IDLE throughout imap_idle().
  This is incorrect. stage should be STAGE_IDLE only when sending
  "IDLE" command and when waiting for asynchronous notification. In
  particular, it should not be set to STAGE_IDLE when sending "NOOP"
  as "NOOP" is a normal imap command and does not idle. Similar
  argument holds true after "DONE" as been sent.

- imap_idle() has 5 return statements. This makes working out what
  happens in which case difficult. Now, there will be only one return
  statement.

- After getting the "EXISTS" asynchronous notification, the "NOOP"
  command is sent *immediately* to get the updated "RECENT" count.
  
======================================================================
Index: fetchmail-6.3/imap.c
===================================================================
--- fetchmail-6.3/imap.c	(revision 4705)
+++ fetchmail-6.3/imap.c	(working copy)
@@ -621,7 +621,6 @@
 {
     int ok;
 
-    stage = STAGE_IDLE;
     saved_timeout = mytimeout;
 
     if (has_idle) {
@@ -629,6 +628,7 @@
 	 * at least every 28 minutes:
 	 * (the server may have an inactivity timeout) */
 	mytimeout = 1680; /* 28 min */
+	stage = STAGE_IDLE;
 	/* enter IDLE mode */
 	ok = gen_transact(sock, "IDLE");
 
@@ -637,37 +637,43 @@
 	    SockWrite(sock, "DONE\r\n", 6);
 	    if (outlevel >= O_MONITOR)
 		report(stdout, "IMAP> DONE\n");
-	} else
-	    /* not idle timeout */
-	    return ok;
+	    /* reset stage and timeout here: we are not idling any more */
+	    mytimeout = saved_timeout;
+	    stage = STAGE_FETCH;
+	    /* get OK IDLE message */
+	    ok = imap_ok(sock, NULL);
+	}
     } else {  /* no idle support, fake it */
-	/* when faking an idle, we can't assume the server will
-	 * send us the new messages out of the blue (RFC2060);
-	 * this timeout is potentially the delay before we notice
-	 * new mail (can be small since NOOP checking is cheap) */
-	mytimeout = 28;
+	/* the server may not send us new mail notification out of the
+	 * blue (RFC2060). Note: stage and timeout have not been
+	 * changed here as NOOP does not idle */
 	ok = gen_transact(sock, "NOOP");
-	/* if there's an error (not likely) or we just found mail (stage 
-	 * has changed, timeout has also been restored), we're done */
-	if (ok != 0 || stage != STAGE_IDLE)
-	    return(ok);
 
-	/* wait (briefly) for an unsolicited status update */
-	ok = imap_ok(sock, NULL);
-	/* again, this is new mail or an error */
-	if (ok != PS_IDLETIMEOUT)
-	    return(ok);
+	/* no error, but no new mail either */
+	if (ok == PS_SUCCESS && recentcount == 0)
+	{
+	    /* There are some servers who do send new mail
+	     * notification out of the blue. Wait for that with a low
+	     * timeout */
+	    mytimeout = 28;
+	    stage = STAGE_IDLE;
+	    /* We are waiting for notification; no tag needed */
+	    tag[0] = '\0';
+	    /* wait (briefly) for an unsolicited status update */
+	    ok = imap_ok(sock, NULL);
+	    if (ok == PS_IDLETIMEOUT) {
+		/* no notification came; ok */
+		ok = PS_SUCCESS;
+	    }
+	}
     }
 
     /* restore normal timeout value */
+    set_timeout(0);
     mytimeout = saved_timeout;
     stage = STAGE_FETCH;
 
-    /* get OK IDLE message */
-    if (has_idle)
-        return imap_ok(sock, NULL);
-
-    return PS_SUCCESS;
+    return(ok);
 }
 
 static int imap_getrange(int sock, 
======================================================================

-- 
Sunil Shetye.

[fetchmail-devel] Re: BUG (?) fetchmail 6.3.2 hangs with 'idle' on OpenBSD

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Wed, Mar 01, 2006:
> Sunil Shetye <sh...@bo...> writes:
> > Well, I am still "NOOP"ing. There is a problem in the current code.
> > What is happening is that gen_transact() is being followed by
> > imap_ok(). Internally, gen_transact() calls gen_recv() which calls
> > imap_ok(). So, imap_ok() is being called twice! The second call to
> > imap_ok() is incorrect as there is actually no "tag" to compare. So,
> > replacing imap_ok() by sleep() is the right thing to do.
> 
> I wonder. This code is ESR's original 
> 
> -----
> r3829 | esr | 2003-07-22 04:32:07 +0200 (Di, 22 Jul 2003) | 2 lines
> 
> Support faked IDLE.
> -----
> 
> (straight from SVN)
> 
> and has apparently never worked.

The credit for this goes to Chris Boyle. Surprising that there is no
mention in the logs and NEWS.

<http://lists.ccil.org/pipermail/fetchmail-friends/2003-July/007705.html>
<http://lists.ccil.org/pipermail/fetchmail-friends/2003-July/007713.html>

[On a side note, I tried searching for this string
"site:lists.ccil.org idle noop patch" on google and did not get the
above links. The same string gave correct results on yahoo.]

-- 
Sunil Shetye.

Re: [fetchmail-devel] [PATCH] handle imap EXPUNGE better

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> A good imap server can handle two email clients accessing the same
> mailbox and reports the mails deleted by one email client correctly to
> the other email client.
>
> However, fetchmail cannot handle this unexpected deletion of mails.
> This is because fetchmail caches which mails are new and the size of
> those new mails. This caching is required to give a better
> performance, but leads to unexpected results when another email client
> is writing to the same mailbox:
>
> - An old mail could be delivered instead of a recent mail.
>
> - fetchmail passes the wrong size to the SMTP server in the MAIL FROM:
>   command. If the SMTP server has size restrictions, a wrong mail will
>   get rejected!
>
> - "fetchmail -vv" reports that there is a mismatch in the expected and
>   received message sizes.
>
> This patch causes fetchmail to quit the poll if there is a mismatch in
> the count of mails being deleted.

Well, OK. We'll take this as a safety measure, but I maintain that the
real solution is to properly cache the mailbox state and update it as
untagged responses come in.

I've merged the patch onto branches/BRANCH_6-3 for the nonce, and will
update the trunk from there after the 6.3.3 release.

> This patch also handles servers which do not report RECENT on EXPUNGE
>in a better way. Now, recentcount is counted down on EXPUNGE of each
>mail along with count. This way, recentcount need not be reset to zero
>if the server does not give a new report of RECENT mails.
>
> This patch has been tested by running two fetchmails on the same
> mailbox!

Sunil Shetye <sh...@bo...> writes:

>> It would be better to update the cache properly (watching out for
>> eventual iterator variables that need to be decremented, too) and
>> continue. From our own cached flags we should be able to deduce if the
>> expunged message was recent or not. If we don't know and the server
>> expunged a different mail.
>> 
>> Now, before you go delve into the code and hack away, take a step back
>> from the blackboard to see the whole picture: the RECENT flags are just
>> crap if several clients are accessing the mailbox. What happens to the
>> RECENT count if we reselect the mailbox? Can we actually use it? Do we
>> need to check \Seen flags instead, for lack of UID support in IMAP?
>
> I plan to implement the recentcount as a difference between EXISTS
> count at end and beginning of poll. I can work on that only after this
> patch is accepted. At that time, your point will also get implemented.

Sounds good.

> fetchmail uses the RECENT only to decide
> a) whether to repoll at the end of mailbox poll
> b) whether to come out of IDLE (or NOOP look) while waiting for new
> mail.

> The UID code is not going to help here as this would require getting
> the full list of UIDs at the end of mailbox poll (and every minute,
> with "idle"). The RECENT code is much cleaner as imap servers reports
> the RECENT count consistently and correctly.
>
> The RECENT count is not being used to decide which mails are new. It
> is only being used at the end of the poll or while idling for new
> mail.

OK.

-- 
Matthias Andree

Re: [fetchmail-devel] Re: BUG (?) fetchmail 6.3.2 hangs with 'idle' on OpenBSD

[Casper G. <cas...@ko...>]

On Wed, Mar 01, 2006 at 01:35:16PM +0530, Sunil Shetye wrote:
> As far as the delay part goes, even with IMAP servers supporting IDLE,
> a delay of one minute is expected as IMAP servers checkpoint the
> folder every one minute or so. Of course, it will actually depend on
> the IMAP server configuration and implementation, 

Ok I see. On the IMAP server I use the mail delivery is instant. 
I tried it with direct telnet to the smtp server, and as soon as 
I send the mail I see it on the IMAP side. There is absolutely 
no delay.

> but the point is
> that a small delay is acceptable. Here, the maximum delay will be 28
> seconds, which should be acceptable.
>
> Could you log to syslog and send the output (with your patch)? I want
> to see the time aspect of this. In particular, I want to see if there
> is a delay between the "NOOP" response and the "EXISTS" response. If
> there is a delay, then your average delay will actually be 14 seconds.

Here is the output. I cleaned it up a bit, and it includes my
mta output too, so you can see when my testmail leaves my machine.

11:51:10 : 0 messages waiting after first poll
11:51:10 : IMAP> A0004 NOOP
11:51:10 : IMAP< A0004 OK NOOP completed
11:51:38 : IMAP> A0005 NOOP
11:51:38 : IMAP< A0005 OK NOOP completed
11:51:54 sendmail[16018]: k219psZR016018: from=casper, size=60, class=0, nrcpts=1, msgid=<200603010951.k219psZR016018@xxxxx>, relay=casper@localhost
11:51:55 sm-mta[30673]: k219psb5030673: from=<casper@xxxxx>, size=392, class=0, nrcpts=1, msgid=<200603010951.k219psZR016018@xxxxx>, proto=ESMTP, daemon=MTA, relay=casper@localhost [127.0.0.1]
11:51:55 sendmail[16018]: k219psZR016018: to=cas...@ko..., ctladdr=casper (1000/1000), delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=30050, relay=localhost. [127.0.0.1], dsn=2.0.0, stat=Sent (k219psb5030673 Message accepted for delivery)
11:51:55 : IMAP< * 1 EXISTS
11:51:55 : IMAP> A0006 EXPUNGE
11:51:55 : IMAP< * 1 RECENT
11:51:55 : IMAP< A0006 OK EXPUNGE completed
11:51:55 : 1 message waiting after expunge
11:51:55 : IMAP> A0007 SEARCH UNSEEN NOT DELETED
11:51:55 : IMAP< * SEARCH 1
11:51:55 : 1 is unseen
11:51:55 : IMAP< A0007 OK SEARCH completed
11:51:55 : 1 is first unseen
11:51:55 : 1 message for xxxxx at mail.kolumbus.fi.

> The problem with your patch is that it forces recentcount to be equal
> to count. You have a simple setup where all mails are always
> downloaded and deleted from the server. In this scenario, the number
> before EXISTS and RECENT will always be equal. However, there are
> people who have a setup where some seen mails are there on the
> mailserver and fetchmail is running with "no fetchall". So, forcing
> recentcount to count is not correct.

Ok. Right. It was a shoot from the hip fix. I just wanted to avoid the
delay if possible. But if it's not avoidable in any proper way then
that is the way it is. I guess people can live with getting their 
e-mails a few seconds later, it's not THAT critical :) But for me
the delay in there will still bother me..can't help it, sorry :)
For anyone else, I guess they will never know. I would have never
known hadn't I started investigating the problem anyway.

Casper

Re: [fetchmail-devel] Re: BUG (?) fetchmail 6.3.2 hangs with 'idle' on OpenBSD

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Well, I am still "NOOP"ing. There is a problem in the current code.
> What is happening is that gen_transact() is being followed by
> imap_ok(). Internally, gen_transact() calls gen_recv() which calls
> imap_ok(). So, imap_ok() is being called twice! The second call to
> imap_ok() is incorrect as there is actually no "tag" to compare. So,
> replacing imap_ok() by sleep() is the right thing to do.

I wonder. This code is ESR's original 

-----
r3829 | esr | 2003-07-22 04:32:07 +0200 (Di, 22 Jul 2003) | 2 lines

Support faked IDLE.
-----

(straight from SVN)

and has apparently never worked.

So right, gen_transact() followed by imap_ok() deserves more commentary
than just:

  3829        esr       /* wait (briefly) for an unsolicited status update */
  3829        esr       ok = imap_ok(sock, NULL);
  3829        esr       /* again, this is new mail or an error */
  3829        esr       if (ok != PS_IDLETIMEOUT)
  3829        esr           return(ok);

It appears that letting imap_ok time out is deliberate here, this is in
line with setting the timeout to 28 seconds and ignoring
PS_IDLETIMEOUT.

> Note that your server is sending the "EXISTS" response asynchronously,
> but fetchmail requires the "RECENT" response to come out of the idle
> loop. Your server is anyway sending "RECENT" only as a reply to the
> next "NOOP" command. So, as far as fetchmail is concerned, there is no
> delay anyway.

So the only question that remains is if fetchmail loses an entirely
asynchronous RECENT reply, or will pick it up next time waiting for the
tagged NOOP response, right?

-- 
Matthias Andree

[fetchmail-devel] Re: BUG (?) fetchmail 6.3.2 hangs with 'idle' on OpenBSD

[Sunil S. <sh...@bo...>]

Quoting from Casper Gripenberg's mail on Tue, Feb 28, 2006:
> Ok. Here's what happens with your patch:
> 
> fetchmail: IMAP> A0014 NOOP
> fetchmail: IMAP< A0014 OK NOOP completed
> fetchmail: IMAP> A0015 NOOP
> fetchmail: IMAP< * 1 EXISTS
> fetchmail: IMAP< * 1 RECENT
> fetchmail: IMAP< A0015 OK NOOP completed
> fetchmail: 1 message waiting after re-poll
> fetchmail: IMAP> A0016 SEARCH UNSEEN NOT DELETED
> fetchmail: IMAP< * SEARCH 1
> fetchmail: 1 is unseen
> ...
> 
> To me it looks like you are deliberately sleeping instead
> of "NOOP":ing. To get the EXISTS message to appear after
> the NOOP, right? In that sense I like my fix better, because
> it returns the mails immidiately. Your fix introduces an 
> unnecessary delay(?), but in the debug output it does
> now looks like everything is working like it should. 

Well, I am still "NOOP"ing. There is a problem in the current code.
What is happening is that gen_transact() is being followed by
imap_ok(). Internally, gen_transact() calls gen_recv() which calls
imap_ok(). So, imap_ok() is being called twice! The second call to
imap_ok() is incorrect as there is actually no "tag" to compare. So,
replacing imap_ok() by sleep() is the right thing to do.

As far as the delay part goes, even with IMAP servers supporting IDLE,
a delay of one minute is expected as IMAP servers checkpoint the
folder every one minute or so. Of course, it will actually depend on
the IMAP server configuration and implementation, but the point is
that a small delay is acceptable. Here, the maximum delay will be 28
seconds, which should be acceptable.

Could you log to syslog and send the output (with your patch)? I want
to see the time aspect of this. In particular, I want to see if there
is a delay between the "NOOP" response and the "EXISTS" response. If
there is a delay, then your average delay will actually be 14 seconds.

The problem with your patch is that it forces recentcount to be equal
to count. You have a simple setup where all mails are always
downloaded and deleted from the server. In this scenario, the number
before EXISTS and RECENT will always be equal. However, there are
people who have a setup where some seen mails are there on the
mailserver and fetchmail is running with "no fetchall". So, forcing
recentcount to count is not correct.

Note that your server is sending the "EXISTS" response asynchronously,
but fetchmail requires the "RECENT" response to come out of the idle
loop. Your server is anyway sending "RECENT" only as a reply to the
next "NOOP" command. So, as far as fetchmail is concerned, there is no
delay anyway.

-- 
Sunil Shetye.

Re: [fetchmail-devel] [PATCH] wrong option named in manual

[Matthias A. <mat...@gm...>]

Nico Golde <ni...@ng...> writes:

> in the manual the ssl option is mentioned as a server option 
> but its a user option.
> Patch attached.

Thanks, applied.

-- 
Matthias Andree