Re: [fetchmail-devel] imap: introduce imapsearchmax parameter

[Andrea R. <rig...@gm...>]

On Tue, Apr 26, 2011 at 01:56:35AM +0530, Sunil Shetye wrote:
> Hi Andrea,
> 
> Could you please send your imap server CAPABILITY string?

* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN
* X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE

> 
> > > You are missing the point of why there is a limit of
> > 1000 in the IMAP SEARCH command. There is a risk of response
> > truncation when there are more than 1860 new messages in the
> > IMAP folder if the limit is not imposed.
> > 
> > OK, thanks for the clarification, I didn't know about this
> > risk.
> > 
> > Maybe a simple and safe improvement could be to check if
> > there are more
> > than 1000 new messages in the folder using a single IMAP
> > SEARCH query
> > and only in this case download the new messages all at
> > once. Otherwise
> > we can fallback to the default behaviour.
> > 
> > What do you think?
> 
> Unfortunately, the information as to how many unseen messages are there is coming from the IMAP SEARCH command itself. So, to find out how to run IMAP SEARCH, you need a count of unseen messages and to get that count, you need to run IMAP SEARCH. Quite a chicken-and-egg problem...
> 
> Is there another way to get the information on the count of unseen messages? There are a few:
> 
> ==================================================================
> 1. Parse the response to the SELECT command:
> 
> A0001 SELECT "INBOX"
> * EXISTS 10000
> * RECENT 1
> * OK [UNSEEN 9988] Message 9988 is the first unseen message
> A0001 OK [READ-WRITE] SELECT COMPLETED
> 
> (As 9988 is the first unseen message, we know that there are 10000-9988+1=13 or less unseen messages)
> 
> Status: Required by RFC 3501, Optional in RFC 2060, RFC 1730.
> 
> Comments: RFC 3501 mentions that if this is missing in the response, client should not make any assumptions and use SEARCH. This means that we are back to square one.
> 
> 2. Send a STATUS command
> 
> A0002 STATUS "INBOX" (UNSEEN)
> * STATUS "INBOX (UNSEEN 13)
> A0002 OK STATUS COMPLETED
> 
> Status: Required by RFC 3501, RFC 2060. Not mentioned in RFC 1730.
> 
> Comments: As it is not supported by all IMAP servers, we cannot rely on this.
> 
> 3. Send an extended SEARCH command
> 
> A0003 SEARCH RETURN (COUNT MAX MIN) UNSEEN
> * ESEARCH (TAG "A0003") MIN 9988 COUNT 13 MAX 10000
> A0003 OK SEARCH COMPLETED
> 
> Status: Supported if CAPABILITY string contains ESEARCH. Mentioned by RFC 4731 in conjunction with RFC 4466. Not mentioned in RFC 3501, RFC 2060, RFC 1730.
> 
> Comments: As this is an extension, servers supporting this may be rare.
> ==================================================================
> 
> As you can see, getting that tiny piece of information in a manner supported by everybody is not going to be easy. As far as possible, fetchmail only uses techniques supported by almost all the imap servers.

I see. :( Any of the solution above should fix my problem, but none of
them are generic.

Do you think it's too difficult to detect if the server supports one of
the method above? In that case we could try to parse the response of
these commands and only if we find that at least one works, try to
reduce the number of submitted IMAP SEARCH queries.

Thanks,
-Andrea