[fetchmail-devel] [PATCH] Re: imap search improvements

[Sunil S. <sh...@bo...>]

Hi Matthias,

Please check the attached patch for improving imap search.

Quoting from Matthias Andree's mail on Mon, Feb 01, 2010:
> > 1. In r3838, the "SEARCH UNSEEN" was changed to "SEARCH UNSEEN NOT
> > DELETED". Some imap servers do not support such a complex search
> > phrase. I could find only one reference for that:
> > 
> > http://lists.ccil.org/pipermail/fetchmail-friends/2004-June/008897.html
> > 
> > I propose that that change should be undone.

...

> The issue reported 5 years ago refers to a regression on IMAP2 servers, and a
> regression fix is also always acceptable (though 6.2.X remains unsupported by
> me, I'm not releasing another 6.2.X - the fix will go into 6.3.X then).
> 
> In the implementation of the fix, could you switch by protocol version to use
> the "... NOT DELETED" form for newer servers (IMAP4r1, not sure if IMAP4 also
> supports it), and the older "SEARCH UNSEEN" only in IMAP2?  That would give us
> the best of both approaches -- or perhaps we could just fall back to SEARCH
> UNSEEN if SEARCH UNSEEN NOT DELETED fails?

Ok, I have ensured that "SEARCH UNSEEN" is used for IMAP2 servers.

Also, I have also changed "NOT DELETED" to "UNDELETED" assuming that
"SEARCH UNSEEN UNDELETED" will be faster than "SEARCH UNSEEN NOT
DELETED". This syntax is acceptable as per IMAP4 and IMAP4r1 RFCs.

> > 2. Some servers do not support "SEARCH" at all. Here are some
> > references for that:
> > 
> > http://lists.ccil.org/pipermail/fetchmail-friends/2004-May/008675.html
> > http://lists.ccil.org/pipermail/fetchmail-friends/2005-January/009351.html
> > 
> > In this case, as a fallback, fetchmail should use something like:
> > 
> > FETCH 1:1000 FLAGS
> > 
> > and process all mails without \Seen as unseen.
> 
> Much appreciated. OK since it extends fetchmail's audience.

...

> > 3. The reply to "SEARCH UNSEEN" overflows fetchmail buffer when there
> > are more than 1860 unseen mails. fetchmail should do a range search:
> > 
> > SEARCH 1:1000 UNSEEN
> > SEARCH 1001:2000 UNSEEN
> > ...
> 
> Desirable bug fix.

...

> Looking forward to your patches!

Thanks a lot for that. I am submitting only one patch to fix all the
mentioned issues. It is not practical for me to submit separate
patches since all the issues are interlinked. Hope that that is
acceptable to you.

Here are the expected fallbacks this patch will do. This means that if
one search command fails, this patch will use the next command as per
the following list:

Case 1)
    no fetchall
    keep
    IMAP4 server or higher
    2000 mails in mailbox

IMAP> SEARCH 1:1000 UNSEEN UNDELETED
IMAP> SEARCH 1:1000 UNSEEN                   # fallback 1
IMAP> SEARCH UNSEEN                          # fallback 2
IMAP> FETCH 1:2000 FLAGS                     # fallback 3

Case 2)
    no fetchall
    keep
    IMAP4 server or higher
    300 mails in mailbox

IMAP> SEARCH UNSEEN UNDELETED
IMAP> SEARCH UNSEEN                          # fallback 1
IMAP> FETCH 1:300 FLAGS                      # fallback 2

Case 3)
    no fetchall
    no keep or IMAP2 server or both
    4000 mails in mailbox

IMAP> SEARCH 1:1000 UNSEEN
IMAP> SEARCH UNSEEN                          # fallback 1
IMAP> FETCH 1:4000 FLAGS                     # fallback 2

Case 4)
    no fetchall
    no keep or IMAP2 server or both
    500 mails in mailbox

IMAP> SEARCH UNSEEN
IMAP> FETCH 1:500 FLAGS                      # fallback 1

Also, is there an IMAP2 server available which can be used for testing
this patch?

-- 
Sunil Shetye.