Thread: [Davmail-users] gnus/davmail/O365: works, but is slow
Brought to you by:
mguessan
Menu
▾
▴
davmail-users
|
From: Henrik G. B. <hg...@if...> - 2019-01-28 15:05:16
|
Hi! My company recently decided to switch to O365, and to turn off imap (apparently sending credentials to MS is a no-go, but giving all our emails to them is fine), so I need to Do Something to continue to use gnus. I've set up davmail with this davmail.properties file: davmail.server=true davmail.mode=O365Interactive davmail.url=https://outlook.office365.com/EWS/Exchange.asmx davmail.oauth.clientId=<company-specific appid> davmail.oauth.redirectUri=https://login.microsoftonline.com/common/oauth2/nativeclient davmail.caldavPort=1080 davmail.imapPort=1143 davmail.popPort= davmail.ldapPort= davmail.allowRemote=false davmail.disableUpdateCheck=true # Send keepalive character during large folder and messages download davmail.enableKeepalive=false # Message count limit on folder retrieval davmail.folderSizeLimit=0 # Delete messages immediately on IMAP STORE \Deleted flag davmail.imapAutoExpunge=true # Enable IDLE support, set polling delay in minutes davmail.imapIdleDelay= # Always reply to IMAP RFC822.SIZE requests with Exchange approximate message size for performance reasons davmail.imapAlwaysApproxMsgSize= # log file path, leave empty for default path davmail.logFilePath=/opt/davmail/davmail.log # maximum log file size, use Log4J syntax, set to 0 to use an external rotation mechanism, e.g. logrotate davmail.logFileSize=1MB # log levels log4j.logger.davmail=WARN log4j.logger.httpclient.wire=WARN log4j.logger.org.apache.commons.httpclient=WARN log4j.rootLogger=WARN This works, I just swapped out the hostname with localhost and the port with 1143 (and disabled ssl) in my .gnus, and BANG! got email. Excellent! There is one snag, though. Every time I want to read a new email (or enter the INBOX *Summary* buffer), I see this in the terminal from which I started davmail: 2019-01-28 15:07:45,128 DEBUG [ImapConnection-59468] davmail - < 485 SELECT "INBOX" 2019-01-28 15:07:45,591 DEBUG [ImapConnection-59468] davmail.imap.ImapConnection - * 2019-01-28 15:07:45,989 DEBUG [ImapConnection-59468-LoadFolder] davmail - Downloaded 128 KBytes from /ews/exchange.asmx 2019-01-28 15:07:46,028 DEBUG [ImapConnection-59468-LoadFolder] davmail - Downloaded 256 KBytes from /ews/exchange.asmx 2019-01-28 15:07:46,068 DEBUG [ImapConnection-59468-LoadFolder] davmail - Downloaded 384 KBytes from /ews/exchange.asmx 2019-01-28 15:07:46,106 DEBUG [ImapConnection-59468-LoadFolder] davmail - Downloaded 512 KBytes from /ews/exchange.asmx 2019-01-28 15:07:46,130 DEBUG [ImapConnection-59468-LoadFolder] davmail.exchange.ExchangeSession - Folder INBOX - Search items current count: 500 fetchCount: 500 highest uid: 510 lowest uid: 1 2019-01-28 15:07:46,490 DEBUG [ImapConnection-59468-LoadFolder] davmail - Downloaded 128 KBytes from /ews/exchange.asmx 2019-01-28 15:07:46,532 DEBUG [ImapConnection-59468-LoadFolder] davmail - Downloaded 256 KBytes from /ews/exchange.asmx 2019-01-28 15:07:46,572 DEBUG [ImapConnection-59468-LoadFolder] davmail - Downloaded 384 KBytes from /ews/exchange.asmx 2019-01-28 15:07:46,612 DEBUG [ImapConnection-59468-LoadFolder] davmail - Downloaded 512 KBytes from /ews/exchange.asmx 2019-01-28 15:07:46,636 DEBUG [ImapConnection-59468-LoadFolder] davmail.exchange.ExchangeSession - Folder INBOX - Search items current count: 1000 fetchCount: 500 highest uid: 1030 lowest uid: 1 Then it goes on like this for a while (a few minutes, I believe), then: 2019-01-28 15:13:14,946 DEBUG [ImapConnection-59468-LoadFolder] davmail.exchange.ExchangeSession - Message IMAP uid: 10722901 uid: AAAAAGufGX2HSZRLqcnMSzPDCFwBACoE1msNNg5PnuYq14Wc9dAAAk6o2KsAAA== ItemId: AAMkAGIwZmIxZWU3LWU3YjQtNDk5Ni04YWZmLTU1YzJmZDgzYjdhNABGAAAAAABrnxl9h0mUS6nJzEszwwhcBwDjgRJS8nUeQJK5dWPzxNz1AAAAHaM0AAAqBNZrDTYOT57mKteFnPXQAAJOqNirAAA= ChangeKey: CQAAABYAAACoMZGAI1dkS6FNVpDZ5oLxAAAC28tw 2019-01-28 15:13:14,946 DEBUG [ImapConnection-59468-LoadFolder] davmail.exchange.ExchangeSession - Message IMAP uid: 10722902 uid: AAAAAGufGX2HSZRLqcnMSzPDCFwBACoE1msNNg5PnuYq14Wc9dAAAk6o2KoAAA== ItemId: AAMkAGIwZmIxZWU3LWU3YjQtNDk5Ni04YWZmLTU1YzJmZDgzYjdhNABGAAAAAABrnxl9h0mUS6nJzEszwwhcBwDjgRJS8nUeQJK5dWPzxNz1AAAAHaM0AAAqBNZrDTYOT57mKteFnPXQAAJOqNiqAAA= ChangeKey: CQAAABYAAACoMZGAI1dkS6FNVpDZ5oLxAAAC28ty 2019-01-28 15:13:14,946 DEBUG [ImapConnection-59468-LoadFolder] davmail.exchange.ExchangeSession - Message IMAP uid: 10722903 uid: AAAAAGufGX2HSZRLqcnMSzPDCFwBACoE1msNNg5PnuYq14Wc9dAAAk6o2KkAAA== ItemId: AAMkAGIwZmIxZWU3LWU3YjQtNDk5Ni04YWZmLTU1YzJmZDgzYjdhNABGAAAAAABrnxl9h0mUS6nJzEszwwhcBwDjgRJS8nUeQJK5dWPzxNz1AAAAHaM0AAAqBNZrDTYOT57mKteFnPXQAAJOqNipAAA= ChangeKey: CQAAABYAAACoMZGAI1dkS6FNVpDZ5oLxAAAC28t0 A whole bunch of these (thousands), then 2019-01-28 15:13:15,059 DEBUG [ImapConnection-59468] davmail.exchange.FolderLoadThread - Still loading INBOX (127147 messages) 2019-01-28 15:13:15,060 DEBUG [ImapConnection-59468] davmail - > 127147 EXISTS 2019-01-28 15:13:15,060 DEBUG [ImapConnection-59468] davmail - > * 46 RECENT 2019-01-28 15:13:15,060 DEBUG [ImapConnection-59468] davmail - > * OK [UIDVALIDITY 1] 2019-01-28 15:13:15,060 DEBUG [ImapConnection-59468] davmail - > * OK [UIDNEXT 10723287] 2019-01-28 15:13:15,060 DEBUG [ImapConnection-59468] davmail - > * FLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded Junk) 2019-01-28 15:13:15,060 DEBUG [ImapConnection-59468] davmail - > * OK [PERMANENTFLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded Junk \*)] 2019-01-28 15:13:15,060 DEBUG [ImapConnection-59468] davmail - > 490 OK [READ-WRITE] SELECT completed 2019-01-28 15:13:15,101 DEBUG [ImapConnection-59468] davmail - < 491 UID FETCH 10719121 BODY.PEEK[] 2019-01-28 15:13:15,403 DEBUG [ImapConnection-59468] davmail.imap.ImapConnection - * 123228 FETCH (UID 10719121 2019-01-28 15:13:15,767 DEBUG [ImapConnection-59468] davmail.exchange.ExchangeSession - Downloaded full message content for IMAP UID 10719121 (22936 bytes) 2019-01-28 15:13:15,767 DEBUG [ImapConnection-59468] davmail - > BODY[] {22936} 2019-01-28 15:13:15,767 DEBUG [ImapConnection-59468] davmail - > ) 2019-01-28 15:13:15,767 DEBUG [ImapConnection-59468] davmail - > 491 OK UID FETCH completed The last part seems to be the one that actually fetches my email, and it's fast and nice (yes, it's a big INBOX). Any ideas as to how I could go about speeding things up? Oh, btw, this is davmail-5.1.0-2891. -- Henrik Grindal Bakken <hg...@if...> PGP ID: 8D436E52 Fingerprint: 131D 9590 F0CF 47EF 7963 02AF 9236 D25A 8D43 6E52 |
|
From: Henrik G. B. <hg...@if...> - 2019-01-28 16:01:09
|
Henrik Grindal Bakken <hg...@if...> writes: > Hi! My company recently decided to switch to O365, and to turn off imap > (apparently sending credentials to MS is a no-go, but giving all our > emails to them is fine), so I need to Do Something to continue to use > gnus. A quick follow-up. Using telnet to port 1143, I also see that a02 SELECT INBOX * 127154 EXISTS * 18 RECENT * OK [UIDVALIDITY 1] * OK [UIDNEXT 10723304] * FLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded Junk) * OK [PERMANENTFLAGS (\Answered \Deleted \Draft \Flagged \Seen $Forwarded Junk \*)] a02 OK [READ-WRITE] SELECT completed takes... ages. Several minutes. It's a big INBOX, so I guess this isn't so surprising first time, but it never speeds up. -- Henrik Grindal Bakken <hg...@if...> PGP ID: 8D436E52 Fingerprint: 131D 9590 F0CF 47EF 7963 02AF 9236 D25A 8D43 6E52 |
|
From: Mickaël G. <mgu...@fr...> - 2019-01-30 22:14:53
|
Le 28/01/2019 à 17:00, Henrik Grindal Bakken a écrit : > takes... ages. Several minutes. It's a big INBOX, so I guess this > isn't so surprising first time, but it never speeds up. => a simple way to get fast folder is to limit message count with: davmail.folderSizeLimit=100 I would also suggest the good old way: use folders instead of a single large INBOX... I have a 6k messages folder here and it's still usable, a bit slow but usable. Regards, -- Mickael Guessant mailto:mgu...@fr... |
|
From: Henrik G. B. <hg...@if...> - 2019-01-31 09:22:08
|
Mickaël Guessant <mgu...@fr...> writes: > Le 28/01/2019 à 17:00, Henrik Grindal Bakken a écrit : >> takes... ages. Several minutes. It's a big INBOX, so I guess this >> isn't so surprising first time, but it never speeds up. > > => a simple way to get fast folder is to limit message count with: > > davmail.folderSizeLimit=100 I tried that, but it still goes through the same loop, I don't see any difference at all, to be honest. > I would also suggest the good old way: use folders instead of a single > large INBOX... Yeah, I could do that. > I have a 6k messages folder here and it's still usable, a bit slow but > usable. What's surprising to me is that it loops through this every time the client does "SELECT INBOX", which gnus happens to do when I read an email even, so I get the same delay over and over. If it was just for fetching new emails, I could possibly live with it. I'm trying to add mbsync on top of davmail. Will see how that goes. -- Henrik Grindal Bakken <hg...@if...> PGP ID: 8D436E52 Fingerprint: 131D 9590 F0CF 47EF 7963 02AF 9236 D25A 8D43 6E52 |
|
From: Henrik G. B. <hg...@if...> - 2019-01-31 08:58:48
|
Mickaël Guessant <mgu...@fr...> writes: > Le 28/01/2019 à 17:00, Henrik Grindal Bakken a écrit : >> takes... ages. Several minutes. It's a big INBOX, so I guess this >> isn't so surprising first time, but it never speeds up. > > => a simple way to get fast folder is to limit message count with: > > davmail.folderSizeLimit=100 I tried that, but it still goes through the same loop, I don't see any difference at all, to be honest. > I would also suggest the good old way: use folders instead of a single > large INBOX... Yeah, I could do that. > I have a 6k messages folder here and it's still usable, a bit slow but > usable. What's surprising to me is that it loops through this every time the client does "SELECT INBOX", which gnus happens to do when I read an email even, so I get the same delay over and over. If it was just for fetching new emails, I could possibly live with it. I'm trying to add mbsync on top of davmail. Will see how that goes. -- Henrik Grindal Bakken <hg...@if...> PGP ID: 8D436E52 Fingerprint: 131D 9590 F0CF 47EF 7963 02AF 9236 D25A 8D43 6E52 |
|
From: Mickaël G. <mgu...@fr...> - 2019-01-31 21:14:54
|
Le 31/01/2019 à 09:45, Henrik Grindal Bakken a écrit :
> Mickaël Guessant<mgu...@fr...> writes:
>
>> Le 28/01/2019 à 17:00, Henrik Grindal Bakken a écrit :
>>> takes... ages. Several minutes. It's a big INBOX, so I guess this
>>> isn't so surprising first time, but it never speeds up.
>> => a simple way to get fast folder is to limit message count with:
>>
>> davmail.folderSizeLimit=100
> I tried that, but it still goes through the same loop, I don't see any
> difference at all, to be honest.
>
>> I would also suggest the good old way: use folders instead of a single
>> large INBOX...
> Yeah, I could do that.
>
>> I have a 6k messages folder here and it's still usable, a bit slow but
>> usable.
> What's surprising to me is that it loops through this every time the
> client does "SELECT INBOX", which gnus happens to do when I read an
> email even, so I get the same delay over and over. If it was just for
> fetching new emails, I could possibly live with it.
>
> I'm trying to add mbsync on top of davmail. Will see how that goes.
=> I don't know gnus, but your issue may be that it triggers the message
load on fetch:
if ( ( parameters.contains("BODY.PEEK[HEADER.FIELDS (")
// exclude mutt header request && !parameters.contains("X-LABEL") )
|| parameters.equals("RFC822.SIZE RFC822.HEADER FLAGS")// icedove || Settings.getBooleanProperty("davmail.imapAlwaysApproxMsgSize") )
=> If you don't match this filter DavMail has to download all message
bodies to get exact message size.
=> can you please provide more details on IMAP commands ?
--
Mickael Guessant
mailto:mgu...@fr...
|
|
From: Henrik G. B. <hg...@if...> - 2019-02-01 10:27:38
|
Mickaël Guessant <mgu...@fr...> writes:
>> What's surprising to me is that it loops through this every time the
>> client does "SELECT INBOX", which gnus happens to do when I read an
>> email even, so I get the same delay over and over. If it was just for
>> fetching new emails, I could possibly live with it.
>>
>> I'm trying to add mbsync on top of davmail. Will see how that goes.
>
>
> => I don't know gnus, but your issue may be that it triggers the
> message load on fetch:
>
> if ( ( parameters.contains("BODY.PEEK[HEADER.FIELDS (")
> // exclude mutt header request &&
> !parameters.contains("X-LABEL") )
> || parameters.equals("RFC822.SIZE RFC822.HEADER
> FLAGS")// icedove ||
> Settings.getBooleanProperty("davmail.imapAlwaysApproxMsgSize") )
>
> => If you don't match this filter DavMail has to download all message
> bodies to get exact message size.
To get the ~4 minute 500 messages-at-a-time download stuff, all I need
to do (and all both gnus and mbsync does) is 'SELECT "INBOX"'.
But then it (mbsync, now, didn't do that before) starts doing
'FETCH (UID 6 FLAGS (\Seen) BODY[HEADER.FIELDS (MESSAGE-ID)] {55}'
which takes forever, and won't complete anyway, since it always bails
out after getting a 'bogus FETCH' at some point.
--
Henrik Grindal Bakken <hg...@if...>
PGP ID: 8D436E52
Fingerprint: 131D 9590 F0CF 47EF 7963 02AF 9236 D25A 8D43 6E52
|
|
From: Mickaël G. <mgu...@fr...> - 2019-02-01 17:31:22
|
Le 01/02/2019 à 11:27, Henrik Grindal Bakken a écrit :
> To get the ~4 minute 500 messages-at-a-time download stuff, all I need
> to do (and all both gnus and mbsync does) is 'SELECT "INBOX"'.
=> This is indeed slow, here I select a 6k messages folder in less than 30s:
2019-02-01 13:59:41,353 DEBUG [ImapConnection-57397] davmail - < 11 noop
2019-02-01 13:59:41,626 DEBUG [ImapConnection-57397] davmail - noop on XXX
2019-02-01 14:01:07,967 DEBUG [ImapConnection-57397] davmail - > * 6823
EXISTS
2019-02-01 14:01:07,968 DEBUG [ImapConnection-57397] davmail - > * 0 RECENT
2019-02-01 14:01:07,968 DEBUG [ImapConnection-57397] davmail - > 11 OK
noop completed
=> Anyway with folderSizeLimit at 100 you will only load latest 100 messages
=> Example with folderSizeLimit=10:
2019-02-01 18:29:24 DEBUG davmail:97 - < . SELECT INBOX
2019-02-01 18:29:24 DEBUG ExchangeSession:778 - Folder INBOX - Search
items count: 10 maxCount: 10 highest uid: 724 lowest uid: 361
2019-02-01 18:29:24 DEBUG davmail:97 - > * 10 EXISTS
2019-02-01 18:29:24 DEBUG davmail:97 - > * 0 RECENT
2019-02-01 18:29:24 DEBUG davmail:97 - > * OK [UIDVALIDITY 1]
2019-02-01 18:29:24 DEBUG davmail:97 - > * OK [UIDNEXT 739]
2019-02-01 18:29:24 DEBUG davmail:97 - > * FLAGS (\Answered \Deleted
\Draft \Flagged \Seen $Forwarded Junk)
2019-02-01 18:29:24 DEBUG davmail:97 - > * OK [PERMANENTFLAGS (\Answered
\Deleted \Draft \Flagged \Seen $Forwarded Junk \*)]
2019-02-01 18:29:24 DEBUG davmail:97 - > . OK [READ-WRITE] SELECT completed
> But then it (mbsync, now, didn't do that before) starts doing
> 'FETCH (UID 6 FLAGS (\Seen) BODY[HEADER.FIELDS (MESSAGE-ID)] {55}'
=> MESSAGE-ID field is not in basic message info loaded on select
=> This triggers full mime body download to get a single header
> which takes forever, and won't complete anyway, since it always bails
> out after getting a 'bogus FETCH' at some point.
=> See above, try not to fetch headers on initial fetch flags step...
Regards,
--
Mickael Guessant
mailto:mgu...@fr...
|
|
From: Henrik G. B. <hg...@if...> - 2019-02-05 18:16:58
|
Mickaël Guessant <mgu...@fr...> writes: > Le 01/02/2019 à 11:27, Henrik Grindal Bakken a écrit : >> To get the ~4 minute 500 messages-at-a-time download stuff, all I need >> to do (and all both gnus and mbsync does) is 'SELECT "INBOX"'. > > => This is indeed slow, here I select a 6k messages folder in less than 30s: [...] Hi again, sorry for the late reply, I was away over the weekend. I've now set up a system based on davmail+mbsync+dovecot+gnus, which actually works pretty well! I've cut my INBOX down, and I'll cut it down further, which keeps things relatively manageable. Beats the hell out of OWA, that's for sure! Initial sync is dog slow, but that doesn't matter too much. -- Henrik Grindal Bakken <hg...@if...> PGP ID: 8D436E52 Fingerprint: 131D 9590 F0CF 47EF 7963 02AF 9236 D25A 8D43 6E52 |
Thanks for helping keep SourceForge clean.
X