#62 Network traffic reduction


Davmail (EWS) is causing some heavy traffic on my network. I could see quite a few duplicated "Downloaded full message content for IMAP UID" messages. At the moment davmail only caches only the last message. I could try to create a patch to do this caching with memcached, if you have interest in merging it.

Also it seems as if Exchange 2003 (and later?) has support for gzip compression (http://support.microsoft.com/kb/830827). Is there a reason why it is only added to DavExchangeSession, but not to EWS?


  • Mickael Guessant

    Caching message content is impractical, you would have to keep all messages in a given folder in cache to avoid the download twice effect, and some users have thousands of messages in their inbox.

    The best way to avoid downloading messages twice would be to find a way to retrieve all requested message information in the initial FETCH without actually downloading message content.

    Basically all IMAP clients use the following sequence:
    - select folder to get message count and highest uid: select "Sent"
    - list messages with flags: UID FETCH 1:* (FLAGS)
    - get message information: UID fetch 4935:4965 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type)])
    - get message content: UID fetch 4965 (UID RFC822.SIZE BODY[])

    The issue is that the only way to get exact RFC822.SIZE is to actually download MIME message content, as Exchange does not use MIME but MAPI to store messages.
    => the get message information request triggers a full message dowload.

    Would need to try to send an approximate message size (MAPI message size) in this request to avoid full message download, but this may break IMAP client implementation.

  • michi

    michi - 2011-06-17


    In my case, the problem is not the "download twice" effect which you mentioned. I have added a small extension so that header fetching does not cause complete email loading (see patch). However this requires changing the client too, so I thought it's not that exciting to other users. But I have seen the mail client evolution avoiding the download twice as well by relying on UIDVALIDITY/UIDNEXT/... fields. Maybe others do so too. This leaves a pretty small active set size. But messages still get downloaded several times. The clients do not do much caching and the caching they do operates on parts, not mails...

    However I think that even with the "download twice" effect the cache will be effective. Maybe even more so, because then the traffic of the message contents is even higher. The active set size cannot be that big either. After all, it must go though your internet connection (except davmail+exchange are on the same LAN - in this case caching will likely be pretty pointless).

    Anyway what about gzip compression?

  • michi

    michi - 2011-06-17
  • Mickael Guessant

    About gzip: you can test it with latest svn revision, I did not implement this until now as this is not enabled on my company Exchange server.

  • Mickael Guessant

    Your patch is definitely not a general purpose patch...

    Not sure I understand your comment on client side caching: if you want to minimize network traffic on the go, you should do the initial sync on a fast connection with your IMAP client set to store all messages locally.

    Thus, when you are using a slow connection, only new messages will pass through the wire (twice but this is the best I can achieve without breaking IMAP).

  • Mickael Guessant

    • status: open --> pending

Log in to post a comment.