#246 Calendar issues after Exchange migration

[Julius]

I've been using DavMail for a while without any issues. This weekend our IT migrated to a newer Exchange server, with a different domain, different user names, etc. but keeping the mailbox and calendar contents.

I've changed the settings in DavMail to reflect the new infrastructure, but now I receive a lot of errors that a lot of items are missing:
Item <some base64="">.EML not available: /users/myname@company/calendar/

In Outlook everything appears ok. Also with the 'Exchange EWS Provider' addon for Thunderbird, I can receive more items than with DavMail.

What could be going wrong here?

[Old Techie]

This sounds similar to an issue I have. Some appointments from some users don't show up via Davmail. (mine were usually recurring) I think it may have to do with a field in the appointment being null - just guessing based on the debug errors in the log. It's definitely some unhandled condition.

[Julius]

I've got the same issue here (using the latest revision from the repo). It concerns a Exchange 2010 to 2013 migration here. In the log file I see this exception:

2016-01-14 14:45:34,468 DEBUG [CaldavConnection-41350] davmail.caldav.CaldavConnection  - ErrorNonExistentMailbox No mailbox with such guid.
 request: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages"><soap:Header><t:RequestServerVersion Version="Exchange2013"/></soap:Header><soap:Body><m:GetItem><m:ItemShape><t:BaseShape>IdOnly</t:BaseShape><t:AdditionalProperties><t:ExtendedFieldURI PropertyTag="0x37" PropertyType="String"/><t:ExtendedFieldURI PropertyTag="0x3008" PropertyType="SystemTime"/><t:ExtendedFieldURI PropertyTag="0x10f3" PropertyType="String"/><t:ExtendedFieldURI PropertyTag="0x670e" PropertyType="String"/><t:ExtendedFieldURI PropertyTag="0x3001" PropertyType="String"/></t:AdditionalProperties></m:ItemShape><m:ItemIds><t:ItemId Id="AAMkADFjZjEwNDdlLTZkZmUtNDUxOS05YmQ2LTBiY2UyODgzMWIwNwBGAAAAAAAiBAJJJkGES4rgn2DgnjTgBwAfAulv5bbRTpxvOgZOBPaAAAAAy+edAAAfAulv5bbRTpxvOgZOBPaAAAAZeS/SAAA="/></m:ItemIds></m:GetItem></soap:Body></soap:Envelope>
davmail.exchange.ews.EWSException: ErrorNonExistentMailbox No mailbox with such guid.
 request: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages"><soap:Header><t:RequestServerVersion Version="Exchange2013"/></soap:Header><soap:Body><m:GetItem><m:ItemShape><t:BaseShape>IdOnly</t:BaseShape><t:AdditionalProperties><t:ExtendedFieldURI PropertyTag="0x37" PropertyType="String"/><t:ExtendedFieldURI PropertyTag="0x3008" PropertyType="SystemTime"/><t:ExtendedFieldURI PropertyTag="0x10f3" PropertyType="String"/><t:ExtendedFieldURI PropertyTag="0x670e" PropertyType="String"/><t:ExtendedFieldURI PropertyTag="0x3001" PropertyType="String"/></t:AdditionalProperties></m:ItemShape><m:ItemIds><t:ItemId Id="AAMkADFjZjEwNDdlLTZkZmUtNDUxOS05YmQ2LTBiY2UyODgzMWIwNwBGAAAAAAAiBAJJJkGES4rgn2DgnjTgBwAfAulv5bbRTpxvOgZOBPaAAAAAy+edAAAfAulv5bbRTpxvOgZOBPaAAAAZeS/SAAA="/></m:ItemIds></m:GetItem></soap:Body></soap:Envelope>
    at davmail.exchange.ews.EWSMethod.checkSuccess(EWSMethod.java:745)
    at davmail.exchange.ews.EwsExchangeSession.executeMethod(EwsExchangeSession.java:2388)
    at davmail.exchange.ews.EwsExchangeSession.getEwsItem(EwsExchangeSession.java:2027)
    at davmail.exchange.ews.EwsExchangeSession.getItem(EwsExchangeSession.java:2043)
    at davmail.caldav.CaldavConnection.reportItems(CaldavConnection.java:749)
    at davmail.caldav.CaldavConnection.handleFolderOrItem(CaldavConnection.java:282)
    at davmail.caldav.CaldavConnection.handleRequest(CaldavConnection.java:225)
    at davmail.caldav.CaldavConnection.run(CaldavConnection.java:169)

Checking the code I see that in EwsExchangeSession, there's an if-statement with a comment "workaround for missing urlcompname in Exchange 2010". This includes line 2027 which is also in the excepion above.
It's not clear whether this workaround should only apply to Exchange 2010 servers or also to newer servers. Maybe it tries to apply the workaround because the item came originally from an Exchange 2010 server, but it fails to work on the new Exchange 2013 server?

[Mickael Guessant]

urlcompname is the old way to identify folder items, For Exchange created items we now rely on item id that is supposed to never change.

My guess would be that those ids can change during an Exchange server migration => any old item id is now invalid. If this is the case you just have to reset your calendar cache on the client side to get new item ids

[Julius]

Would the client be DavMail itself or the caldav client? I tried multiple caldav clients with DavMail, but the missing events do not show up there either and DavMail returns a 503 to them.

I also notice that the urlcompname variable is filled when I add extra logging there.

[Mickael Guessant]

Ok, next step would be to try to download calendar in a web browser (use calendar path with a trailing slash) and check davmail.log (at level WIRE DEBUG).

The logic should be:
iterate over items in folder
for each item check urlcompname
if urlcompname is available, use it as item name
if not, use itemId as item name

When the caldav client tries to get item content, the input is the url that ends with item name
if item name looks like an itemid we try to get item content by item id
if not we try to get irem by urlcompname

Somehow your Exchange migration broke this logic.

[Julius]

Should I also see the exception balloons pop-up when I do that? Because that did not happen this time.

In the logging I do not see anything special, but it also doesn't log that much for some reason. Even though I have set the log file size to 102400, there are only three log files of around 100 kB.

[Julius]

I did some more analysis. First I looked up the GUIDs of my mailbox in the old environment and in the new environment. Then I decoded the base64 from the item names.

I turns out that the items about which it complains, indeed contain the GUID of my mailbox on the old server. So it seems it's trying to use this old GUID, which may still be stored in the item on the new server, directly and then runs into the "No mailbox with such guid." error.

Would it be possible that the item name needs to be explicitly composed based on the current (new) mailbox GUID?

[Julius]

I just tried it on a completely fresh installation of DavMail + Thunderbird in a VM, just to see if there could be any caching anywhere. I get exactly the same error message then though. Which bit in the Java code does the "iterate over items in folder"?

Maybe I could do something similar through the EWS Editor?

[Mickael Guessant]

If you look at EWS convertid API, it seems that the ItemId maps to a mailbox and a PR_ENTRYD => maybe your events still have the previous ItemId.

You can check whether EWS Editor is able to send GetItem requests on those events, and compare this request with DavMail generated GetItem requests

[Julius]

Both in Evolution and EWS Editor those items show up. I'm not really sure how to obtain the ItemId when looking in EWS Editor. I looked through the values when I just click through and open an item, but I can't find "ItemId" anywhere and none of the values that are listed for an item seem to have base64 which refers to the old GUID.

I'm not really familiar with EWS Editor, so if you could direct me a little bit on which steps I could take to send a GetItem request, that would really help me.

I was looking at this page about the ConvertId API:
https://msdn.microsoft.com/en-us/library/office/bb799665%28v=exchg.150%29.aspx

Here they do write about a specific situation with Exchange 2013 in which only the GUID and a few more bytes should be overwritten to get things to work properly. Maybe this could be related?

[Mickael Guessant]

Just check Tools/Log Viewer

When you open the calendar folder you should see a folder request
<m:FindItem Traversal="Shallow">
response:
<m:FindItemResponseMessage
... with itemids

Then when you select an item you will have:
<m:GetItem>
with <t:ItemId value

=> This will give you the ItemId used by EWSEditor to get item details

[Julius]

Thanks! With your help it was indeed easy to find. I get quite a few EwsResponse entries which contain all the items in my calendar. In the <t:CalendarItem> elements there are <t:ItemId> elements with an attribute id. Also for the pre-migration events, these contain new IDs however.

I can't find a reference to the old IDs anywhere in this output.

[Mickael Guessant]

Ok, we should also check urlcompname. In EWS editor View -> Configure detailed property set.
Add extended property 0x10f3, type String

=> this should contain ics item name for client created items

Do you see anything in pre migration events ?

[Julius]

In some of the events (most seem to miss a urlcompname property) I indeed see the base64 encoding for the old GUID there. The string ends with '.EML'.

[Mickael Guessant]

Well, we eventually found the root cause: the urlcompname is used as item name in initial DavMail response, but as those item names are valid item ids event retrieval fails.

=> quickest workaround would be to remove those urlcompnames from events.

[Julius]

I changed the DavMail source itself now (to not use urlcompname). Most appointments show up now, somehow there are still a few missing.
Is it possible that DavMail itself wrote those urlcompname properties?

[Mickael Guessant]

Well, I need to give a bit more info on urlcompname:

• for events created by Exchange we use ItemId as item name
• for events created by a Caldav client, we need to store user provided item name to retrieve it later => we store item name in urlcompname

Just for the record: over WebDav Exchange used to do exactly this.

Most of the time item names are short and Exchange itemIds are long => it's easy to determine wheter client provided item name is an ItemId or an item name

We do not expect itemIds in urlcompname attribute !

=> you should probably just do a quick batch to clear itemIds from urlcompnames

[Julius]

I see. So the item name is actually generated by my Caldav client, in my case Thunderbird/Lightening? And the check in this case fails, because Thunderbird generated item names that are too similar to item IDs for DavMail?

What is the reason that a user provided name needs to be stored for Caldav client generated items?

Do you think something could be done on the DavMail side to prevent this specific issue?

Thanks!

[Mickael Guessant]

Should work better with latest release, we now always use ItemId as item name (although the initial request with urlcompname should work fine).

[Sandro Santilli]

What version would "latest release" be ? I'm having this problem with 5.4.0: https://sourceforge.net/p/davmail/bugs/728/