I am trying to export calendars and tasks from Exchange, among other data, using a "proxy-user" (Full mailbox permission). While calendars mostly come out okay (there is one account with so much data that export usually fails for many attempts in a row), there is nearly no task data.
The data is there in OWA for many of the users, there are even recent tasks (within the defaul 90-day limit, though I've expanded that in DAVmail config to many years), but the only one I've seen exported by DAVmail (VTODO tags present) was on my test account which is the proxy-user to access other accounts.
Logs are not very helpful, or I don't know what to look for. Can anyone advise how to troubleshoot - where is the problem (config, davmail code, anything else)?
Just in case it helps - "bump!" :)
The problem is, again, with the impersonating user not seeing Tasks for anyone but his own account via DavMail. This data is however accessible in the interactive OWA interface (so likely not a permissions problem), and the impersonating user sees Calendar and Messages for other users (also confirms that permissions for those users work, as well as that DavMail works with this server and with impersonation).
I would be grateful even for pointers where to dig (and a lot more grateful for a made solution); I am not a Java developer though I can try to fix something apparent in the code - if I can find it ;)
Ideas welcome :)
Partial progress: while digging in the code, I found the setting davmail.caldavEnableLegacyTasks=true which can according to comment "return tasks created in calendar folder". Going from here, I found this in the release log (for 3.9.7):
- Caldav: Create a new davmail.caldavEnableLegacyTasks to allow access to tasks created in calendar folder by previous DavMail versions
- Caldav: drop davmail.caldavDisableTasks setting, retrieve only events from calendar
Quite possibly, what I want is the reverse of the second bullet-point :) Gonna try older releases now.
Enabling the new setting did export a few VTODO's when accessing users' "/Tasks/" resource via DavMail (with another proxy-auth user); however, these were some strange entries that I couldn't locate in original OWA interface. Possibly (by context), this was a task assigned to people by their manager - not their own task in their own mailbox/container... it all gets weirder and weirder ;)
Also, it seems that at least part of the problem may be in the server's/user's locale - the "Tasks" folder is not named in English, but in Russian for most of the users and has to be requested as such (/%D0%9A%D0%BE%D0%BD%D1%82%D0%B0%D0%BA%D1%82%D1%8B and so on). This does not seem to cause problems with Contacts and Calendar, but maybe if (I couldn't prove by practice or code, so just a guess) - if Tasks are recognized by fixed string-matching and anything else is a Calendar - that may be the problem?..
Also, DAV searches in MSExchange server are done by DAVmail with "binary" unicode paths, I am not sure this is valid as well (maybe should be percent-string as above?) and result in "0 found item(s)", according to the DavMail log:
... FROM SCOPE('Shallow TRAVERSAL OF "/firstname.lastname@example.org/????????"') WHERE
However, changing a user's interface language to English and restarting DavMail did not help, seemingly - while interface labels for folders did become English, objects are still russian-named and only accessible with the same concerns as outlined above.
So... there is still the quest to solve...
And when I found that flag - I felt so happy that it's over! ;) It isn't :(
Log shows that the "legacy" tasks were indeed extracted using the "binary-UTF8" name of the Tasks folder, and one task was found that way; so it is not an illegal/invalid access path. I also found it in OWA interface... somewhat.
When I click on the Tasks tab for this user, I only see one (another) task which is marked started and overdue. When I click the "Tasks" folder among the list of email folders, I get into the same-looking interface, but with two tasks now - that started one, and the one which DavMail finds (unstarted, no due date).
Ah, on the left I now see a selection between "Flagged items and tasks" and "Tasks" (folder name in russian, as it is in the mail-folder list). "Tasks" shows both items, and "Flagged" shows the item which is not exported by DavMail.
This still does not explain a bit to me, but maybe people more acquainted with Exchange might get some ideas on what is wrong? ;)
Found a toggle to rename "system folders" along with the language... making them into English did not change anything (legacy enabled or disabled works the same as with russian), even though object accesses are now to "/Tasks" instead of binary-UTF and "percented" Russian-named paths.
However, got one more correlation: the tasks which are exported correctly are not marked as "Private", while tasks which are not exported do have this mark set in the couple of users I've reviewed so far. I wonder if this is related to checks for "DAV:ishidden" = false" in the DavMail lookups?..
Yup, it was verified to influence things, at least - unmarking "Private" on a few tasks got them into the export stream, IFF the legacy setting was true (if false - no export either).
The "Flagged" vs. "Tasks" lists of items do not seem relevant ("Flagged" includes mails from assorted folders as well as tasks, at least those which have dates set).
Alas, this check ("DAV:ishidden = false") is also not it. I've built a version of DavMail which removes this clause from Events lookups, but it did not begin to automagically see Tasks marked as Private (which become exportable as soon as unmarked). Instead, it now "finds" and exports an (exclusive or additional) empty VTODO with only an UID in it.
Now I'm stumped... and should go to sleep not victorious once again ;(
Older version (3.9.6) did not help.
One more idea (supporting an earlier guess): the DavMail log has entries like:
DEBUG [CaldavConnection-47065] davmail.exchange.ExchangeSession - Inbox URL: https://email@example.com/Inbox Trash URL: https://firstname.lastname@example.org/Deleted Items Sent URL: https://email@example.com/Sent Items Send URL: https://firstname.lastname@example.org/##DavMailSubmissionURI##/ Drafts URL: https://email@example.com/Drafts Calendar URL: https://firstname.lastname@example.org/Calendar Tasks URL: https://email@example.com/Tasks Contacts URL: https://firstname.lastname@example.org/Contacts Outbox URL: https://email@example.com/Outbox Public folder URL: https://mail.domain.ru/public/
These lines are built with variables like "tasksUrl", and are valid for the proxyauth user (adminauth above) who has an english interface - but wrong for a Russian-interfaced user.
However, just like I checked above by changing a Russian user into English interface, now I changed the adminauth user into Russian interface - this did not change returned values from queries by a bit (log entries with URLs equivalent to the ones listed here did change, into binary-UTF text, as somewhat expected now).
So, as promising as this idea looked (to be The Reason) - it seems a dead-end. One down... infinity to go...
Remember I earlier said that this adminauth user was the only one who had any VTODO's in his exported data?
I tested now - the "davmail.caldavEnableLegacyTasks" flag toggles whether his tasks would be seen in /Tasks or as part of /Calendar (leaving /Tasks empty), and the Private checkmark has no influence at all - he sees his tasks, this mark is not even part of the resulting ICS markup.
So there is some double-standard somewhere... :\
Also tried EWS instead of OWA access (and while at it - also varied access to MS Exchange directly or via reverse-proxy for easier sniffing; EWS doesn't like that, OWA shows no difference) - overall results are the same.
However, noted an entry in the logs (which was there all the time in history since I enabled debugging):
"DEBUG ... davmail.exchange.ExchangeSession - Shared or public calendar: exclude private events"
That... rings a bell against yesterday's findings :)
Will go dig some code...
Aaaaand, it seems we have a winner! ;)
Grepping this logged message I found that there is another undocumented option, davmail.excludePrivateEvents (true by default); though this one is hidden on purpose:
Reverting it to "false" and enabling back OWA (even over an HTTP reverse-proxy as my sniffing/logging man-in-the-middle for this conversion), with (tested required) these options in place:
... I did extract all the expected amount of Tasks entries by requests to the /Tasks URL on DavMail.
If only someone ""in the know" had read this thread sooner and told me about such flags! Eh, since it was the log file that led me to one good discovery after another, I can't even complain "why didn't anyone tell me!?" ;)
A few nits that remain in this subject are:
1) Documentation. Especially for migration scenarios like mine, but also perhaps for a secretary/assistant taking care of another person's schedule, access even to "private" tasks of another user may be required. You don't have to add it to GUI, but at least tell about these options!
2) With both access to own or another account's private tasks, there is no ICS tag apparent to mark the task as private. In the target messaging product for migration there is a drop-list "Task is: Public/Private/Only show date and time" - so I believe it is a pretty common attribute, not an MS extension. In fact, in RFC 5545 I see examples with "CLASS:PRIVATE", so I think this flag should be exposed in the markup.
I'll try to do that and see where it takes me :)
And one more nit, which I am not going to fix but leave for the project to consider:
3) Rename davmail.excludePrivateEvents into davmail.caldavExcludePrivateEvents for clarity. Maybe support legacy name in a quiet manner.
Ok, I thought I was smarter than you - wrong again :)
The code, at least for OWA, does include CLASS attribute based on values of "sensitivity" - IF that is returned by the server, I assume. EWS code seems different, I can only guess that it maps the server's returned markip somehow else.
In my fetched test entries I see some CLASS values - but only for /Calendar queries, none for /Tasks.
The query code which was disabled for shared folders by that flag by default adds a clause for "sensitivity==0" so as to find only public entries; however in the requests (log) it translates into a more cryptic 'And "http://schemas.microsoft.com/mapi/proptag/x00360003" = 0'
I wonder if there's some simple fix to get values of this property for an entry and translate it back into "sensitivity"? Or would I have to request all possible values and fill in the CLASS value accordingly?..
And wrong again, I was smart enough after all ;p
The original code (for OWA) did include the logic, but only for VEVENTs explicitly. I copy-pasted the lines to VTODOs and added logging (debug) to be sure; now it works the way I want it! :)
(Possibly, it would be cleaner to not duplicate the code, but move those lines up to DESCRIPTION and other common attributes, before the "if" which splits to VEVENT or VTODO or possibly other entry types - up to you...)
Patch against release 4.3.2 (r2138) attached.