#126 Add an "O365SemiInteractive"(?) mode

[Dave Dykstra]

The O365Interactive mode is working for me with my company's MFA on my Mac laptop. However, I would like to avoid having to fill in username/password each time. I would like to use those from the client (which has an encrypted storage for the password already), and prompt for MFA after that when it is needed (the IdP only asks for it when not on the company LAN). I tried O365Modern, but it is always denied, I'm pretty sure because the IdP (from Ping Federate) is not set up to have any integration with a phone app. What do you think about the possibility of a hybrid mode, called something like "O365SemiInteractive", which takes the username & password from the client and then does the rest like O365Interactive?

[Dave Dykstra]

I played with a tool a few years ago https://github.com/UWIT-IAM/webisoget that was able to immediately respond to an SSO's username & password prompt so I don't think that part is very difficult.

[Mickael Guessant]

DavMail already sends client provided username in login_hint, O365Modern is the best choice for non interactive authentication, however as you found out it does not cover all use cases with external IdPs.

If you are looking for less interaction, you should probably try to enable davmail.oauth.persistToken=true
=> this will store refresh token to avoid frequent reauthentications.

Note that this setting is enabled by default in latest trunk build.

[Dave Dykstra]

persistToken sounds like it will help, but I am talking about every time DavMail starts, like after reboots. I would like to only have to enter the MFA response and not the usename and password again. Thunderbird stores them in an encrypted storage, and DavMail doesn't so I'd like to use what comes from the client, yet still use an interactive popup for the MFA response. Hence my idea for a semi-interactive mode.

[Mickael Guessant]

Please try it, persistent tokens survive reboots :-)

[Dave Dykstra]

That does sound promising. If I only had to enter my username and password very rarely, that might not be so bad. It depends on how long the refresh token lasts.

So I installed the Mac trunk build. Unfortunately after the first connection & stop & start of DavMail I'm getting an authorization failure (with no popup). The log file has a stack trace in it:

2019-12-15 20:43:12,861 DEBUG [main] davmail.DavGateway - Start DavMail in GUI mode
2019-12-15 20:43:12,878 DEBUG [main] davmail - OS Name: Mac OS X Java version: 1.8.0_231 64 System tray supported
2019-12-15 20:43:12,878 INFO [main] davmail - O365Interactive is not compatible with SWT, do not try to create SWT tray
2019-12-15 20:43:12,985 INFO [main] davmail - DavMail Gateway 5.4.0-trunk listening on SMTP port 1025 POP port 1110 IMAP port 1143 CALDAV port 1080 LDAP port 1389
2019-12-15 20:43:13,365 DEBUG [CheckRelease] davmail.DavGateway - DavMail released version: 5.4.0-3135
2019-12-15 20:43:18,510 DEBUG [davmail.imap.ImapServer] davmail - Connection from /0:0:0:0:0:0:0:1 on port 1143
2019-12-15 20:43:18,525 INFO [davmail.imap.ImapServer] davmail.connection - CONNECT - 0:0:0:0:0:0:0:1:52648
2019-12-15 20:43:19,081 INFO [ImapConnection-52648] org.apache.commons.httpclient.auth.AuthChallengeProcessor - basic authentication scheme selected
2019-12-15 20:43:19,084 INFO [ImapConnection-52648] org.apache.commons.httpclient.HttpMethodDirector - No credentials available for BASIC ''@outlook.office365.com:443
2019-12-15 20:43:19,089 DEBUG [ImapConnection-52648] davmail.exchange.ExchangeSession - Test configuration status: 401
2019-12-15 20:43:19,090 DEBUG [ImapConnection-52648] davmail - > * OK [CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE SPECIAL-USE] IMAP4rev1 DavMail 5.4.0-trunk server ready
2019-12-15 20:43:19,091 DEBUG [ImapConnection-52648] davmail - < A0001 CAPABILITY
2019-12-15 20:43:19,094 DEBUG [ImapConnection-52648] davmail - > * CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE SPECIAL-USE
2019-12-15 20:43:19,096 DEBUG [ImapConnection-52648] davmail - > A0001 OK CAPABILITY completed
2019-12-15 20:43:19,097 DEBUG [ImapConnection-52648] davmail - < LOGIN **
2019-12-15 20:43:19,151 DEBUG [ImapConnection-52648] davmail.exchange.auth.O365Token - Loaded stored token for dwd@fermitest.fnal.gov
2019-12-15 20:43:19,178 DEBUG [ImapConnection-52648] davmail.http.DavGatewaySSLSocketFactory - createSocket login.microsoftonline.com 443
2019-12-15 20:43:19,402 DEBUG [ImapConnection-52648] davmail.exchange.auth.O365Token - Access token expires Sun Dec 15 21:43:19 CST 2019
2019-12-15 20:43:19,404 WARN [ImapConnection-52648] davmail.exchange.ExchangeSession - All network interfaces down or host unreachable !
2019-12-15 20:43:19,404 DEBUG [ImapConnection-52648] davmail.exchange.ExchangeSession - java.lang.StringIndexOutOfBoundsException: String index out of range: -1
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1960)
at davmail.exchange.auth.O365Token.setJsonToken(O365Token.java:95)
at davmail.exchange.auth.O365Token.executeRequest(O365Token.java:160)
at davmail.exchange.auth.O365Token.refreshToken(O365Token.java:153)
at davmail.exchange.auth.O365Token.load(O365Token.java:189)
at davmail.exchange.auth.O365InteractiveAuthenticator.authenticate(O365InteractiveAuthenticator.java:90)
at davmail.exchange.ExchangeSessionFactory.getInstance(ExchangeSessionFactory.java:181)
at davmail.exchange.ExchangeSessionFactory.getInstance(ExchangeSessionFactory.java:97)
at davmail.imap.ImapConnection.run(ImapConnection.java:113)
2019-12-15 20:43:19,406 INFO [ImapConnection-52648] davmail.connection - FAILED - 0:0:0:0:0:0:0:1:52648 dwd@fermitest.fnal.gov
2019-12-15 20:43:19,407 ERROR [ImapConnection-52648] davmail - All network interfaces down or host unreachable !
2019-12-15 20:43:19,407 DEBUG [ImapConnection-52648] davmail - > A0002 NO LOGIN failed
2019-12-15 20:43:19,408 DEBUG [ImapConnection-52648] davmail - < A0003 LOGOUT
2019-12-15 20:43:19,408 DEBUG [ImapConnection-52648] davmail - > * BYE Closing connection
2019-12-15 20:43:19,408 DEBUG [ImapConnection-52648] davmail - > A0003 OK LOGOUT completed
2019-12-15 20:43:19,408 INFO [ImapConnection-52648] davmail.connection - DISCONNECT - 0:0:0:0:0:0:0:1:52648

Although it reports network interfaces down, the network is not down.

[Mickael Guessant]

Sorry this was a regression in trunk builds, should be fixed now

[Dave Dykstra]

I updated to the latest trunk a couple of days ago and this morning it crashed. There wasn't a stacktrace in the log this time, but instead there was an apple crash report. I will attach it to this message.

[Dave Dykstra]

And another one that looks pretty similar.

[Mickael Guessant]

Definitely a JDK issue, see:
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8133783

=> we could try to switch to latest OpenJDK build, this may help.

[Dave Dykstra]

I updated from "Java 8 build 231" to "Java 8 build 241" according to these instructions https://www.java.com/en/download/help/mac_java_update.xml and restarted DavMail. Is that the same thing? I googled for openjdk mac and it says it is from Oracle but I'm not sure if the one I updated is OpenJDK.

[Mickael Guessant]

Java on OSX used to be simple: only one binary provided by Apple.

Now we have the choice between Oracle JDK, Amazon Corretto, Azul...

Not sure which on is the best / most robust.
On windows I tried Azul to validate OpenJFX behavior:
https://www.azul.com/downloads/zulu-community/?&os=&os=macos&architecture=x86-64-bit&package=jre-fx

[Dave Dykstra]

Ok I installed Zulu 11 in /Library/Java/JavaVirtualMachines/zulu11.35.15-ca-fx-jre11.0.5-macosx_x64/. What's a good way to get DavMail to use it without changing the system default? I looked through Contents/MacOS/davmail and didn't come to a good conclusion. It seems like setting JAVA_HOME might be easiest but I'm not sure where to set that without editing the script itself. I tried adding it with PlistBuddy but it doesn't seem to have taken effect.

[Mickael Guessant]

Java MacOS package is based on UniversalJavaApplicationStub, you should be able to embed jre inside the application, see:
https://github.com/tofi86/universalJavaApplicationStub#bundle-a-jrejdk-with-your-app

Never tested it myself though.

[Dave Dykstra]

Thanks that was helpful. I found I also had to run /System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -v -f /Applications/DavMail.app and it has to be under my own user id, not root.

I do worry however about getting security updates. It does not seem like a reasonably good long term general solution. I looked for any means of getting automatic updates of Zulu and didn't find one, although I did at least find some Mac native installers at https://www.azul.com/downloads/zulu-community/?&os=&os=macos&architecture=x86-64-bit&package=jdk and have also installed zulu-11.jdk and zulu-13.jdk. When I tried zulu-11.jdk DavMail gave me a different kind of authentication prompt, asking me to click on an office 365 login link and past back the resulting URL (I have selected O365Interactive Exchange Protocol). I then switched to zulu-13.jdk and it didn't prompt at all so I guess the refresh token was preserved.

[Mickael Guessant]

Thanks for your feedback. I can answer on the difference between Zulu jdk and Zulu FX jdk: the second one includes the OpenJFX implementation required for embedded browser support. With non FX jdks you get the failover dialog... which is just fine know that we are able to persist Oauth refresh token for a long time.

[Dave Dykstra]

I wrote the previous comment a couple of weeks ago and just now noticed that I had apparently never hit the Post button, I am sorry about that.

Today I am trying more deliberately to test the davmail.oauth.persistToken=true feature, and I do not see it working. If I stop Davmail and restart it, each time it prompts for username and password. At first I did not have the option set at all since you said it was default in the trunk, but I explicitly added it to .davmail.properties just now and it didn't help. My last update from trunk was a month ago.

My company's MFA policy is scheduled to be set in a couple of months to add the MFA prompt but only when in the office (I'm an early adopter for testing), so I'd really like to avoid those username/ password prompts. I would only need to fill in anything (the MFA prompt) when the laptop is away from the office.

[Mickael Guessant]

Ok, so you have conditional MFA enabled on your Office 365 tenant.

In order to understand what happens, you need to check davmail.log, please also have a look at .davmail.properties after a successfull authentication: you should find a davmail.oauth.xxxx line with an encrypted refresh token.

Then stop DavMail and restart it, try to connect to your mailbox
=> you should get messages related to oauth token load

[Dave Dykstra]

The davmail.oauth.xxxx line is there. However the log shows this message every time it tries to use it:

2020-02-23 22:33:22,218 WARN [ImapConnection-52835] davmail.exchange.auth.O365Token - refresh token failed javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

I tried removing the davmail.oauth.xxxx line and re-initializing it, but keep having the same error. Would it help for me to show you more of the log or contents of .davmail.properties?

[Dave Dykstra]

Where does the decryption key come from?

[Dave Dykstra]

Ok I think I found that it is coming from the smtp password. The password has to match the real password in order for the refresh key to work. Everything else worked with a random password so I didn't know that was needed.

[Mickael Guessant]

Sorry I didn't answer faster: yes the key is derived from client provided password. Thus a rogue client will not be able to benefit from cached Oauth token.

[Peter Krútel]

When reading this topic, how is planned to support MFA with Microsoft Authenticator? It can be some combination of O365Modern (username+password stored in mail client) with O365Interactive - popup information window "Please allow login" (with link to other backup MFA possibilites) which automatically disappers when login is allowed.