#633 Compatibility with javamail 1.5.6

[Antonio Ospite]

Hi,

I am working on packaging DavMail for inclusion in the official Debian
archives and for that to happen DavMail has to depend on the system java
libraries.

In particular Debian provides javamail 1.5.6 from http://javamail.java.net and
DavMail seems to have some compatibility problems with it, apparently due to
an assumption which seemed to be valid in the past but not anymore. Or it may
be a new javamail bug, I am not sure.

Anyways, to the end user this results in IMAP messages with no headers when
retrieved with a command like "FETCH 18 (RFC822)".

I tracked this down to a difference behavior of MimeMessage() between the
provided mail-1.4.3.jar and the javax.mail.jar from Debian, see the attached
MimeMessageTest.java program, the comments in the code try to describe the
issue and how the test code represents what DavMail does.

I came up with a quick and dirty workaround in DavMail for my particular setup
(IMAP + Sylpheed), but I don't know DavMail (and Java) well enough to develop
a proper solution.

Can you plese help?

Thanks,
Antonio

[Mickael Guessant]

Interesting, did you managed to track what changed in MimeMessage ?

As I understand if the whole idea of SharedByteArrayInputStream is to be able to reuse the inputstream => we should not have to create multiple instances !

[Antonio Ospite]

Hi Mickael,

I took a loot at https://javamail.java.net/docs/CHANGES.txt and https://javamail.java.net/docs/COMPAT.txt but I didn't find anything obvious.

No mention of the issue on the bug tracker either https://kenai.com/bugzilla/buglist.cgi?product=javamail .

I didn't look deeply at the development history in the mercurial repo https://java.net/projects/javamail/sources/mercurial/show because Java is not really my thing.

I could file a bug report and ask the javamail devs, or are you willing to do that yourself?
Could you reproduce the issue using the test program I sent?

Thanks,
Antonio

[Mickael Guessant]

Found the root cause at:
https://kenai.com/bugzilla/show_bug.cgi?id=3539
and
https://java.net/projects/javamail/sources/mercurial/revision/317

Basically now LineInputStream now relies on mark / reset to avoid creating a PushBackInputStream
=> when DavMail tries to reset inputStream it only gets to latest mark, not to stream start.

[Antonio Ospite]

So LineInputStream.readLine() sets a mark (with a read limit of two bytes) but the mark does not always get invalidated?

I see that the invalidation of the mark depends on how the particular InputStream implements the mark() and read() methods, some classes just ignore the readlimit argument (e.g. ByteArrayInputStream), while some other classes respect it and reset the mark (e.g. BufferedInputStream).

However LineInputStream seems to rely on the automatic invalidation logic.

So I'd say that if that mark is used for internal purposes it should be reset when readLine() returns, and now it's not.

If this reasoning is correct, the issue we are seeing is a bug in javamail.

Can you send the bug report of should I?

Thanks,
Antonio

[Antonio Ospite]

Ping.

Mickael if dont' have time I can file the bug report with javamail, however it would be good if you piched in anyways.

Thanks,
Antonio

[Antonio Ospite]

Javamail report filed: https://kenai.com/bugzilla/show_bug.cgi?id=8562

[Antonio Ospite]

The javamail developer does not consider this a bug:
https://kenai.com/bugzilla/show_bug.cgi?id=8562#c3

So DavMail should pass a copy of the Input Stream to MimeMessage.
I'll propose a patch later.

Ciao,
Antonio

[Antonio Ospite]

I am attaching the patch, it does basically what I was doing in my original report, but using the newStream() method as suggested by the javamail dev.

Mickael, could you verify if the same is needed in other places in DavMail?

I also noticed that sometimes you call a variable mimeBody even if it's supposed to contain the full message, not only the "body" part, would you accept a cleanup patch to improve the naming in such cases?

Thanks,
Antonio

[Mickael Guessant]

I took a slightly different approach: I store the byte array content instead of a SharedByteArrayInputStream, which is created as needed.

Note for future reference: SharedByteArrayInputStream means byte array is shared by multiple input streams, not that the SharedByteArrayInputStream instance is shareable.

[Antonio Ospite]

Thanks a lot Mickael, but with the latest code from svn I get this exception:

2016-12-20 13:00:32,220 DEBUG [ImapConnection-59022] davmail.exchange.ExchangeSession  - Downloaded full message content for IMAP UID 232 (8836 bytes)
2016-12-20 13:00:32,221 ERROR [ImapConnection-59022] davmail  - java.lang.NullPointerException
java.lang.NullPointerException
        at java.io.ByteArrayInputStream.<init>(ByteArrayInputStream.java:106)
        at javax.mail.util.SharedByteArrayInputStream.<init>(SharedByteArrayInputStream.java:68)
        at davmail.exchange.ExchangeSession$Message.getRawInputStream(ExchangeSession.java:2073)
        at davmail.imap.ImapConnection$MessageWrapper.getRawInputStream(ImapConnection.java:797)
        at davmail.imap.ImapConnection.handleFetch(ImapConnection.java:879)
        at davmail.imap.ImapConnection.run(ImapConnection.java:308)
2016-12-20 13:00:32,225 DEBUG [ImapConnection-59022] davmail  - > 38 BAD unable to handle request: java.lang.NullPointerException

BTW, after this gets fixed a possible further cleanup could be to avoid using the word "body" in variables referring to the full message. In my mind the "body" is only the part after the headers.

I'd avoid using the word "content" as well because, in MimeMessage, "content" seems to mean the body part, headers excluded.

So maybe the byte array could be called "rawMessage" or something like that, to communicate that it hasn't been parsed as mime yet.

[Mickael Guessant]

Indeed, wrong merge from my unit test code to main class => fixed

Also renamed mimeBody to mimeContent, as the EWS option to get full MIME message is named IncludeMimeContent... at least now it's consistent.

Thanks for your quick feedback.

[Antonio Ospite]

And thank you for the quick fixup Mickael.

It all looks good now, I think the bug report can be closed.

Ciao ciao,
Antonio