mstor

File Release Notes and Changelog

Release Name: 0.9.9

Notes:
=======================
 mstor - Release Notes
=======================

 mstor is a local store provider for JavaMail. mstor builds on the mbox format and
 provides the following benefits:

 * implemented using Java "New I/O" (java.nio.*) to (theoretically) provide better
 performance
 
 * uses the mbox format for storing messages
 
 * incorporates a "metafile" for each mbox folder to store additional metadata for
 messages (and alleviate the need to modify the mbox file)
 
 * provides basic caching configuration to allow control over the performance vs.
 memory usage equation
 
 * better performance for large mailboxes

 Detailed descriptions of changes included in each release may be found
 in the CHANGELOG.


==============
 How to build
==============
 
 If you have downloaded the source distribution, you should be able to package a JAR
 file simply by running ant in the root directory. e.g:
 
   C:\Libs\mstor-0.9.6-src\>ant
 
 If for some reason you would like to override the default build classpath, I would
 suggest creating a "build.properties" file (see the provided sample) in the root directory
 and add overridden properties to this. You can also override properties via Java system
 properties (e.g. -Dproject.classpath="..."). You shouldn't need to modify the "build.xml" at all,
 so if you do find a need let me know and I'll try to rectify this.


===================
 System Properties
===================

 A number of system properties may be specified in order to configure the operation
 of mstor to suit your purpose. These are as follows:
 
  - mstor.meta.enabled (enables mstor-specific metadata to provide full JavaMail
  functionality and performance enhancement. Default value: "true").
  
  - mstor.mbox.encoding (specifies the file encoding used to read the mbox file.
  Default value: "ISO-8859-1").
  
  - mstor.mbox.useNioMapping (specifies whether to use java.nio maps to read
  messages from the mbox file. This may improve performance for reading, however
  due to the implementation of java.nio maps it is not recommended to enable
  maps when you plan to modify the underlying mbox file. Default value: "false").
  
  - mstor.mbox.cacheBuffers (message data from an mbox file is read into a buffer
  prior to returning the underlying data stream. This property specifies whether
  to cache these buffers in memory. This option may improve performance in reading
  messages, but will increase memory usage. Default value: "true").
  
 
==========
 Metadata
==========

 By default mstor provides the ability to extend the standard JavaMail features
 through the use of metadata. This metadata is stored in an XML-based document
 for each folder in the store. Current metadata includes the following:
 
  - received date
  - flags
  - headers (NOTE: although headers are already saved to the underlying mbox
  file, duplicating in metadata allows them to be read without needing to parse
  the entire message content - thus increasing performance..hopefully!)
 
 If you decide not to use mstor's metadata feature, you can disable it by
 specifying the following session property:
 
 	mstor.meta.enabled=false
 	
 e.g.
 
 	Properties p = new Properties();
 	p.setProperty("mstor.meta.enabled", "false");
 	Session session = Session.getDefaultInstance(p);

====================
 Mbox File Encoding
====================

 The mbox format is essentially a concatenation of RFC822 (or RFC2822) messages
 with an additional "From_" line inserted at the start of each message. Instances
 of "From_" within the message body are also escaped with a preceding ">"
 character.
 
 Although mstor doesn't encode/decode message content (it is assumed appended
 messages are valid RFC822 messages), we still need to use an encoding to
 interpret and create the "From_" line.
 
 Because mbox is just RFC822 messages, file encoding should always be "US-ASCII",
 however JavaMail seems to use "ISO-8859-1" encoding and as such mstor will also
 use this encoding as the default.
 
 It is possible however, to override the mbox file encoding used by specifying the
 following system property:
 
     -Dmstor.mbox.encoding=<some_encoding>
 

==================
 OutOfMemoryError
==================

 You may encounter errors when trying to load a large mailbox into memory all
 at once. Here are some pointers to help avoid this problem:
 
 - As mstor messages subclass javax.mail.internet.MimeMessage a "lazy" loading
 scheme is employed. This means that message data is only parsed on request
 (e.g. message.getSubject()) as opposed to upon instantiation.
 	
 Memory problems may still be encountered however, in circumstances where you
 require data from all messages in a folder (e.g. displaying message subject
 line in a table). If using a swing table this may be alleviated somewhat by
 using an appropriate table model implementation to only retrieve message
 data for visible messages.
 
 Mstor helps in this respect as it should only cache a limited number of
 messages in memory (the default is one hundred (100)) allowing other messages
 to be garbage collected (*).
 
 * Side Note: mstor doesn't always conform to the following comment on
 javax.mail.Folder.getMessage() due to caching:
 
 	"Unlike Folder objects, repeated calls to getMessage with the same
 	 message number will return the same Message object, as long as no
 	 messages in this folder have been expunged."

 - Try to avoid using javax.mail.Folder.getMessages() on large folders as you
 will be operating on all available messages - effectively overriding the
 cache mechanism. The method itself is not that memory intensive, but parsing
 message data for all messages in the array may be.

Changes:
=================
 mstor - Changes
=================

0.9.9:

 - Replaced missing javamail.providers file in JAR
 
 - Refactored tags into seperate package
 
 - Added CapabilityHints to centralise flags used to control mstor capabilities
 
 - Cleaned up MboxFile buffer usage
 
 - Added some improvements in memory usage for large files
 
 - Fixed bug with metadata message removal
 
 - Removed redundant MetaDateFormat
 
 - Fixed package build target to include META-INF/javamail.providers in JAR
 
 - Updated commons logging to 1.1
 
 - Added capability hint for additional mozilla (thunderbird, seamonkey)
 compatibility
 

0.9.8:

 - Added option for disabling java.nio direct buffers
 
 - Added support for user flags containing spaces

 - Added Tags support for messages (user flags)
 
 - Added search term persistence
 
 - Various bug fixes
 
 
0.9.7:

 - Fixed bug where message count not updated after appending message to MboxFile
 
 - Added assertions to MStorFolder
 
 - Updated checkstyle configuration and clean up
 
 - Added event broadcasting for opened, closed, created, deleted and renamed
 folders
 
 - Added event broadcasting for messages added or removed from a folder
 
 
0.9.6:

 - Fixed bug introduced in 0.9.5 causing message reads from incorrect postion
 in mbox file.
 
 - Fixed date lines (message headers, metadata) to US-specific locale
 
 - Removed invalid tag from maven project descriptor
 
 - Now throws FolderNotFoundException when opening a folder that doesn't exist
 
 - Fixed appendMessages() retrieval of metadata
 
 
0.9.5:

 - Provided system properties to configure the use of java.nio mapping and caching
 (see README for details).
 
 - Folders automatically closed when underlying store is closed.
 
 - Fixed unit tests to correctly release resources (i.e. close the mbox file)
 
 - File encoding defaults to "ISO-8859-1" again (may be overridden with system
 property).
 
 - Restored compatibility with Java 1.4.
 

0.9.4:

 - Reimplemented aspects of MboxFile to reduce the amount of encoding/decoding
 performed. Now decoding is only used to determine message positions and check
 for existing "From_" lines. Message data is now returned in raw byte format.
 Encoding is now only used for adding "From_" lines to appended messages, with
 message data written in raw bytes.
 
 - Added system property "mstor.mbox.encoding" to specify a non-default file
 encoding.
 
 - Removed functionality to strip "From " escaping when returning messages. It
 seems that this cannot be performed reliably (various references on the Web)
 and until confirmed otherwise I thought it best to disable this feature.
 
 - Improved memory performance in MstorFolder.appendMessages(): messages passed
 to this method are now garbage collected on the fly to allow for very large
 messages to be copied without running out of heap space.


0.9.3:

 - Added validation of mbox files to MStorFolder.list()
 
 - Support for headers in metadata. This has some implications:
 
 	* metadata is only created for existing mailboxes if its
 	contents are changed (i.e. headers, flags, etc.)
 	
 	* if metadata exists, headers are read from metadata in an
 	attempt to delay loading of the entire message
 	
 	* all headers are currently save which means the metadata
 	file can get quite large. Will review this at a later date
 	
 - metadata reworked to use message number as identifier as opposed
 to message id. This means that:
 
    * the message content no longer requires loading to get the
    message id and obtain the metadata (sort of defeated the purpose
    previously..)
 
    * non-MIME messages can now save metadata (MIME messages don't
    have a message id)
    
    * when messages are expunged other messages' metadata must have their
    message numbers updated to stay in synch with the folder
 
 - Added error handling to MboxFile to "fall back" on using a random access
 file where java.nio fails. This seems to occur on Windows 98 consistently.
 

0.9.2_01:

 - Recompiled with JDK 1.4.2 for compatibility with pre-1.5 JVM


0.9.2:

 - Added patches to MboxFile.purge() and MstorFolder.expunge()
 to work on Unix/Linux - provided by Paul Legato
 
 - Added fix to mask/unmask patterns in message content that
 match the "From_" line pattern - provided by Michael G. Kaiser
 
 - From_ line now specifies date message is appended to the
 mbox file (rather than "1 Jan 1970")
 
 - Added support for folders that hold both messages and subfolders
 
 
0.9.1:

 - Improved meta-data implementation
 
 
0.9:

 - Initial Release
 
 - Reading from/appending to mbox files
 
 - Basic message caching