mstor / Discussion / Help: Unique id per message when stored

Norman Maurer - 2006-09-04

Hi Ben,

we develop a java mail server named james ( james.apache.org ). We thought about to use mstor to provide a backend for mbox. We need to have a persist unique uid (key) per message we add to retrieve the mail later by using the "unique" id. Any idea how to adapted this ?

Here his the interface we using for such needs:

package org.apache.james.mailrepository.javamail;

import javax.mail.Message;
import javax.mail.MessagingException;
import javax.mail.UIDFolder;

/**
* Interim interface to provide access to UID PLUS methods reflecting RFC 2359,
* until official Javamail API offers this.
*/

public interface UIDPlusFolder extends UIDFolder, FolderInterface {
    /**
     * Appends the given messages to the folder and returns corresponding uids.<br>
     * Implementations may require the folder to be open.
     *
     * @see javax.mail.Folder#appendMessages(javax.mail.Message[])
     *
     * @param msgs
     *            messages to append
     * @return array of same size and sequenze of msgs containing corresponding
     *         uids or -1, if something went wrong
     * @throws MessagingException
     * @throws IllegalStateException when folder has to be open
     */
    public long[] addUIDMessages(Message[] msgs) throws MessagingException;

    /**
     * Appends the given messages to the folder and returns corresponding
     * instances of the appended messages.<br>
     * Implementations may require the folder to be open.
     *
     * @see javax.mail.Folder#appendMessages(javax.mail.Message[])
     *
     * @param msgs
     *            messages to append
     * @return array of same size and sequenze of msgs containing corresponding
     *         added messages or null, if something went wrong
     * @throws MessagingException
     * @throws IllegalStateException when folder has to be open
     */
    public Message[] addMessages(Message[] msgs) throws MessagingException;

}

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Ben Fortuna - 2006-09-05
  
  Hi Norman,
  
  I would assume the best place to store the message UID would be as a custom header in the message. eg:
  
  X-Apache-James-UID: JVBERi0xLjQKMyAwIG9iaiA8PA
  
  Then to retrieve a messages you could use a search term like this:
  
  HeaderTerm uidSearch = new HeaderTerm("X-Apache-James-UID", "JVBERi0xLjQKMyAwIG9iaiA8PA");
  
  // messages should contain only one (1) message..
  Message[] messages = folder.search(uidSearch);
  
  Your implementation of UIDPlusFolder could encapsulate this functionality along with a way to generate the unique UIDs. The benefit of using the standard JavaMail APIs like this is that you wouldn't be tied to any particular mail store provider, so you could choose to use mbox, MailDir, etc. without needing code changes.
  
  Let me know if this is not what you meant.
  
  regards,
  ben
  
  ps. I've used strings for the UID, but longs would also work here.
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Norman Maurer - 2006-09-05
  
  I talked about this with a other james developer. We think the main idea is cool.
  
  But maybe we able todo this with Flags (Not sure yet). Cause we don't want to "modify" the original message.
  
  Any feedback is welcome ;-)
  
  Bye
  Norman
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
  - Ben Fortuna - 2006-09-06
    
    Yup you could also do this with flags.
    
    You might want to look at how I have implemented Tags (like GMail Labels) in mstor, which I store as flags with a common prefix ("tag_").
    
    If a message UID is standard mail server functionality (i.e. is defined in an RFC somewhere) I'd be willing to consider adding support for UID flags to mstor.
    
    regards,
    ben
    
    If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
    - Joachim Draeger - 2006-09-25
      
      Hi Ben,
      
      UIDFolder is a standart JavaMail Interface.
      http://java.sun.com/products/javamail/javadocs/javax/mail/UIDFolder.html
      
      UIDPLUS allows to return the UID of an appended message immediately, which is very useful.
      
      Joachim
      
      If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
      - Ben Fortuna - 2006-09-26
        
        Hi Joachim,
        
        Ok, so do you think it would be sufficient enough if MstorFolder implements UIDFolder? I think that should be ok as the interface is relatively simple.
        
        I could possibly add another attribute (UID) to folder metadata in a similar way the message number attribute is stored, but from the description it appears that the UID doesn't change once it has been assigned (message numbers will be reallocated when deleted messages are expunged).
        
        Let me know what you think.
        
        regards,
        ben
        
        If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
        
        Joachim Draeger - 2006-09-28
        
        Hi ben,
        
        right, the uid doesn't change when messages are expunged. It belongs to the messages like the flags.
        And you have to save the last assigned uid.
        A new uid has do be greater then the last used, no uid is allowed to get assigned twice and so on.
        In one word: uid++. :-)
        
        In addition to the UIDFolder: It is a missing feature in UIDFolder to return the uid of a just appended message. This would be very useful, too.
        
        Joachim
        
        If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
        
        Ben Fortuna - 2006-10-01
        
        Joachim,
        
        I've implemented initial support for UIDFolder in mstor. Message UIDs are allocated upon being appended to a folder (Folder.appendMessages()), and are stored in the metadata for each message (note that the metadata for a folder also stores the last used UID to ensure there is no conflict between UIDs in a folder).
        
        All changes are committed to CVS, and I've also deployed a maven snapshot release for mstor 0.9.10, which you will find here:
        
        http://m2.modularity.net.au/snapshots/net/fortuna/mstor/0.9.10-SNAPSHOT/
        
        If you are also using maven you might like to look at the Wiki for instructions on using the snapshot dependency:
        
        http://wiki.modularity.net.au/mstor/index.php?title=Maven2
        
        Let me know if you see any problems.
        
        regards,
        ben
        
        If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
        
        Joachim Draeger - 2006-10-01
        
        Hi Ben,
        
        Looks good!
        No, I'm not using maven. For testing I wire the dependencies in eclipse by hand and use eclipse to run the tests.
        Together with the last uid you should store the uidvalidity value.
        The uidvalidity is a simple long that is created randomly on mailbox creation. Although JavaMail uses a long here I would limit it to a positive int value. IMAP RFC says it has to be 32 bit, and I don't know whether clients can deal with negative values.
        Why uidvalidity? Imagine the metadata gets lost or the mailbox gets deleted and recreated. Then the uids start again from 1 and the client can notice that by comparing uidvalidity.
        
        Thank you for starting work on UIDs, but note that obviously the cache bug is the worst blocker.
        For inclusion in Apache James we would need either the checksum save message content stability (the additional \r\n bug) or UIDPLUS support.
        
        Joachim
        
        If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
        
        Ben Fortuna - 2006-10-01
        
        Hi Joachim,
        
        Thanks for the feedback. Yeah I obviously hadn't figured out what the UIDValidity was for, so thanks for the info regarding that.
        
        I haven't had a chance to look at the bugs yet, but hopefully now the unit tests are in a bit better shape (although they still require some work) it should be easier to track down the cause of the problems.
        
        regards,
        ben
        
        If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Norman Maurer - 2006-09-06
  
  Hi Ben,
  
  thx for the tip.
  
  The UID is used by the UIDPLUS Method refered by RFC 2359
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
  - Ben Fortuna - 2006-10-08
    
    Guys,
    
    I've now added support for UID validity to mstor. If there is further functionality you require regarding UIDFolder support let me know.
    
    NOTE: The changes are available in the latest snapshot build here:
    
    http://m2.modularity.net.au/snapshots/net/fortuna/mstor/0.9.10-SNAPSHOT/
    
    regards,
    ben
    
    If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Antoni Mylka - 2008-02-02
  
  I'd like to resurrect this discussion.
  
  I'd like to detect if a folder has been changed or not, but can't use listeners. In my project (http://aperture.sourceforge.net) we already have an IMAP crawler. The IMAPFolder class from Sun has a getUIDNext(). It returns the next UID that would be assigned to a new message. We crawl a folder repeatedly and remember the nextUID, the message count and the subfolder list between crawls. This allows us to check if a folder has been changed with a following algorithm
  
  boolean changed = true;
  if (folder.getUIDNext() == oldUidNext) {
     // no messages have been added
     if (folder.messageCount() == oldMessageCount) {
         // no messages have been removed
         if (twoArraysHaveEqualContent(folder.list[], oldFolderList) {
             // the subfolder list is the same
             changed = false;
         }
     }
  }
  
  MstorFolder doesn't have the getUIDNext(). Do we have to store the UID's of each and every message in a folder to achieve the same result, or is there some other way to achieve this?
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
  - Antoni Mylka - 2008-02-03
    
    A followup, this could actually achieve the same if the lastModified timestamp of the underlying file was exposed. I've posted a patch. I'd be grateful for any other ideas.
    
    If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
    - Ben Fortuna - 2008-02-08
      
      I've applied this patch, but I think you may require an additional method in MstorFolder to use it..
      
      Note that you could also call FolderDelegate.getLastUid(), which returns the last UID allocated for delegates that support UID allocation (currently this is only MetaFolder, which is enabled using the META_ENABLED flag).
      
      regards,
      ben
      
      If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Unique id per message when stored

Forums

Help

Unique id per message when stored document.SUBSCRIPTION_OPTIONS = { "thing": "topic", "subscribed": false, "url": "subscribe", "icon": { "css": "fa fa-envelope-o" } };

Unique id per message when stored