mailsync-list — discussion about development, config, bugs etc.

[Mailsync-list] fix to md5 digest patch

[Bob L. <li...@is...>]

I made a slight update to my patch at http://www.isi.edu/~lindell/mailsync/

It now stores the md5 digests in the msinfo messages prefixed with
the '<' character (e.g. "<000806bd44723e283e6f5c79fce98dc0").  I realized
that without the '<' prefix, the parsing code in read_lasttime() was
confused.

Bob

Re: [Mailsync-list] Re: [mailsync - Help] Recursion through subfolders seems to fail.

[Tomas P. <tp...@so...>]

On Mon, 16 Sep 2002, Tim Culver wrote:

> On Mon, 16 Sep 2002, Tomas Pospisek wrote:
>
> > regarding the recursion - what currently does not work with mailsync (at
> > least my experience says so) is having folders that contain a) subfolders
> > _and_ b) messages
>
> This does not surprise me.  Thank you for pointing it out.  It is
> *possible* that c-client does not support folders that contain both
> messages and subfolders.  I don't remember for sure, though.

The FAQ in the latest imap-2002.RC5/docs says this:

1.8 Are hierarchical mailboxes supported?
1.9 Are "dual-use" mailboxes supported?
1.10 Can I have a mailbox that has both messages and sub-mailboxes?

          Yes. However, there is one important caveat.

          Some mailbox formats, including the default which is the
          traditional UNIX mailbox format, are stored as a single file
          containing all the messages. UNIX does not permit a name in the
          filesystem to be both a file and a directory; consequently you
          can not have a sub-mailbox within a mailbox that is in one of
          these formats.

          This is not a limitation of the software; this is a limitation
          of UNIX. For example, there are mailbox formats in which the
          name is a directory and each message is a file within that
          directory; these formats support sub-mailboxes within such
          mailboxes. However, for technical reasons, the "flat file"
          formats are generally preferred since they perform better. Read
          imap-2002/docs/formats.txt for more information on this topic.

          It is always permissible to create a directory that is not a
          mailbox, and have sub-mailboxes under it. The easiest way to
          create a directory is to create a new mailbox inside a
          directory that doesn't already exist. For example, if you
          create "Mail/testbox" on UNIX, the directory "Mail/" will
          automatically be created and then the mailbox "testbox" will be
          created as a sub-mailbox of "Mail/".

          It is also possible to create the name "Mail/" directly. Check
          the documentation for your client software to see how to do
          this with that software.

          Of course, on Windows systems you would use "\" instead of "/".

I have read this twice now and AFAI understands it says: yes we support it
but no we do not support it and btw. actually it's UNIX' fault. Meaybe my
english or just my brain is somewhat impaired and I'll have to reparse
that :-/

> If there is a need, I'm sure mailsync could work around this deficiency in
> c-client.

Maybe you understand the text above better than me and you can figure out
what we need to do from there on...

*t

--
--------------------------------------------------------
     Tomas Pospisek
     sourcepole    -   Linux & Open Source Solutions
     http://sourcepole.com
     Elestastrasse 18,  7310 Bad Ragaz,  Switzerland
     Tel:+41 (81) 330 77 13,  Fax:+41 (81) 330 77 12
--------------------------------------------------------

Re: [Mailsync-list] Re: [mailsync - Help] Recursion through subfolders seems to fail.

[Tim C. <ful...@do...>]

On Mon, 16 Sep 2002, Tomas Pospisek wrote:

> regarding the recursion - what currently does not work with mailsync (at
> least my experience says so) is having folders that contain a) subfolders
> _and_ b) messages

This does not surprise me.  Thank you for pointing it out.  It is
*possible* that c-client does not support folders that contain both
messages and subfolders.  I don't remember for sure, though.

If there is a need, I'm sure mailsync could work around this deficiency in 
c-client.

> PS: Tim, are you now subscribed to the list?

Yes, and I find it much easier than the forum.  I don't think there is a 
mail->forum gateway.  ;-)

Tim

Re: [Mailsync-list] Re: [mailsync - Help] Recursion through subfolders seems to fail.

[Tomas P. <tp...@so...>]

wcattey, could you please confirm that the mailbox deletion problem you
reported to the BTS is now fixed in current CVS?

regarding the recursion - what currently does not work with mailsync (at
least my experience says so) is having folders that contain a) subfolders
_and_ b) messages
*t

PS: Tim, are you now subscribed to the list?

--
--------------------------------------------------------
     Tomas Pospisek
     sourcepole    -   Linux & Open Source Solutions
     http://sourcepole.com
     Elestastrasse 18,  7310 Bad Ragaz,  Switzerland
     Tel:+41 (81) 330 77 13,  Fax:+41 (81) 330 77 12
--------------------------------------------------------

[Mailsync-list] Re: [mailsync - Help] Recursion through subfolders seems to fail.

[Tim C. <ful...@do...>]

On Fri, 13 Sep 2002 no...@so... wrote:

> I have extremely complex archives with subfolders and sub-subfolders.  I've
> as yet been unable to craft a .mailsync file that does the right thing.
> 
> The current rev has the interesting and annoying behavior:
> 
> For each new subfolder, copy messages from remote to local.
> Then copy all  messages seen for all folders up to this point
> from local to remote.
> 
> So:
> 
> Remote:
> 
> Able
>    1 able
>    2  able
>     Baker
>         1 baker
>          2 baker
>    
> Will mutilate Remote to become:
> 
> Able
>    1 able
>    2 able
>     Baker
>         1 baker
>         2 baker
>         1 able
>         2 able
> 
> ----
> 
> Is recursion SUPPOSED to work?  Or must I create separate
> non-recursing store entries for each subfolder?

Recursion is supposed to work.  It has the following limitations, though:

 * It must be supported by your imap server.
 * You do it by ending your pattern with % instead of *.  This is the 
syntax for local files; some imap servers use the same convention, but it 
isn't part of the spec, so you have to find the server docs or experiment. 
 * Config file syntax limitations mean that you have to choose one root 
directory and synchronize all subfolders of that directory (per channel).

I don't understand your able/baker example.  I do notice some interesting 
spacing; if this means you have spaces in your mailbox names, then that 
could be your problem.  More likely, the problem is that mailsync is not 
getting the mailbox names exactly right.  For this, mailsync has a "list 
mode" which you get by specifying a store on the command line (instead of 
a channel).  So you could try:

% mailsync Testing
able
able/baker

% mailsync l-Testing
able
able/baker

If there is any difference at all in these two results, you will get lots
of duplicate messages.

If the only difference is the delimiter character, and this is what's 
causing problems, then I'm very sad because I thought I fixed that bug.  
:-(

> The relevant lines from my .mailsync are:
> 
> store Testing {
> 	server	{po12.mit.edu/user=wdc}
> 	ref	{po12.mit.edu}
> 	pat	INBOX.Testing.*
> 	prefix	INBOX.Testing.
> }
> 
> store l-Testing {
> 	pat	MESSAGE_ARCHIVE/local-Testing/*
> 	prefix	MESSAGE_ARCHIVE/local-Testing/
> }
> 
> channel save-Testing Testing l-Testing {
> 	msinfo MESSAGE_ARCHIVE/local-Testing/msinfo
> }

This looks correct to me, except you need to use % instead of * for 
recursion.

Tim

[Mailsync-list] md5 digest patch

[Bob L. <li...@is...>]

I've posted my md5 digest patch at http://www.isi.edu/~lindell/mailsync/.

For the benefit of the list, I'll try and recap the problem I'm trying
to solve and the outstanding issues.

I encountered a problem synchronzing my draft mail folder.  Messages in
my draft folder do not have a Message-Id field.

After giving the problem some thought, I decided it would be good
to have a "strong" digest for each message, rather than relying on
the message-id.  I made a modification to mailsync to compute the md5
digest over the fields From:, To:, Subject:, Date:, and Message-Id:.
I ignore any of these fields that are not present and compute the digest
over the remaining fields.

Although this method requires some additional computation, it seems to
be more reliable than assuming the Message-Id is unique.  Instead of
the Message-Id, I store the ascii hex representation of the md5 digest
in the msinfo message, which is 32 bytes long.

Information about the uniqueness of Message Ids:

	http://cr.yp.to/immhf/thread.html

        In practice, Message-IDs are not necessarily unique. For example,
        Internet Mail Service 5.0.1457.3 reportedly copies Message-ID into
        a bounce message from the message being bounced; and Microsoft
        Internet Mail reportedly uses the same Message-ID for every
        message. Furthermore, from a security perspective, an attacker
        can easily forge a message with a duplicate Message-ID.

I'm still not convinced that mailsync should rely exclusively on
the Message-Id field, since I believe that RFC 822 doesn't require
a message to contain this field.  Bernstein's site also mentions:

        Any message that starts or continues a thread needs a
        Message-ID. Not all messages contain Message-ID; for example,
        bounce messages from qmail do not contain Message-ID, and the
        Bell Labs upas mailer never creates Message-ID.

Outstanding issues for my patch:

1) Mail clients may modify some message envelope fields due to character
set conversions.  This would cause the message to a different digest
value.  If the mail client modifies the Message Id field, the existing
mailsync algorithms would also have difficulties.

2) The current code ignores a msinfo message since it has no Message-Id
field.  My modifications operate on messages without Message-Id fields,
so a different mechanism is needed to exclude the msinfo message from
the synchronization process.

3) Using the digest should probably be an option.  My patch currently
replaces the Message Id with the digest.

4) Currently, mailsync warns users when it sees two messages that have
the same Message Id.  In a similar vein, mailsync could warn users
when two messages have a different digest but have the same Message Id
(for messages with Message Id fields).  Those messages could be ignored
for syncing purposes unless the user asks to force these to be synced
(with a new command line option).

5) How would one do a transition from the current Message Id based scheme to
the md5 one?

Assuming that you can disambiguate a md5 digest string from a Message Id
string in the msinfo list, it should be straightforward.  If the msinfo
previous contained the Message Id field, it is matched against the
message during the sync process and written back out as a digest instead.

6) The msinfo list entry should probably be of the form:

	md5digest DELIMITER_CHAR Message_ID

with no spaces between the 3 elements.  The DELIMITER_CHAR should be
chosen as a character that cannot appear in a Message Id or md5digest.
A test for the old or new format would check the 33rd character of the
list element.  If it is the DELIMITER_CHAR, it is the new format,
otherwise it is the old format (e.g. Message Id).

Bob

[Mailsync-list] Re: mailsync cvs

[Tim C. <ful...@do...>]

Hi all,

(Please use my spam-protected address "ful...@do..." on the list.)

On Tue, 10 Sep 2002, Tomas Pospisek wrote:

> I went through the forums and deleted those that were obsolete. In the one
> forum that remains are some open questions. So I'll close the forum as
> soon as they're cleared.

Thanks!

> > > * why does mailsync need a procedure (simplify_message_id) that is
> > >   removing spaces in messages at all? AFAI can see a message id is defined
> > >   by the regular expression:
> > >
> > >         ^message-id: .*
> > >
> > >   So to manipulate a message I just have to process line by line, strip
> > >   the string "^message-id:" from them if necessary (it wouldn't be
> > >   necessary if we'd be reading it from a msinfo file) and take the rest,
> > >   whatever it's composed of, as the message id.
> >
> > This is important.  The .msinfo parser is very stupid and relies on
> > whitespace to separate message-ids, so message-ids can't be allowed to
> > have spaces.  Since they are compared to each other, it's crucial that if
> > I modify message-ids, I always modify them in exactly the same way.
> 
> Yeah, I've seen that. It's a very creative hack ;-)

A hack, at least.

> > Eventually we are going to have to change the .msinfo format.  When we do,
> > we should probably use... (wait for it...) XML.  I like the fact that it
> > has such a small number of magic characters (just <, >, &, is that right?)
> > which is important for us since message-ids have all manner of
> > punctuation.
> 
> Yeah - on the other side, the msinfo format is trivial enough, I do not
> think we need the syntactic candy if the structure is _that_ simple.

You're right.  I think the only thing wrong with the config file is that 
it might be difficult to specify folders with interesting punctuation.

> Additionaly since message-ids contain a lot of <,>,"'s we'll have to
> escape them. Yuck!!

I'm hoping it's just three characters to escape: <, >, &.  In fact we 
could strip off the standard <, > from the message-id!  s/^<|>$//g.

Note for the record: I'm talking about the hypothetical XML format here.

> Have you ever tried to read XML config files? IMHO it's not easy to read.
> So wrt to msinfo I'm against XML.

OK.

> message-ids can't start with a space. So we could use starting spaces to
> determine what a message-id is.

I like that idea!

> Since IMAP is so "flexible" I'm not sure which character can _not_ be
> contained in a mailbox name. In case there is one such character, then we
> could use it as a syntactic element. F.ex.:
> 
> private/tim
>   <Pin...@pa...>

Yes, I don't know whether such a character exists.  Perhaps NUL is the 
only one!

> If we decide to put all message-ids into <> (a fabulous idea). Then we can
> define whatever we want later. F.ex.:
> 
> private/tim
>   mid: <Pin...@pa...>
>   md5: 1234456
>   md5, mid: <Pin...@pa...>, 983745
> 
> etc.

Yes, that's good.

> I think the config parser is the lesser evil now. Mailsync has quite some
> bugs. I want to kill those first. Then it'd be nice to get the code into
> some shape (a global variable can pop up at any line right now in
> the code ;-)

An excellent idea.

One kind of global variable is the container used to collect results from
c-client callbacks.  These could be made static variables in a separate
c-client adaptor file.

> AFAIK (I'll have to check in the code to verify this) the whole header is
> downloaded anyway, so it doesn't hurt if we compare md5's as well.

It won't hurt performance, true, I'm just worried that the algorithm might 
get a little too complicated.  

> True, we can make it optional and/or warn the user if mailsync finds two
> identical message-ids with differing headers.
> 
> > The more I think about it, the less I like md5ing for a unique identifier.
> > If we don't get the header selection heuristic exactly right, we won't be
> > able to fix it without invalidating everybody's msinfo.
> 
> Ack.

Well, we can alleviate this by versioning the header-selection algorithm:

<mailbox>
 <name>mail/foo</name>
 <message>
  <md5 version="1">325252ae267a</md5>
  <message-id>my_lame_client&lt;&amp;&gt;@patroclus.doppke.com</message-id>
 </message>
 ...
</mailbox>

(I assume if we did <mailbox name="mail/foo"> we'd just have one more 
character to escape in mailbox names.)

> > How about incorporating some functionality to add message-ids where
> > necessary?  The new message-ids could be computed by an md5, so that
> > duplicate messages will tend to get the same message-ids, and when the md5
> > heuristic fails (some header changed in transit), the result will be no
> > worse than duplicate messages.
> 
> Do you want to retransmit the message with a newly inserted message id or
> what are you thinking about here?

Unfortunately, imap would require retransmitting the message in this 
scenario.  :-(

> > Seriously: the < may be required by the RFC, I can't remember.  Anyway,
> > can the clean_message_id() function just add < > when necessary?  Not many
> > mailsync users can be depending on its buggy behavior here.
> 
> Yes, I think we could do that. Good idea.

For the record, we're talking about *adding* < > to the current msinfo 
format, while *removing* them from the hypothetical future XML format.

> > Possibly for the < > problem, we won't have to change the format.  But
> > eventually we will, and it won't be a big deal.  Just tag the new format,
> > allow mailsync to permanently read both formats, but only write the new
> > one.  Msinfo is always rewritten from scratch each time anyway.
> 
> I'll have to look at the code to find where it's rewritten. In such
> a case we'd have less problems (well, downgrading mailsync wouldn't be
> possible, but in case we get it right, who cares to do that anyway).

Microsoft Word supports downgrading, I think.  :-)

Tim

[Mailsync-list] Re: mailsync cvs

[Tomas P. <tp...@so...>]

I'm Cc:ing Bob with this and also the mailing list. I'm inviting you both
to subscribe the mailing list so we can continue there.

On Mon, 9 Sep 2002, Tim Culver wrote:

> > I suggest merging the existing forums into the mailing list and deleting
> > the three forums. I don't know about you, but I don't ever come around
> > monitoring what's going on in the forums. And there's not as much traffic
> > that would justify 3 of them (IMHO). Once they're merged into the mailing
> > list, I'd delete them.
>
> That sounds great.  I think the 3 forums was the default setup at the
> time.  Forum posts are cc:ed to me, and sometimes I'm lazy and just reply
> by email.

I went through the forums and deleted those that were obsolete. In the one
forum that remains are some open questions. So I'll close the forum as
soon as they're cleared.

> > * why does mailsync need a procedure (simplify_message_id) that is
> >   removing spaces in messages at all? AFAI can see a message id is defined
> >   by the regular expression:
> >
> >         ^message-id: .*
> >
> >   So to manipulate a message I just have to process line by line, strip
> >   the string "^message-id:" from them if necessary (it wouldn't be
> >   necessary if we'd be reading it from a msinfo file) and take the rest,
> >   whatever it's composed of, as the message id.
>
> This is important.  The .msinfo parser is very stupid and relies on
> whitespace to separate message-ids, so message-ids can't be allowed to
> have spaces.  Since they are compared to each other, it's crucial that if
> I modify message-ids, I always modify them in exactly the same way.

Yeah, I've seen that. It's a very creative hack ;-)

> Eventually we are going to have to change the .msinfo format.  When we do,
> we should probably use... (wait for it...) XML.  I like the fact that it
> has such a small number of magic characters (just <, >, &, is that right?)
> which is important for us since message-ids have all manner of
> punctuation.

Yeah - on the other side, the msinfo format is trivial enough, I do not
think we need the syntactic candy if the structure is _that_ simple.

Additionaly since message-ids contain a lot of <,>,"'s we'll have to
escape them. Yuck!!

>  Also I assume there is a decent free parser out there (though I haven't
> looked).

certainly. The most simple XML parser will be 10 lines of C, and the most
complex (being able to parse all the DTD candy etc.) will be.

Have you ever tried to read XML config files? IMHO it's not easy to read.
So wrt to msinfo I'm against XML.

message-ids can't start with a space. So we could use starting spaces to
determine what a message-id is.

Since IMAP is so "flexible" I'm not sure which character can _not_ be
contained in a mailbox name. In case there is one such character, then we
could use it as a syntactic element. F.ex.:

private/tim
  <Pin...@pa...>

If we decide to put all message-ids into <> (a fabulous idea). Then we can
define whatever we want later. F.ex.:

private/tim
  mid: <Pin...@pa...>
  md5: 1234456
  md5, mid: <Pin...@pa...>, 983745

etc.

> While we're at it, let's make the config file XML too and get rid of all
> of my cheesy parsing code.

Your parsing code is IMO OK, I've commented it. Now it's pretty readable
and understandable. But getting rid of the msinfo parser would be a good
thing (IMHO).

> > "Store", "Channel" etc.
> >   clearly look like objects, but they are structures. This results in some
> >   ugly code:
> >
> > 	struct ConfigItem {
> > 	  int is_Store;
> > 	  Store* store;
> > 	  Channel* channel;
> > 	  ...
> >
> >   Is there something that I'm missing that doesn't allow those structures
> >   to be real classes (like f.ex. some interdependency between them and the
> >   c-client code which I guess is C only).
> >   If there's nothing that speaks against it, I'm going to slowly hack the
> >   code into object oriented form ;-)
>
> Please do, but be aware that I may have strong opinions about how OO code
> should work.  ConfigItem is ugly, but I don't think that store and channel
> have enough in common to require a common base class.  The only thing they
> have in common is that they are specified in a config file.  If we get rid
> of my cheesy parser, we probably won't need it at all.

I think the config parser is the lesser evil now. Mailsync has quite some
bugs. I want to kill those first. Then it'd be nice to get the code into
some shape (a global variable can pop up at any line right now in
the code ;-)

> > * what does "tdc" mean (as in tdc_mail_list_dest, tdc_mail_list_store,
> >   tdc_mail_list etc.)?
>
> Those are my initials.  :-)  Probably those functions were slight
> variations on things that existed in the c-client example programs.  Feel
> free to come up with better names.

I see. I've started some renaming already.

> > * Regarding the message-id vs. md5 proble I'm proposing the following
> >   solution:
> >
> >   if ( enabled_md5 or message_id_not_available)
> >         calculate_compare_and_save_md5
> >   if ( enabled_message_id and message_id_available)
> >         compare_and_save_message_id
> >
> >   What do you think?
>
> I don't like fallbacks.  It should use message-id or md5sum at the user's
> discretion.  Also, the user should be able to switch back and forth
> without worrying about his msinfo acting corrupted.  If md5sum turns out
> to work better after some experience, we'll make it the default.

Yeah. But did you read Bob Lindell's email regarding message ids? It seems
that a) message id's aren't even required by any RFC and b) that there is
software that produces the same message-id for all emails.

AFAIK (I'll have to check in the code to verify this) the whole header is
downloaded anyway, so it doesn't hurt if we compare md5's as well.

True, we can make it optional and/or warn the user if mailsync finds two
identical message-ids with differing headers.

> The more I think about it, the less I like md5ing for a unique identifier.
> If we don't get the header selection heuristic exactly right, we won't be
> able to fix it without invalidating everybody's msinfo.

Ack.

> How about incorporating some functionality to add message-ids where
> necessary?  The new message-ids could be computed by an md5, so that
> duplicate messages will tend to get the same message-ids, and when the md5
> heuristic fails (some header changed in transit), the result will be no
> worse than duplicate messages.

Do you want to retransmit the message with a newly inserted message id or
what are you thinking about here?

> > * Regarding the msinfo format. Currently it looks like this:
> >
> > 	  Headers: blabla
> > 	  More_Headers: blabli
> >
> > 	  mailbox/submailbox
> > 	  WHATEVER_MESSAGE_ID
> > 	  mailbox/submailbox2
> > 	  ...
> >
> >   You are distinguishing between a mailbox name and a message id this way:
> >
> > 	  if (text[k] != '<') { /* Mailbox name */ }
> > 	  else                { /* Message id */ }
> >
> >   The problem is, that there are mailclients that don't give a damn
> >   about that convention. An excerpt from my msinfo:
> >
> > 	j-spin
> > 	TCPSMTP_GEN.12.550@194.235.177.92
> >
> >   Let's look at the email message in question:
> >
> > 	From xx...@go... Sat Aug 30 08:24:23 1997
> > 	Received: from bbs.datacomm.ch (datacomm.ch [194.148.11.200]) by spin.ch (8.7.5/
> > 	New SPIN) with SMTP id IAA29983 for <xx...@sp...>; Sat, 30 Aug 1997 08:24:19 GMT
> > 	X-ROUTED: Sat, 30 Aug 1997 08:23:06 -0200
> > 	X-TCP-IDENTITY: Edo
> > 	Received: from octum-880-mhz- [194.235.177.92] by bbs.datacomm.ch with smtp
> > 	        id AIBGBHBL ; Sat, 30 Aug 1997 08:22:22 -0200
> > 	From: "ed" <xx...@go...>
> > 	To: <xx...@sp...>
> > 	Subject: Es ist doch ganz klar...
> > 	Date: Sat, 30 Aug 1997 08:13:22 +0200
> > 	X-MSMail-Priority: Normal
> > 	X-Priority: 3
> > 	X-Mailer: Microsoft Internet Mail 4.70.1161
> > 	MIME-Version: 1.0
> > 	Content-Type: text/plain; charset=Windows-1250
> > 	Content-Transfer-Encoding: 8bit
> > 	message-id: TCPSMTP_GEN.12.550@194.235.177.92
> > 	Status: RO
> > 	X-Status:
> > 	X-Keywords:
> > 	X-UID: 123
>
> See, there's your problem.  You must not accept any incoming mail from
> Outlook Express.

Heh :-)

> Seriously: the < may be required by the RFC, I can't remember.  Anyway,
> can the clean_message_id() function just add < > when necessary?  Not many
> mailsync users can be depending on its buggy behavior here.

Yes, I think we could do that. Good idea.

> >   AFAI can see, the only thing we can do here is to define a new msinfo
> >   format, release a new mailsync and test for the new format and if it
> >   doesn't fit either transform it automaticall (blegh) or tell the user to
> >   please transform it by had (using some nice script, that catches some
> >   misstakes).
>
> Possibly for the < > problem, we won't have to change the format.  But
> eventually we will, and it won't be a big deal.  Just tag the new format,
> allow mailsync to permanently read both formats, but only write the new
> one.  Msinfo is always rewritten from scratch each time anyway.

I'll have to look at the code to find where it's rewritten. In such
a case we'd have less problems (well, downgrading mailsync wouldn't be
possible, but in case we get it right, who cares to do that anyway).
*t

-----------------------------------------------------------------------
     Tomas Pospisek
     sourcepole    -   Linux & Open Source Solutions
     http://sourcepole.com
     Elestastrasse 18,  7310 Bad Ragaz,  Switzerland
     Tel:+41 (81) 330 77 13,  Fax:+41 (81) 330 77 12
------------------------------------------------------------------------