International Components for Unicode

enh <enh@...> wrote on 08/24/2010 05:17:43 PM:

> yoshito_umaoka@... wrote:
> > At least, there is a user using ICU for JDK implementation. I'm not 
sure I
> > understand your statement correctly, but I think if we change the 
default
> > behavior to autoQuoteApostorphe() enabled, it looks it's not trivial 
to
> > disable the behavior in the nested formatter.
> 
> If you mean Android, we do use ICU4C, but we don't use it for 
MessageFormat.
> 

I was not sure about Android - I meant Apache Harmony (my understanding is 
Android is based on Harmony code base).

> (Java's MessageFormat is basically dead anyway. non-legacy code uses
> Formatter or one of its many wrappers.)
> 

Not sure Java's MessageFormat is dead or not. But, if this is true, I'm 
not sure we should improve ICU's MessageFormat like this way now, because 
ICU's MessageFormat is nothing much different from Java's. We know there 
are some issues in the design of MessageFormat. If we believe Java's 
MessageFormat is dead (because of various issues), then we should also 
consider new function separated from MessageFormat in ICU.

-Yoshito

I think there are two issues at hand.

Short-term, the changes to MessageFormat solve a current problem, and the
possibility of an a compatibility problem are so minuscule that I think we
should make the changes. After all, the phrases like "I don''t like {0}"
will still work; in addition, "I don't like {0}" will.

Longer-term, I do think we should look seriously at supporting different
syntaxes for message formatting, including the Java Format class, so that
people can use syntax that they are familiar with, but then also get correct
behavior for all the supported languages.

Mark

*— Il meglio è l’inimico del bene —*


On Wed, Aug 25, 2010 at 08:22, <yoshito_umaoka@...> wrote:

> Not sure Java's MessageFormat is dead or not. But, if this is true, I'm
> not sure we should improve ICU's MessageFormat like this way now, because
> ICU's MessageFormat is nothing much different from Java's. We know there
> are some issues in the design of MessageFormat. If we believe Java's
> MessageFormat is dead (because of various issues), then we should also
> consider new function separated from MessageFormat in ICU.
>

Mark Davis ☕ <mark@...> wrote on 08/25/2010 01:00:48 PM:

> I think there are two issues at hand.
> 
> Short-term, the changes to MessageFormat solve a current problem, 
> and the possibility of an a compatibility problem are so minuscule 
> that I think we should make the changes. After all, the phrases like
> "I don''t like {0}" will still work; in addition, "I don't like {0}" 
will.

My point is simple. The current MessageFormat pattern syntax itself has no 
logical flaws. It's just error prone.
If the new behavior introduces any issues with existing code, even it's 
very rare, we should provide a way to provide capability to support it. 
Especially, this specific code is parallel to Java implementation in 
ICU4J, we do not want to introduce any compatibility issues without any 
reasonable workaround.

> 
> Longer-term, I do think we should look seriously at supporting 
> different syntaxes for message formatting, including the Java Format
> class, so that people can use syntax that they are familiar with, 
> but then also get correct behavior for all the supported languages.
> 

A while ago, we discussed about separation of formatter and parser. In 
many use cases, people do not need "parse" capability and want light 
weight formatter. Also, the parse capability in our existing Format 
services is not enough for some use cases (taking free text input). We 
learnt a lot from the original Format design/implementation and I think we 
can provide much better formatter/parser if we design them from scratch. 
Can we simply leave existing implementation as is and move on?

-Yoshito

On Wed, Aug 25, 2010 at 10:00 AM, Mark Davis ☕ <mark@...> wrote:

> Longer-term, I do think we should look seriously at supporting different
> syntaxes for message formatting, including the Java Format class, so that
> people can use syntax that they are familiar with, but then also get correct
> behavior for all the supported languages.
>

I like the Java Formatter
syntax<http://download-llnw.oracle.com/javase/6/docs/api/java/util/Formatter.html#syntax>,
but I don't want to throw out the baby with the bathwater.

   - I assume that the Formatter code is a quite sizable chunk of code, thus
   work if we duplicate it. It certainly would add a lot of API.
   - The MessageFormat syntax, despite its issues, lends itself more to
   interesting and readable argument styles. A DateFormat pattern is a bit more
   readable and flexible than a single letter after "t".
   - We use plural and select formats (someone might use choice), and those
   would not lend themselves to extensions of the Java Formatter syntax.

I also think the MessageFormat quoting is fixable, although the necessary
implementation looks less simple than I thought yesterday.

If there are strong concerns with compatibility, I think we could live with
not changing the default MessageFormat quoting behavior; we here would
advise our users to always use wrapper functions that fix the quoting. (But
I do think we would serve most users better if we changed the default.)

markus

On Wed, Aug 25, 2010 at 10:30 AM, <yoshito_umaoka@...> wrote:

> A while ago, we discussed about separation of formatter and parser. In
> many use cases, people do not need "parse" capability and want light
> weight formatter. Also, the parse capability in our existing Format
> services is not enough for some use cases (taking free text input).


Correct. Parsing on a NumberFormat or DateFormat makes sense. On a
MessageFormat, it does not.

We
> learnt a lot from the original Format design/implementation and I think we
> can provide much better formatter/parser if we design them from scratch.
>

I have ideas for simpler formatting support that would use the MessageFormat
syntax, help with modularization and would exclude parsing of "messages".

I am mostly done with writing from scratch a MessageFormat syntax parser in
Java that allows us to deal with the quoting issue, and could serve as the
core of such a simplified formatting implementation. (I would not expose the
parser itself as public API, although it's useful for analyzing message
strings.)

Can we simply leave existing implementation as is and move on?
>

No. We depend on it, at least in the short term, and do not want to have to
teach developers and translators that they have to double or avoid
apostrophes in some set of translatable messages.

markus

yoshito_umaoka@... wrote:
> I was not sure about Android - I meant Apache Harmony (my understanding is
> Android is based on Harmony code base).

Harmony doesn't use ICU4C at all, and they don't use ICU4J to
implement their MessageFormat. Android and Harmony use basically the
same implementation of MessageFormat. we've rewritten most of
Formatter, because people really care about Formatter and use it
heavily, but there's been no reason to touch MessageFormat.

Markus Scherer wrote:
>   - The MessageFormat syntax, despite its issues, lends itself more to
>  interesting and readable argument styles. A DateFormat pattern is a bit more
>  readable and flexible than a single letter after "t".

i don't think that's used much either. SimpleDateFormat is the usual
choice, and if that needs embedding in a Formatter-produced string, %s
suffices. we explicitly recommend against using Formatter to format
dates/times.

to substantiate my earlier hand-waving about MessageFormat's
obsolescence, i actually checked... outside of JDK logging classes
that are defined to use it (classes which are also rarely used on
Android, which has its own logger), and one caller in our Eclipse
plugin, MessageFormat is _literally_ unused in Android. it's basically
just there in case a third-party app drags in a third-party library
that's still using it.

you have other users, of course, but as far as Android's concerned,
there's no reason to spend engineering effort on MessageFormat.
(making ICU4C's DecimalFormat's double formatting faster would
probably have the largest impact, as far as formatting goes.)

-- 
Elliott Hughes - http://jessies.org/~enh/

We discussed this in the ICU team meeting today, and reached the following
consensus:

   - Change the default parsing behavior but implement both variants.
   - In Java, the old behavior can be chosen via an ICUConfig class backed
   by a .properties file. (Essentially at build time.)
   - In C++, the old behavior can be chosen via a #define to uconfig.h .
   (Also at build time.)

Best regards,
markus

Hi all,

I need to amend the auto-quoting behavior. The existing
autoQuoteApostrophe() function only considers apostrophes as quoting literal
text if they immediately precede a curly brace {}. However, this function
only operates on the top level of a message string, where {} are the only
syntax that needs quoting (aside from the apostrophe itself).

Now that we are extending the auto-quoting behavior to message fragments in
choice/plural/select arguments, there is more quotable syntax to consider.

In a choice argument style, a message fragment is terminated with a | pipe
symbol. For example, "a_{0,choice,-99 #-inf|  5≤ five | 99 # ninety'|'nine
 }_z" with a value of 99 formats to "a_ ninety|nine  _z". (Current ICU4J
behavior which uses java.text.ChoiceFormat.)

In a plural argument style, a # pound sign in a message fragment is replaced
with the number (minus the offset once we implement the extensions). (There
is also a bug (#7917) <http://bugs.icu-project.org/trac/ticket/7917>:
currently, quoting does not protect # from being replaced.)

Nothing special happens in a select message fragment.

As a logical extension of our plan, I should at least treat '|...' as a
quoted literal in a choice message fragment, and '#...' as a quoted literal
in a plural message fragment.

I could leave it minimalistic like that, or I could *always* start a quoted
literal when an apostrophe is followed by any of {}|#.

The former (minimalistic approach) would be "safer" in that only characters
that are syntactically meaningful in their context are quoted.

The latter (inclusive approach) might be slightly easier to explain (and
would lend itself to shorter RBNF of the whole MessageFormat syntax). The
inclusive approach also makes me slightly nervous with regard to possible
future complex-argument types beyond choice/plural/select where we might
introduce new syntax. It would not be so good at that time if apostrophes
followed by new syntax characters started quoted literals where they didn't
used to.

I am leaning towards the minimalistic approach. Let me know if you disagree.

markus

On Wed, Aug 25, 2010 at 12:24 PM, Markus Scherer <markus.icu@...:

> We discussed this in the ICU team meeting today, and reached the following
> consensus:
>
>    - Change the default parsing behavior but implement both variants.
>    - In Java, the old behavior can be chosen via an ICUConfig class backed
>    by a .properties file. (Essentially at build time.)
>    - In C++, the old behavior can be chosen via a #define to uconfig.h .
>    (Also at build time.)
>
> I am implementing this now and need to decide:

A) In the new MessagePattern class (new shared pattern parser/handler) I can
add a static field initialized from ICUConfig, so that by default all users
of the MessagePattern get the ICUConfig'ured behavior.

B) In the new MessagePattern class I can always default to the "auto-quote"
behavior, but the Message/Choice/Plural/SelectFormat classes would always
pass the ICUConfig'ured mode into the MessagePattern constructor.

In either case, I will have a MessagePattern constructor that takes an
ApostropheMode parameter, so that it's possible to get particular behavior
regardless of the ICUConfig setting.

Differences between A and B:

   - We might want to keep MessagePattern very light-weight, without loading
   the ICUConfig.properties. (This argues for B.)
   - It would be nice to have one single spot where this option is
   evaluated. (This argues for A.)

Related: The enum ApostropheMode could be a nested type in MessagePattern
vs. MessageFormat. Since it is an (optional) constructor parameter on
MessagePattern, it probably wants to be MessagePattern.ApostropheMode.

Opinions?

markus

Markus Scherer <markus.icu@...> wrote on 03/02/2011 07:52:47 PM:

> From: Markus Scherer <markus.icu@...>
> To: icu-design@...
> Date: 03/02/2011 07:57 PM
> Subject: Re: [icu-design] ICU API proposal: MessageFormat with 
> autoQuoteApostrophe() behavior by default
> 
> On Wed, Aug 25, 2010 at 12:24 PM, Markus Scherer <markus.icu@...> 
wrote:
> We discussed this in the ICU team meeting today, and reached the 
> following consensus:
> Change the default parsing behavior but implement both variants.
> In Java, the old behavior can be chosen via an ICUConfig class 
> backed by a .properties file. (Essentially at build time.)

ICUConfig class use JVM system properties (-Dprop=value) first. So you can 
flip the setting without .properties if you want.

> In C++, the old behavior can be chosen via a #define to uconfig.h . 
> (Also at build time.)
> I am implementing this now and need to decide:
> 
> A) In the new MessagePattern class (new shared pattern parser/
> handler) I can add a static field initialized from ICUConfig, so 
> that by default all users of the MessagePattern get the 
> ICUConfig'ured behavior.
> 
> B) In the new MessagePattern class I can always default to the 
> "auto-quote" behavior, but the Message/Choice/Plural/SelectFormat 
> classes would always pass the ICUConfig'ured mode into the 
> MessagePattern constructor.
> 
> In either case, I will have a MessagePattern constructor that takes 
> an ApostropheMode parameter, so that it's possible to get particular
> behavior regardless of the ICUConfig setting.
> 
> Differences between A and B:
> We might want to keep MessagePattern very light-weight, without 
> loading the ICUConfig.properties. (This argues for B.)

I believe this overhead is minimal - it should be one time initialization 
from .properties included in icu4j.jar. icu4j.jar has to be loaded first 
anyway, so there is no extra OS level file I/O overhead, which is usually 
pretty expensive at cold start.

> It would be nice to have one single spot where this option is 
> evaluated. (This argues for A.)
> Related: The enum ApostropheMode could be a nested type in 
> MessagePattern vs. MessageFormat. Since it is an (optional) 
> constructor parameter on MessagePattern, it probably wants to 
> be MessagePattern.ApostropheMode.
> 
> Opinions?
> 
> markus

On Tue, Mar 8, 2011 at 7:04 AM, <yoshito_umaoka@...> wrote:

> > In C++, the old behavior can be chosen via a #define to uconfig.h .
> > (Also at build time.)
> > I am implementing this now and need to decide:
> >
> > A) In the new MessagePattern class (new shared pattern parser/
> > handler) I can add a static field initialized from ICUConfig, so
> > that by default all users of the MessagePattern get the
> > ICUConfig'ured behavior.
> >
> > B) In the new MessagePattern class I can always default to the
> > "auto-quote" behavior, but the Message/Choice/Plural/SelectFormat
> > classes would always pass the ICUConfig'ured mode into the
> > MessagePattern constructor.
> >
> > In either case, I will have a MessagePattern constructor that takes
> > an ApostropheMode parameter, so that it's possible to get particular
> > behavior regardless of the ICUConfig setting.
> >
> > Differences between A and B:
> > We might want to keep MessagePattern very light-weight, without
> > loading the ICUConfig.properties. (This argues for B.)
>
> I believe this overhead is minimal - it should be one time initialization
> from .properties included in icu4j.jar. icu4j.jar has to be loaded first
> anyway, so there is no extra OS level file I/O overhead, which is usually
> pretty expensive at cold start.


Yes, I think that's right. I did implement this option A).

Thanks,
markus

More details for the ongoing MessageFormat work. Please speak up if you
disagree.

*The JDK parses and removes syntactic apostrophes several times, but it's
more complicated than just "once per nesting level":*

"ab{0,choice,0#1'2''3'''4''''.}yz" -> "ab12'3'4''.yz"
but
"ab{0,choice,0#1'2''3'''4''''.*{0}*}yz" -> "ab1234'.0yz"

This is because MessageFormat.format() treats the ChoiceFormat's result
differently depending on whether the result contains the character '{'. See
the JDK MessageFormat.format() API docs:
instanceof ChoiceFormat*any*subformat.format(argument).indexOf('{') >= 0 ?
(new MessageFormat(subformat.format(argument),
getLocale())).format(argument) : subformat.format(argument)

It could have been a simple optimization with consistent behavior if it also
checked for the presence of an apostrophe, but it doesn't.

*I plan to do the following for ICU 4.8:*

The ICU 4.8 default behavior (ApostropheMode.DOUBLE_OPTIONAL) will parse and
remove syntactic apostrophes only once, as agreed.
As a consequence, ChoiceFormat (& plural/select) .format(arg) will not
remove syntactic apostrophes, for consistency with using the resulting
string to instantiate another MessageFormat.
(This is already done on my branch.)

The *behavior with ApostropheMode.DOUBLE_REQUIRED* will mimic the JDK as
much as possible. In particular,

   - MessageFormat.format() will remove half of the result's apostrophes,
   check for '{' and either append the modified subformat result or instantiate
   a temporary MessageFormat instance.
   - Choice/Plural/SelectFormat will remove half of the result's
   apostrophes.

That is, the compatibility-apostrophe-mode will not just modify the pattern
parsing behavior but also the formatting behavior.

Note: It is not possible to just apply changes to MessagePattern alone,
because the JDK behavior requires that different parsing passes see parts of
the pattern string in different parsing/quoting states.

Note: The JDK fails to parse a MessageFormat pattern that has two choice
format arguments nested inside each other. ICU 4.8 will have no trouble with
that. (While this particular combination is probably not useful, we
definitely use patterns with two or three nesting levels of select+plural.)

markus