International Components for Unicode

This is a proposal to add functions to BreakIterator in ICU4C to inform ICU
that the underlying string being operated on has moved in memory.  This is
needed when working on strings that originate in a garbage
collected environment - a gc operation will typically move everything around
in memory.

The proposed functions are modeled after similar functions that were added
to ICU regular expressions for the same purpose.

Ticket:  8490
Reviewer:  Markus
Comment by: 5/17
Note:  this is proposed for ICU 5.0, not 4.8

In file ubrk.h,

/**
  *  Set the subject text string upon which the break iterator is operating
  *  without changing any other aspect of the state.
  *  The new and previous text strings must have the same content.
  *
  *  This function is intended for use in environments where ICU is
operating on
  *  strings that may move around in memory.  It provides a mechanism for
notifying
  *  ICU that the string has been relocated, and providing a new UText to
access the
  *  string in its new position.
  *
  *  Note that the break iterator never copies the underlying text
  *  of a string being processed, but always operates directly on the
original text
  *  provided by the user. Refreshing simply drops the references to the old
text
  *  and replaces them with references to the new.
  *
  *  Caution:  this function is normally used only by very specialized
  *            system-level code.   One example use case is with garbage
collection
  *            that moves the text in memory.
  *
  * @param bi         The break iterator.
  * @param text       The new (moved) text string.
  * @param status     Receives errors detected by this function.
  *
  * @draft ICU 5.0
  */
void ubrk_refreshUText(UBreakIterator *bi,
                       UText          *text,
                       UErrorCode     *status);


In file brkiter.h, in class BreakIterator:

  /**
    *  Set the subject text string upon which the break iterator is
operating
    *  without changing any other aspect of the matching state.
    *  The new and previous text strings must have the same content.
    *
    *  This function is intended for use in environments where ICU is
operating on
    *  strings that may move around in memory.  It provides a mechanism for
notifying
    *  ICU that the string has been relocated, and providing a new UText to
access the
    *  string in its new position.
    *
    *  Note that the break iterator implementation never copies the
underlying text
    *  of a string being processed, but always operates directly on the
original text
    *  provided by the user. Refreshing simply drops the references to the
old text
    *  and replaces them with references to the new.
    *
    *  Caution:  this function is normally used only by very specialized,
    *  system-level code.  One example use case is with garbage collection
that moves
    *  the text in memory.
    *
    * @param input      The new (moved) text string.
    * @param status     Receives errors detected by this function.
    *
    * @draft ICU 5.0
    */
    virtual BreakIterator &refreshInputText(UText *input, UErrorCode
&status);

(bcc: icu-announce)

Dear friends and users of ICU,

We are pleased to announce the release of ICU 4.8.

This version includes the latest locale data from CLDR
2.0<http://cldr.unicode.org/index/downloads/cldr-2-0>which was also
released today, as well as a rewritten MessageFormat class,
API for collation script reordering, a number of other new features, and bug
fixes.

More information about this release can be found on the following page:
http://site.icu-project.org/download/48

The API reference documents for both ICU4C and ICU4J are published at the
following locations:
  ICU4C: http://icu-project.org/apiref/icu4c/
  ICU4J: http://icu-project.org/apiref/icu4j/

Best regards,
Markus Scherer for the ICU Project

Dear ICU team & users,

I propose that, starting with the next release after 4.8, we require
namespace support in C++.

* Background:

ICU is still written to work with or without namespaces. autoconf finds out
whether namespaces are supported. The ICU namespace tends to be hidden,
although we have been recommending to turn off the "using namespace
icu"<http://source.icu-project.org/repos/icu/icu/trunk/readme.html#RecBuild>
(set
U_USING_ICU_NAMESPACE to 0).

* Rationale:

Namespaces are useful :-)

I have collected configure'd platform.h files from the platforms we
frequently test on, and they all support namespaces. (U_HAVE_NAMESPACE=1)

Namespace support has been part of the C++ standard "forever" (well, since
1998).

I would like to reduce the set of possible build variations, and I would
like to get away from generating a header file (platform.h) differently for
different platforms. The more platform-dependent switches we can remove the
better. This will also help with cross-compiling.

Function declarations outside the ICU namespace
require U_NAMESPACE_QUALIFIER which might be empty; it would be better and
clearer if we could just write "icu::". (I believe this works even if
namespace icu is an alias for the renamed namespace icu_<version>. I will
verify.)

Entry point renaming (library versioning, unicode/urename.h) requires
renaming of each C++ class if there is no namespace support, while otherwise
the renamed namespace suffices.

Please provide feedback by Tuesday, 2011-05-31.

Sincerely,
markus

The following draft ICU4J API needs review per ticket #8144:

http://icu-project.org/apiref/icu4j/com/ibm/icu/text/CurrencyDisplayNames.html
http://icu-project.org/apiref/icu4j/com/ibm/icu/text/CurrencyMetaInfo.html
http://icu-project.org/apiref/icu4j/com/ibm/icu/text/CurrencyMetaInfo.CurrencyDigits.html
http://icu-project.org/apiref/icu4j/com/ibm/icu/text/CurrencyMetaInfo.CurrencyFilter.html
http://icu-project.org/apiref/icu4j/com/ibm/icu/text/CurrencyMetaInfo.CurrencyInfo.html

Doug

Dear friends and users of ICU,

We are pleased to announce the *Release Candidate* for ICU 4.8. *Please test
* this Release Candidate if possible.

The final release of ICU 4.8 is planned for next Wednesday, May 25. ICU 4.8
is now in a code freeze state: There will be no further changes to the code
unless showstopper bugs are reported. Please use the icu-support mailing
list <http://site.icu-project.org/contacts> for error reports. As a
reminder, this Release Candidate build is NOT recommended for production
use.

More information about this release can be found on the following page:
http://site.icu-project.org/download/48

The API reference documents for both ICU4C and ICU4J are published at the
following locations:
  ICU4C: http://icu-project.org/apiref/icu4c48rc1/
  ICU4J: http://icu-project.org/apiref/icu4j48rc1/

Best regards,
Markus Scherer for the ICU Project

I am out of the office until 11/05/2011.

Please note that 10 May is the public holiday in HK.

I will respond to you when i am back.


Note: This is an automated response to your message  "icu-design Digest,
Vol 58, Issue 2" sent on 10/5/11 7:12:02.

This is the only notification you will receive while this person is away.

-----------------------------------------
*******************************************************************
This e-mail is confidential. It may also be legally privileged.
If you are not the addressee you may not copy, forward, disclose
or use any part of it. If you have received this message in error,
please delete it and all copies from your system and notify the
sender immediately by return e-mail.

Internet communications cannot be guaranteed to be timely,
secure, error or virus-free. The sender does not accept liability
for any errors or omissions.
*******************************************************************
"SAVE PAPER - THINK BEFORE YOU PRINT!"

On Fri, Jan 28, 2011 at 5:10 PM, Steven R. Loomis <srl@...:

>  Re-proposing on behalf of Mati.
> Reviewer: Markus (who filed #3660 a long time ago)
> Please respond by Feb 9th, 2011
> Target: 4.8 - yes, we really want to commit this, this time.
>
> [...]
>
> a) This proposal only addresses ICU4C.  Once there is an agreement on
> ICU4C, I will submit a parallel proposal for ICU4J.
>

FYI: We ended up with the new com.ibm.icu.text.Bidi.java method copied
below. I don't think the Java signature had been sent to this list before.

Best regards,
markus

    /**
     * Set the context before a call to setPara().<p>
     *
     * setPara() computes the left-right directionality for a given piece
     * of text which is supplied as one of its arguments. Sometimes this
piece
     * of text (the "main text") should be considered in context, because
text
     * appearing before ("prologue") and/or after ("epilogue") the main text
     * may affect the result of this computation.<p>
     *
     * This function specifies the prologue and/or the epilogue for the next
     * call to setPara(). If successive calls to setPara()
     * all need specification of a context, setContext() must be called
     * before each call to setPara(). In other words, a context is not
     * "remembered" after the following successful call to setPara().<p>
     *
     * If a call to setPara() specifies DEFAULT_LTR or
     * DEFAULT_RTL as paraLevel and is preceded by a call to
     * setContext() which specifies a prologue, the paragraph level will
     * be computed taking in consideration the text in the prologue.<p>
     *
     * When setPara() is called without a previous call to
     * setContext, the main text is handled as if preceded and followed
     * by strong directional characters at the current paragraph level.
     * Calling setContext() with specification of a prologue will change
     * this behavior by handling the main text as if preceded by the last
     * strong character appearing in the prologue, if any.
     * Calling setContext() with specification of an epilogue will change
     * the behavior of setPara() by handling the main text as if followed
     * by the first strong character or digit appearing in the epilogue, if
any.<p>
     *
     * Note 1: if <code>setContext</code> is called repeatedly without
     *         calling <code>setPara</code>, the earlier calls have no
effect,
     *         only the last call will be remembered for the next call to
     *         <code>setPara</code>.<p>
     *
     * Note 2: calling <code>setContext(null, null)</code>
     *         cancels any previous setting of non-empty prologue or
epilogue.
     *         The next call to <code>setPara()</code> will process no
     *         prologue or epilogue.<p>
     *
     * Note 3: users must be aware that even after setting the context
     *         before a call to setPara() to perform e.g. a logical to
visual
     *         transformation, the resulting string may not be identical to
what it
     *         would have been if all the text, including prologue and
epilogue, had
     *         been processed together.<br>
     * Example (upper case letters represent RTL characters):<br>
     * &nbsp;&nbsp;prologue = "<code>abc DE</code>"<br>
     * &nbsp;&nbsp;epilogue = none<br>
     * &nbsp;&nbsp;main text = "<code>FGH xyz</code>"<br>
     * &nbsp;&nbsp;paraLevel = LTR<br>
     * &nbsp;&nbsp;display without prologue = "<code>HGF xyz</code>"
     *             ("HGF" is adjacent to "xyz")<br>
     * &nbsp;&nbsp;display with prologue = "<code>abc HGFED xyz</code>"
     *             ("HGF" is not adjacent to "xyz")<br>
     *
     * @param prologue is the text which precedes the text that
     *        will be specified in a coming call to setPara().
     *        If there is no prologue to consider,
     *        this parameter can be <code>null</code>.
     *
     * @param epilogue is the text which follows the text that
     *        will be specified in a coming call to setPara().
     *        If there is no epilogue to consider,
     *        this parameter can be <code>null</code>.
     *
     * @see #setPara
     * @draft ICU 4.8
     * @provisional This API might change or be removed in a future release.
     */
    public void setContext(String prologue, String epilogue);

Still problems with http://bugs.icu-project.org/trac/ticket/8006

On Sat, Mar 26, 2011 at 2:08 PM, Markus Scherer <markus.icu@...:

>
>>  * Another example is UTF-32 which is always 4 bytes per charset
>> character.  UTF-32 code units
>>
>  * may represent more than one UTF-16 code point (e.g. surrogate pairs)
>> but always
>>
>
> fix to:
> may be represented by more than one UTF-8 or UTF-16 code unit but always
>

Ping...


>   * have size of 4 bytes.
>>  * This method is not intended to be used to determine whether the charset
>> has a fixed ratio
>>  * of bytes to Unicode codes units for any particular Unicode encoding
>> form.
>>  * FALSE is returned with the UErrorCode if error occurs or cnv is NULL.
>>  * @param cnv       The converter to be tested
>>  * @param status    ICU error code in/out paramter
>>  * @return TRUE if the converter is fixed-width
>>  * @draft ICU 4.8
>>  */
>> U_DRAFT UBool U_EXPORT2
>> ucnv_isFixedWidth(UConverter *cnv, UErrorCode *status);
>>
>
> Otherwise ok.
>

The behavior is not as documented. As the test shows,
ucnv_isFixedWidth(UTF-16) = TRUE which contradicts your definition. This is
because in the implementation you write minBytesPerChar == maxBytesPerChar
which you said is not what you want. (And you should write "return
condition" rather than the overly redundant "return condition ? TRUE :
FALSE")

To implement what you want, you could probably do something like
switch(ucnv_getType()) {
case UCNV_SBCS:
case UCNV_DBCS:
case some-other-types-that-are-fixed-width-if-any:
    return TRUE;
default:
    return FALSE;
}

Same in Java.

In the C test, please ucnv_close(cnv) at the end of each loop body. And
please use LENGTHOF(array) rather than the error-prone hardcoding of the
array length.

markus