International Components for Unicode

I see no fundamental problem with this API.  A few remarks.

1) The name of the function "ubidi_getParagraphLevelDirectionUBA" is 
really long.  There is no need to mention UBA, everything Bidi in ICU 
follows the UBA.  On the other hand, "Default" should be mentioned, since 
this is the paragraph level which would be assigned with the existing API. 
 I suggest "ubidi_resolveDefaultBaseDirection" or even "
ubidi_resolveDefaultDirection".

2) It is worth mentioning in the comments why this function exists: it is 
a lightweight function when only the direction is needed and no further 
Bidi processing of the text is needed.

3) Instead of "paragraph level direction", I would say "base direction", 
this is the term used in the comments for UBIDI_DEFAULT_xxx.


Shalom (Regards),  Mati
           Bidi Architect
           Globalization Center Of Competency - Bidirectional Scripts
           IBM Israel
           Phone: +972 2 5888802    Fax: +972 2 5870333    Mobile: +972 52 
2554160




From:   Norbert Lindenberg ♻ <norbert.lindenberg@...>
To:     icu-design@...
Date:   30/06/2010 03:56
Subject:        [icu-design] ICU API proposal: Fast string direction 
detection



Dear ICU team and users,

I would like to propose the addition of a fast function for string 
direction detection to the Bidi API, for ICU 4.6. This function 
determines the paragraph level direction of a string without creating 
a UBidi object. This direction can currently be derived from 
ubidi_getParaLevel, but at the substantial cost of initializing a 
UBidi object. To give callers control over the handling of strings 
with no strong direction, the new function allows for a NEUTRAL result 
value in addition to LTR and RTL values.

Ticket: http://bugs.icu-project.org/trac/ticket/7772
Designated API reviewer: Mark Davis

Please provide feedback by Wednesday, 2010-07-07T18:00:00Z.

Best regards,
Norbert



ICU4C API changes - bidi.h:
http://icu-project.org/apiref/icu4c/ubidi_8h.html

- add constant UBIDI_NEUTRAL to enum UBiDiDirection:

UBIDI_NEUTRAL /**< No strongly directional text. */

- in ubidi_getDirection, clarify that UBIDI_NEUTRAL is not returned.

- in ubidi_getVisualRun, clarify that UBIDI_NEUTRAL is not returned.

- add ubidi_getParagraphLevelDirectionUBA:

/**
  * Gets the paragraph level direction of the text provided according 
to the Unicode
  * Bidirectional Algorithm. The paragraph level direction is derived 
from the first
  * character in the string with bidirectional character type L, R, or 
AL. If the first such
  * character has type L, UBIDI_LTR is returned. If the first such 
character has type R or AL,
  * UBIDI_RTL is returned. If the string does not contain any 
character of these types, then
  * UBIDI_NEUTRAL is returned.
  * @param text is a pointer to the text whose paragraph level 
direction is needed.
  * Note: the text must be (at least) @c length long.
  * @param length is the length of the text; if <code>length==-1</ 
code> then the text
  * must be zero-terminated.
  * @return  UBIDI_LTR, UBIDI_RTL, UBIDI_NEUTRAL
  */
UBiDiDirection ubidi_getParagraphLevelDirectionUBA(const UChar *text,
int32_t length)


ICU4J API changes - Bidi.java:
http://icu-project.org/apiref/icu4j/com/ibm/icu/text/Bidi.html

- add constant Bidi.NEUTRAL:

/**
  * No strongly directional text.
  */
public static final byte NEUTRAL;

- add Bidi.getParagraphLevelDirectionUBA:

/**
  * Gets the paragraph level direction of the text provided according 
to the Unicode
  * Bidirectional Algorithm. The paragraph level direction is derived 
from the first
  * character in the string with bidirectional character type L, R, or 
AL. If the first such
  * character has type L, LTR is returned. If the first such character 
has type R or AL,
  * RTL is returned. If the string does not contain any character of 
these types, then
  * NEUTRAL is returned.
  * @param paragraph the text whose paragraph level direction is needed.
  * @return LTR, RTL, NEUTRAL
  * @see LTR
  * @see RTL
  * @see NEUTRAL
  * /
public static byte getParagraphLevelDirectionUBA(String paragraph)


-------------------------------------
Norbert Lindenberg
Yahoo! Internationalization Architect

I would like to propose the addition of an API to facilitate the 
presentation of specialized text (like a file path, an e-mail address or 
lines of a programming language) including RTL characters.

I have opened a ticket for this purpose, please see 
http://bugs.icu-project.org/trac/ticket/7797

I have designated Markus Scherer as reviewer because of his great 
involvement with the UBA implementation. I hope that he will accept this 
burden.

I have attached 2 files to the ticket:

1. A design document, which is mostly the body of a standard to be 
published by SII (the standards bureau for Israel).  It is a bit long, but 
the first sections should be enough to understand the need and the 
suggested design for a solution.

2. Javadoc for an ICU4J implementation.

I have a working and tested Java implementation of the API, which I have 
not submitted yet to the ICU repository.  The reason is that I am still 
discussing the API with people from Eclipse development, and some changes 
to the API are expected based on their review.  The base lines should 
remain the same, but I expect the number of public methods to be reduced 
and the distribution in packages to be modified.

Assuming that this proposal is accepted in principle, and after the Java 
API is finalized, I plan to create an equivalent API for ICU4C.

Please provide feedback until Sunday, 2010-07-11.


Shalom (Regards),  Mati
           Bidi Architect
           Globalization Center Of Competency - Bidirectional Scripts
           IBM Israel
           Phone: +972 2 5888802    Fax: +972 2 5870333    Mobile: +972 52 
2554160

Thanks, Mati.

I did a very quick look at it. Functionality looks good overall.

A key issue will be to specify exactly what each thing does, because there
map be reasonable options that all satisfy, for example:
          Constructor for a complex expressions processor with support for
Java statements.


Some items:

   1. We are using Java enums instead of static final ints.
   2. For modularization, we might want a more flexible dispatch.
   3. I think fullToLeanText / leanToFullText would be better as
   set/getFullText, and set/getLeanText. The class needs to have both for all
   of the positional stuff to work.
   4. Then you can have getLeanPosition(fullPosition), etc.
   5. recall/assume need to be set/get, etc.



Even though methods are protected, you need to document what they are for,
eg in "
          This method is not supposed to be invoked directly by users of
this class.
"
Mark

— Il meglio è l’inimico del bene —


On Tue, Jun 29, 2010 at 23:49, Matitiahu Allouche <matial@...> wrote:

> I would like to propose the addition of an API to facilitate the
> presentation of specialized text (like a file path, an e-mail address or
> lines of a programming language) including RTL characters.
>
> I have opened a ticket for this purpose, please see
> http://bugs.icu-project.org/trac/ticket/7797
>
> I have designated Markus Scherer as reviewer because of his great
> involvement with the UBA implementation. I hope that he will accept this
> burden.
>
> I have attached 2 files to the ticket:
>
> 1. A design document, which is mostly the body of a standard to be
> published by SII (the standards bureau for Israel).  It is a bit long, but
> the first sections should be enough to understand the need and the suggested
> design for a solution.
>
> 2. Javadoc for an ICU4J implementation.
>
> I have a working and tested Java implementation of the API, which I have
> not submitted yet to the ICU repository.  The reason is that I am still
> discussing the API with people from Eclipse development, and some changes to
> the API are expected based on their review.  The base lines should remain
> the same, but I expect the number of public methods to be reduced and the
> distribution in packages to be modified.
>
> Assuming that this proposal is accepted in principle, and after the Java
> API is finalized, I plan to create an equivalent API for ICU4C.
>
> Please provide feedback until Sunday, 2010-07-11.
>
>
> Shalom (Regards),  Mati
>           Bidi Architect
>           Globalization Center Of Competency - Bidirectional Scripts
>           IBM Israel
>           Phone: +972 2 5888802    Fax: +972 2 5870333    Mobile: +972 52
> 2554160
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Sprint
> What will you do first with EVO, the first 4G phone?
> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
> _______________________________________________
> icu-design mailing list
> icu-design@...
> To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design
>

LGTM

Mark

— Il meglio è l’inimico del bene —


On Tue, Jun 29, 2010 at 13:21, Andy Heninger <andy.heninger@...:

> Here is a proposal that will allow ICU regular expressions to operate over
> strings that may move in memory.
>
> The specific use for this is with Android.  The strings come from its
> Java-like environment, and a garbage collection there may result in them
> moving in memory while a regex Matcher has an active reference to them.  The
> obvious work around is to copy the strings from Java space into the C heap
> in the JNI wrapper code, but doing that would be very wasteful.
>
> Implementation of this is very straight forward.  Between calls, the match
> engine does not retain any pointers into the text except for those in the
> pair of UText objects that wrap the input text.  So all that needs to be
> done is to update those while retaining the current position.  Nothing in
> the existing code path for a match operation is touched.
>
>   - Andy
>
>
> In file uregex.h:
>
> /**
>   *  Set the subject text string upon which the regular expression is
> looking for matches
>   *  without changing any other aspect of the matching state.
>   *  The new and previous text strings must have the same content.
>   *
>   *  This function is intended for use in environments where ICU is
> operating on
>   *  strings that may move around in memory.  It provides a mechanism for
> notifying
>   *  ICU that the string has been relocated, and providing a new UText to
> access the
>   *  string in its new position.
>   *
>   *   Caution:  this function is normally used only by very specialized
>   *             system-level code.
>   *
>   * @param regexp     The compiled regular expression.
>   * @param text       The new (moved) text string.
>   * @param status     Receives errors detected by this function.
>   *
>   * @internal ICU 4.6
>   */
> U_INTERNAL void U_EXPORT2
> uregex_refreshUText(URegularExpression *regexp,
>                     UText              *text,
>                     UErrorCode         *status);
>
>
> In file regex.h:
>
>
>  /**
>     *  Set the subject text string upon which the regular expression is
> looking for matches
>     *  without changing any other aspect of the matching state.
>     *  The new and previous text strings must have the same content.
>     *
>     *  This function is intended for use in environments where ICU is
> operating on
>     *  strings that may move around in memory.  It provides a mechanism for
> notifying
>     *  ICU that the string has been relocated, and providing a new UText to
> access the
>     *  string in its new position.
>     *
>     *  Caution:  this function is normally used only by very specialized,
>     *  system-level code.
>     *
>     * @param input      The new (moved) text string.
>     * @param status     Receives errors detected by this function.
>     *
>     * @internal ICU 4.6
>     */
>     virtual RegexMatcher &refreshInputText(UText *input, UErrorCode
> &status);
>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Sprint
> What will you do first with EVO, the first 4G phone?
> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
> _______________________________________________
> icu-design mailing list
> icu-design@...
> To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design
>

works for me.

Mark

— Il meglio è l’inimico del bene —


On Tue, Jun 22, 2010 at 14:22, Andy Heninger <andy.heninger@...:

> I am proposing that we drop support for the old RBBI binary data format
> that was used with ICU 3.2 and earlier.
>
> I have been sifting through compiler warnings and other cruft, and the code
> to support the old format came up.
>
> Reasons for dropping the old RBBI data format:
> - There are no tests for it.  It could be dead already, and we would never
> know.
> - The old code is producing warnings about breaking strict aliasing rules
> with recent gcc compilers.  Getting rid of them isn't completely trivial.
> These you can't just cast away - it's casts that create the (false)
> problems.
> - ICU 3.2 is very old.
>
>  - Andy
>
>
>
>
>
> ------------------------------------------------------------------------------
> ThinkGeek and WIRED's GeekDad team up for the Ultimate
> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> lucky parental unit.  See the prize list and enter to win:
> http://p.sf.net/sfu/thinkgeek-promo
> _______________________________________________
> icu-core mailing list
> icu-core@...
> https://lists.sourceforge.net/lists/listinfo/icu-core
>
>

Sounds fine to me also. Please update the readme and download page to 
note it.

Steven

On 6/22/10 3:09 PM, Mark Davis ☕ wrote:
> works for me.
>
> Mark
>
> — Il meglio è l’inimico del bene —
>
>
> On Tue, Jun 22, 2010 at 14:22, Andy Heninger <andy.heninger@... 
> <mailto:andy.heninger@...>> wrote:
>
>     I am proposing that we drop support for the old RBBI binary data
>     format that was used with ICU 3.2 and earlier.
>
>     I have been sifting through compiler warnings and other cruft, and
>     the code to support the old format came up.
>
>     Reasons for dropping the old RBBI data format:
>     - There are no tests for it.  It could be dead already, and we
>     would never know.
>     - The old code is producing warnings about breaking strict
>     aliasing rules with recent gcc compilers.  Getting rid of them
>     isn't completely trivial.  These you can't just cast away - it's
>     casts that create the (false) problems.
>     - ICU 3.2 is very old.
>
>      - Andy
>
>
>
>
>     ------------------------------------------------------------------------------
>     ThinkGeek and WIRED's GeekDad team up for the Ultimate
>     GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
>     lucky parental unit.  See the prize list and enter to win:
>     http://p.sf.net/sfu/thinkgeek-promo
>     _______________________________________________
>     icu-core mailing list
>     icu-core@... <mailto:icu-core@...>
>     https://lists.sourceforge.net/lists/listinfo/icu-core
>
>
>
> ------------------------------------------------------------------------------
> ThinkGeek and WIRED's GeekDad team up for the Ultimate
> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> lucky parental unit.  See the prize list and enter to win:
> http://p.sf.net/sfu/thinkgeek-promo
>
>
> _______________________________________________
> icu-design mailing list
> icu-design@...
> To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design


-- 
Steven R. Loomis  srl@...
Technical Lead, ICU for C/C++<http://icu-project.org>
IBM San José Globalization Center of Competency
<http://ibm.com/software/globalization>

Hi Jason,

First of all, I think the goal of #7524 is to provide the functionalities 
which are not accessible from C APIs.  I think these are:

- detecting transition time (backward/forward) - BasicTimeZone
- creating a time zone from rule text in VTIMEZONE format - VTimeZone

TimeZone itself has two main features

- create a time zone for a given ID
- get UTC offset of a zone at a given time

I'd expect these features are covered in this proposal.  But it looks some 
important aspects (BasicTimeZone/VTimeZone) are missing while some much 
less important things are included.

It looks your approach is trying to provide simple C API wrapper for C++ 
TimeZone class.  I personally do not think we need to expose every C++ 
TimeZone APIs in C.


For now, I disagree with the proposal.  We need to brush up APIs - correct 
errors, remove unnecessary APIs and add necessary APIs.
We should also decide whether we really want to use utimezone_ prefix or 
use ucal_ prefix instead.


I added comments below.


Jason J Spieth <spieth@...> wrote on 06/09/2010 04:17:02 PM:

> 
> Dear ICU team & users, 
> 
> Originally I have a user request to create a wrapper for the 
> VTimeZone class.  However it was determined that such a wrapper for 
> this specific time zone subclass would be rather restrictive.  A 
> more useful solution is to create a wrapper for the base time zone 
> class.  The wrapper for the base class will expose some basic 
> functions.  However, Some APIs in the current TimeZone class were 
> designed when we did not support historic rules, functions such as 
> getRawOffset, getDSTSavings and some others. These are not included 
> in the wrapper at this time.  Also the DisplayName APIs will not be 
> wrapped as they have similar restrictions in that a time zone name 
> can change over time.  These out of date interfaces are not wrapped. 
> 
> For details see http://www.icu-project.org/trac/ticket/7524 
> 
> Please provide feedback by Wed. June 16. 
> I would like to designate Yoshito as reviewer. 
> 
> Here is the specification for the wrapper: 
> 
> 
> #ifndef UCNV_H 
> struct UTimeZone; 
> /** 
>  * Use the utimezone_* API to manipulate.  Create with 
>  * utimezone_open*, and destroy with utimezone_close. 

There is no "utimezone_open*" in your proposal.

>  * @draft ICU 4.6 
>  */ 
> typedef struct UTimeZone UTimeZone; 
> #endif 
> 
> /** 
>  * Creates a TimeZone for the given ID. 
>  * @param ID: the ID for a TimeZone, such as "America/Los_Angeles", 
>  * or a custom ID such as "GMT-8:00". 
>  * @param IDLength: the length of the ID 
>  * @return the specified TimeZone, or the GMT zone if the given ID 
>  * cannot be understood.  Return result guaranteed to be non-null.  If 
you 
>  * require that the specific zone asked for be returned, check the ID of 
the 
>  * return result. 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT UTimeZone* U_EXPORT2 
> utimezone_createTimeZone(const UChar* ID, 
>                          int32_t* IDLength); 

I always feel C++ TimeZone::createTimeZone is problematic.  It silently 
fallback to GMT if the given ID is unknown.  I do not think we need to 
duplicate the bad API from C++ TimeZone here.  In fact, we should probably 
add another createTimeZone in C++ which returns status, instead of 
silently fallback to GMT.

> 
> 
> /** 
>  * Creates a new copy of the default TimeZone for this host. Unless 
> the default time 
>  * zone has already been set using adoptDefault() or setDefault(), 
> the default is 
>  * determined by querying the system using methods in 
> TPlatformUtilities. If the 
>  * system routines fail, or if they specify a TimeZone or TimeZone 
> offset which is not 
>  * recognized, the TimeZone indicated by the ID kLastResortID is 
instantiated
>  * and made the default. 

What is kLastResourceID?  I know you copied API doc from C++ API.  But I 
think it's an error.  We do not expose anything like kLastResourceID if my 
understanding is correct.

>  * 
>  * @return   A default TimeZone. Clients are responsible for 
> deleting the time zone 
>  *           object returned. 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT UTimeZone* U_EXPORT2 
> utimezone_createDefault(void); 

I'm not sure you need to "create" a new instance.  If we exclude any 
"setters", this API can return "const UTimeZone*".

> 
> 
> /** 
>  * Disposes of the storage used by a UTimeZone object.  This function 
should 
>  * be called exactly once for objects returned by utimezone_open*. 
>  * @param zone: the object to dispose of 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT void U_EXPORT2 
> utimezone_close(UTimeZone* zone); 
> 
> 
> /** 
>  * Clones TimeZone objects polymorphically. Clients are responsible 
> for deleting 
>  * the TimeZone object cloned. 
>  * @param zone: the time zone to clone 
>  * @return   A new copy of this TimeZone object. 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT UTimeZone* U_EXPORT2 
> utimezone_clone(UTimeZone* zone); 
> 

I do not think we need this function.  In which case you actually want to 
create a copy?

> 
> /** 
>  * The GMT time zone has a raw offset of zero and does not use daylight 
>  * savings time. This is a commonly used time zone. 
>  * @param: zone: the time zone to use 
>  * @return the GMT time zone. 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT UTimeZone* U_EXPORT2 
> utimezone_getGMT(UTimeZone* zone); 
> 

I do not understand how the input arg "UTimeZone* zone" is used.
I think this API should be "const UTimeZone* utimezone_getGMT(void)"

> 
> /** 
>  * Returns the number of IDs in the equivalency group that 
>  * includes the given ID.  An equivalency group contains zones 
>  * that have the same GMT offset and rules. 
>  * 
>  * <p>The returned count includes the given ID; it is always >= 1. 
>  * The given ID must be a system time zone.  If it is not, returns 
>  * zero. 
>  * 
>  * @param: zone: the time zone to use 
>  * @param id: a system time zone ID 
>  * @param idLength: length of the id 
>  * @return the number of zones in the equivalency group containing 
>  * 'id', or zero if 'id' is not a valid system ID 
>  * @see #getEquivalentID 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT int32_t U_EXPORT2 
> utimezone_countEquivalentIDs(UTimeZone* zone, 
>                              const UChar* ID, 
>                              int32_t* IDLength); 
> 
> 
> /** 
>  * Returns an ID in the equivalency group that 
>  * includes the given ID.  An equivalency group contains zones 
>  * that have the same GMT offset and rules. 
>  * 
>  * <p>The given index must be in the range 0..n-1, where n is the 
>  * value returned by <code>countEquivalentIDs(id)</code>.  For 
>  * some value of 'index', the returned value will be equal to the 
>  * given id.  If the given id is not a valid system time zone, or 
>  * if 'index' is out of range, then returns an empty string. 
>  * 
>  * @param: zone: the time zone to use 
>  * @param id: a system time zone ID 
>  * @param idLength: the length of the id 
>  * @param index a value from 0 to n-1, where n is the value 
>  * returned by <code>countEquivalentIDs(id)</code> 
>  * @param: equivId: the returned id of the index-th zone in the 
>  * equivalency group containing 'id', or an empty string if 'id' is 
> not a valid 
>  * system ID or 'index' is out of range 
>  * @param: equivId: the length of the retuned equiv id.  If equivId is 
null, 
>  * this will contain the number of bytes to contain the equivId. 
>  * 
>  * @see #countEquivalentIDs 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT void U_EXPORT2 
> utimezone_getEquivalentID(UTimeZone* zone, 
>                           const UChar* id, 
>                           int32_t* idLength, 
>                           int32_t index, 
>                           UChar* equivId, 
>                           int32_t* equivIdLength); 
> 
> 

The semantics of "equivalent" is somewhat ambiguous.  In the existing C++ 
implementation, "equivalent" means that one time zone is a "Link" of 
another time zone in the tz database.  But, we also have another 
"equivalent" derrived from CLDR alias mapping.  Unfortunately, they are 
not guranteed to match.  I think "Link" in the tz database just specify 
that one zone use the same set of rules with another.  Technically, you 
can specify these two time zones without using "Link" statement.  In this 
case, ICU TimeZone implementation assume they are not equivalent.  Thus, 
even now, C++ implementation has some potential problems.

I do not think this API is really important for most of ICU users.  Of 
course, we cannot remove the APIs in TimeZone C++ implementation.  But, I 
do not think it is worth adding this to C API set.  So I propose to remove 
these from the proposal.


> /** 
>  * Sets the default time zone (i.e., what's returned by 
> createDefault()) to be the 
>  * specified time zone.  If NULL is specified for the time zone, the
> default time 
>  * zone is set to the default host time zone.  This call adopts the 
> TimeZone object 
>  * passed in; the clent is no longer responsible for deleting it. 
>  * 
>  * @param zone: the time zone to use 
>  * @param dftZone:  A pointer to the new TimeZone object to use as 
> the default. 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT void U_EXPORT2 
> utimezone_adoptDefault(UTimeZone* zone, 
>                        UTimeZone* dftZone); 
> 
> 
> 
> /** 
>  * Same as adoptDefault(), except that the TimeZone object passed in
> is NOT adopted; 
>  * the caller remains responsible for deleting it. 
>  * 
>  * @param zone: the time zone to use 
>  * @param dftZone: a pointer to the TimeZone object to use as the 
default. 
>  * @system 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT void U_EXPORT2 
> utimezone_setDefault(UTimeZone* zone, 
>                      const UTimeZone* dftZone); 
> 
> 

First of all, these API signatures are wrong.  These APIs simply take one 
"UTimeZone*" argument.

Also, I feel these APIs are not critical.  I do not see practical use case 
to "set" default time zone in regular applications.  Even we want such, 
one is good enough.

> /** 
>  * Returns the timezone data version currently used by ICU. 
>  * @param zone: the time zone to use 
>  * @param status: Output param to filled in with a success or an error. 
>  * @return the data version 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT const char* U_EXPORT2 
> utimezone_getTZDataVersion(UTimeZone* zone, 
>                            UErrorCode* status); 
> 
>

We alredy have ucal_getTZDataVersion.  This API also does not make sense - 
how do you use the first argument "UTimeZone* zone"?
 
> /** 
>  * Returns the canonical system timezone ID or the normalized 
>  * custom time zone ID for the given time zone ID. 
>  * @param zone: the time zone to use 
>  * @param id            The input time zone ID to be canonicalized. 
>  * @param idLength:     The length of the id 
>  * @param canonicalID   Receives the canonical system time zone ID 
>  *                      or the custom time zone ID in normalized format. 

>  * @param canonicalIDLen: Length of the canonicalID.  If the canonicalID 
is 
>  *                      null this will contain the number of bytes 
needed 
>  *                      to hold the canonicalID 
>  * @param status        Recevies the status.  When the given time zone 
ID 
>  *                      is neither a known system time zone ID nor a 
>  *                      valid custom time zone ID, 
U_ILLEGAL_ARGUMENT_ERROR 
>  *                      is set. 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT void U_EXPORT2 
> utimezone_getCanonicalID(UTimeZone* zone, 
>                          const UChar* id, 
>                          int32_t* idLength, 
>                          UChar* canonicalID, 
>                          int32_t* canonicalIDLen, 
>                          UErrorCode* status); 
> 
> 
> /** 
>  * Returns the canonical system time zone ID or the normalized 
>  * custom time zone ID for the given time zone ID. 
>  * @param zone: the time zone to use 
>  * @param id            The input time zone ID to be canonicalized. 
>  * @param idLength:     The length of the id 
>  * @param canonicalID   Receives the canonical system time zone ID 
>  *                      or the custom time zone ID in normalized format. 

>  * @param canonicalIDLen: Length of the canonicalID.  If the canonicalID 
is 
>  *                      null this will contain the number of bytes 
needed 
>  *                      to hold the canonicalID 
>  * @param isSystemID    Receives if the given ID is a known system 
>  *                      time zone ID. 
>  * @param status        Recevies the status.  When the given time zone 
ID 
>  *                      is neither a known system time zone ID nor a 
>  *                      valid custom time zone ID, 
U_ILLEGAL_ARGUMENT_ERROR 
>  *                      is set. 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT void U_EXPORT2 
> utimezone_getCanonicalIDSys(UTimeZone* zone, 
>                          const UChar* id, 
>                          int32_t* idLength, 
>                          UChar* canonicalID, 
>                          int32_t* canonicalIDLen, 
>                          UBool* isSystemID, 
>                          UErrorCode* status); 
> 

We already have ucal_getCanonicalTimeZoneID.  Same question - how do you 
use the first arg "UTimeZone*" in these functions?

> 
> /** 
>  * Returns true if the two TimeZones are equal.  (The TimeZone 
> version only compares 
>  * IDs, but subclasses are expected to also compare the fields they 
add.) 

subclasses?

>  * 
>  * @param zone1: the time zone to be checked for equality 
>  * @param zone2: the time zone to be checked for equality 
>  * @return true if the test condition is met 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT UBool U_EXPORT2 
> utimezone_equals(const UTimeZone* zone1, 
>                  const UTimeZone* zone2); 
> 

I do not see much need for this API.  I would remove this from the 
proposal.

> 
> /** 
>  * Returns the time zone raw and GMT offset for the given moment 
>  * in time.  Upon return, local-millis = GMT-millis + rawOffset + 
>  * dstOffset.  All computations are performed in the proleptic 
>  * Gregorian calendar.  The default implementation in the TimeZone 
>  * class delegates to the 8-argument getOffset(). 
>  * 
>  * @param: zone: the time zone to use 
>  * @param date moment in time for which to return offsets, in 
>  * units of milliseconds from January 1, 1970 0:00 GMT, either GMT 
>  * time or local wall time, depending on `local'. 
>  * @param local if true, `date' is local wall time; otherwise it 
>  * is in GMT time. 
>  * @param rawOffset output parameter to receive the raw offset, that 
>  * is, the offset not including DST adjustments 
>  * @param dstOffset output parameter to receive the DST offset, 
>  * that is, the offset to be added to `rawOffset' to obtain the 
>  * total offset between local and GMT time. If DST is not in 
>  * effect, this value is zero; otherwise it is a positive value, 
>  * typically one hour. 
>  * @param ec input-output error code 
>  * 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT void U_EXPORT2 
> utimezone_getOffset(UTimeZone* zone, 
>                     UDate date, 
>                     UBool local, 
>                     int32_t* rawOffset, 
>                     int32_t* dstOffset, 
>                     UErrorCode* ec); 
> 
> /** 
>  * Fills in "ID" with the TimeZone's ID. 
>  * 
>  * @param zone: the time zone to use 
>  * @param ID  Receives this TimeZone's ID. 
>  * @param IDLength: the length of ID.  If ID is null 
>  * this will return the number of bytes needed for ID 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT void U_EXPORT2 
> utimezone_getID(UTimeZone* zone, 
>                 UChar* ID, 
>                 int32_t* IDLength); 
> 
> 
> /** 
>  * Sets the TimeZone's ID to the specified value.  This doesn't 
> affect any other 
>  * fields. 
>  * 
>  * @param zone: the time zone to use 
>  * @param ID  The new time zone ID. 
>  * @param IDLength: the length of ID. 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT void U_EXPORT2 
> utimezone_setID(UTimeZone* zone, 
>                 UChar* ID, 
>                 int32_t* IDLength); 
> 

I do not think we need utimezone_setID.  I strongly feel that time zone 
should not have any state.
For getID - I'm not sure.  I think it's not important.  I would remove 
both of above from the proposal.  But someone else thinks it's important, 
I can live with it.


> /** 
>  * Returns true if this zone has the same rule and offset as another 
zone. 
>  * That is, if this zone differs only in ID, if at all. 
>  * @param zone: the time zone to use 
>  * @param other the <code>TimeZone</code> object to be compared with 

Above description should be updated for C API.

>  * @return true if the given zone is the same as this one, 
>  * with the possible exception of the ID 
>  * @draft ICU 4.6 
>  */ 
> U_DRAFT UBool U_EXPORT2 
> utimezone_hasSameRules(UTimeZone* zone, 
>                        UTimeZone* other);

I think this is probably more useful than getEquivalentID.  But I do not 
think any critical use case.  I would remove this for this proposal until 
someone really requests this.



-Yoshito

It might be useful if the API did not guarantee a callback on every character, even if the implementation usually provides one. 

-Doug

On Jun 14, 2010, at 6:52 PM, Andy Heninger <andy.heninger@...> wrote:

> Review comments for progress callback on regexp find().
> 
> It looks fine.  I don't see anything that needs fixing before putting it in.
> 
> Possible tweaks:
> 
> It's doing the callback on every character of the input string being searched.  For finds() where the match engine is being invoked at each character this is probably a good thing.  For some of the other more optimized loops, where the pattern is known to only start, and the match engine invoked, at certain character(s), the test for a call-back might be a significant part of the running time. Maybe it's ok.
> 
> If you do choose to (potentially) report on each character, making the ReportFindProgress() function an inline might be good.  (For the case where there is no callback registered, to get all function calls out of the loop.)
> 
> It might be worth noting in the documentation that the callback will be invoked for functions that indirectly use find(), such as split(), some replaces
> 
> That's about it.  I think it's a good addition.
> 
>   - Andy
> 
> ------------------------------------------------------------------------------
> ThinkGeek and WIRED's GeekDad team up for the Ultimate 
> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the 
> lucky parental unit.  See the prize list and enter to win: 
> http://p.sf.net/sfu/thinkgeek-promo
> _______________________________________________
> icu-design mailing list
> icu-design@...
> To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design

I'll get the changes from our branch onto trunk today, and file a ticket with all your feedback, to be investigated soon.  The suggestion to inline the test for callback seems especially important.
Thanks all,
Michael Grady

On Jun 14, 2010, at 10:33 PM, Douglas Davidson wrote:

> It might be useful if the API did not guarantee a callback on every character, even if the implementation usually provides one. 
> 
> -Doug
> 
> On Jun 14, 2010, at 6:52 PM, Andy Heninger <andy.heninger@...> wrote:
> 
>> Review comments for progress callback on regexp find().
>> 
>> It looks fine.  I don't see anything that needs fixing before putting it in.
>> 
>> Possible tweaks:
>> 
>> It's doing the callback on every character of the input string being searched.  For finds() where the match engine is being invoked at each character this is probably a good thing.  For some of the other more optimized loops, where the pattern is known to only start, and the match engine invoked, at certain character(s), the test for a call-back might be a significant part of the running time. Maybe it's ok.
>> 
>> If you do choose to (potentially) report on each character, making the ReportFindProgress() function an inline might be good.  (For the case where there is no callback registered, to get all function calls out of the loop.)
>> 
>> It might be worth noting in the documentation that the callback will be invoked for functions that indirectly use find(), such as split(), some replaces
>> 
>> That's about it.  I think it's a good addition.
>> 
>>  - Andy
>> 
>> ------------------------------------------------------------------------------
>> ThinkGeek and WIRED's GeekDad team up for the Ultimate 
>> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the 
>> lucky parental unit.  See the prize list and enter to win: 
>> http://p.sf.net/sfu/thinkgeek-promo
>> _______________________________________________
>> icu-design mailing list
>> icu-design@...
>> To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design

I made a couple of minor fixes and added a paragraph about the before-script
weight we need in invuca.icu to make tailoring-before-first-script-letter
work with script reordering.

markus

Looks good to me.

-Yoshito

Markus Scherer <markus.icu@...> wrote on 06/09/2010 01:56:53 PM:

> Dear ICU team & users,
> 
> It looks like I did not formally propose what we had discussed, so I
> would like to now propose that we add the C++ API 
> DecimalFormatSymbols::createWithLastResortData(UErrorCode) as in 
> the attached email below.
> 
> Ticket: http://bugs.icu-project.org/trac/ticket/7392
> Designated API reviewer: Yoshito
> Please provide feedback by next Wednesday, jun16.
> 
> Sincerely,
> markus

> On Wed, Feb 3, 2010 at 1:58 PM, Markus Scherer <markus.icu@...> 
wrote:
> This is what we arrived at in today's ICU meeting:

> Add createWithLastResortData(UErrorCode) which creates a symbols 
> object with "last resort data". Internally, it would call the 
> existing initialize() function.
> 
> We would document the symbols values and say in the API doc that the
> user should set all symbols.
> 
> Here are the "last resort symbols" set by initialize(), which are 
> currently used when the normal construction fails:
> 
> void
> DecimalFormatSymbols::initialize() {
>     fSymbols[kDecimalSeparatorSymbol] = (UChar)0x2e;    // '.' 
> decimal separator
>     fSymbols[kGroupingSeparatorSymbol].remove();        //     group
> (thousands) separator
>     fSymbols[kPatternSeparatorSymbol] = (UChar)0x3b;    // ';' 
> pattern separator
>     fSymbols[kPercentSymbol] = (UChar)0x25;             // '%' percent 
sign
>     fSymbols[kZeroDigitSymbol] = (UChar)0x30;           // '0' native 0 
digit
>     fSymbols[kDigitSymbol] = (UChar)0x23;               // '#' pattern 
digit
>     fSymbols[kPlusSignSymbol] = (UChar)0x002b;          // '+' plus sign
>     fSymbols[kMinusSignSymbol] = (UChar)0x2d;           // '-' minus 
sign
>     fSymbols[kCurrencySymbol] = (UChar)0xa4;            // 'OX' 
> currency symbol
>     fSymbols[kIntlCurrencySymbol] = INTL_CURRENCY_SYMBOL_STR; // = 
> U+00A4 U+00A4
>     fSymbols[kMonetarySeparatorSymbol] = (UChar)0x2e;   // '.' 
> monetary decimal separator
>     fSymbols[kExponentialSymbol] = (UChar)0x45;         // 'E' 
exponential
>     fSymbols[kPerMillSymbol] = (UChar)0x2030;           // '%o' per mill
>     fSymbols[kPadEscapeSymbol] = (UChar)0x2a;           // '*' pad 
> escape symbol
>     fSymbols[kInfinitySymbol] = (UChar)0x221e;          // 'oo' infinite
>     fSymbols[kNaNSymbol] = (UChar)0xfffd;               // SUB NaN
>     fSymbols[kSignificantDigitSymbol] = (UChar)0x0040;  // '@' 
> significant digit
> }
> 
> markus
> 
------------------------------------------------------------------------------
> ThinkGeek and WIRED's GeekDad team up for the Ultimate 
> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the 
> lucky parental unit.  See the prize list and enter to win: 
> http://p.sf.net/sfu/thinkgeek-promo
> _______________________________________________
> icu-design mailing list
> icu-design@...
> To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design

I was just reminded I omitted a designated reviewer and deadline.
If it's OK, I'd like to pick on Andy Heninger and any feedback by Monday 6/14/10  :)

TIA, Michael Grady

On Jun 6, 2010, at 7:26 PM, Michael Grady wrote:

> Hi all,
> 
> This is a proposal for something we've implemented here at Apple, a regex "find progress" callback that can be called periodically with monotonically increasing indices into the target search string.  A regex match can take a long time due to evaluation of a complex expression (e.g. backtracking takes exponential time) or simply because of the length of the target string.  The goal is to interrupt/cancel at some target string index, and reissue the search at that index at a later time.
> 
> 	http://bugs.icu-project.org/trac/ticket/7666
> 
> I've passed the URegexFindProgressCallback model by Andy Heninger who has suggested putting the callback sites into find() methods instead of inside the matcher engine MatchAt().  Works great.  find() has its own search loop, and moves forward consistently.
> 
> As you know, there is precedent for such callbacks with a combination of MatchCallback, TimeLimit, and StackLimit notions, but for this purpose they are a bit unreliable since they are based on a fixed operation count and do not convey a good starting point for later/interrupted search as it implements the various states of a compiled pattern applied to a target string.  What we need is a callback with a target index at which it's OK to re-start a later regex search without risk of skipping a potential match.
> 
> The URegexFindProgressCallback model also diverges from URegexMatchCallback in that, when the callback interrupts a match, the internal "fDeferredStatus" [in rematch.cpp] is _not_ set to  U_REGEX_STOPPED_BY_CALLER.
> Because the intention is to re-initiate a find/findNext operation later, leaving "fDeferredStatus" set to something causes problems.
> 	1.  Several reset() operations call utext_clone which currently pass around "fDeferredStatus", which kills the cloning and cause crashes.
> 	2.  There are no accessors (setter) to reset the internal "fDeferredStatus" without resetting all the other matcher state
> 
> For these reasons, the new URegexFindProgressCallback does not set error state, except to interrupt find() with a UBool, and the last invocation of the callback gives the client the search index at which to retry, after which various operations like reset() are valid.
> 
> ---------  Here's an overview (details in the .patch files attached)
> 
> test/intltest/regextst.cpp
> test/intltest/regextst.h
> i18n/rematch.cpp
> i18n/uregex.cpp
> 
> i18n/unicode/uregex.h
> 
> 	typedef UBool U_CALLCONV URegexFindProgressCallback (
>                    const void *context,
>                    int64_t     matchIndex);
>  
> U_STABLE void U_EXPORT2
> uregex_setFindProgressCallback(URegularExpression              *regexp,
>                                 URegexFindProgressCallback      *callback,
>                                 const void                      *context,
>                                 UErrorCode                      *status);
> 
> U_STABLE void U_EXPORT2
> uregex_getFindProgressCallback(const URegularExpression          *regexp,
>                                 URegexFindProgressCallback        **callback,
>                                 const void                        **context,
>                                 UErrorCode                        *status);
> 
> i18n/unicode/regex.h
> class U_I18N_API RegexMatcher: public UObject {
> public:
>     virtual void setFindProgressCallback(URegexFindProgressCallback      *callback,
>                                               const void                              *context,
>                                               UErrorCode                              &status);
> 
>     virtual void getFindProgressCallback(URegexFindProgressCallback      *&callback,
>                                               const void                      *&context,
>                                               UErrorCode                      &status);
> 
> private:
> 
>     URegexFindProgressCallback  *fFindProgressCallbackFn;  // Pointer to match progress callback funct.
>                                                            //   NULL if there is no callback.
>     const void         *fFindProgressCallbackContext;      // User Context ptr for callback function.
> 
> ------ patch files
> 
> > cd sources/i18n/
> > patch < regex_progressCallback.patch
> >cd sources/test/intltest/
> > patch < test_progressCallback.patch
> 
> <regex_progressCallback.patch>
> <test_progressCallback.patch>
> 
>

I support this.

Mark

— Il meglio è l’inimico del bene —


On Fri, Jun 4, 2010 at 17:06, Markus Scherer <markus.icu@...> wrote:

> Dear ICU team & users,
>
> I would like to propose the following addition to the Normalizer2 class,
> for ICU 4.6:
>
> Add a public abstract getDecomposition(c) method to Normalizer2,
> independent of the normalizer's mode. Equivalent to
> normalize(UnicodeString(c)) on a UNORM2_DECOMPOSE Normalizer2 instance, but
> much faster.
>
> This would replace several uses of Normalizer2Impl::getDecomposition() in
> the ICU code with calls to the then-public Normalizer2 method.
>
> Java (returns null if no decomposition):
>
> public abstract String getDecomposition(int c);
>
> C++ (returns FALSE and leaves the string unchanged if no decomposition):
>
> UBool getDecomposition(UChar32 c, UnicodeString &decomposition) const = 0;
>
> C (returns decomposition length; returns -1 and leaves the buffer unchanged
> if no decomposition):
>
> int32_t unorm2_getDecomposition(UChar32 c, UChar *decomposition, int32_t capacity, UErrorCode *pErrorCode);
>
> Ticket: http://bugs.icu-project.org/trac/ticket/7736
> Designated API reviewer: Mark
>
> Please provide feedback by next Friday, 2010-06-11
>
> Sincerely,
> markus
>
>
>
> ------------------------------------------------------------------------------
> ThinkGeek and WIRED's GeekDad team up for the Ultimate
> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> lucky parental unit.  See the prize list and enter to win:
> http://p.sf.net/sfu/thinkgeek-promo
> _______________________________________________
> icu-design mailing list
> icu-design@...
> To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design
>