International Components for Unicode

icu-design-bounces@... wrote on 10/25/2006 07:53:46 PM:

> b) ICU has Asia/Riyadh87 and similar (3 zones with aliases) but CLDR
> does not. (tzdata files solar87..solar89)
> 
> These represent an abandoned experiment. In these zones, each day of
> 1987-1989 has its own rule, adjusting official time to astronomical
> time. Therefore, they have 365/366 transitions times each in ICU's
> zoneinfo.txt.
> 
> I propose to remove the Riyadh<yy> names from ICU.
> 

As you know, ICU uses Olson tz database data.  Although these zones seems 
not so important (because they are historic zones, not used currently), I 
can not see any strong reason to exclude them from ICU.  I think what we 
should do is - import Olson tz data first, then if there are zones which 
are only in CLDR, capture them in ICU, then ICU timezone is a super set of 
both Olson and CLDR.

-Yoshito Umaoka

We excluded them from CLDR, because they are essentially deprecated (an odd
failure), and we don't want people translating them.

Mark

On 10/26/06, yoshito_umaoka@... <yoshito_umaoka@...> wrote:
>
> icu-design-bounces@... wrote on 10/25/2006 07:53:46 PM:
>
> > b) ICU has Asia/Riyadh87 and similar (3 zones with aliases) but CLDR
> > does not. (tzdata files solar87..solar89)
> >
> > These represent an abandoned experiment. In these zones, each day of
> > 1987-1989 has its own rule, adjusting official time to astronomical
> > time. Therefore, they have 365/366 transitions times each in ICU's
> > zoneinfo.txt.
> >
> > I propose to remove the Riyadh<yy> names from ICU.
> >
>
> As you know, ICU uses Olson tz database data.  Although these zones seems
> not so important (because they are historic zones, not used currently), I
> can not see any strong reason to exclude them from ICU.  I think what we
> should do is - import Olson tz data first, then if there are zones which
> are only in CLDR, capture them in ICU, then ICU timezone is a super set of
> both Olson and CLDR.
>
> -Yoshito Umaoka
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> icu-design mailing list
> icu-design@...
> To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design
>

It should also be noted that these deprecated timezones are usable Olson=20
timezone based systems, like Linux.  They don't need to be translated, but =

they don't need to be removed either.  I have a test to verify that all=20
timezones on a system work under ICU, and it would be nice if I didn't get =

any spurious test failures due to missing timezones.

George Rhoten
IBM Globalization Center of Competency/ICU  San Jos=E9, CA, USA
http://www.icu-project.org/
http://icu.sourceforge.net/



"Mark Davis" <mark.davis@...
Sent by: icu-design-bounces@...
10/26/2006 07:53 AM
Please respond to
icu-design@...


To
icu-design@...
cc

Subject
Re: [icu-design] ICU proposal: better alignment of time zone names ICU vs. =

CLDR






We excluded them from CLDR, because they are essentially deprecated (an=20
odd failure), and we don't want people translating them.

Mark

On 10/26/06, yoshito=5Fumaoka@... <yoshito=5Fumaoka@...> wrot=
e:
icu-design-bounces@... wrote on 10/25/2006 07:53:46 PM:

> b) ICU has Asia/Riyadh87 and similar (3 zones with aliases) but CLDR
> does not. (tzdata files solar87..solar89)=20
>
> These represent an abandoned experiment. In these zones, each day of
> 1987-1989 has its own rule, adjusting official time to astronomical
> time. Therefore, they have 365/366 transitions times each in ICU's=20
> zoneinfo.txt.
>
> I propose to remove the Riyadh<yy> names from ICU.
>

As you know, ICU uses Olson tz database data.  Although these zones seems
not so important (because they are historic zones, not used currently), I=20
can not see any strong reason to exclude them from ICU.  I think what we
should do is - import Olson tz data first, then if there are zones which
are only in CLDR, capture them in ICU, then ICU timezone is a super set of =


both Olson and CLDR.

-Yoshito Umaoka

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job=20
easier=20
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D1=
21642=20
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
icu-design mailing list
icu-design@...
To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job=20
easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D1=
21642
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
icu-design mailing list
icu-design@...
To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design

As I see it, for the Riyadh<yy> zones to be used, someone would have
to select the right zone (e.g., Riyadh87) for the right year (e.g.,
1987). There are only three of them (87..89). You don't get this
behavior by selecting the regular Asia/Riyadh time zone. In other
words, I think these are pretty useless. (And they take up some 11.5kB
in zoneinfo.res, some 8% of the total.)

Having said that, Yoshito is right that these may not count as strong
reasons to drop the Riyadh<yy> zones.

If we remove this part b) of the proposal, we are left with
a) add SystemV/XXX
c) add Etc/Unknown
d) remove Factory

markus

On 10/26/06, Mark Davis <mark.davis@...> wrote:
> We excluded them from CLDR, because they are essentially deprecated (an odd
> failure), and we don't want people translating them.
>
> On 10/26/06, yoshito_umaoka@... <yoshito_umaoka@...> wrote:
> > > I propose to remove the Riyadh<yy> names from ICU.
> >
> > As you know, ICU uses Olson tz database data.  Although these zones seems
> > not so important (because they are historic zones, not used currently), I
> > can not see any strong reason to exclude them from ICU.  I think what we
> > should do is - import Olson tz data first, then if there are zones which
> > are only in CLDR, capture them in ICU, then ICU timezone is a super set of
> > both Olson and CLDR.

-- 
Opinions expressed here may not reflect my company's positions unless
otherwise noted.

On 10/26/06, George Rhoten <grhoten@...> wrote:
> ... I have a test to verify that all
> timezones on a system work under ICU, and it would be nice if I didn't get
> any spurious test failures due to missing timezones.

Good! (Is the test in the public repository? Is it part of the BRS?)

Could you please try to remove the Factory time zone from zoneinfo.txt
and see if you get any test failures?

,  /* Factory */ :array {
    :intvector {  }
    :intvector { 0, 0 }
    :bin { "" }
  } //Z#458

Thanks,
markus

The first incarnation of the test has issues.  Vladimir probably remembers =

the person that first worked on it.  I'm going to rewrite the test so that =

it doesn't require you to change your system time, which usually requires=20
root access.

I'm going to check in the new test under jitterbug 5423.  The concept of=20
the new test probably won't change very much.  It iterates through all=20
timezones installed on a system, and it compares ICU's local time against=20
the system local time.  You can view jitterbug 5423 for a longer=20
description of the problem I'm trying to fix.

George Rhoten
IBM Globalization Center of Competency/ICU  San Jos=E9, CA, USA
http://www.icu-project.org/
http://icu.sourceforge.net/



"Markus Scherer" <markus.icu@...
Sent by: icu-design-bounces@...
10/26/2006 09:01 AM
Please respond to
icu-design@...


To
icu-design@...
cc

Subject
Re: [icu-design] ICU proposal: better alignment of time zone names ICU vs. =

CLDR






On 10/26/06, George Rhoten <grhoten@...> wrote:
> ... I have a test to verify that all
> timezones on a system work under ICU, and it would be nice if I didn't=20
get
> any spurious test failures due to missing timezones.

Good! (Is the test in the public repository? Is it part of the BRS?)

Could you please try to remove the Factory time zone from zoneinfo.txt
and see if you get any test failures?

,  /* Factory */ :array {
    :intvector {  }
    :intvector { 0, 0 }
    :bin { "" }
  } //Z#458

Thanks,
markus

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job=20
easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D1=
21642
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
icu-design mailing list
icu-design@...
To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design

It looks like the "Factory" time zone name is "supported" on Linux.
When used with the Linux date command, it prints the "Local time zone
must be set--see zic manual page" string from the tzdata factory file
as the time zone display name (and behaves like GMT). It is "more
supported" than a bogus name like "foo".

$ date
Thu Oct 26 10:17:55 PDT 2006
$ TZ=Europe/Berlin date
Thu Oct 26 19:18:04 CEST 2006
$ TZ=Asia/Amman date
Thu Oct 26 19:18:18 EET 2006
$ TZ=Factory date
Thu Oct 26 17:18:27 Local time zone must be set--see zic manual page 2006
$ TZ=foo date
Thu Oct 26 17:25:43 foo 2006

I guess that would argue for keeping the "Factory" time zone name in ICU.

We would be left with
a) add SystemV/XXX
c) add Etc/Unknown

Opinions?

markus

Andy and I are against this API proposal.  We both think that it=20
complicates the overall API, and it exposes too much of the internals that =

don't need to be exposed.  General end users won't normally be using this=20
API, and knowledge about when to use it is not clear (yes, I did read the=20
API documentation).  Andy can elaborate more if he wants to do that.

My personal opinion about this API is that it's not needed, and the ICU=20
transliterator code that should be using this API can't use it because it=20
changes how the rules come out of a transliterator.  This causes some=20
transliterator test failures.  At the moment, I don't see a real problem=20
being solved by this API proposal.  It can be proposed later when we have=20
a concrete need for it.

Also ICU4C already has a several release functions (see UnicodeSet,=20
UnicodeString and some charset conversion code).  So when we decide that=20
we need this function, it should be called releasePattern and not=20
clearPattern.

George Rhoten
IBM Globalization Center of Competency/ICU  San Jos=E9, CA, USA
http://www.icu-project.org/
http://icu.sourceforge.net/



"Mark Davis" <mark.davis@...
Sent by: icu-design-bounces@...
10/14/2006 11:27 AM
Please respond to
icu-design@...


To
icu-design@...
cc

Subject
[icu-design] API proposal






There is currently no way to clear a pattern. There are two times you want =

to do this; one is to remove the extra storage, and the other is when you=20
want the resulting pattern to be expressed as ranges instead of=20
combinations of properties. The following proposes having an extra method=20
-- the alternative is to change compact(). That has the disadvantage that=20
it doesn't allow someone to compact without throwing away the pattern, but =

the issue needs further dicsussion.=20

    /**
     * Clears the pattern used to create this set, if there was one. The=20
effect
     * of this is that the next time that toString() or toPattern() is=20
called,
     * the pattern will be constructed. Thus:=20
     * <pre>
     * System.out.println(new UnicodeSet(&quot;[:whitespace:]&quot;));
     * System.out.println(new=20
UnicodeSet(&quot;[:whitespace:]&quot;).clearPattern());
     * </pre>=20
     * produces:
     * <pre>
     * [:whitespace:]
     *=20
[\\u0009-\\u0085\\u00A0\\u1680\\u180E\\u2000-\\u200A\\u2028\\u2029\\u202F\\=
u205F\\u3000]
     * </pre>
     * @return this, for chaining=20
     * @draft
     */
    public final UnicodeSet clearPattern() {
      pat =3D null;
      return this;
    }
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job=20
easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat=3D1=
21642
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
icu-design mailing list
icu-design@...
To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design

On 10/14/06, Mark Davis <mark.davis@...> wrote:
> There is currently no way to clear a pattern. There are two times you want
> to do this; one is to remove the extra storage, and the other is when you
> want the resulting pattern to be expressed as ranges instead of combinations
> of properties.

I'm not wholly convinced this proposed new function is a good idea.

1.   The existing function generatePattern() will already produce the
pattern without reference to the saved initial pattern.

2.  The memory saving aspect of the proposed function is not something
that users of the class should have to think about.  I suspect that
memory consumption isn't normally any kind of issue for UnicodeSets,
but if it turns out to be a substantial issue after all, we should
find an approach that don't require API and user intervention.

3.  The UnicodeSet API is really big for what it does, so much so that
the fundamental operations are somewhat obscured.  I think that it's
time to stop being quite so free in adding everything that seems like
it might possibly be useful in some context.

  -- Andy


> The following proposes having an extra method -- the
> alternative is to change compact(). That has the disadvantage that it
> doesn't allow someone to compact without throwing away the pattern, but the
> issue needs further dicsussion.
>
>     /**
>      * Clears the pattern used to create this set, if there was one. The
> effect
>      * of this is that the next time that toString() or toPattern() is
> called,
>      * the pattern will be constructed. Thus:
>      * <pre>
>      * System.out.println(new
> UnicodeSet(&quot;[:whitespace:]&quot;));
>      * System.out.println(new
> UnicodeSet(&quot;[:whitespace:]&quot;).clearPattern());
>      * </pre>
>      * produces:
>      * <pre>
>      * [:whitespace:]
>      *
> [\\u0009-\\u0085\\u00A0\\u1680\\u180E\\u2000-\\u200A\\u2028\\u2029\\u202F\\u205F\\u3000]
>      * </pre>
>      * @return this, for chaining
>      * @draft
>      */
>     public final UnicodeSet clearPattern() {
>       pat = null;
>       return this;
>     }
>

On 10/16/06, Andy Heninger <andy.heninger@...> wrote:
> [...]  we should
> find an approach that don't require API and user intervention.

Err, doesn't.

The hazards of editing and not reading.

On 10/5/06, Deborah Goldsmith <goldsmit@...> wrote:
> Since RuleBasedBreakIterator::checkDictionary will become protected
> in ICU4C 3.8, it would be a good idea to fix it to handle 64-bit
> offsets, so that the implementation of RBBI can eventually handle 64
> bit offsets.

The thing is, nothing  else  in the break iterator  API can deal with
64 bit index positions,  so are we actually gaining anything by
enabling them for this one function only?

  -- Andy


> I propose changing the signature from:
>
>      int32_t checkDictionary(int32_t startPos, int32_t endPos, UBool
> reverse);
>
> to
>
>      int64_t checkDictionary(int64_t startPos, int64_t endPos, UBool
> reverse);
>
> Deborah
>

sounds good.

---------- Forwarded message ----------
From: Deborah Goldsmith <goldsmit@...>
Date: Oct 5, 2006 8:19 PM
Subject: [icu-design] API proposal: change signature for
RBBI::checkDictionary for 64-bit
To: icu-design@...

Since RuleBasedBreakIterator::checkDictionary will become protected
in ICU4C 3.8, it would be a good idea to fix it to handle 64-bit
offsets, so that the implementation of RBBI can eventually handle 64
bit offsets. I propose changing the signature from:

     int32_t checkDictionary(int32_t startPos, int32_t endPos, UBool
reverse);

to

     int64_t checkDictionary(int64_t startPos, int64_t endPos, UBool
reverse);

Deborah


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
icu-design mailing list
icu-design@...
To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design

The thing we're gaining is not having to introduce a 64-bit version  
down the road. It's one less thing to clean up.

Also, because it's called internally, having two versions would make  
the implementation of RBBI::handleNext() and RBBI::handlePrevious()  
somewhat problematic. This way, we can avoid proliferating  
implementations.

I agree it doesn't solve the entire problem, but it solves one tiny  
little bit.

Deborah

On Oct 6, 2006, at 10:38 AM, Andy Heninger wrote:

> On 10/5/06, Deborah Goldsmith <goldsmit@...> wrote:
>> Since RuleBasedBreakIterator::checkDictionary will become protected
>> in ICU4C 3.8, it would be a good idea to fix it to handle 64-bit
>> offsets, so that the implementation of RBBI can eventually handle 64
>> bit offsets.
>
> The thing is, nothing  else  in the break iterator  API can deal with
> 64 bit index positions,  so are we actually gaining anything by
> enabling them for this one function only?
>
>   -- Andy
>
>
>> I propose changing the signature from:
>>
>>      int32_t checkDictionary(int32_t startPos, int32_t endPos, UBool
>> reverse);
>>
>> to
>>
>>      int64_t checkDictionary(int64_t startPos, int64_t endPos, UBool
>> reverse);
>>
>> Deborah
>>
>
> ---------------------------------------------------------------------- 
> ---
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to  
> share your
> opinions on IT & business topics through brief surveys -- and earn  
> cash
> http://www.techsay.com/default.php? 
> page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> icu-design mailing list
> icu-design@...
> To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu- 
> design

On 10/5/06, George Rhoten <grhoten@...> wrote:
> I'm not sure I follow the reasoning behind the change.

Proximity and convenience. Not a technical must, but a readability request.

Technically harmless because the earlier-defined set of macros won't
work without the later-defined U_ICU_NAMESPACE.

> When you define
> U_USING_ICU_NAMESPACE to 0, you disable the using namespace declaration.
> There shouldn't be anything else you should be doing.

With this set to 0, U_NAMESPACE_USE disappears from uversion.h, but we
still use the begin/end/qualifier macros. (We are turning off the
"using" directive, but we are not turning off the namespace.)

When I was hunting down a related bug in unistr.h, I went to
uversion.h to get the spelling of the namespace qualifier macro, but
it's not defined there.

> Why are you doing so much searching?  Am I missing certain steps during
> your modification?

You are missing steps my brain takes in finding which part of the
namespace puzzle is defined in which header file.

markus

On 10/5/06, Markus Scherer <markus.icu@...> wrote:
> On 10/5/06, George Rhoten <grhoten@...> wrote:
> > I'm not sure I follow the reasoning behind the change.
>
> Proximity and convenience. Not a technical must, but a readability request.

I vote for the change.

I always have trouble keeping straight in my head where all of our
various configuration macros are defined and how they interact.
Anything that makes things easier to figure out is a Good Thing.

  --- Andy


>
> Technically harmless because the earlier-defined set of macros won't
> work without the later-defined U_ICU_NAMESPACE.
>
> > When you define
> > U_USING_ICU_NAMESPACE to 0, you disable the using namespace declaration.
> > There shouldn't be anything else you should be doing.
>
> With this set to 0, U_NAMESPACE_USE disappears from uversion.h, but we
> still use the begin/end/qualifier macros. (We are turning off the
> "using" directive, but we are not turning off the namespace.)
>
> When I was hunting down a related bug in unistr.h, I went to
> uversion.h to get the spelling of the namespace qualifier macro, but
> it's not defined there.
>
> > Why are you doing so much searching?  Am I missing certain steps during
> > your modification?
>
> You are missing steps my brain takes in finding which part of the
> namespace puzzle is defined in which header file.
>
> markus
>

It's the Hanyu pinyin. If you are interested in adding the TW variant, that
can be done. These are drawn from CLDR, so file a bug there. (
unicode.org/cldr)

Mark

On 10/6/06, Cindy Conlin <ConlinC@...> wrote:
>
>  Hello,
> I see that one of the Transforms supported by ICU is NumericPinyin-Latin.
>
> My question is: which Pinyin variant is supported by this Transform?
> There's the Hanyu Pinyin adopted by the PRC, and also a Tongyon Pinyin used
> in Taiwan  (see http://en.wikipedia.org/wiki/Pinyin#Pinyin_in_Taiwan).
> I'm guessing that ICU supports the Hanyu Pinyin, but just wanted to check to
> be sure.
>
> Thanks in advance for your help!
> Cindy
> ------------------------------
> NOTICE: This email message is for the sole use of the intended
> recipient(s) and may contain confidential and privileged information. Any
> unauthorized review, use, disclosure or distribution is prohibited. If you
> are not the intended recipient, please contact the sender by reply email and
> destroy all copies of the original message.
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share
> your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>
> _______________________________________________
> icu-design mailing list
> icu-design@...
> To Un/Subscribe: https://lists.sourceforge.net/lists/listinfo/icu-design
>
>