Re: [Sbcl-devel] unicode support

[William H. N. <wil...@ai...>]

On Mon, Sep 24, 2001 at 01:05:49PM +1000, Brian Spilsbury wrote:
> I've just implemented some preliminary unicode support in sbcl, so far 
> this consists of;
> 
> Adding a new immediate primitive type UNICODE-CHAR.
> 
> Adding a bunch of functions, and patching a bunch of existing ones with (etypecase) mostly.

Neat.
 
> In the longer term I'm tempted to look at phasing out base-char 
> completely, and look at perhaps a (character 0 255) type to replace it, 
> along a similar line to (integer 0 255). Then we can define UCS-2  (ugly 
> thing) as (char 0 #xffff), etc.

BASE-CHAR is required by the ANSI standard, so I don't think it should
be phased out (?). Perhaps you mean eliminating the distinction between
BASE-CHAR and this (CHARACTER #.(CODE-CHAR 0) #.(CODE-CHAR 255)) type?
or eliminating the distinction between BASE-CHAR and UNICODE-CHAR?

If UCS-2 is the scheme where special digraphs are used to refer to
characters numbered >64K, I'm not sure it'll be practical to support
it cleanly in Common Lisp strings. The assumption of uniform indexing
of characters seems to be built pretty deeply into the Common Lisp
string operations. But that's just my superficial impression, not
something I've thought about deeply.

> Unicode-Char is currently 21 bit, and covers the range of 0-#x10FFFF as 
> per the unicode 3.1 specification.

That should fit nicely into the Common Lisp string operations, I'd
think, although of course it will tend to make strings a little large.
 
> When this settles down a little I'm intending to implement utf-8 string 
> support, and general immutable string/vector support. Fix the reader to 
> handle unicode properly, add input method support, etc.

What extensions do you plan to use to express immutable strings and 
vectors? Is this a feature set in some other Lisp that you want to port,
or have you designed it yourself?

> What do people think wrt Base-Char and who should I talk to about vops?

I think at this point Dan Barlow may know more about VOPs than anyone
else. And anyone who's worked on a port of SBCL or CMU CL from one CPU
architecture to another probably knows more than me, but I might be
able to answer some questions too.

-- 
William Harold Newman <wil...@ai...>
"Sometimes if you have a cappuccino and then try again it will work OK."
  -- Dr. Brian Reid, 1992, quoted by mjr
"Sometimes one cappucino isn't enough."
  -- mjr = Marcus J. "will do TCP/IP for food" Ranum <mj...@hu...>
PGP key fingerprint 85 CE 1C BA 79 8D 51 8C  B9 25 FB EE E0 C3 E5 7C