clisp-devel

Re: UTF-16 min/max_ bytes_per_char must be 2

[Hoehle, Joerg-C. <Joe...@t-...>]

Hi

I'm restarting a very old thread:

Bruno Haible wrote on 21. Mai 2002
>> > Obviously, having correct min/max is the "better" solution.
>Yes, it would, but it is too much maintenance overhead, given that
>iconv by itself doesn't provide this information.

It's time to raise this thread again because whoever (Sam?) added the non-1:1 support to the FFI will run into this issue.
For example, encoding.d:SET-FOREIGN-ENCODING has code checking max_bytes_per_char.

My observation & report, back in 2002, was that those min/max fields of the encoding structure were not all reliable. Bruno's more precise answer IIRC was that they were ok for CLISP's built-in encodings and junk for iconv() based ones.

That's why I invented SYS::ENCODING-ZEROES, which I needed for WITH-FOREIGN-STRING.

Now, in 2005, what's changed? Is SET-FOREIGN-ENCODING operating on sensible data?
a) nothing changed, thus code must not depend on min/max_bytes_per_char
   and the 1:1 check set-foreign-encoding does is questionable
   -- or needs precise documentation about why it is expected to work.
b) the min/max fields are ok nowadays, thus encoding-zeroes belongs to the past.
   Then, WITH-FOREIGN-STRING could take advantage of these fields.

Looking at encoding.d, I'd guess situation a) applies.

Regards,
	Jorg Hohle.

Re: UTF-16 min/max_ bytes_per_char must be 2

[Hoehle, Joerg-C. <Joe...@t-...>]

Samm Steingold wrote:
>my position holds:
>ENCODING-ZEROES must be implemented in C and use & update
>min/max_bytes_per_char.
This sounds reasonable.

There's another issue.
The FFI will likely have to document that (C-ARRAY CHARACTER N) is measured in bytes, which can cause some surprises to people manipulating only e.g. UTF-16. It's rather different from (C-ARRAY UINT16 N) and may cause surprises!

Yet I find it reasonable to have it set at a known size and not dependent on the current encoding. That would be worrying.

Regards,
	Jorg Hohle.

Re: UTF-16 min/max_ bytes_per_char must be 2

[Sam S. <sd...@gn...>]

> * Hoehle, Joerg-Cyril <Wbr...@g-...> [2005-06-24 15:34:23 +0200]:
>
> Bruno Haible wrote on 21. Mai 2002
>>> > Obviously, having correct min/max is the "better" solution.
>>Yes, it would, but it is too much maintenance overhead, given that
>>iconv by itself doesn't provide this information.
>
> It's time to raise this thread again because whoever (Sam?) added the
> non-1:1 support to the FFI will run into this issue.

2004-12-24  Bruno Haible  <br...@cl...>

	Support multibyte encodings in FFI:*FOREIGN-ENCODING*.
	* encoding.d (init_dependent_encodings): Remove warning if
	*FOREIGN-ENCODING* is not 1:1. Initialize O(foreign_8bit_encoding).
	(SYSTEM::SET-FOREIGN-ENCODING): Update O(foreign_8bit_encoding).
	* foreign.d (convert_from_foreign_array_fill): For characters, use
	O(foreign_8bit_encoding).
	(convert_from_foreign): Likewise. Add special conversion routines for
	(C-ARRAY CHARACTER dim1), (C-ARRAY-MAX CHARACTER maxdim),
	(C-ARRAY-PTR CHARACTER).
	(count_walk_pre): For C-STRING and (C-ARRAY-PTR CHARACTER ...), compute
	the amount of needed memory.
	(convert_to_foreign): For characters, use O(foreign_8bit_encoding).
	Add special conversion routines for (C-ARRAY CHARACTER dim1),
	(C-ARRAY-MAX CHARACTER maxdim), (C-ARRAY-PTR CHARACTER).

> That's why I invented SYS::ENCODING-ZEROES, which I needed for
> WITH-FOREIGN-STRING.

my position holds:
ENCODING-ZEROES must be implemented in C and use & update
min/max_bytes_per_char.

specifically:

1. ENCODING-ZEROES should be removed
2. code similar to that in ENCODING-ZEROES should be invoked by
   MAKE-ENCODING and fill the min/max_bytes_per_char slots.
3. call-with-foreign-string should not accept "extra-zeroes" but use
   min/max_bytes_per_char.

the more radical approach would be to incorporate the extra-zeroes into
cstombs at al.

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
<http://www.memri.org/> <http://www.dhimmi.com/> <http://www.camera.org>
The world will end in 5 minutes.  Please log out.