[Sbcl-help] SBCL 0.8.17 binaries

[Brian Mastenbrook <bmastenb@cs...>]

Hello all,

It's time to make the binaries for SBCL 0.8.17, and because of the new 
sb-unicode feature, I need to decide how to make the binaries. Right 
now, sb-md5 fails its tests on a sb-unicode binary. The way I see it, 
there are five options for how to make the binaries:

1. Don't make any.
2. Make the binaries without sb-unicode.
Or, if we make the binaries with unicode, there are two options:
3. Touch test-passed in the sb-md5 directory and ship it, knowing that 
users who use sb-md5 will see their MD5 computed values change.
4. Don't touch test-passed and distribute without sb-md5.
5. Some combination of #3 or #4 and #2.

My personal feeling is that option #3 is the right option: I'd like to 
ship with unicode to get users testing, and it doesn't make sense to 
not ship sb-md5 just because the tests haven't been fixed. However, I 
could see an argument for #2 - waiting until there are encode and 
decode to string functions before shipping binaries with unicode. #5 
would be a lot of work.

Advice?
--
Brian Mastenbrook
http://www.iscblog.info/
http://www.cs.indiana.edu/~bmastenb/

[Sbcl-help] SBCL 0.8.17 binaries

[Brian Mastenbrook <bmastenb@cs...>]

Hello all,

It's time to make the binaries for SBCL 0.8.17, and because of the new 
sb-unicode feature, I need to decide how to make the binaries. Right 
now, sb-md5 fails its tests on a sb-unicode binary. The way I see it, 
there are five options for how to make the binaries:

1. Don't make any.
2. Make the binaries without sb-unicode.
Or, if we make the binaries with unicode, there are two options:
3. Touch test-passed in the sb-md5 directory and ship it, knowing that 
users who use sb-md5 will see their MD5 computed values change.
4. Don't touch test-passed and distribute without sb-md5.
5. Some combination of #3 or #4 and #2.

My personal feeling is that option #3 is the right option: I'd like to 
ship with unicode to get users testing, and it doesn't make sense to 
not ship sb-md5 just because the tests haven't been fixed. However, I 
could see an argument for #2 - waiting until there are encode and 
decode to string functions before shipping binaries with unicode. #5 
would be a lot of work.

Advice?
--
Brian Mastenbrook
http://www.iscblog.info/
http://www.cs.indiana.edu/~bmastenb/

[Sbcl-help] Re: [Sbcl-devel] SBCL 0.8.17 binaries

[Christophe Rhodes <csr21@ca...>]

Brian Mastenbrook <bmastenb@...> writes:

> 1. Don't make any.
> 2. Make the binaries without sb-unicode.
> Or, if we make the binaries with unicode, there are two options:
> 3. Touch test-passed in the sb-md5 directory and ship it, knowing that
> users who use sb-md5 will see their MD5 computed values change.
> 4. Don't touch test-passed and distribute without sb-md5.
> 5. Some combination of #3 or #4 and #2.
>
> My personal feeling is that option #3 is the right option: I'd like to
> ship with unicode to get users testing, and it doesn't make sense to
> not ship sb-md5 just because the tests haven't been fixed. However, I
> could see an argument for #2 - waiting until there are encode and
> decode to string functions before shipping binaries with unicode. #5
> would be a lot of work.

Again, speaking personally, I'd rather #4; that way, if anyone feels
that they need sb-md5 in their binaries, they have an incentive to get
down and make it happen.  I see #3 as dishonest -- what is the point
of having tests for contrib modules which we have always said have a
slightly less supported status than the main body of code, if at the
least provocation we circumvent those tests?  It's not just a question
of fixing the tests: the interface itself is broken, and despite two
calls for contributions remains so.

I believe Kevin has chosen option #2 for his Debian uploads -- in some
sense, I think that covers that base; it seems to me that #4 is the
option that best reflects the state of SBCL development at release
time.

Cheers,

Christophe

[Sbcl-help] Re: [Sbcl-devel] SBCL 0.8.17 binaries

[Adam Warner <lists@co...>]

Hi Christophe Rhodes,

> I believe Kevin has chosen option #2 for his Debian uploads -- in some
> sense, I think that covers that base; it seems to me that #4 is the
> option that best reflects the state of SBCL development at release
> time.

Indeed I've just installed SBCL 0.8.17-2 from Debian Unstable and
unfortunately it doesn't have large character support compiled in.

I find 8 bit strings that can store arbitrary octets useful and I'm
wondering if you will consider continuing support for them in preference
to the ASCII support you've mooted for the FFI. The fantastic property of
ISO-8859-1 and Unicode is that the first 256 characters of Unicode
map exactly onto ISO-8859-1. So if you define BASE-CHAR to be octets
(of ostensibly ISO-8859-1 encoded characters) then EXTENDED-CHAR naturally
maps from character code 256 onwards. In practice the only two useful
encodings will be BASE-CHAR and CHARACTER.

I have one situation where I'm encoding binary within strings over a
socket connection that can only transfer text. The lowest common
denominator is ASCII so I have 7 bits (effectively 6 bits to avoid control
characters). If the implementation supports strings of octets then I'm
only using (8-6)/6 = 33% more memory. If one implementation only supports
32 bit characters then I'm using (32-6)/6 = 433% more memory.

It also sounds like ongoing support for strings of octets would help the
porting of sb-md5 (add :element-type 'base-char when making strings).

People could also store UTF-8 encoded text in the strings of octets.
There's no decoding overhead. You just store the octets as read instead of
the implementation having to decode the stream of octets and store
them as code points within the CHARACTER data structure.

It's also possible to encode extra information within the string such as
an escape character. Some values in UTF-8 are undefined such as 255.
Such strings with undefined UTF-8 sequences can be portably transferred as
ISO-8859-1 while additionally interpreting and unescaping the UTF-8
sequence at the other end.

There are many reasons why strings of octets would continue to be
useful in an implementation that has Unicode support. This advice may be
helpful but feel free to ignore it without explanation.

Regards,
Adam