Steel Bank Common Lisp

On Fri, May 27, 2005 at 12:35:58PM +0100, Gustavo Felisberto wrote:
> http://www.sbcl.org/manual/Threading.html#Threading
> 
> that in itself is a bit contradictory as it says it needs a NPTL system
> and then says:
> "On Linux x86, threading is implemented using |clone()| and does not
> involve pthreads. This is not because there is anything wrong with

This is not contradictory at all.  It requires NPTL because pre-NPTL
systems don't (usually) have futex support - and it's more likely
that people know whether they have NPTL than whether they have a
kernel that supports some syscall they've probably never heard of.

> pthreads /per se/, but there is plenty wrong (from our perspective) with
> LinuxThreads. SBCL threads are mapped 1:1 onto Linux tasks which share a
> VM but nothing else - each has its own process id and can be seen in
> e.g. |ps| output."
> in
> http://www.sbcl.org/manual/Implementation--Linux-x86-.html#Implementation%20(Linux%20x86)
> 
> which is not true.

What's not true?

* (sb-thread:make-thread (lambda () (sleep 300)))

12554
* (sb-thread:make-thread (lambda () (sleep 300)))

12555
* (sb-thread:make-thread (lambda () (sleep 300)))

12556
* (asdf:run-shell-command "ps axf | grep sbcl >/dev/tty")
12553 pts/1    S+     0:01              |       \_ sbcl
12554 pts/1    S+     0:00              |           \_ sbcl
12555 pts/1    S+     0:00              |           \_ sbcl
12556 pts/1    S+     0:00              |           \_ sbcl
12560 pts/1    R      0:00              |           \_ /bin/sh -c ps axf | grep sbcl >/dev/tty

(OK, technically I think they may be sharing the file descriptor table
as well as the VM, but they're not calling pthreads functions and they
definitely have distinct pids)


-dan

When i has reading the docs i saw "On Linux x86, threading is
implemented using clone()" this is refering to "On Linux x86, sbcl is
using threads using clone()" When i read that i understood that the docs
said that linux used clone for threads (as in the old pre nptl).

It was my mistake, i should have read with more attention. I'm sory if i
caused so much commotion.

-- 
Gustavo Felisberto
(HumpBack)
Web: http://dev.gentoo.org/~humpback
Blog: http://blog.felisberto.net/
------------
It's most certainly GNU/Linux, not Linux. Read more at
http://www.gnu.org/gnu/why-gnu-linux.html .
-------------

On Fri, May 27, 2005 at 01:19:06PM +0100, Gustavo Felisberto wrote:
> When i has reading the docs i saw "On Linux x86, threading is
> implemented using clone()" this is refering to "On Linux x86, sbcl is
> using threads using clone()" When i read that i understood that the docs
> said that linux used clone for threads (as in the old pre nptl).

You seem still to be confused.  For the record:

- there are two threading libraries for Linux: LinuxThreads (the old one)
  and NPTL (the new one)

- They both provide a POSIX Threads ("pthreads") interface, and are both
  implemented with use of clone()

- however, they typically use different options to clone(); a LinuxThreads
  system will create a separate pid for each thread, whereas a system
  with NPTL libraries will share the pid between threads.  

- SBCL does not currently[*] use the pthreads interface at all, because when
  the threading support was added NPTL was either in early development or 
  did not yet exist, and LinuxThreads is badly broken.  Instead it calls
  clone() directly.  The options to clone() that it uses cause it to
  create a separate pid for each thread

- However, it does require kernel support for futexes (for its mutex 
  implementation).  As not many people knwo whether they've got this,
  and as it's available on pretty much exactly the same systems as 
  NPTL (it is part of the kernel support required for NPTL), we describe
  SBCL as needing NPTL



[*] There's a currently-stalled branch of SBCL in CVS which uses the pthread
interfaces, but it's stuck on an interaction between POSIX 1003.1b semaphores
and copying GC, which I cannot see how to resolve in a non-icky way.  See
http://ww.telent.net/diary/2005/3/#6.793

On Fri, 27 May 2005 13:54:50 +0100, <dan@...> wrote:
>
> - However, it does require kernel support for futexes (for its mutex 
>   implementation).  As not many people knwo whether they've got this,
>   and as it's available on pretty much exactly the same systems as 
>   NPTL (it is part of the kernel support required for NPTL), we describe
>   SBCL as needing NPTL

This is a poor approach, it would be better to state:

... with NPTL kernel support, (more specifically futex support), ...

Once a linux system has been converted to NPTL, it is stuck in 2.6
kernel land.  There are many products, for instance VMware that do not
support 2.6 kernels. With NPTL in place, I cannot drop back into 2.4
to provide a test case to VMware.  NPTL requires more than a kernel
change, if I recall correctly, I had to rebuild libc and a few other
tools.  A futex changes sounds like I have the opportunity to return
to 2.4 kernels to deal with support issues.


-- 
With sufficient thrust, pigs fly fine.

GP lisper <spambait@...> writes:

> This is a poor approach, it would be better to state:
>
> ... with NPTL kernel support, (more specifically futex support), ...

This may be a fair point: I suppose we could say something more like

| This requires x86 and Linux kernel 2.6 or systems with NPTL backports.

or

| Queues require the sys_futex() system call to be available: this is
| the reason for the NPTL requirement. We test at runtime that this
| system call exists.

Oh, but look!  WE ALREADY DO!  

> Once a linux system has been converted to NPTL, it is stuck in 2.6
> kernel land.  There are many products, for instance VMware that do not
> support 2.6 kernels. 

The VMWare web site lists several 2.6-series kernels that it works
with: <http://www.vmware.com/support/ws5/doc/intro_hostreq_ws.html>.
I don't know what problems you're experiencing, but you should probably
take them up with vmware support.



-dan

On Sat, 28 May 2005 02:28:07 +0100, <dan@...> wrote:
> GP lisper <spambait@...> writes:
>
>| Queues require the sys_futex() system call to be available: this is
>| the reason for the NPTL requirement. We test at runtime that this
>| system call exists.
>
> Oh, but look!  WE ALREADY DO!  

Do what Dan?  Previously you have stated:

This is not contradictory at all.  It requires NPTL because pre-NPTL
systems don't (usually) have futex support - and it's more likely
that people know whether they have NPTL than whether they have a
kernel that supports some syscall they've probably never heard of.

and

- However, it does require kernel support for futexes (for its mutex 
  implementation).  As not many people knwo whether they've got this,
  and as it's available on pretty much exactly the same systems as 
  NPTL (it is part of the kernel support required for NPTL), we describe
  SBCL as needing NPTL


So which is it, just futex (i.e. kernel only) support or futex plus
NPTL (additional libs) ?

For that matter, I don't even see an xconfig setting for futex.
CONFIG_FUTEX=y appears in General Setup of .config, I know it I've
toggled it in the past (x86).


> The VMWare web site lists several 2.6-series kernels that it works

Oh good, thanks, my friend that works for VMware didn't mention it at
lunch.  Odds are it's the broken kernels, but at least they moved
forward.  2.6 wouldn't support full screen here on multiple displays.
Too bad it wasn't months ago when I first needed it.

-- 
With sufficient thrust, pigs fly fine.

Someone else pointed out my original post appeared negative toward sbcl
developers. Sorry about that. I definitely harbour no negative feelings
toward sbcl developers. You guys are doing great work and SBCL rocks my
world. I was just being much more whingy than necessary because I was in
panic mode. So Dan and any other developers, I apologise if it seemed my
negativity was directed toward you in any way.

And I'm sorry I posted to sbcl-devel instead of sbcl-help. I wasn't
thinking.

Now, with those apologies out of the way, all I'm really asking for is a
nice consise list of known issues with SBCL threads, their nature,
possible fixes etc etc so I have somewhere to start looking. I know I'm
probably dreaming else I'd have found it by now, but whatever you guys
use for bugtracking would be all I need if it's publically accessible in
any way. I'm kinda hoping there's something I can use other than the
sbcl-devel archives with more info than the BUGS file. 

(Right now what I've been doing is google for thread related bugs on the
mailing list, give up because the word "thread" appears in every email
on the web, ask a developer whether symptoms I'm experiencing are due to
a known bug, ask a developer what's known about the bug and ask a
developer if there's any known fix. Given Dan's the threads guy he tends
to be that developer a lot so I'm probably getting up his nose (Sorry
Dan!). So if there's any simple way I can find this information directly
that'd be great. If not, oh well.)


-- 
Robert Marlow <bobstopper@...>

On Fri, 20 May 2005, John Morrison wrote:

> * (SB-SYS:GET-DYNAMIC-FOREIGN-SYMBOL-ADDRESS "_ZNSt9exceptionC1Ev")
> STYLE-WARNING: Undefined alien: "_ZNSt9exceptionC1Ev"

> Longer Version: GCCXML emits records for everything, including the
> above (constructor for std::exception). However, when I try to
> define-alien-routine, it errors.

What do you mean by errors? The intention is that DEFINE-ALIEN-ROUTINE 
should just give the same undefined-alien style-warning, and only actually
_calling_ it should signal an error.

What version of SBCL are you using?

This is SBCL 0.9.0.28, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
* (define-alien-routine foobar int)
;
; caught STYLE-WARNING:
;   Undefined alien: "foobar"
;
; compilation unit finished
;   caught 1 STYLE-WARNING condition

FOOBAR
* (foobar)

debugger invoked on a SB-KERNEL::UNDEFINED-ALIEN-FUNCTION-ERROR in thread 
5590:
   Attempt to call an undefined alien function.

As for you actual question, I think maybe we should and could provide 
SB-ALIEN:ALIEN-SYMBOL-ADDRESS (or -SAP?), that would accept an optional 
argument to indicate what should be done if the symbol cannot be found.

Currently, you would need to write your own using 
GET-DYNAMIC-FOREIGN-SYMBOL-ADDRESS as a starting point.

Cheers,

  -- Nikodemus              Schemer: "Buddha is small, clean, and serious."
                   Lispnik: "Buddha is big, has hairy armpits, and laughs."

On Tuesday 17 May 2005 02:57 pm, John Morrison wrote:
> SBCL complains about not finding vintage GLIBC_2.0 _h_errno in
> /lib/libpthread.  However, lsof says that /lib/tls/libpthread-0.61.so

[ snip snip snip ]

Oops.  Never mind.  Part of the problem is a stale fasl file.  (The
LD_DEBUG death, I'm not so sure of yet, and I still cannot save a core
image with my alien bindings in it, so I think there's Yet Another
something screwy I'm doing.)

-jm


-- 
==== John Morrison
==== MAK Technologies Inc.
==== 10 Fawcett Street, Cambridge, MA 02138
==== http://www.mak.com/
==== vox:617-876-8085 x115
==== fax:617-876-9208
==== jm@...

On Tue, 17 May 2005, John Morrison wrote:

> LD_DEBUG death, I'm not so sure of yet, and I still cannot save a core
> image with my alien bindings in it, so I think there's Yet Another
> something screwy I'm doing.)

Which internal bits are you twiddling with? (As I've a vague recollection 
of pointing you towards some, I'm expecting this to be the case.)

Also, does this inability to save a core file manifest in not being able 
to save-lisp-and-die, or errors on startup? If the latter, a backtrace
from the error would be good.

As a totally unrelated thing, once you emerge victorious from these 
tribulations it would be very interesting to hear what all this has been 
for, and what SBCL (and other) hackery has been involved.

Cheers,

  -- Nikodemus              Schemer: "Buddha is small, clean, and serious."
                   Lispnik: "Buddha is big, has hairy armpits, and laughs."

On Fri, May 13, 2005 at 03:52:21PM +0100, Christophe Rhodes wrote:
> "R. Mattes" <rm@...> writes:
> 
> > Hmm, so now we have a small but important semantic difference between
> > sb-md5:md5sum-sequence and md5:md5sum-sequence. 
> 
> Who is the maintainer of md5:md5sum-sequence, and do they know the
> implications of multiple string representations?

Kevin M. Rosenberg is the maintainer of the Debian package and since the
ASDF link on Cliki poiunts to his site i _assume_ it's him. The asdf file
mentins him as an author.

> > And, since (typep "string" 'sequence) is true it makes conditional
> > code rather more elaborate than necessary. What's the design
> > rationale behind this? After all strings _are_ sequences. 
> 
> Strings are sequences, but the md5 algorithm is defined over octets,
> not characters.  As such, the name md5sum-sequence is a little
> misleading, but is a relic of the days when sbcl only knew about 256
> characters, and did not have any idea that there might be more than
> one possible encoding. 

Yes, that name itself seems to come from Pierre Mai and his code originally
written for CMUCL.

> The sb-md5:md5sum-string entry point
> acknowledges the existence of multiple encodings for character data,
> and the sb-md5:md5sum-sequence entry point will shortly no longer
> accept general strings to prevent the user from shooting themselves in
> the foot.

Thanks, i think this is what i would have expected. I didn't shoot intentionally :-)

> 
> Just to put this on a concrete footing, let me throw the question back
> to you: what, if md5sum-sequence works on strings, should 
>   (md5sum-sequence "?" return)
> [ where the character in the string is a euro sign ]

I'd expect one of two possible results: 

 - first: throw an error/condition (maybe only iff the codepoints of the string's characters
   don't fit into 8 bits (taking advantage of the code point overlay of ASCII, latin-1
   and Unicode).

 - second: create the hash of the internal representation of the string. After all
   the md5 algorithm is _always_ senitive to the binary representation. Will there be 
   a possible case in SBCL where the binary representation of to strings equal under
   string= will differ? If not then i'd vote dor this solution.
   One drawback of this solution: the md5 sum of a string would not necessarily match
   that of a file containing the same string.

 - third  (just to make a mathematician nervous): have md5sum-sequence accept a keyword
   :encoding. This would actually be backward compatible and (with :default as the default
   encoding) would work as solution 2.
   
> 
> > The "mysterious" thing is that md5sum-sequence does provide a result
> > (and doesn't throw a condition) - just a strange one.
> 
> Probably it has been compiled with a low safety setting relative to
> speed, so that the usual checks are not performed.

Hmmm, beats me. This is on a "standard" Debian box. How would i check this?

> 
> > Given the use of md5 in security relevant code i'm a bit worried.
> 
> md5 in security-relevant code should probably be avoided, as it has
> "only" 128 bits of hash, which wouldn't be a problem were it not for
> the fact that it has been broken fairly comprehensively.

Still, it's used in several security relevant spots. Luckily, in my case it "only"
totally messed up a knowlege base (where object IDs are generated with a hash of
important properties (all 7-bit URIs).

Thanks for your input

  ralfd

> Cheers,
> 
> Christophe

rm@... writes:

>> Just to put this on a concrete footing, let me throw the question back
>> to you: what, if md5sum-sequence works on strings, should 
>>   (md5sum-sequence "?" return)
>> [ where the character in the string is a euro sign ]
>
> I'd expect one of two possible results: 
>
>  - first: throw an error/condition (maybe only iff the codepoints of
>    the string's characters don't fit into 8 bits (taking advantage
>    of the code point overlay of ASCII, latin-1 and Unicode).

OK, this is basically what will happen, apart from the DWIMish part,
which is not really in the spirit of SBCL.  (For more information on
this, you might take a look at the PRINCIPLES file distributed with
the sbcl source.)

>  - second: create the hash of the internal representation of the string. 
>    After all the md5 algorithm is _always_ senitive to the binary
>    representation. Will there be a possible case in SBCL where the
>    binary representation of to strings equal under string= will
>    differ?

Yes, there are.  BASE-STRINGs and (ARRAY CHARACTER (*)) can have the
same contents with different in-memory representations.

>    If not then i'd vote dor this solution.  One drawback of this
>    solution: the md5 sum of a string would not necessarily match
>    that of a file containing the same string.

Right, and with the added potential confusion over different in-memory
representations, this is a non-starter.

>  - third (just to make a mathematician nervous): have
>    md5sum-sequence accept a keyword :encoding. This would actually
>    be backward compatible and (with :default as the default
>    encoding) would work as solution 2.

Backwards compatibility, at this point, is not even remotely
interesting to me.  I'd much rather get an interface that we can
collectively be happy to support in the long term, than deal with the
headaches involved in supporting half-baked ones.  I'm sorry if that
causes our current users problems, but our current users have made
that choice by using a 0.x piece of software where the development
culture is not focussed on interface stability.  (Again, see the
PRINCIPLES file, as well as, well, five years of commit logs :-)

Cheers,

Christophe

rm@... wrote:
> Kevin M. Rosenberg is the maintainer of the Debian package and since the
> ASDF link on Cliki poiunts to his site i _assume_ it's him. The asdf file
> mentins him as an author.

I'll fix the assumption: I'm the author of the only the .asd file, not
of the md5sum program. I host the package on my server only so that it
can be downloadable with asdf-install for my CLSQL package. I am no
longer the maintainer of the Debian package.

-- 
Kevin Rosenberg
kevin@...

On Fri, May 13, 2005 at 01:42:38PM -0600, Kevin Rosenberg wrote:
> rm@... wrote:
> > Kevin M. Rosenberg is the maintainer of the Debian package and since the
> > ASDF link on Cliki poiunts to his site i _assume_ it's him. The asdf file
> > mentins him as an author.
> 
> I'll fix the assumption: I'm the author of the only the .asd file, not
> of the md5sum program. I host the package on my server only so that it
> can be downloadable with asdf-install for my CLSQL package. I am no
> longer the maintainer of the Debian package.

Ok, so Peter took over the responsibilties. Oh gosh, yet another bug report to
him.

 Thanks  Ralf Mattes

> -- 
> Kevin Rosenberg
> kevin@...

On Fri, 13 May 2005 20:14:58 +0100, Christophe Rhodes wrote:

> rm@... writes:
> 
>> [...snip ...]
>>  - first: throw an error/condition (maybe only iff the codepoints of
>>    the string's characters don't fit into 8 bits (taking advantage
>>    of the code point overlay of ASCII, latin-1 and Unicode).
> 
> OK, this is basically what will happen, apart from the DWIMish part,
> which is not really in the spirit of SBCL.  (For more information on
> this, you might take a look at the PRINCIPLES file distributed with
> the sbcl source.)
> 
>>  - second: create the hash of the internal representation of the string. 
>>    After all the md5 algorithm is _always_ senitive to the binary
>>    representation. Will there be a possible case in SBCL where the
>>    binary representation of to strings equal under string= will
>>    differ?
> 
> Yes, there are.  BASE-STRINGs and (ARRAY CHARACTER (*)) can have the
> same contents with different in-memory representations.

How is that? Doesn't a base-string consist entirely of base-chars (with
code-points <= 127)? How _can_ i construct an array of characters with
code-point <= 127 that has a different internal representation?

>>    If not then i'd vote dor this solution.  One drawback of this
>>    solution: the md5 sum of a string would not necessarily match
>>    that of a file containing the same string.
>
> Right, and with the added potential confusion over different in-memory
> representations, this is a non-starter.
> 
>>  - third (just to make a mathematician nervous): have
>>    md5sum-sequence accept a keyword :encoding. This would actually
>>    be backward compatible and (with :default as the default
>>    encoding) would work as solution 2.
> 
> Backwards compatibility, at this point, is not even remotely
> interesting to me.  

Backwards compatibility isn't what i aim for, but if it can be achieved
easily i don't mind :)

> I'd much rather get an interface that we can
> collectively be happy to support in the long term, than deal with the
> headaches involved in supporting half-baked ones.  

I _hope_ i don't sound stubborn but i somehow miss to see the half-
bakedness of this interface [1]. Somehow i expect 

 (sb-md5:md5sum-sequence "Blah") to act equivalent to
 (sb:md5sum-sequence (string-to-octets "Blah" :encoding :default))

but i might be wrong.

> I'm sorry if that
> causes our current users problems, but our current users have made
> that choice by using a 0.x piece of software where the development
> culture is not focussed on interface stability.  (Again, see the
> PRINCIPLES file, as well as, well, five years of commit logs :-)

I never complained about non-stable interfaces. I'm actually _very_ 
thankful for the Unicode support. I was taken by surprise because
the semantics of a public function changed (and from what i understand
then missing warning/error was just an accident). 

 Thanks RalfD

[1] i purposely won't quote "the other" languages as a reference or 
 guide.
> Cheers,
> 
> Christophe
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by Oracle Space Sweepstakes
> Want to be the first software developer in space?
> Enter now for the Oracle Space Sweepstakes!
> http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click

On Sat, May 14, 2005 at 12:17:40AM +0200, R. Mattes wrote:
> How is that? Doesn't a base-string consist entirely of base-chars (with
> code-points <= 127)? How _can_ i construct an array of characters with
> code-point <= 127 that has a different internal representation?

(Assuming a SB-UNICODEd SBCL) You do this all the time, simply by
typing strings at the REPL:

CL-USER> (type-of "DAD")
(SIMPLE-ARRAY CHARACTER (3))

Such a string has a layout that looks roughly like:

[tag] [length] [00 00 00 44] [00 00 00 41] [00 00 00 44] [00 00 00 00]

where [...] is a 32-bit quantity, with values written in hexadecimal
when necessary.  If you instead said something like:

CL-USER (type-of (coerce "DAD" '(simple-array base-char (3)))
(SIMPLE-BASE-STRING (3))

The memory layout of such a string would look like:

[tag] [length] [44 41 44 00]

which is more like what you are expecting.  But that doesn't mean that
you get to pun such a string into being a sequence of bytes like C.

> I _hope_ i don't sound stubborn but i somehow miss to see the half-
> bakedness of this interface [1]. Somehow i expect 
> 
>  (sb-md5:md5sum-sequence "Blah") to act equivalent to
>  (sb:md5sum-sequence (string-to-octets "Blah" :encoding :default))
> 
> but i might be wrong.

Don't expect; read the documentation! :)

CL-USER> (documentation 'sb-md5:md5sum-sequence 'function)
"Calculate the MD5 message-digest of data bounded by START and END
in SEQUENCE , which must be a vector with element-type (UNSIGNED-BYTE
8)."
-- 
Nathan | From Man's effeminate slackness it begins.  --Paradise Lost

The last good thing written in C was Franz Schubert's Symphony Number 9.
                                               --Erwin Dieterich

On Fri, May 13, 2005 at 05:57:14PM -0500, Nathan Froyd wrote:
> On Sat, May 14, 2005 at 12:17:40AM +0200, R. Mattes wrote:
> > How is that? Doesn't a base-string consist entirely of base-chars (with
> > code-points <= 127)? How _can_ i construct an array of characters with
> > code-point <= 127 that has a different internal representation?
> 
> (Assuming a SB-UNICODEd SBCL) You do this all the time, simply by
> typing strings at the REPL:
> 
> CL-USER> (type-of "DAD")
> (SIMPLE-ARRAY CHARACTER (3))
> 
> Such a string has a layout that looks roughly like:
> 
> [tag] [length] [00 00 00 44] [00 00 00 41] [00 00 00 44] [00 00 00 00]
> 
> where [...] is a 32-bit quantity, with values written in hexadecimal
> when necessary.  If you instead said something like:
> 
> CL-USER (type-of (coerce "DAD" '(simple-array base-char (3)))
> (SIMPLE-BASE-STRING (3))
> 
> The memory layout of such a string would look like:
> 
> [tag] [length] [44 41 44 00]
> 
> which is more like what you are expecting.  

I was expecting (falsely) expecting sbcl to use utf-8 encoding (where the first
string would look like the second).  Where actually can i find notes on the 
implementation of unicode in sbcl? I found a  page on the sbcl-internals wiki and
listened to Christophe's talk in Amsterdam. Is there more?

> But that doesn't mean that
> you get to pun such a string into being a sequence of bytes like C.
> 
> > I _hope_ i don't sound stubborn but i somehow miss to see the half-
> > bakedness of this interface [1]. Somehow i expect 
> > 
> >  (sb-md5:md5sum-sequence "Blah") to act equivalent to
> >  (sb:md5sum-sequence (string-to-octets "Blah" :encoding :default))
> > 
> > but i might be wrong.
> 
> Don't expect; read the documentation! :)
> 
> CL-USER> (documentation 'sb-md5:md5sum-sequence 'function)
> "Calculate the MD5 message-digest of data bounded by START and END
> in SEQUENCE , which must be a vector with element-type (UNSIGNED-BYTE
> 8)."

Yes, three bonus points to the SBCL developers for updating the documentation
together with the code :-) When i wrote my code it read:

 "Calculate the MD5 message-digest of data in sequence.  On CMU CL
 this works for all sequences whose element-type is supported by the
 underlying MD5 routines, on other implementations it only works for 1d
 simple-arrays with such element types."
 
And, from the README:

 the "high-level" entry points to the md5 algorithm are
 MD5SUM-FILE, MD5SUM-STREAM and MD5SUM-SEQUENCE (despite its name,
 the last only acts on vectors).

This is not "... The basic criteria are that the introduction of Unicode
should be invisible to existing code ..." (from the wike page). Since we
all seem to agree that an MD5 digest of a string depends on the sequence
of characters (code points) as well as their encoding why _do_ we have to
treat strings (which _are_ sequences according to the CL specs) different.
Christophe seems to fear that the encoding will confuse users, i think
a user will/would expect the equivalence of:

(sb-md5:md5sum-sequence "Blah") == (sb:md5sum-sequence (string-to-octets "Blah" :encoding :default))

[being probably a bit to pragmatic i'd add an :encoding keyword to md5sum-sequence to
help in those cases where a digest needs to be compared against one of a file with
a known encoding - but that is syntactic sweetener].

 Thanks, Ralf Mattes 
> -- 
> Nathan | From Man's effeminate slackness it begins.  --Paradise Lost
> 
> The last good thing written in C was Franz Schubert's Symphony Number 9.
>                                                --Erwin Dieterich

rm@... (Le grand pinguin) writes:

> On Fri, May 13, 2005 at 05:57:14PM -0500, Nathan Froyd wrote:
>> [tag] [length] [00 00 00 44] [00 00 00 41] [00 00 00 44] [00 00 00 00]
>> 
>> where [...] is a 32-bit quantity, with values written in hexadecimal
>> when necessary.  If you instead said something like:
>> 
>> CL-USER (type-of (coerce "DAD" '(simple-array base-char (3)))
>> (SIMPLE-BASE-STRING (3))
>> 
>> The memory layout of such a string would look like:
>> 
>> [tag] [length] [44 41 44 00]
>> 
>> which is more like what you are expecting.  
>
> I was expecting (falsely) expecting sbcl to use utf-8 encoding (where the first
> string would look like the second).  Where actually can i find notes on the 
> implementation of unicode in sbcl? I found a  page on the sbcl-internals wiki and
> listened to Christophe's talk in Amsterdam. Is there more?

Well,
  <http://www.doc.gold.ac.uk/~mas01cr/talks/2005-04-24%20Amsterdam/presentation.html>
are my slides from that talk, sadly minus the deadpan asides.  The section from 
  <http://www.doc.gold.ac.uk/~mas01cr/talks/2005-04-24%20Amsterdam/img15.html>
is of particular interest, with img17.html discussing exactly this point.

(Actually, I would never expect anyone to use utf-8 as an internal
representation for mutable strings, because you lose the O(1) access
time; that's why it doesn't appear as one of the implementation
options.)

> (sb-md5:md5sum-sequence "Blah") == (sb:md5sum-sequence (string-to-octets "Blah" :encoding :default))

Well, in my world, probably md5sum-sequence will end up going away
altogether, being replaced by md5sum-octet-vector or somesuch.  As
you've observed, the name md5sum-sequence is confusing.  I would ask
you what you actually _gain_ from the genericity you're asking for in
the above -- how often do you call a function on X, where X is either
a string or a vector of (unsigned-byte 8) and you don't know which?
This would seem to be only useful if you're confused in the first
place.

Cheers,

Christophe