Re: [clisp-list] reading character and binary input from a socket

[Mike P. <mic...@gm...>]

Hi Pascal

Thanks for the pointers to Babel, your Informatimago libraryt,
Usockets and flexi-streams. I will certainly look into them.

My original approach did involve opening a binary stream and
converting bytes to characters as required. However,  my  reason for
wanting to open a character stream and then convert it to binary is as
follows. I am writing a web server interface abstraction/wrapper which
will allow a plain old CGI script to run transparently over CGI or
SCGI (and potentially FastCGI), similar to WSCGI, Rack, Plack, Clack
etc. Whereas these other gateway abstractions require the application
(or framework) to read from a binary input stream and generate a
triplet of status, headers and body, I want my abstraction to be
completely compatible with CGI, i.e.  headers accessible via
environment variables and body (for REQUEST_METHOD = POST) accessible
via a *standard-input* stream read as either bytes or characters.  The
web framework (or application) layer _may_ convert the standard input
stream to binary, e.g.for multipart/form-data but I did not want to
make that a requirement. I realise I am swimming against the tide and
maybe I'll end up trying to turn my code into an SCGI handler for
Clack.In the meantime, I want to persevere with the possibility of
switching from character to binary on the fly.

Coming back to my original problem, if I there is no way to get
ext:make-stream to duplicate a socket stream with a different element
type then guess I'll try flexi-streams. Looking at the docs, it
appears to allow setf on element-type
(http://weitz.de/flexi-streams/#flexi-stream-element-type).

Thanks for the help.

Mike


On 19/07/2013, Pascal J. Bourguignon <pj...@in...> wrote:
> Mike Palmer <mic...@gm...> writes:
>
>> I am trying to change the element type of an open socket-server stream
>> from 'character to '(unsigned-byte 8) to allow me to read both
>> characters and binary from the same socket. I have a working using
>> setf directly on the socket's stream-element-type. However, I
>> understand this is a non-portable specific clisp extension, so would
>> prefer to write something more portable, e.g. using ext:make-stream to
>> duplicate the stream with a different element-type.  When I try to do
>> this however, I keep getting the error "Invalid direction :INPUT".
>> I've also tried specifying :DIRECTION :IO but it makes no difference.
>> […]
>> 1. Have I misunderstood something fundamental - am I correct to try to
>> do this? I am going by the clisp implementation notes for
>> ext:make-stream.
>> 2. If the make-stream approach is flawed, is there another portable
>> way to change the :DIRECTION on an open socket stream?
>
> Ok, since you're considering a portable solution, I'll mention usockets
> and iolib, or even flexi-streams.
>
> But I would even go as far as writing conforming code (once you have the
> socket stream).  My advice would be to open a binary socket stream to
> begin with, and to convert the bytes to characters when you need to, or
> just keep them as bytes.
>
> While ASCII or even UTF-8 is often used in Internet protocols, they're
> almost always actually BINARY protocols!
>
> In particular, in the way they handle the bytes 13 and 10.
>
> To convert between bytes and characters, you may use the clisp specific
> EXT:CONVERT-STRING-FROM-BYTES and EXT:CONVERT-STRING-TO-BYTES, but you
> may prefer to use a portable library such as BABYL ('big' since it
> supports amongst other encoding Unicode UTF-8), or
> COM.INFORMATIMAGO.COMMON-LISP.CESARUM.ASCII ('big' since quickloading
> :com.informatimago.common-lisp.cesarum will fetch a whole set of package
> unrelated to ASCII encoding/decoding, I'm sorry about that, but I'm lazy
> and provide a global systems for my code; but you could just copy the
> single file you need (AGPL3)).
>
>
>
> Notice how COM.INFORMATIMAGO.COMMON-LISP.CESARUM.ASCII also provides
> functions to work with ASCII _bytes_, for example:
>
>     (use-package :com.informatimago.common-lisp.cesarum.ascii)
>
>     (let ((buffer #(72 69 76 79 32 118 111 121 97 103 101 114 46 105 110
> 102
>                     111 114 109 97 116 105 109 97 103 111 46 99 111 109)))
>       (bytes= buffer #.(ascii-bytes "HELO ") :end1 (min 5 (length
> buffer))))
>
>     (set-dispatch-macro-character #\# #\" (function ascii-dispatch-macro)
> *readtable*)
>     (let ((buffer  (read-ascii-line binary-stream)))
>       (bytes= buffer #"HELO " :end1 (min 5 (length buffer))))
>
>
> In conclusion: with a few convenience functions, you can manipulate
> vectors of bytes as easily as vectors of characters aka. strings.
>
>
> --
> __Pascal Bourguignon__                     http://www.informatimago.com/
> A bad day in () is better than a good day in {}.
> You know you've been lisping too long when you see a recent picture of
> George
> Lucas and think "Wait, I thought John McCarthy was dead!" -- Dalek_Baldwin
>