[clisp-list] bugs in (setf (stream-element-type x) ...)

[Todd S. <ta...@we...>]

Hi,

This is with clisp-2.27, on RH Linux 6.2.

So, I was trying to write some sample code that would act as an HTTP
client.  What I did was (socket-connect ...), (format ...), then
(read-line ...) until I got the empty line.  At that point, I did
(setf (stream-element-type *socket*) '(unsigned byte 8)), and then
(read-sequence *array* *socket*), where *array* was obtained from
make-array with the size specified by the Content-Length: header.

Everything worked, except that the first element in *array* got the
value 10 (LF), and the rest of the data was shifted by one byte,
leaving one byte of the actual data unread.  I realized that somehow I
was getting the final LF from the headers, even though it should have
already been read.  Indeed, if instead of doing the (setf
(stream-element-type ...)), I just did (read-char *socket*), I would
get the first char of the data, and not #\Newline.

I figured that it probably had something to do with CRLF -> #\Newline
translation, and that maybe it wouldn't be a problem with buffering.
So, I tried:

[1]> (setq *my-sock* (socket-connect 80 "jetcar.qnz.org" :buffered t))
#<IO BUFFERED SOCKET-STREAM CHARACTER jetcar.qnz.org:80>
[2]> (format *my-sock* "GET /icons/apache_pb.gif HTTP/1.0

")
NIL
[3]> (force-output *my-sock*)
NIL
[4]> (read-line *my-sock*)
"HTTP/1.1 200 OK" ;
NIL
[5]> (setf (stream-element-type *my-sock*) '(unsigned-byte 8))

*** - SYSTEM::BUILT-IN-STREAM-SET-ELEMENT-TYPE: argument 
*** - handle_fault error2 ! address = 0x40490824 not in [0x2029E000,0x2035A410) !
SIGSEGV cannot be cured. Fault address = 0x40490824.
Segmentation fault (core dumped)

Well, I tracked the crash down, and this patch fixes it:

--- stream.d	Mon Jul 16 10:35:54 2001
+++ ../../clisp-2.27-tas/src/stream.d	Fri Dec 28 23:31:44 2001
@@ -16238,9 +16238,9 @@
       #ifdef SOCKET_STREAMS
       case strmtype_twoway_socket:
         # Apply to the input and output side individually.
-        pushSTACK(TheStream(STACK_1)->strm_twoway_socket_input); pushSTACK(STACK_(0+1));
+        pushSTACK(TheStream(stream)->strm_twoway_socket_input); pushSTACK(STACK_(0+1));
         funcall(L(built_in_stream_set_element_type),2);
-        pushSTACK(TheStream(STACK_1)->strm_twoway_socket_output); pushSTACK(STACK_(0+1));
+        pushSTACK(TheStream(stream)->strm_twoway_socket_output); pushSTACK(STACK_(0+1));
         funcall(L(built_in_stream_set_element_type),2);
         break;
       #endif

I'm not sure why the code was using STACK_1 instead of stream to begin
with, but if there was a reason to reference the stack directly, it
should have been STACK_2, not STACK_1.  With the patch, the buffered
case works as expected, no extra LF.

The problem in the unbuffered case turns out to be the way CRLF is
handled in the unbuffered case.  If read-line encounters a CRLF in the
input stream, the LF remains unread, and the ignore_next_LF flag is
set.  When (setf (stream-element-type ...)) is used, that flag is lost
or discarded, which causes the problem.

There seem to be a few ways around it, but I haven't attempted any of
them.

 o The flag could be done away with by reading the next char from
 input, and discarding it if it's a LF, and unreading it if not.  In
 theory, this might cause an app to hang where it wouldn't have before,
 but I think the risk of that is pretty low.

 o The flag could be preserved across the (setf (stream-element-type
 ...)), but then all the read-byte functions would need to be aware of
 it.

 o apps can do a (peek-char *stream*) before
 (setf (stream-element-type ...)) to workaround it.  Seems gross to
 require that of apps, though.

Anyway, this is the first I've looked at clisp source code, so I'm
sure you'll have a better idea of how to handle this.


Todd

p.s.  I looked, but didn't find, some instructions for developers on
how to debug clisp.  I managed to build it with debugging info by
adding the appropriate "-g"s to the Makefile, but it crashed with
sigsegv whenever I tried to do a step or next in gdb.  Is there a
secret to debugging clisp in gdb?  I had to resort to printfs.  I saw
the Makefile.devel, but it didn't seem to have any support for doing a
debug build.

p.p.s.  Is there any way not to do the CRLF translation?  As it
stands, if I wanted to, e.g., write a network server that verified
strict compliance of its clients, i.e., make sure there are both a CR
and a LF at the end of lines, there's no way to do it with clisp,
unless I treat everything as bytes instead of chars, which makes lots
of other things unwieldy.