[ clisp-Bugs-1564818 ] CRLF handling does not work for UNICODE-16

[SourceForge.net <no...@so...>]

Bugs item #1564818, was opened at 2006-09-24 22:54
Message generated for change (Comment added) made by sds
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=101355&aid=1564818&group_id=1355

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: clisp
Group: lisp error
>Status: Closed
>Resolution: Fixed
Priority: 5
Private: No
Submitted By: Mike McMahon (mmcm)
>Assigned to: Sam Steingold (sds)
Summary: CRLF handling does not work for UNICODE-16

Initial Comment:
CRLF is not converted to #\Newline by READ-CHAR and
READ-LINE from 16-bit Unicode files.  (clisp 2.39 MinGW
no-readline XP.)

For -BIG-ENDIAN, the LF is not recognized and an extra
blank line is input.

For -LITTLE-ENDIAN, only the LF byte is removed and an
"incomplete byte sequence at end of buffer" failure
results.

The culprit is code marked "FIXME: This is wrong." in
stream.d.

The straightforward fix is to move the ignore_next_LF
flag from BufferedStream to ChannelStream and use it in
the two marked cases.  This seems to work fine for me
locally.  Perhaps I am overlooking some problem with
this approach.

----------------------------------------------------------------------

Comment By: Sam Steingold (sds)
Date: 2008-05-19 11:44

Message:
Logged In: YES 
user_id=5735
Originator: NO

thank you for your bug report.
the bug has been fixed in the CVS tree.
you can either wait for the next release (recommended)
or check out the current CVS tree (see http://clisp.cons.org)
and build CLISP from the sources (be advised that between
releases the CVS tree is very unstable and may not even build
on your platform).

----------------------------------------------------------------------

Comment By: Sam Steingold (sds)
Date: 2007-01-08 22:29

Message:
Logged In: YES 
user_id=5735
Originator: NO

your patch only fixed the buffered case.
unbuffered streams are stil broken.
(let ((f "unicode-file"))
  (unwind-protect
       (loop :for e1 :in '(charset:unicode-16-big-endian
                           charset:unicode-16-little-endian
                           charset:unicode-32-big-endian
                           charset:unicode-32-little-endian)
         :for e = (ext:make-encoding :charset e1 :line-terminator :dos)
         :do (with-open-file (o f :direction :output :external-format e)
               (write-line f o))
         :nconc
         (loop :for b :in '(nil t) :nconc
           (with-open-file (i f :direction :input :external-format e
                              :buffered b)
             (handler-case (let ((l (read-line i)))
                             (if (string= f l) ()
                                 (list (list e b 'read-line l))))
               (error (c) (list (list e b 'read-line (princ-to-string
c)))))
             (handler-case (let ((c (read-char i nil nil)))
                             (and c (list (list e b 'read-char c))))
               (error (c) (list (list e b 'read-char (princ-to-string
c))))))))
    (delete-file f)))


----------------------------------------------------------------------

Comment By: Sam Steingold (sds)
Date: 2006-12-28 11:10

Message:
Logged In: YES 
user_id=5735
Originator: NO

sorry for taking so long - we are not notified of new attachments unless
they are accompanied by a comment.
could you please also attach a test case that demonstrates the bug?
(please also make a comment when attaching the test case so that we will
notice it)
thanks!

----------------------------------------------------------------------

Comment By: Sam Steingold (sds)
Date: 2006-09-25 12:37

Message:
Logged In: YES 
user_id=5735

could you please attach your patch to this issue?
thanks

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=101355&aid=1564818&group_id=1355