Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

#366 CRLF handling does not work for UNICODE-16

lisp error
closed-fixed
Sam Steingold
clisp (525)
5
2008-05-19
2006-09-25
Mike McMahon
No

CRLF is not converted to #\Newline by READ-CHAR and
READ-LINE from 16-bit Unicode files. (clisp 2.39 MinGW
no-readline XP.)

For -BIG-ENDIAN, the LF is not recognized and an extra
blank line is input.

For -LITTLE-ENDIAN, only the LF byte is removed and an
"incomplete byte sequence at end of buffer" failure
results.

The culprit is code marked "FIXME: This is wrong." in
stream.d.

The straightforward fix is to move the ignore_next_LF
flag from BufferedStream to ChannelStream and use it in
the two marked cases. This seems to work fine for me
locally. Perhaps I am overlooking some problem with
this approach.

Discussion

  • Sam Steingold
    Sam Steingold
    2006-09-25

    Logged In: YES
    user_id=5735

    could you please attach your patch to this issue?
    thanks

     
  • Mike McMahon
    Mike McMahon
    2006-09-26

    My changes as described.

     
    Attachments
  • Sam Steingold
    Sam Steingold
    2006-12-28

    Logged In: YES
    user_id=5735
    Originator: NO

    sorry for taking so long - we are not notified of new attachments unless they are accompanied by a comment.
    could you please also attach a test case that demonstrates the bug?
    (please also make a comment when attaching the test case so that we will notice it)
    thanks!

     
  • Sam Steingold
    Sam Steingold
    2007-01-09

    Logged In: YES
    user_id=5735
    Originator: NO

    your patch only fixed the buffered case.
    unbuffered streams are stil broken.
    (let ((f "unicode-file"))
    (unwind-protect
    (loop :for e1 :in '(charset:unicode-16-big-endian
    charset:unicode-16-little-endian
    charset:unicode-32-big-endian
    charset:unicode-32-little-endian)
    :for e = (ext:make-encoding :charset e1 :line-terminator :dos)
    :do (with-open-file (o f :direction :output :external-format e)
    (write-line f o))
    :nconc
    (loop :for b :in '(nil t) :nconc
    (with-open-file (i f :direction :input :external-format e
    :buffered b)
    (handler-case (let ((l (read-line i)))
    (if (string= f l) ()
    (list (list e b 'read-line l))))
    (error (c) (list (list e b 'read-line (princ-to-string c)))))
    (handler-case (let ((c (read-char i nil nil)))
    (and c (list (list e b 'read-char c))))
    (error (c) (list (list e b 'read-char (princ-to-string c))))))))
    (delete-file f)))

     
  • Sam Steingold
    Sam Steingold
    2008-05-19

    Logged In: YES
    user_id=5735
    Originator: NO

    thank you for your bug report.
    the bug has been fixed in the CVS tree.
    you can either wait for the next release (recommended)
    or check out the current CVS tree (see http://clisp.cons.org\)
    and build CLISP from the sources (be advised that between
    releases the CVS tree is very unstable and may not even build
    on your platform).

     
  • Sam Steingold
    Sam Steingold
    2008-05-19

    • assigned_to: haible --> sds
    • status: open --> closed-fixed