#366 CRLF handling does not work for UNICODE-16

lisp error
closed-fixed
clisp (525)
5
2008-05-19
2006-09-25
No

CRLF is not converted to #\Newline by READ-CHAR and
READ-LINE from 16-bit Unicode files. (clisp 2.39 MinGW
no-readline XP.)

For -BIG-ENDIAN, the LF is not recognized and an extra
blank line is input.

For -LITTLE-ENDIAN, only the LF byte is removed and an
"incomplete byte sequence at end of buffer" failure
results.

The culprit is code marked "FIXME: This is wrong." in
stream.d.

The straightforward fix is to move the ignore_next_LF
flag from BufferedStream to ChannelStream and use it in
the two marked cases. This seems to work fine for me
locally. Perhaps I am overlooking some problem with
this approach.

Discussion

  • Sam Steingold

    Sam Steingold - 2006-09-25

    Logged In: YES
    user_id=5735

    could you please attach your patch to this issue?
    thanks

     
  • Mike McMahon

    Mike McMahon - 2006-09-26

    My changes as described.

     
  • Sam Steingold

    Sam Steingold - 2006-12-28

    Logged In: YES
    user_id=5735
    Originator: NO

    sorry for taking so long - we are not notified of new attachments unless they are accompanied by a comment.
    could you please also attach a test case that demonstrates the bug?
    (please also make a comment when attaching the test case so that we will notice it)
    thanks!

     
  • Sam Steingold

    Sam Steingold - 2007-01-09

    Logged In: YES
    user_id=5735
    Originator: NO

    your patch only fixed the buffered case.
    unbuffered streams are stil broken.
    (let ((f "unicode-file"))
    (unwind-protect
    (loop :for e1 :in '(charset:unicode-16-big-endian
    charset:unicode-16-little-endian
    charset:unicode-32-big-endian
    charset:unicode-32-little-endian)
    :for e = (ext:make-encoding :charset e1 :line-terminator :dos)
    :do (with-open-file (o f :direction :output :external-format e)
    (write-line f o))
    :nconc
    (loop :for b :in '(nil t) :nconc
    (with-open-file (i f :direction :input :external-format e
    :buffered b)
    (handler-case (let ((l (read-line i)))
    (if (string= f l) ()
    (list (list e b 'read-line l))))
    (error (c) (list (list e b 'read-line (princ-to-string c)))))
    (handler-case (let ((c (read-char i nil nil)))
    (and c (list (list e b 'read-char c))))
    (error (c) (list (list e b 'read-char (princ-to-string c))))))))
    (delete-file f)))

     
  • Sam Steingold

    Sam Steingold - 2008-05-19

    Logged In: YES
    user_id=5735
    Originator: NO

    thank you for your bug report.
    the bug has been fixed in the CVS tree.
    you can either wait for the next release (recommended)
    or check out the current CVS tree (see http://clisp.cons.org\)
    and build CLISP from the sources (be advised that between
    releases the CVS tree is very unstable and may not even build
    on your platform).

     
  • Sam Steingold

    Sam Steingold - 2008-05-19
    • assigned_to: haible --> sds
    • status: open --> closed-fixed
     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks