#261 win32 console copy&paste yields errors about 'L'

lisp error
open
clisp (525)
5
2006-01-31
2005-07-05
Jörg Höhle
No

Hi,

there have been mentions in the mailing list here and
there, but no use of the bug tracker.

Using a DOS console under MS-Windows-2k, it can happen
that CLISP complains about a spurious L, as in
*** - Invalid option in
(FFI:DEF-CALL-OUT GETPID (:NAME
"GetCurrentProcessId")
(:LIBRARY "kernel32.dll") L (:RETURN-TYPE
FFI:UINT32))
: L

I just happened to me now, so I investigated the
symptoms a little bit.
(defun foo() (loop for c = (read-char *standard-input*
nil nil nil) while c until (char= c #\%) collect c))
(foo) (coerce * 'string)

"(ffi:def-call-out getpid (:name
\"GetCurrentProcessId\")
l (:library \"kernel32.dll\") (:return-type
ffi:uint32))

It now appears that the spurious L is overwriting the
last line of copy&paste. Here's a sample test:
(a
b
c
d
f)
-> "(a
b
c
d
lf)
"
Further observations:
o The command line history of the MS-Windows console
does not show the junk, it looks correct.
o It happens with Copy&Paste from Netscape or from
Emacs-20.7.

o Work-around -- It does not happen when the copy&paste
block ends at the start of a new line (instead of after
the last character of things to copy). I had never
heard of a work-around before!

Using clisp-cvs, built with MS-VC 6.0 and
libsigsegv/cvs.
Regards,
Jörg Höhle.

Discussion

  • Jörg Höhle
    Jörg Höhle
    2005-07-06

    Logged In: YES
    user_id=377168

    Thanks for the reminder, I had forgotten those
    particular e-mails.

    Sam wrote:
    >OK, so the symptom is:
    >in a clisp build with mingw
    >when we paste a multi-line expression into a console
    >and the expression does _NOT_ end with a newline,
    >the last newline (and only it?) is read by CLISP as #\l (or
    #\L ?)

    I'd phrase it slightly differently:
    With either a mingw build, using a MS-XP console,
    or an MS-VC-6.0 build, using a MS-2k console,
    the first character of the last line is overwritten with #\l
    (and not #\L as my read-char tests shows) -- unless the line
    is terminated by a newline.

    I don't know how to choose among pasting CR or CRLF or LF
    terminated text into the console. I don't know whether the
    clipboard transforms line-terminators. I just select text
    from another MS-Windows GUI app and paste it.

    It still does not rule out a bug in CLISP, instead of
    MS-windows consoles IMHO.

     
  • Jörg Höhle
    Jörg Höhle
    2005-07-07

    Logged In: YES
    user_id=377168

    The bug does not appear in RAW mode, i.e.
    (with-keyboard (loop (print(read-char *keyboard-input*))))

     
  • Jörg Höhle
    Jörg Höhle
    2005-07-07

    Logged In: YES
    user_id=377168

    The bug appears when doing 1 character/byte at a time I/O to
    the MS-Windows console. Thus obviously, it's a long-standing
    bug in MS-Windows. However CLISP must question itself
    whether 1 by 1 I/O is that great... (It remembers me how it
    killed performance on AmigaOS).

    I tried to mimic CLISP's stream.d/win32aux.d using the FFI
    (def-c-type sword sint32)
    (def-call-out GetStdHandle (:name"GetStdHandle")(:library
    "kernel32.dll")
    (:return-type handle) (:language :stdc-stdcall)
    (:arguments (stdhandle sword)))
    (setq conin (GetStdHandle -10)) ; input handle
    http://msdn.microsoft.com/library/default.asp?url=/library/e
    n-us/dllproc/base/getstdhandle.asp

    (setq buf (allocate-shallow 'character :count 400))
    (def-call-out ReadFile (:name"ReadFile")(:library
    "kernel32.dll")
    (:return-type boolean) (:language :stdc-stdcall)
    (:arguments (handle handle) (buffer c-pointer)
    (bytestoread dword) (bytesread (c-ptr dword)
    :out)
    (overlapped c-pointer)))
    (defun foo1 ()
    (loop do (multiple-value-setq (ok num) (ReadFile conin buf
    1 nil))
    while ok do
    (prin1 (setq read (foreign-value(foreign-variable
    buf(parse-c-type `(c-array uint8 ,num))))))
    until (position (char-code #\%) read)))

    #(108)#(32)#(32)#(32)#(32)#(32)#(32)#(32)#(119)#(104)#(105)#
    (108)#(101)#(32)#(111)#(107)#(32)#(100)#(111)#(13)#(10)%

    Note that a priori, 1by1 is a waste for the console, since
    ReadFile() returns for each Return (even when using
    copy&paste), at least on my MS-Windows-2k box.
    So a line-buffered approach to console I/O would be a much
    better approach.
    I vaguely remember that the AmigaCLISP used a line buffer
    also?? or was that one of the three HAVE_TERMINAL1/2/3 which
    did this? I could't find HAVE_TERMINAL2 anymore in stream.d

    Experimenting a little more, reading 5 by 5, instead of #\l
    I get 5 completely junk characters
    #(108 0 105 0 115)#(32 32 32 119 104)
    overwriting 5 spaces from Copy&Paste... Looks like UNICODE
    junk from somewhere -- with an even longer buffer, it says
    "lisp.exe"(!)
    Now, how can one get MS to fix this??
    Have my colleagues create a CERT buffer overflow alert??

    The bug even showed up reading by blocks of 30 characters.
    But not with 100. It's probably silent when the buffer can
    hold a whole line.

    Regards,
    Jrg

     
  • Sam Steingold
    Sam Steingold
    2005-07-07

    Logged In: YES
    user_id=5735

    I confirm.
    [
    you also need
    (def-c-type handle c-pointer)
    (def-c-type dword uint32)
    ]

    I suggest that you get this information to CERT ASAP.
    we also need to switch to buffered consile i/o.
    terminal2 is no longer used and ChangeLog offers no
    explanations.
    maybe you could try enabling it in woe32?

     
  • Sam Steingold
    Sam Steingold
    2005-07-08

    Logged In: YES
    user_id=5735

    BTW, Jorg, could you please add all woe32 FFI forms you
    write to clisp/modules/bindings/win32/win32.lisp?
    let up grow the module!
    Thanks!

     
  • Jörg Höhle
    Jörg Höhle
    2005-07-12

    Logged In: YES
    user_id=377168

    Nope. I have very little time and am already very stressed
    already trying to do the following:
    o fix the c-struct bug #xx correctly
    o liasing with Kevin Rosenberg about UFFI open design issues
    and CLISP support
    o some other FFI stuff I'm even to stressed to remember
    about, possibly trying to have reasonable support fo strings
    -- raising the encoding_min/max issue again, but would be
    required for a general FOREIGN-STRING-LENGTH

    Others:
    o review update_library() & startup code
    o review and fix *foreign-encoding* in non-1:1 case --
    probably requires correct FOREIGN-STRING-LENGTH
    functionality, instead of asciz_string() which is only good
    for extended ASCII like UTF-8, but not UNICODE-16.
    o fix postgres module (that code has been sitting for 6
    month on one of the Laptops I can use)

    And I've always been an enemy of writing interfaces to
    hundreds of unused functions. The only one that is of
    interest to me is some callback test involving :Stdcall,
    possibly a directory scan filtering function -- I have yet
    to identify such a callback in the MS-Windows dll.
    Sorry

     
  • Sam Steingold
    Sam Steingold
    2005-07-12

    Logged In: YES
    user_id=5735

    you misunderstood me.
    all I am asking you about is to save whatever FFI forms you
    write anyway (for testing purposes or whatever) into CVS
    under clisp/modules/bindings/win32/win32.lisp.
    I am not asking you to interface to functions you do not need,
    just save the code you already wrote.

     
  • Sam Steingold
    Sam Steingold
    2006-01-31

    • assigned_to: sds --> ampy
     
  • It's funny. Just reproduced this in C. Certainly a Windows bug. I thought it has something to do with threads but it is not. To avoid the bug, one shouldn't use ReadFile on console with buffer less than 41 byte. Probably it's related to that 40 is minimal terminal width in DOS? I remember CGA had this line width...

     
  • Sam Steingold
    Sam Steingold
    2010-05-13

    so, I guess the fix is to ensure that we read from the console with a sufficiently large buffer, right?

     
  • This is the call stack in the old CLISP which I can build in MSVC (I'm not used to GDB yet).

    DoInterruptible
    fd_read
    low_read_unbuffered_handle
    rd_ch_unbuffered
    rd_ch_terminal1
    read_line
    C_read_line
    funcall_subr
    funcall
    read_form

    So to implement read_line clisp reads characters one by one on the rather high level, extracting handle from stream each time etc. When it comes to ReadFile, there's no other choice other than to give it the buffer of unary length. I'm not ready yet to hack this complex machinery.