#529 Command line and unicode characters

[Kalith]

Using Linux Mint 14 (Ubuntu 12.11) and calling GDL from gnome-terminal, the command line gets crazy when a non ASCII character is entered. For examples (the position of the caret is symbolized by a pipe '|') :

Case A:
1) type:
> abcù|
2) press the 'backspace' key to remove the 'ù':
> abc|
3) press 'enter' to evaluate the expression:
% Token stream error: unexpected char: 0xC3

The encoding here is probably UTF8, meaning that the 'ù' character is coded on two bytes (1: escape character, 2: code corresponding to 'ù'). It seems that, when pressing backspace, only the second of those two bytes is removed, and the escape character (0xC3) is still there, although invisible because not printable. I usually types such characters by mistake, and it gets very annoying because it is hard (if not impossible?) to fix an input line that has been corrupted like that.

Case B:
1) type:
> abcù|
2) move the carret left by three characters:
> a|bcù
3) move it right once:
> ac|cù

The text is modified: 'abc' -> 'acc'.

[Alain C.]

the bad new:
1/ I have the same problem, in GDL but also in bash ...
2/ no idea how to solve such problem in the code

the good new:
in readline in GDL, we do have a lot of Ctrl commands, like ctrl-u, ctrl-a, ctrl-k
I use to use to clean up the line !

Alain

[giloo]

works for me, mandriva 2010.2, KDE + Konsole.
IDL has some problems: translates uncorrectly:
IDL> abcù
% Attempt to call undefined procedure/function: 'ABCY'.
so if there was a function ABCY it would have been called (wrongly)

GDL does not like it either but i believe the behaviour is better since no atte,pt to call a function is done:

GDL> abcù
% Token stream error: unexpected char: 0xF9
GDL> abc
% Procedure not found: ABC
% Execution halted at: $MAIN$

[Kalith]

There must be someting wrong in GDL, because this simple example runs fine (non ASCII characters are correctly handled and printed) :

#include <iostream>
#include <cstdlib>
#include <readline/readline.h>
#include <readline/history.h>

int main() {
rl_initialize();

char* buf;

while ((buf = readline("\n> ")) != 0) {
std::cout << buf << std::endl;
if (buf[0] != '\0') add_history(buf);
}

free(buf);

return 0;
}

I have tried to look at GDL's source code, but couldn't find anything strange (plus readline is rather straightforward to use, except if there is a custom configuration file that is loaded for GDL?).

[Sylwester Arabas]

Perhaps a hint: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=725957

Seems that changing from autoconf to CMake in Debian causes gdl to pick ncurses (no unicode support) instead of ncursesw (with unicode support). I haven't looked into it - just postig here for the record. Seems like it could be a related issue.

Sylwester

[Sylwester Arabas]

For the record, the lack of unicode-curses support in CMake has been identified as a CMake bug yet in Feb 2010:
http://www.cmake.org/Bug/view.php?id=10347
I've asked there for a best workaround.

Sylwester

[Sylwester Arabas]

Moving the discussion to github: https://github.com/gnudatalanguage/gdl/issues/219