Re: [Freedos-devel] ASCII to unicode table

[Eric A. <e....@jp...>]

Hi Christian,

Using UTF-8 CON with a codepage based app or vice versa
is worse for block graphics than just using the wrong
codepage, as not only the shape but also the number of
displayed characters will change: Everything outside of
basic ASCII takes 2 or more bytes in UTF-8, so display
on codepage-CON will show block graphics "too wide".

In the other direction, sequences can contain invalid
groups or invalid start bytes, so trying to show block
graphics from codepage-apps on UTF8-CON will typically
show as many or fewer "bad char" chars than the number
of block graphics chars that the app wanted to display.



>> A possible workaround would be dosver-style, to make
 >> a per-app decision who uses Unicode.

Because DOS is not multitasking, you do not have to put
status flags in the PSP...  You just switch to codepage
(or whatever default you want) mode when anything exits
and switch to UTF8 mode (...) when either an app starts
which you know to be UTF8 tolerant or when a modern app
explicitly switches to UTF8 mode. You are right that a
TSR pop-up would not fit in that scheme BUT as far as I
remember, pop-ups always write to the VGA directly, so
they cannot use the UTF8 CON. If the UTF8 CON uses the
graphics mode to render text (because otherwise you can
only keep a small 512 char font in hardware) it is very
possible that you will not see your TSR pop-up at all.



> I'd propose to use a new interface instead - this new interface
 > then always uses UTF-8, the normal one will use code pages (or
 > reject CP-dependent characters). (Of course using only ASCII it
 > doesn't matter which interface you use.)

What that could mean is having a UNICODE$ char device,
similar to the existing MORE$ device which you already
know: It forwards text to CON but waits for a keypress
after every 25 line breaks, so MORE$ (moresys) shows
text immediately where MORE (the app) has to wait for
all text to arrive first before starting to show any
of it (because DOS does not have real | pipelines...).

Well... Coming back from this blatant ad ;-) A driver
which provides this UNICODE$ device could either do a
"best effort" translation of incoming text to whatever
the current codepage is at that moment or it could do
a graphical rendering of the text. In the latter case,
you can only show text while the VGA is in graphics
mode which is acceptable for classic CON (just slow)
but which, as said, will break your TSR pop-up text.



> The DOS LFN API works with code page encoded strings.

Wow. Well at least the DOS LFN directory item data is
based on Unicode already. So it could have been worse.

Eric