mlterm-dev-en — mlterm mailing list(English)

Re: [Mlterm-dev-en] mlterm, tibetan, and bitmap fonts?

[MINAMI H. <mi...@mi...>]

On Wed, 2006-03-08 at 03:48 -0500, Rich Felker wrote:
> On Tue, Mar 07, 2006 at 02:45:57AM +0900, MINAMI Hirokazu wrote:
> > > I finally succeeded in getting mlterm to display Tibetan on my laptop,
> > > but it's horribly misrendered. Here's a screenshot:
> > > 
> > > http://brightrain.aerifal.cx/~dalias/tibetan_misrender.png
> > 
> > You have to enable "variable column width" and Xft support
> > to render some character as zero-width.
> > Under the configuration, simple over-striking with vertical
> > shift can be performed.
> > 
> > What I saw was:
> > http://mistfall.net/minami/tmp/tibetan_w3m.png
> > 
> > which is not looks so bad to me.
> 
> Nope, it's horribly misrendered. All of the combining characters are
> displayed a whole character-cell to the left of where they belong. I
> don't see any characters that require a special form for combining,
> but I expect those would be wrong too.

Hmm, it shows how Xft renders Tibetan represented in UCS4.  
So 
 - All of the combining characters are displayed a whole
   character-cell to the left of where they belong.
may mean there's a bug in whether Xft or freetype or the Tibetan font
that glyph position is not correctly calculated. 

Since some of "a special form for combining" may not exist in Unicode,
it's not surprising if they cannot be displayed using UCS4.

Anyway, if you believe the result if far from the acceptable one,
we have to consider to use more advanced features of a font,
instead of just hacking Xft.

> > > I also got around to reading the mlterm source, and it seems that
> > > there's no effort made to position combining characters; they're just
> > > blindly displayed relative to the same origin as the base character
> > > with overstrike.
> > 
> > For combined character in Arabic, mlterm perform combining by
> > converting a group of non-combined UCS4 codes to a combined form.
> > Since they cannot be rendered simply vertically stacking each glyphs.
> 
> AFAIK these are not combining characters but simply ligatures.
> > The code and its conversion table is in ml_shape.c.
> > 
> > For Tibetan, however, the same approach cannot be used because
> > combined form of Tibetan glyphs are not registered in Unicode.
> 
> Combining and shaping are two completely different issues as far as
> unicode/ucs semantics go.

I'm not sure why we should distinguish liguatures/combining/shaping
here. We just have to convert a special sequence of UCS4 into
corresponding glyph in all cases.

Do you think combining characters need more complex processing?

However, the Unicode's semantics is not usable to combine Tibetan.
Because Unicode do not contain pre-combined glyphs for Tibetan,
we can't write a rule to map a (un-combined) Tibetan UCS4 sequence
into a already-combined one.

# And the same is true for some combined-form of Arabic glyphs.
# At present, mlterm cannot use such glyphs unless they are 
# accessible using Unicode.

Re: [Mlterm-dev-en] mlterm, tibetan, and bitmap fonts?

[MINAMI H. <mi...@mi...>]

On Wed, 2006-03-08 at 03:32 -0500, Rich Felker wrote:...

> > With old x core font, I don't think we can use glyphs outside of
> > Unicode.
> > 
> > Do you think we can force to use Xft?
> 
> My question was about _bitmap_ fonts, which are totally outside the
> domain of Xft. This is a terminal, not a typesetting system. High
> quality scalable fonts have their place, but tiny character-cell
> systems is not one of them, IMHO.

Using Xft is not a mechanism to handle anti-aliased/scalable fonts.
Xft *can* handle embedded bitmap glyphs.

So what I care was about the overhead required to use Xft.
Does Xft work for you with acceptable load? 

> > The problem is, mlterm can't know whether an application running on it
> > knows Tibetan or not. When both of mlterm and the application perform
> > conversion, the result will be horrible. But some problems also occur
> > if none of them cares about combined characters.
> 
> Conversion? There is no conversion, and nothing at all
> Tibetan-specific. The above applies to all combining characters
> equally, in all scripts including Latin.

I agree processing of combined chars are not Tibetan specific.

But it means most existing console applications do not care about
combining characters including Latin's ones.
If you want to make such programs work on mlterm, you may need
some tricks like sending a few, instead of one, "^?" when BS key
was pressed.  


> > > > - What should we do if part of combined character was overwritten? 
> > > 
> > > I don't think this is possible. The terminal control codes to move to
> > > a position are based on character cells, not characters, right?
> > 
> > There are no standard to handle combined characters on terminal.
> > So an application may be implemented in either way. 
> 
> Any terminal implementation that uses character-count positioning
> rather than character-cell positioning is hopelessly broken for both
> wide and combining characters..


Not really. (Maybe-good) old days when a real terminal was developed,
everyone can believe that character-count ==  character-cell position.
So there's no standard for terminals describes about how to count 
characters in column.
Since a emulator is not required to do more than what it emulates did,
we can't blame a terminal emulator for it is not conforming particular
specs.



> > > On the other hand, what happens when you overwrite just half of a cjk
> > > doublewidth character?
> > 
> > If the application is wise enough, entire character will be erased.
> > Otherwise, we see garbage.
> 
> I think you mean the terminal, not the application..

Ah yes, form my point of view, a terminal emulator is
nothing but an application :)

regards,

[Mlterm-dev-en] Re: variable width X font configuration

[Seiichi S. <ss...@sh...>]

On Mon, Mar 27, 2006 at 01:05:32AM -0800,
 John Magolske wrote:

> in ~/.mlterm/vfont I've tried each of the following in turn:
> 
> ISO8859_1=12,-*-helvetica-medium-r-*-*-12-*-*-*-*-*-iso8859-1
> ISO8859_1=12,-helvetica-medium-r-*-12-*-iso8859-1
> ISO8859_1=12,-helvetica-*-medium-r-*-12-*-iso8859*
> ISO10646_1=12,-helvetica-medium-r-*-12-*-iso10646-1
> 
> but mlterm -V does not bring up the desired helvetica font,
> only a default mono-spaced font instead.

If you want to use a 12 pixel font, give -w option to specify
font size.

--
Seiichi

[Mlterm-dev-en] variable width X font configuration

[John M. <lis...@b7...>]

Hi,

I'd like to have a proportional X font display in MLterm.
xfontsel tells me :

-*-helvetica-medium-r-*-*-12-*-*-*-*-*-iso8859-1

in ~/.mlterm/vfont I've tried each of the following in turn:

ISO8859_1=12,-*-helvetica-medium-r-*-*-12-*-*-*-*-*-iso8859-1
ISO8859_1=12,-helvetica-medium-r-*-12-*-iso8859-1
ISO8859_1=12,-helvetica-*-medium-r-*-12-*-iso8859*
ISO10646_1=12,-helvetica-medium-r-*-12-*-iso10646-1

but mlterm -V does not bring up the desired helvetica font,
only a default mono-spaced font instead.

Checking & unchecking "Anti Alias" in the control panel will
cause the font in aafont to be selected & de-selected,
but I don't see any change when checking & un-checking
the "Variable column width" option.

This is MLterm v2.9.2 on Mac OS X 10.3.9

Thanks for any suggestions,

John

Re: [Mlterm-dev-en] mlterm, tibetan, and bitmap fonts?

[Rich F. <da...@ae...>]

On Wed, Mar 08, 2006 at 03:48:47AM -0500, Rich Felker wrote:
> On Tue, Mar 07, 2006 at 02:45:57AM +0900, MINAMI Hirokazu wrote:
> > The plan I had was:
> > 
> > By assuming specific font, the conversion table can be hard-coded.
> > i.e. when the glyph ordering of the font to be used is known,
> > we can write conversion rules like following:
> > 0x0f40, 0x0f72 => glyph ID xxxx
> > 0x0f40, 0x0f7f => glyph ID yyyy
> 
> Yes... Maybe the most appropriate solution is to make a special
> encoding for precombined Tibetan and include the mappings to that
> encoding in mlterm so it could use fonts in that encoding. Actually...
> there are already such encodings for nasty pre-Unicode hacks
> (basically they were fake latin1 fonts with Tibetan characters in
> place of the letters). For the sake of being able to use these legacy
> fonts too, they could be used as the basis for such an encoding.

After reviewing the situation, it seems the the TCRC font (the most
popular one) uses overstriking hacks wherever possible to fit all
glyphs in the single-byte range. This is obviously not acceptable as a
glyph encoding because the ability to use overstrike is very
font-specific and generally incorrect. There's another somewhat
established encoding, from the "Tibetan Machine" font, which could be
used instead. My understanding is that it contains all possible
stacks and does not use overstriking.

> > With Xft, drawing for non USASCII/ISO8859-1 characters are processed in 
> > xwindow/x_window.c:x_window_xft_draw_string32().
> > the function takes an array of UCS4 codes like 0x0f40, 0x0f72... .
> > 
> > So it may be possible to hack the function to watch input
> > and if the input sequence were Tibetan chars to be combined,
> > intercept them and call XftDrawGlyphs() instead.
> 
> Again I don't think special-casing Tibetan is appropriate. The same
> issues apply to all combining characters, including most South Asian
> languages, mathematical combining marks, etc. and they can all be
> handled in a unified way IMO.

Hmm, it seems like I said two rather conflicting things here. What I
meant to say is that, while supporting script-specific glyph encodings
separate from character encodings is a possible solution (and probably
results in maximal font quality), what we really need is a mechanism
for all fonts (including bitmap fonts) to carry with them anchor point
information for combining, so that the terminal emulator can perform
arbitrary combining/stacking rather than just a fixed set of glyphs.

Rich

Re: [Mlterm-dev-en] mlterm, tibetan, and bitmap fonts?

[Rich F. <da...@ae...>]

On Tue, Mar 07, 2006 at 02:45:57AM +0900, MINAMI Hirokazu wrote:
> > I finally succeeded in getting mlterm to display Tibetan on my laptop,
> > but it's horribly misrendered. Here's a screenshot:
> > 
> > http://brightrain.aerifal.cx/~dalias/tibetan_misrender.png
> 
> You have to enable "variable column width" and Xft support
> to render some character as zero-width.
> Under the configuration, simple over-striking with vertical
> shift can be performed.
> 
> What I saw was:
> http://mistfall.net/minami/tmp/tibetan_w3m.png
> 
> which is not looks so bad to me.

Nope, it's horribly misrendered. All of the combining characters are
displayed a whole character-cell to the left of where they belong. I
don't see any characters that require a special form for combining,
but I expect those would be wrong too.

> > I also got around to reading the mlterm source, and it seems that
> > there's no effort made to position combining characters; they're just
> > blindly displayed relative to the same origin as the base character
> > with overstrike.
> 
> For combined character in Arabic, mlterm perform combining by
> converting a group of non-combined UCS4 codes to a combined form.
> Since they cannot be rendered simply vertically stacking each glyphs.

AFAIK these are not combining characters but simply ligatures.

> The code and its conversion table is in ml_shape.c.
> 
> For Tibetan, however, the same approach cannot be used because
> combined form of Tibetan glyphs are not registered in Unicode.

Combining and shaping are two completely different issues as far as
unicode/ucs semantics go.

> > Personally I'm of the opinion that, except for accents on latin
> > characters and other simple diacritics, automatic generation of
> > combined glyphs will never look great, so I think we need some sort of
> > solution for using precombined glyphs. If there were an artificial
> > non-unicode encoding, then mlterm could get to the glyphs that way
> > similarly to how it can use legacy-encoded Japanese, etc. fonts even
> > in unicode mode. And of course it would be a big bonus if precombined
> > bitmap glyphs could be used as well.
> 
> The plan I had was:
> 
> By assuming specific font, the conversion table can be hard-coded.
> i.e. when the glyph ordering of the font to be used is known,
> we can write conversion rules like following:
> 0x0f40, 0x0f72 => glyph ID xxxx
> 0x0f40, 0x0f7f => glyph ID yyyy

Yes... Maybe the most appropriate solution is to make a special
encoding for precombined Tibetan and include the mappings to that
encoding in mlterm so it could use fonts in that encoding. Actually...
there are already such encodings for nasty pre-Unicode hacks
(basically they were fake latin1 fonts with Tibetan characters in
place of the letters). For the sake of being able to use these legacy
fonts too, they could be used as the basis for such an encoding.

> With Xft, drawing for non USASCII/ISO8859-1 characters are processed in 
> xwindow/x_window.c:x_window_xft_draw_string32().
> the function takes an array of UCS4 codes like 0x0f40, 0x0f72... .
> 
> So it may be possible to hack the function to watch input
> and if the input sequence were Tibetan chars to be combined,
> intercept them and call XftDrawGlyphs() instead.

Again I don't think special-casing Tibetan is appropriate. The same
issues apply to all combining characters, including most South Asian
languages, mathematical combining marks, etc. and they can all be
handled in a unified way IMO.

> ... and we can replace the converter from UCS4 to glyph ID 
> to be able to handle any font using libotf/freetype/ or something.

According to someone I've been working with on unrelated stuff,
freetype will automatically do all the combined glyph rendering for
you using the opentype info in the font file. Will Xft make use of
that? Or is some huge bloat like pango needed?

BTW, this won't work for bitmap fonts, and ideally terminals should be
using bitmap fonts..

> Unfortunately, I don't have enough time to do this for a while.
> Patches are welcome, of course.

:)
In the meantime I'm working on other components of my system overhaul,
but I'll look at this again when I get to it. Thanks for the replies.

Rich

Re: [Mlterm-dev-en] mlterm, tibetan, and bitmap fonts?

[Rich F. <da...@ae...>]

On Tue, Mar 07, 2006 at 01:13:07AM +0900, MINAMI Hirokazu wrote:
> Sorry for late reply.
> I've just started to process my pile of To-Do...
> 
> On Sun, 2005-12-18 at 21:40 -0500, Rich Felker wrote:
> 
> > Is there any way to use a glyph substitution table from an external
> > file and still use the fonts through X? That is to say: does X make
> > the precombined glyphs available in any way at all, or are they
> > completely inaccessible? If there's a sane way to make them accessible
> > and just use a separate subst table it would work for bitmap fonts
> > too, which would be a big plus.
> 
> If you can convert a UCS4 sequence to corresponding id and can use Xft,
> XftDrawGlyphs() which takes an array of glyph-id may be used to draw
> any glyph in a font. The conversion is not trivial, though.
> 
> With old x core font, I don't think we can use glyphs outside of
> Unicode.
> 
> Do you think we can force to use Xft?

My question was about _bitmap_ fonts, which are totally outside the
domain of Xft. This is a terminal, not a typesetting system. High
quality scalable fonts have their place, but tiny character-cell
systems is not one of them, IMHO.

> > > - When you press Delete/Backspace key in a combined character,
> > > Should we delete entire character or part of it?
> > > - When you press right/left key, should a cursor move one consonant or
> > > one character?
> > 
> > Hard to say. Either way would be usable. This is definitely left to
> > the application however (or when using cooked line-based input at the
> > terminal, the kernel tty driver).
> 
> The problem is, mlterm can't know whether an application running on it
> knows Tibetan or not. When both of mlterm and the application perform
> conversion, the result will be horrible. But some problems also occur
> if none of them cares about combined characters.

Conversion? There is no conversion, and nothing at all
Tibetan-specific. The above applies to all combining characters
equally, in all scripts including Latin.

> > > - What should we do if part of combined character was overwritten? 
> > 
> > I don't think this is possible. The terminal control codes to move to
> > a position are based on character cells, not characters, right?
> 
> There are no standard to handle combined characters on terminal.
> So an application may be implemented in either way. 

Any terminal implementation that uses character-count positioning
rather than character-cell positioning is hopelessly broken for both
wide and combining characters..

> > On the other hand, what happens when you overwrite just half of a cjk
> > doublewidth character?
> 
> If the application is wise enough, entire character will be erased.
> Otherwise, we see garbage.

I think you mean the terminal, not the application..

Rich

[Mlterm-dev-en] Re: tis620 encoding

[Seiichi S. <ss...@sh...>]

On Thu, Mar 02, 2006 at 07:07:23PM -0500,
 George Cross wrote:

> xlsfonts
> -adobe-courier-bold-o-normal--0-0-100-100-m-0-iso8859-1
[snip]
> xif-s25

There are no fonts for tis620.2533-1, tis620.2529-1, iso8859-11 and
iso10646-1. Please make sure thai fonts have been installed correctly.

-- 
Seiichi

Re: [Mlterm-dev-en] scim dlopen problem

[MINAMI H. <mi...@mi...>]

On Wed, 2006-01-18 at 16:43 +0100, wagner frederic wrote:
> the binary debian package is achieving to load scim so why not if
> I compile it myself from source ?

The package in debian unstable seems to be built
using the source fetched from CVS.

Plaese try to use 'apt-get source mlterm' to rebuild mlterm
in debian's way or checkout CVS HEAD by yurself.

# I know we should release 2.9.3 someday...

--
MINAMI Hirokazu <mi...@mi...>

Re: [Mlterm-dev-en] mlterm, tibetan, and bitmap fonts?

[MINAMI H. <mi...@mi...>]

On Mon, 2005-12-19 at 15:45 -0500, Rich Felker wrote:
> On Mon, Dec 19, 2005 at 01:30:47AM +0900, mi...@mi... wrote:
> > Hi. 
> > 
> > I've tried to see Tibetan text by following procedure.
> > Please let me know you have used some different way or
> > you are using another font. 
> > 
> > 1. install "Tibetan Machine Uni" from www.thdl.org and register the font to 
> > fontconfig library. 
> > 
> > 2. create ~/.mlterm/aafont and add:
> > ISO10646_UCS4_1=Tibetan Machine Uni-iso10646-1; 
> > 
> > 3. run mlterm with anti-alias enabled:
> > mlterm -A 
> > 
> > 4. display a Tibetan web page by w3m on mlterm:
> > w3m "http://www.thdl.org/xml/show.php?xml=test/tibetnew/thdlhp.xml&lng=tib" 
> 
> I finally succeeded in getting mlterm to display Tibetan on my laptop,
> but it's horribly misrendered. Here's a screenshot:
> 
> http://brightrain.aerifal.cx/~dalias/tibetan_misrender.png

You have to enable "variable column width" and Xft support
to render some character as zero-width.
Under the configuration, simple over-striking with vertical
shift can be performed.

What I saw was:
http://mistfall.net/minami/tmp/tibetan_w3m.png

which is not looks so bad to me.

> I also got around to reading the mlterm source, and it seems that
> there's no effort made to position combining characters; they're just
> blindly displayed relative to the same origin as the base character
> with overstrike.

For combined character in Arabic, mlterm perform combining by
converting a group of non-combined UCS4 codes to a combined form.
Since they cannot be rendered simply vertically stacking each glyphs.
 
The code and its conversion table is in ml_shape.c.

For Tibetan, however, the same approach cannot be used because
combined form of Tibetan glyphs are not registered in Unicode.


> With that in mind, I'm trying to decide what the best way to fix this
> is, short of having mlterm use FreeType directly. I could either:
> 
> - build some sort of tables to load in mlterm that tell it how to
>   correctly combine characters in a given font.
> - make an artificial encoding of some sort to allow access to the
>   precombined glyphs.
> 
> Personally I'm of the opinion that, except for accents on latin
> characters and other simple diacritics, automatic generation of
> combined glyphs will never look great, so I think we need some sort of
> solution for using precombined glyphs. If there were an artificial
> non-unicode encoding, then mlterm could get to the glyphs that way
> similarly to how it can use legacy-encoded Japanese, etc. fonts even
> in unicode mode. And of course it would be a big bonus if precombined
> bitmap glyphs could be used as well.

The plan I had was:

By assuming specific font, the conversion table can be hard-coded.
i.e. when the glyph ordering of the font to be used is known,
we can write conversion rules like following:
0x0f40, 0x0f72 => glyph ID xxxx
0x0f40, 0x0f7f => glyph ID yyyy

With Xft, drawing for non USASCII/ISO8859-1 characters are processed in 
xwindow/x_window.c:x_window_xft_draw_string32().
the function takes an array of UCS4 codes like 0x0f40, 0x0f72... .

So it may be possible to hack the function to watch input
and if the input sequence were Tibetan chars to be combined,
intercept them and call XftDrawGlyphs() instead.

... and we can replace the converter from UCS4 to glyph ID 
to be able to handle any font using libotf/freetype/ or something.


Unfortunately, I don't have enough time to do this for a while.
Patches are welcome, of course.

--
MINAMI Hirokazu <mi...@mi...>

Re: [Mlterm-dev-en] mlterm, tibetan, and bitmap fonts?

[MINAMI H. <mi...@mi...>]

On Mon, 2005-12-19 at 08:34 -0500, Rich Felker wrote:
> With traditional character cell storage, a fixed size window holds a
> finite fixed number of characters (W*H). However with combining
> characters in unicode, it seems that a utf8 terminal has no upper
> bound on the storage requirement. Is it appropriate to just choose an
> arbitrary limit?

In mlterm, we have assumed that a combined char can be represented
by up to seven UCS4 codes.

Since reserving (W*7) chars instead of (W*H) for a line 
is considered to be too wasteful,
extra memory for combined chars are dynamically allocated
only when necessary.

We could use linked-list to support arbitrary long sequence.
But I don't think we can bear to store a pointer to next node 
(=extra 4 or 8 bytes) for every character.


--
MINAMI Hirokazu <mi...@mi...> 

Re: [Mlterm-dev-en] mlterm, tibetan, and bitmap fonts?

[MINAMI H. <mi...@mi...>]

Sorry for late reply.
I've just started to process my pile of To-Do...

On Sun, 2005-12-18 at 21:40 -0500, Rich Felker wrote:

> Is there any way to use a glyph substitution table from an external
> file and still use the fonts through X? That is to say: does X make
> the precombined glyphs available in any way at all, or are they
> completely inaccessible? If there's a sane way to make them accessible
> and just use a separate subst table it would work for bitmap fonts
> too, which would be a big plus.

If you can convert a UCS4 sequence to corresponding id and can use Xft,
XftDrawGlyphs() which takes an array of glyph-id may be used to draw
any glyph in a font. The conversion is not trivial, though.

With old x core font, I don't think we can use glyphs outside of
Unicode.

Do you think we can force to use Xft?


> > Alternatively, we may able to use m17n-lib
> > (http://www.m17n.org/m17n-lib/)
> > as a rendering engine. Though I'm not yet investigated deeply,
> > implementation of mlterm side can simpler in this case. 
> 
> Yes I've been looking at m17n-lib a lot since I wrote. It seems to
> have the only reasonable Tibetan input methods on *nix (altho I was
> just planning on adding key bindings to screen to switch keyboard to
> tibetan mode and back). I haven't looked at its rendering stuff yet. I
> see that mlterm supports m16n-lib already; what does it use it for?

mlterm is using m17n-lib only as one of input frameworks
like XIM/iiimf/uim/scim.

> > Supporting to edit Tibetan on command line / editors like vim can be
> > very difficult. Because there's no standard for handling of combined
> > characters on a terminal, you have to define "the right way" at first.
> > ex.)
> 
> Some of these questions belong at the terminal level, some at the pty
> level, and some at the application level.
> 
> > - When you press Delete/Backspace key in a combined character,
> > Should we delete entire character or part of it?
> > - When you press right/left key, should a cursor move one consonant or
> > one character?
> 
> Hard to say. Either way would be usable. This is definitely left to
> the application however (or when using cooked line-based input at the
> terminal, the kernel tty driver).

The problem is, mlterm can't know whether an application running on it
knows Tibetan or not. When both of mlterm and the application perform
conversion, the result will be horrible. But some problems also occur
if none of them cares about combined characters.

> > - What should we do if part of combined character was overwritten? 
> 
> I don't think this is possible. The terminal control codes to move to
> a position are based on character cells, not characters, right?

There are no standard to handle combined characters on terminal.
So an application may be implemented in either way. 

> On the other hand, what happens when you overwrite just half of a cjk
> doublewidth character?

If the application is wise enough, entire character will be erased.
Otherwise, we see garbage.

--
MINAMI Hirokazu <mi...@mi...>

Re: [Mlterm-dev-en] tis620 encoding

[Seiichi S. <ss...@sh...>]

On Thu, Feb 02, 2006 at 02:22:11PM -0500,
 George Cross wrote:

> I'm on hpux 11.11 with my locale set to th_TH.tis620.
> 
> I'd like to use mlterm, but I get the error "No fonts found for charset
> tis620.2533-1 or tis620.2529-1."
> 
> I've got lots of ucs2 fonts with thai glyphs, and some iso8899-11 (Thai single
> byte encoding) fonts. Can't I use them somehow?

I could not reproduce the problem on my debian box.
Could you send me your ~/.mlterm/*font* files and
'xlsfonts' command results?

-- 
Seiichi

[Mlterm-dev-en] mlterm under Xgl: open_screen_intern() failed

[Alex L. <are...@gm...>]

Hi,

I'm running Ubuntu Dapper (i386) with more or less the latest updates.
Mlterm usually works great, but when I recently gave Xgl a try
(because of the recent Compiz release) I found that mlterm won't start
up, giving the following error: open_screen_intern() failed. If I
switch back to the usual xorg server, everything is fine.

Can anyone give me pointers on how to figure out what's going on and
address this problem? Thanks,

Alex

[Mlterm-dev-en] tis620 encoding

[George C. <geo...@ex...>]

 Hi All, I'm on hpux 11.11 with my locale set to th_TH.tis620. I'd like to use mlterm, but I get the error "No  fonts found for charset tis620.2533-1 or tis620.2529-1." I've got lots of ucs2 fonts with thai glyphs, and some iso8899-11 (Thai single byte encoding) fonts.  Can't I use them somehow?  Thank you much anyone who can give me some advice.  Sincerley, George 

_______________________________________________
Join Excite! - http://www.excite.com
The most personalized portal on the Web!

[Mlterm-dev-en] scim dlopen problem

[wagner f. <Fre...@lo...>]

hello, I currently have a bug in mlterm with combining characters and I
wanted to debug it myself.

I am on debian unstable with scim version 1.4.2-1

so I downloaded mlterm 2.9.2, and started a=20

./configure --enable-debug --enable-fribidi --enable-scim

followed by make

it complained about the following problem :


im_scim_1.2.cpp: In function `int im_scim_initialize(char *)':
im_scim_1.2.cpp:536: no matching function for call to
`scim::ConfigModule::create_config (const char[5])'
/usr/include/scim-1.0/scim_config_module.h:117: candidates are: class
scim::ConfigPointer scim::ConfigModule::create_config() const

but this was quickly fixed by removing the "scim" string in
im_scim_1.2.cc

make again, and everything finished compiling.

the only problem is that mlterm fails loading the scim module

when launching ./xwindow/mlterm I got a=20

*** ERROR HAPPEND ***  scim: Could not load.

strace confirmed me that the right library=20
/usr/local/lib/mlterm/libim-scim.so was being loaded

so I added in the file  kiklib/src/kik_dlfcn_dl.c
a small error message after the dlopen :

   if( ( ret =3D dlopen( path , RTLD_LAZY)))
    {
        return  (kik_dl_handle_t)ret ;
    }
    else {
        fprintf(stderr, "error: %s\n", dlerror());
    }

when running the programm I have :

error: /usr/local/lib/mlterm/libim-scim.so: undefined symbol:
_._Q24scim11Transaction
*** ERROR HAPPEND ***  scim: Could not load.

at this point this is becoming a bit too tough for me.

has anyone ever experienced something like that ? any idea what could go
wrong here ?

i've tried different versions of gcc (2.95 and 4.0) but it has been the
same problem on each case.

also, the binary debian package is achieving to load scim so why not if
I compile it myself from source ?

thanks for the help

Wagner Frederic
--=20
Why fear computers ? They don't byte !

Re: [Mlterm-dev-en] shift shortcuts

[Seiichi S. <ss...@sh...>]

On Tue, Jan 03, 2006 at 12:32:44AM +0100,
 Alexandros Droseltis wrote:

> I started using mlterm and I like it very much. My only problem is that
> all shortcuts that I had defined that include the shift key (in vim,
> screen) do not work. I searched many hours without success. Any idea?
> I would be grateful.

Some key codes, such as Shift+Fn, are not implemented yet.

--
Seiichi

[Mlterm-dev-en] shift shortcuts

[Alexandros D. <dro...@ma...>]

Hallo everyone!

# mlterm --version
mlterm version 2.8.0 post/cvs-1.683

I started using mlterm and I like it very much. My only problem is that
all shortcuts that I had defined that include the shift key (in vim,
screen) do not work. I searched many hours without success. Any idea?
I would be grateful.

My files in .mlterm are:

# cat main
use_transbg=3Dtrue
brightness=3D60
fg_color=3Dwhite
bg_color=3Dblack

# cat termcap
kD=3D=7F
kb=3D=08

# cat challenge=20
1120162910

Alexandros

Re: [Mlterm-dev-en] mlterm, tibetan, and bitmap fonts?

[Rich F. <da...@ae...>]

On Mon, Dec 19, 2005 at 01:30:47AM +0900, mi...@mi... wrote:
> Hi. 
> 
> I've tried to see Tibetan text by following procedure.
> Please let me know you have used some different way or
> you are using another font. 
> 
> 1. install "Tibetan Machine Uni" from www.thdl.org and register the font to 
> fontconfig library. 
> 
> 2. create ~/.mlterm/aafont and add:
> ISO10646_UCS4_1=Tibetan Machine Uni-iso10646-1; 
> 
> 3. run mlterm with anti-alias enabled:
> mlterm -A 
> 
> 4. display a Tibetan web page by w3m on mlterm:
> w3m "http://www.thdl.org/xml/show.php?xml=test/tibetnew/thdlhp.xml&lng=tib" 

I finally succeeded in getting mlterm to display Tibetan on my laptop,
but it's horribly misrendered. Here's a screenshot:

http://brightrain.aerifal.cx/~dalias/tibetan_misrender.png

Notice that the vowel marks and subjoined characters are not aligned
with the base character properly, and that in one instance (multiple
subjoined characters) the glyphs are definitely being rendered in the
same place and completely unreadable.

I also got around to reading the mlterm source, and it seems that
there's no effort made to position combining characters; they're just
blindly displayed relative to the same origin as the base character
with overstrike. Since Tibetan characters sequentially stack, there's
no way this can work..

With that in mind, I'm trying to decide what the best way to fix this
is, short of having mlterm use FreeType directly. I could either:

- build some sort of tables to load in mlterm that tell it how to
  correctly combine characters in a given font.
- make an artificial encoding of some sort to allow access to the
  precombined glyphs.

Personally I'm of the opinion that, except for accents on latin
characters and other simple diacritics, automatic generation of
combined glyphs will never look great, so I think we need some sort of
solution for using precombined glyphs. If there were an artificial
non-unicode encoding, then mlterm could get to the glyphs that way
similarly to how it can use legacy-encoded Japanese, etc. fonts even
in unicode mode. And of course it would be a big bonus if precombined
bitmap glyphs could be used as well.

Incidentally, I was mistaken about screen, and testing it now I see
that it does in fact work. The hackish storage of combining characters
as pairs is recursive, so that a sequence ABCDE is represented as
[[[[AB]C]D]E].

Rich

Re: [Mlterm-dev-en] mlterm, tibetan, and bitmap fonts?

[Rich F. <da...@ae...>]

On Mon, Dec 19, 2005 at 01:30:47AM +0900, mi...@mi... wrote:
> >my second question is off-topic, but maybe someone here can answer: is
> >screen's utf8 support usable? (including combining characters?) a
> >quick rtfs seemed to indicate so but i don't have an environment for
> >testing it at present. i'm quite addicted to screen, so if its utf8
> >support is broken i need to do something about that first...
> 
> In my understanding, screen can pass utf-8 but do not care about
> combined characters. So there should be some breakage.
> I'm not using screen these days and not sure its current state, though. 

In case anyone else cares, the answer is that screen is incredibly
broken in this regard and needs to be fixed. It implements combining
characters for 2-character combinations only, and does so by
dynamically allocating them in the utf-16 surrogates range for storage
in its buffers. In principle it can run out of slots and just forget
about some of the combining characters in one window if you use enough
of them in another window...

I suppose I can fix this horrible abomination correctly (store an
arbitrary-length list of combined characters for each cell) or by
extending screen's hack to store many-character combinations in its
fake character table, but either way this raises an interesting issue:
With traditional character cell storage, a fixed size window holds a
finite fixed number of characters (W*H). However with combining
characters in unicode, it seems that a utf8 terminal has no upper
bound on the storage requirement. Is it appropriate to just choose an
arbitrary limit?

Rich

Re: [Mlterm-dev-en] mlterm, tibetan, and bitmap fonts?

[Rich F. <da...@ae...>]

On Mon, Dec 19, 2005 at 01:30:47AM +0900, mi...@mi... wrote:
> Hi. 
> 
> I've tried to see Tibetan text by following procedure.
> Please let me know you have used some different way or
> you are using another font. 
> 
> 1. install "Tibetan Machine Uni" from www.thdl.org and register the font to 
> fontconfig library. 
> 
> 2. create ~/.mlterm/aafont and add:
> ISO10646_UCS4_1=Tibetan Machine Uni-iso10646-1; 
> 
> 3. run mlterm with anti-alias enabled:
> mlterm -A 
> 
> 4. display a Tibetan web page by w3m on mlterm:
> w3m "http://www.thdl.org/xml/show.php?xml=test/tibetnew/thdlhp.xml&lng=tib" 

I will try this soon. I don't yet have an environment on my own system
where I can test these sort of things, but I do have a laptop I can
test on which might have the prerequisite software.

BTW thanks for mentioning that w3m supports unicode properly. I use
elinks and love it but the support for non-latin encodings is total
crap. :(

> Judging from the result of
> ftview 32 .fonts/TibetanMachineUniAlpha.ttf
> , the "Tibetan Machine Uni" font seems to contain
> many pre-combined glyphs. 
> 
> So it's a good news.
> If these glyph covers all combination of Tibetan consonants,
> you do not have to create another bitmap collection.
> The freetype library should be able to generate
> anti-alias aware images at whatever size you want. 

The problem is that I don't see any way to get readable glyphs in an
8x16 character cell from a scalable font. The hinting would have to be
mindblowingly good for that to work at all. I ended up compromising
with 8x24 with my bitmap fonts even though 8x16 could be made (barely)
readable, but I don't have faith in FreeType/TibetanMachineUni font to
give me something usable even with 8 extra pixels.

> The bad news is, these glyphs do not have unicode code point and
> mlterm can't use them at present since neither Xft nor "X core font"
> works as desired in this case. 
> 
> # For Xft, it can create glyphs by over-striking corresponding glyphs.
> However, the result looks tend to be ugly and do not seems to be correct
> when more than two combining consonants are involved. 

If it can use the alternate-form glyphs when stacking, then it will be
mostly correct, but still ugly enough that it's undesirable. And of
course even more hopeless at small sizes.

> So, it can be done in sane manner.
> If we can make mlterm to use freetype library directly,
> we can access glyph substitution table and pre-combined glyph. 

Is there any way to use a glyph substitution table from an external
file and still use the fonts through X? That is to say: does X make
the precombined glyphs available in any way at all, or are they
completely inaccessible? If there's a sane way to make them accessible
and just use a separate subst table it would work for bitmap fonts
too, which would be a big plus.

> Alternatively, we may able to use m17n-lib
> (http://www.m17n.org/m17n-lib/)
> as a rendering engine. Though I'm not yet investigated deeply,
> implementation of mlterm side can simpler in this case. 

Yes I've been looking at m17n-lib a lot since I wrote. It seems to
have the only reasonable Tibetan input methods on *nix (altho I was
just planning on adding key bindings to screen to switch keyboard to
tibetan mode and back). I haven't looked at its rendering stuff yet. I
see that mlterm supports m16n-lib already; what does it use it for?

> Supporting to edit Tibetan on command line / editors like vim can be
> very difficult. Because there's no standard for handling of combined
> characters on a terminal, you have to define "the right way" at first.
> ex.)

Some of these questions belong at the terminal level, some at the pty
level, and some at the application level.

> - When you press Delete/Backspace key in a combined character,
> Should we delete entire character or part of it?
> - When you press right/left key, should a cursor move one consonant or
> one character?

Hard to say. Either way would be usable. This is definitely left to
the application however (or when using cooked line-based input at the
terminal, the kernel tty driver).

An interesting analogy is with wide cjk characters and wide
'characters' the tty driver generates when you enter control codes. It
shows them as 2 character cells on the terminal, but when you delete
them with backspace they're removed as one unit.

> - What should we do if part of combined character was overwritten? 

I don't think this is possible. The terminal control codes to move to
a position are based on character cells, not characters, right?
Otherwise it would be impossible for programs to use the terminal to
setup tabular data without keeping track of all the character widths
in the line.. You can only position the cursor before or after a
complete combining sequence, not in the middle of it. (Of course an
advanced editor could still have a sense of "in the middle of it" not
representible in the terminal.)

On the other hand, what happens when you overwrite just half of a cjk
doublewidth character?

> So we should consider a bit deeper after finishing to implement proper
> glyph rendering scheme. 

I agree, but I think these are mostly usability issues that could be
improved incrementally once the essentials are in place.

> >my second question is off-topic, but maybe someone here can answer: is
> >screen's utf8 support usable? (including combining characters?) a
> >quick rtfs seemed to indicate so but i don't have an environment for
> >testing it at present. i'm quite addicted to screen, so if its utf8
> >support is broken i need to do something about that first...
> 
> In my understanding, screen can pass utf-8 but do not care about
> combined characters. So there should be some breakage.

Well like I said, the code seems to be aware of character width,
including both doublewidth and zerowidth... Not sure how it works
though, and that's just from RTFS'ing briefly.

> I'm not using screen these days and not sure its current state, though. 

:(

Rich

Re: [Mlterm-dev-en] mlterm, tibetan, and bitmap fonts?

[<mi...@mi...>]

Hi. 

I've tried to see Tibetan text by following procedure.
Please let me know you have used some different way or
you are using another font. 

1. install "Tibetan Machine Uni" from www.thdl.org and register the font to 
fontconfig library. 

2. create ~/.mlterm/aafont and add:
ISO10646_UCS4_1=Tibetan Machine Uni-iso10646-1; 

3. run mlterm with anti-alias enabled:
mlterm -A 

4. display a Tibetan web page by w3m on mlterm:
w3m "http://www.thdl.org/xml/show.php?xml=test/tibetnew/thdlhp.xml&lng=tib" 


On Mon, 2005-12-12 at 08:16 -0500, Rich Felker wrote:
> first: my main motivation for wanting a full utf8 environment is
> Tibetan language support. i want to be able to read and write tibetan
> in email (mutt/emacs), irc/aim (irssi), etc. 
> 
> tibetan makes heavy use of combining characters -- up to 5 characters
> in one cell in ordinary colloquial words. i know that mlterm has
> support for combining characters (unlike the other useless "unicode"
> terminals i've found), but my understanding is that it overlays
> several glyphs to make the combined glyph. this is certainly usable,
> but only at rather large font sizes -- large enough that i wouldn't
> have very many columns on the screen. 
>
> what i'd like is to use a bitmap font with all possible stacks
> precombined and edited for readability. i've already played around
> with this quite a bit and made a fully readable tibetan font at 8x24
> pixels; however, the information i've found online (*nix/unicode faq,
> http://www.cl.cam.ac.uk/~mgk25/unicode.html) seems to indicate that
> the x font system has no decent way of supporting precombined bitmap
> glyphs when there is no precombined unicode character number. 
> 
> so my first question is: is there a way to do this already? or a sane
> way to implement it without introducing hacks that are way too evil?
> i'm willing to do really ugly hacks to support it if necessary (like
> ignoring the x font system and loading the bitmap glyphs directly in
> mlterm as a pixmap) but i'd be a lot more interested in doing it right
> if there's a sane way to do it instead, so that it could get included
> in the official mlterm source.

Judging from the result of
ftview 32 .fonts/TibetanMachineUniAlpha.ttf
, the "Tibetan Machine Uni" font seems to contain
many pre-combined glyphs. 

So it's a good news.
If these glyph covers all combination of Tibetan consonants,
you do not have to create another bitmap collection.
The freetype library should be able to generate
anti-alias aware images at whatever size you want. 

The bad news is, these glyphs do not have unicode code point and
mlterm can't use them at present since neither Xft nor "X core font"
works as desired in this case. 

# For Xft, it can create glyphs by over-striking corresponding glyphs.
However, the result looks tend to be ugly and do not seems to be correct
when more than two combining consonants are involved. 

So, it can be done in sane manner.
If we can make mlterm to use freetype library directly,
we can access glyph substitution table and pre-combined glyph. 

Alternatively, we may able to use m17n-lib
(http://www.m17n.org/m17n-lib/)
as a rendering engine. Though I'm not yet investigated deeply,
implementation of mlterm side can simpler in this case. 


Supporting to edit Tibetan on command line / editors like vim can be
very difficult. Because there's no standard for handling of combined
characters on a terminal, you have to define "the right way" at first.
ex.)
 - When you press Delete/Backspace key in a combined character,
Should we delete entire character or part of it?
 - When you press right/left key, should a cursor move one consonant or
one character?
 - What should we do if part of combined character was overwritten? 

So we should consider a bit deeper after finishing to implement proper
glyph rendering scheme. 

> my second question is off-topic, but maybe someone here can answer: is
> screen's utf8 support usable? (including combining characters?) a
> quick rtfs seemed to indicate so but i don't have an environment for
> testing it at present. i'm quite addicted to screen, so if its utf8
> support is broken i need to do something about that first...

In my understanding, screen can pass utf-8 but do not care about
combined characters. So there should be some breakage.
I'm not using screen these days and not sure its current state, though. 

regards,
 --
minami <mi...@mi...>

[Mlterm-dev-en] mlterm, tibetan, and bitmap fonts?

[Rich F. <da...@ae...>]

hi. i'm preparing to overhaul my system (a very diy linux-based setup)
in the near future, with one of the big goals being complete migration
from ascii to unicode. i tried mlterm recently on another system and
was very pleased with it, and intend for it to be the primary/only
terminal i use on my new setup. however, i have a few questions before
i begin:

first: my main motivation for wanting a full utf8 environment is
tibetan language support. i want to be able to read and write tibetan
in email (mutt/emacs), irc/aim (irssi), etc.

tibetan makes heavy use of combining characters -- up to 5 characters
in one cell in ordinary colloquial words. i know that mlterm has
support for combining characters (unlike the other useless "unicode"
terminals i've found), but my understanding is that it overlays
several glyphs to make the combined glyph. this is certainly usable,
but only at rather large font sizes -- large enough that i wouldn't
have very many columns on the screen.

what i'd like is to use a bitmap font with all possible stacks
precombined and edited for readability. i've already played around
with this quite a bit and made a fully readable tibetan font at 8x24
pixels; however, the information i've found online (*nix/unicode faq,
http://www.cl.cam.ac.uk/~mgk25/unicode.html) seems to indicate that
the x font system has no decent way of supporting precombined bitmap
glyphs when there is no precombined unicode character number.

so my first question is: is there a way to do this already? or a sane
way to implement it without introducing hacks that are way too evil?
i'm willing to do really ugly hacks to support it if necessary (like
ignoring the x font system and loading the bitmap glyphs directly in
mlterm as a pixmap) but i'd be a lot more interested in doing it right
if there's a sane way to do it instead, so that it could get included
in the official mlterm source.

my second question is off-topic, but maybe someone here can answer: is
screen's utf8 support usable? (including combining characters?) a
quick rtfs seemed to indicate so but i don't have an environment for
testing it at present. i'm quite addicted to screen, so if its utf8
support is broken i need to do something about that first...

sorry for the long email. let me know if you have any ideas. (i'm
subscribed to the list so no need to cc me)

rich

Re: [Mlterm-dev-en] Problem with Variable Width and Whitespaces

[Fabian Z. <fa...@xo...>]

MINAMI Hirokazu wrote:
> mlterm only cares the font specified for 10646_UCS4_1,
> not for ISO8859-1  when "only_use_unicode_font=true".
> 
> If you have sanely configured the fontconfig library,
> ISO10646_UCS4_1=monospace=iso10646-1;
> should work.

It does. My hero! :-)

>>>>And the last problem: Eventually underscores like in _test_
>>>>vanish.
>>>
>>>I haven't noticed such cases. Is it reproducible in your environment?
>>
>>Yes, when I select a line in vim forexample. But maybe this has got to
>>do something with my font or with freetype.
> 
> 
> As far as I've tested, underscores sometimes can be difficult to see
> but do not disappear. Would you try to increase "line_spacing"?

Jep, line_spacing was set to zero, now to 1. Works now. (With line
spacing 0 the underscores really vanished, p.e. I type ____ in the bash,
i see what I'm typing, once I press Return it vanishes from the line above.)

greetings and thanks!
fabian


-- 
... On his long walk home, he came up with the four maxims that have
guided his life: most people are fools, most authority is malignant,
God does not exist, and everything is wrong.

          - Biography of Ted Nelson

I prefer signed/encrypted Mail (even if I can't write it myself at
the moment due to OpenWebMail deficiencies):
Fingerprint: CFE8 38A7 0BC4 3CB0 E454  FA8D 04F9 B3B6 E02D 25BA

Re: [Mlterm-dev-en] Problem with Variable Width and Whitespaces

[MINAMI H. <mi...@mi...>]

On Sun, 2005-10-09 at 17:32 +0200, Fabian Zeindl wrote:
> MINAMI Hirokazu wrote:
> > So if you really want to use mlterm without variable width option,
> > you should find/create a properly designed font for now.
> 
> Hm, because you said "really want": Is there a way to circumvent the
> drawbacks variable width brings? Like crappy ncurses interfaces etc.

No. Since curses libraries were designed with fixed-width fonts in mind,
some breakages are inevitable if they are used with variable-width ones.

> I tried to use another font. (Do I need a "monospace" font here?) I put
> lines in aafont like
> ISO8859-1=monospace-ISO8859-1

mlterm only cares the font specified for 10646_UCS4_1,
not for ISO8859-1  when "only_use_unicode_font=true".

If you have sanely configured the fontconfig library,
ISO10646_UCS4_1=monospace=iso10646-1;
should work.

> >> And the last problem: Eventually underscores like in _test_
> >>vanish.
> > 
> > I haven't noticed such cases. Is it reproducible in your environment?
> 
> Yes, when I select a line in vim forexample. But maybe this has got to
> do something with my font or with freetype.

As far as I've tested, underscores sometimes can be difficult to see
but do not disappear. Would you try to increase "line_spacing"?

--
minami <mi...@mi...>