#116 Support true colors

[Egmont Koblinger]

Some terminal emulators (e.g. gnome-terminal, konsole, st) support direct RGB true colors (e.g. \e[38;2;17;34;51m is a foreground of #112233).

It would be nice if joe supported these, in addition to the more widely available 256-color support.

(Unfortunately there's no terminfo entry that would tell if the feature is supported by the terminal. One would need to use a -assume_truecolor or similar cmdline/config option.)

[Egmont Koblinger]

slang-2.3.1 introduced truecolor support, and since the maintainer of ncurses+terminfo expressed that he has no interest whatsoever in truecolor support or even adding a terminfo capability, slang's author decided that he checks for $COLORTERM containing "truecolor" or "24bit". Probably the most well-known slang-based app is Midnight Commander, which will have truecolor support in its next release (4.8.19).

In turn, gnome-terminal, konsole and iTerm2 have all added COLORTERM=truecolor by default to advertise truecolor support.

I guess it could be a reasonable move for Joe to also look for this variable.

[Egmont Koblinger]

One interesting use of truecolor could be to automatically highlight color codes with their exact color (or at least to make it possible to define this behavior for certain contexts in the syntax highlight files).

That is, if e.g. a html source file contains something like <... style="color: #abcdef">, the string #abcdef could automatically be highlighted with #abcdef background color, and an automatically selected white or black foreground.

I understand that this is probably more work than the plain truecolor support, which is hopefully hardly more than a copy-paste of the already available 256-color support.

[Egmont Koblinger]

FYI: a continuously updated page about truecolor availability in terminal emulators and apps: https://gist.github.com/XVilka/8346728

[John J. Jordan]

Thanks for the update, Egmont! The colors branch contains some work to support color schemes in JOE and has the concept of true color built-in (but not the implementation). The immediate plan was to support 256-color terminals first, and true-color in Windows (because I can :-)) and then re-evaluate if termcap ever came around. I'm glad to see some other applications have moved forward and I'll have another look at this.

[John J. Jordan]

I've implemented this in [aa1cd0]. It's currently #ifdef'd out, as I'm not really sure what will be the correct way to determine whether the terminal supports it. COLORTERM isn't a great solution and won't get passed across SSH sessions, or sudo's, or much of anything. Some of the terminals I tried don't set it. And some of those terminals don't really work well at all. gnome-terminal seems to be the most consistent of any that I was able to try. That variable is also not mentioned in the gist. I'd like to see what jhallen has to say about it before potentially enabling it. And I'd really like to see this in terminfo, though I've read the maintainer's stance.

Regardless, I've put it in because substantial parts of the implementation (the palette-building, in particular) need to be in place to implement this in Windows, or even a potential initc-based implementation in the future.

[Egmont Koblinger]

Hi John,

Thanks a lot for your work!

I've added a comment to the truecolor gist about COLORTERM. It indeed suffers from the problems you mentioned, but the user having to set it manually is still better than the user having to modify an #if and recompile joe. Also I'm wondering if you could add config options along the lines of assume_color and assume_256color (including updated .joerc, c.jsf files and manpages), I think truecolor support in joe should (to some extent, as much as reasonable) follow this already existing pattern.

"And some of those terminals don't really work well at all." -- could you please clarify what you mean by this?

I haven't thorougly looked at your patch (and probably won't have time to do it, sorry) but I haven't found any sign of how to use truecolors, and suspiciously lot of palette usage -- the point of the truecolor feature is to get rid of palettes completely, at least wherever these are used.

Let's make sure that we're on the same page. I believe truecolors should work this way:

If support is detected (via COLORTERM and/or a joerc option), in addition to the previous colors new ones such as fg_#00FFC0, bg_#abcdef etc. should become available which map to the corresponding 38;2;0;255;192, 48;2;171;205;239 etc. escape sequences. Legacy palette (16 or 256 color) definitions such as e.g. fg_RED, bg_001 etc. should still map to the palette escape sequences (e.g. 31, 48;5;17 etc.) so that users can still set their favorite palette colors in their terminal emulators which joe would pick up in these cases (that is, the behavior here shouldn't change compared to the pre-truecolor days or when truecolor is not available runtime).

Thanks!

[John J. Jordan]

That branch has quite a few more changes that perhaps I should explain.

It introduces color schemes - isolated ".jcf" files that have all the colors for the environment defined in them. Syntaxes can no longer define colors, and neither can *rc files. Each jcf can have multiple sections - one per number of colors the terminal supports. Conceivably, you might have a section for 16-color terminals, for 256-color terminals and for true-color terminals -- maybe even for 88-color terminals. Color schemes can be changed at runtime and JOE will pick the appropriate section for the terminal currently in use. All that was already implemented.

The palette I speak of is there to keep the attribute data structure at 32-bits, so the color value is an index into the palette to grab the RGB values. There is an extra bit on the FG/BG to indicate to use the palette, so they could be intermingled with 256-color specs. When drawing the screen, if the bit is found, the RGB value is looked up and written with the new escape codes. Most of this implementation was code to build the palette for the color schemes.

The #ifdef is there due to the decision not being made yet, not as a feature flag for the user. I don't think assume_truecolor would be appropriate, since on many terminals (still), it would always force the color scheme code to pick a section of the scheme the terminal doesn't support.

All this said, with the changes I can now use nice blue color schemes. As acceptable as xterm-256 is, there just aren't good blue background colors.

==

As for the terminals... The PuTTY patch had all kinds of issues. Most notably, it wasn't setting background on clr_eol, plus a handful of other visual artifacts that I couldn't make any sense of. Tera Term would always make my blue background gray, and had some weird screen update flickering (not sure if that was the patch or the base version doing it). gnome-terminal worked perfectly, and st was fine minus the weird font (both on Arch).

[Egmont Koblinger]

Hi John,

Thanks for the answer!

On Linux I believe gnome-terminal (actually VTE, and hence all VTE-based emulators), konsole (and hence yakuake etc.) and st support true colors perfectly. It's not supported in mainstream PuTTY and I haven't tried any patch (I know there are patches, maybe the fork called KiTTY has it incorporated), I also haven't tried Tera Term since I'm not using Windows.

Just FYI: A relatively recent poll at https://opensource.com/life/15/11/top-open-source-terminal-emulators (which was even Slashdotted back then) suggests that somewhere between 72% - 89% of the users use a terminal emulator whose newest version (i.e. not the version at the time the poll was live, but really today's newest version) supports true colors. Obviously this poll is not representative, and it's unclear whether it's about Linux or what, the title says "open source terminal emulators" yet there's no record of iTerm2 or PuTTY... But it gives an overall feeling that truecolors is probably already available (or will soon be available) for a quite significant portion of joe users in their favorite emulator.

I absolutely agree that joe should not risk emitting truecolor sequences on terminals that may not support them. That's why I'm suggesting it might check for $COLORTERM as mentioned above (this might err in the safe direction) or have an assume_truecolor config option turned off by default.

[Egmont Koblinger]

FYI: emacs just integrated truecolor support; they went for introducing a new terminfo capability. It's indeed a nicer approach than $COLORTERM, although would require ncurses actually shipping such terminfo descriptions (which I find unlikely to happen, but let's hope).

http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=e463e57

[John J. Jordan]

FWIW, I pulled the #ifdef. It looks like the proposal has enough traction, so we'll support true colors schemes in the next release, whenever that is -- though I'm not sure whether any will be included in the default distro (that don't have reasonable 256-color counterparts).