#2163 linetype incorrectly given TC_RGB breaks fig terminal depth

[Ed Brambley]

This bug is reproducable in gnuplot 5.2.6.

There appears to be a problem with a linetype (TC_LT) turning into a linecolor (TC_RGB). In particular, this breaks the fig terminal using linetype / 1000 as a depth (and the setting of color, etc).

To reproduce the broken fig terminal:

set term fig
set style line 1 linetype 1001
set style line 2 linetype 2002
set output 'test.fig'
plot x title 'depth 11' linestyle 1 , x**2 title 'depth 12' linestyle 2
unset output

This produces test.fig with two curves at depth 10, not one at depth 11 and one at depth 12 as expected.

As mentioned above, I've traced this as far as the fig term code being passed a colorspec with type TC_RGB rather than TC_LT. This appears to be more general than just the fig terminal:

set term fig
set style line 1 linetype 1001
show style line

produces

linestyle 1,  linecolor rgb "dark-violet"  linewidth 1.000 dashtype solid pointtype 1 pointsize default pointinterval 0 pointnumber 0

i.e. linetype 1001 is converted into linecolor rgb "dark-violet".

[Ethan Merritt]

Could you please explain further? I do not understand how linetype has anything to do with "depth". Is this some quirk of fig? caveat: I know virtually nothing about fig.

You are correct that gnuplot version 5 splits the user-level "linetype" into components for color, linewidth, dash pattern, etc that are sent separately to the terminal. The terminal code still sees TC_LT when it indicates a special treatement like "background", "nodraw", "axis" but not for normal lines.

"linestyle" (as opposed to "linetype") is something else again. Let's not go there yet. The core issue seems like it may be a confusion of "depth" with any of these.

[Ed Brambley]

Depth is a fig thing - each line in fig has a depth (0-99, I believe), and lines with a lower depth go over the top of lines with a higher depth. Depth is also very useful for editing figures in something like xfig after being produced in gnuplot - you can individually hide or show each depth, so I like to make each line in a plot a different depth. This used to work (a few years ago), but it doesn't at the moment.

help set term fig

provides the following explanation of how it's supposed to work (and indeed how it used to work):

 `thickness` sets the default line thickness, which is 1 if not specified.  Overriding the thickness can be achieved by adding a multiple of 100 to the `linetype` value for a `plot` command.  In a similar way the `depth` of plot elements (with respect to the default depth) can be controlled by adding a multiple of 1000 to <linetype>.  The depth is then <layer> + <linetype>/1000 and the thickness is (<linetype>%1000)/100 or, if that is zero, the default line thickness. `linewidth` is a synonym for `thickness`
 ...
 Available fill colors are (from 1 to 9): black, blue, green, cyan, red, magenta, yellow, white and dark blue (in monochrome mode: black for 1 to 6 and white for 7 to 9)

So this means I should be able to use linetype 10205 to get a depth of default depth+10, a thickness/linewidth of 2, and a colour of 5 (possibly red), But this doesn't work at all at present.

The code that governs this in fig.trm if FIG_linetype(int linetype), in the switch(linetype) ... default: block. But this doesn't seem to get called. I think it should be getting called by FIG_set_color(t_colorspec* colorspec), under switch (colorspec->type) case TC_LT:, but instead a line type of 10205 (for the example above) gets processed as if it were an rgb triple by the case TC_RGB block.

This made me think that the following are being processed in the same way, but they're not:

set style line 1 linetype 10205
set style line 2 linecolor 10205
set style line 3 linecolor rgbcolor 10205
show style line

gives:

    linestyle 1,  linecolor rgb "#f0e442"  linewidth 1.000 dashtype solid pointtype 1 pointsize default pointinterval 0 pointnumber 0
    linestyle 2,  linecolor 10205 linewidth 1.000 dashtype solid pointtype 2 pointsize default pointinterval 0 pointnumber 0
    linestyle 3,  linecolor rgb "#0027dd"  linewidth 1.000 dashtype solid pointtype 3 pointsize default pointinterval 0 pointnumber 0

But I think all of these get passed to FIG_set_color as a colorspec with type TC_RGB, and fig is expecting a linetype with type TC_LT, so is getting confused.

I'm afraid I don't know enough about the inner workings of gnuplot to fix this myself, but I'm happy to help in whatever way I can.

[Ethan Merritt]

That's all new to me. So that means the fig wants a "linetype" that is a separate property from color, width, dash pattern, etc?

I'll have to think about it, but at the moment I don't see how that is compatible with the way gnuplot now works. As to hiding/showing individual plots, all of the interactive terminals can do that now but they do it based on the plot number not on the line properties. Maybe we could assign a fig linetype property as 1000 + plot number?

[Bastian Märkisch]

Just a thought:. does set colorsequence classic help?

[Ed Brambley]

A good thought! That does indeed fix it.

Ahh, so does that mean that the linetype I was setting was being interpretted as asking for a particular terminal independent colour, and then that colour (as a colorspec) was being passed to the fig terminal driver?

[Ed Brambley]

I guess there are a number of possible fixes, all of which might be ok:

1. Add Bastian's "set colorsequence classic" into the documentation for "set term fig".
2. Default the fig terminal to "set colorsequence classic", if that's possible.
3. Add the plot number to the depth when being used with "set colorsequence default", so that each line can be individually selected, as Ethan describes can be done with the interactive terminals. Ps. Is this the right way round? Should higher plot numbers go behind lower plot numbers? The other way round might be a bit difficult.

What do you suggest?

[Ethan Merritt]

Is there a maximum depth in fig? It would be easy enough to start at depth 1000, say, and decrement by 1 for each plot.

[Ed Brambley]

Unfortunately the depth can only be 0 to 999. That probably rules out giving each line it's own depth (from plot number), as we definitely want to allow for plots with more than 1000 lines.

[Ethan Merritt]

Upon inspection I see that the fig terminal has complicated packing schemes for
linetype ( 1000 x lt + 100 x lw + something-related-to-dashes?)
and pointtype (1000 x depth + complicated mixture of color and shape)
This must have been designed to make up for lack of generic support for colors, linewidth, point type, etc in ancient gnuplot core code.

It could be simplified greatly by ripping all this out and using the generic framework that gnuplot 5 has for all terminals. I can see two paths to accomplish this

1) Try to maintain backwards compatibility with pre-v5 versions triggered by something like the "set colorsequence classic" command

2) Keep the current fig.trm as fig_old.trm or something like that but replace it with a much simpler derivative that only implements the current gnuplot v5 approach.

I'd prefer (2). This would presumably break scripts that assumed the pre-v5 behaviour. However as we see in this bug report they were broken by 5.0 (if not earlier). So that's about 5 years elapsed before the first report of the breakage.

If I were to revise the terminal, is it sufficient to test in Xfig? Or are there other more sophisticated applications that people use to process fig files?

Thoughts?

[Ed Brambley]

The only software I use on fig files is xfig and fig2dev, the latter converting fig files into other formats (including pdf and eps).

I like Ethan's idea (2). However, while I'm clearly not as familiar with gnuplot v5 as Ethan and Bastian, I don't see any mechanism in gnuplot v5 to set the depth, and so the new fig terminal may still need to use some sort of packing scheme to allow user specified depths (which are very useful).

[Ethan Merritt]

Gnuplot v5 has a system of "layers", described in the corresponding documentation section. Objects, labels, etc (basically everything other than the plot lines) can be assigned to a layer.
There is a fixed order of item categories within each layer (e.g. objects are drawn before labels).
The order of same-category items within a layer is determined by their tag. So
set label 4 "Some text" back
will be the 4th label printed into layer "back".

Unfortunately none of this is being preserved by the fig terminal. So far as I can see all text is assigned to the same depth and no attention is paid to the order in which objects other than the plot lines are drawn. So I actually think we can do a better job of handling depth than the current fig driver by assigning depth based on layer. Unfortunately the limit of 1000 depths means we cannot assign every single item or plot a separate depth as is currently done in Qt where there is a 32-bit depth field.

So something like
depth range content

  1-100    reserved for post-editing
101-200   layer "front"  items
201-700   plots by number
701-800   layer "back" items
999       layer "behind"

I am uncertain whether it is practical to assign distinct depths to individual objects, although without this it may be impossible to control their order of placement within a layer. It's unclear to me how fig decides the order of drawing for items with the same depth.

The fig terminal already has an entry routine to track layer. This entire scheme can be implemented by using it to assign the current depth. The code unpacking and applying disparate properties from "linetype" numbering can all go away.

There are some fiddly details like what to do with layers like "hypertext", "key box", "border", "colorbox", etc, and whether or not to wrap plot numbers inside multiplot.

Other points open to discussion:

   - default to "color" rather than "mono"?
   - how to deal with explicit dot/dash line patterns
   - text properties like "Bold" or "Italic" without a full font name
   - enhanced text?
   - utf8?
   - version 5 features like textboxes, user-defined point types

[Ed Brambley]

That all sounds good to me; thank you very much!

500 depths, one per plot number, I guess means we run out of depths at 500 plots and put everything else into depth 700 (or we start at 700 and work down, putting everything else into depth 201).

I don't think it's defined in the fig format what order things are plotted in if they have the same depth.

Regarding your other points for discussion:

default to "color" rather than "mono"?

From my point of view, yes please, let's default to color.

how to deal with explicit dot/dash line patterns

If you mean how to deal with it in the fig format, there are dot, dash, dot-dash, dot-dot-dash, and dot-dot-dot-dash patterns, each of which have a single variable giving the spacing between dots/dashes. See here for the details.

If you mean how to deal with it in gnuplot, I have no idea, but being able to specify the pattern in fig is is very useful. Perhaps take whatever the standard is from other terminals that support dot/dash lines (such as the postscript terminal)?

text properties like "Bold" or "Italic" without a full font name

No idea, but fig supports a "default" font (I think)...

enhanced text? utf8? Version 5 features

No idea, sorry.

If the old fig terminal didn't support these, then (at least initially) there's no need to necessarily support them in the new terminal, in my view. The main objective, as I see it, is to fix some things (like depth) that used to work and have since broken. As long as I get working depths, dash/dot patterns, and colours back again, I'll be extremely happy!

[Ethan Merritt]

There is now a heavily-revised fig terminal in the git repository for the development version (5.3). Please give it a try if possible and provide suggestions. Since the existing fig terminal has apparently been of marginal use in gnuplot version 5, it seems reasonable to back-port this to the next stable release as well.

Here is the commit message:

commit 77db337142577aa8efb36f7507bbc7a230af1b83

heavily revised fig terminal

- Generate fig depth tags from current layer (front/back/etc) and plot number.
- Get rid of fig-specific scheme that encoded linewidth and solid/dash status
  and depth all in the linetype.
- Support 5 native dashtypes.
    Unfortunately Xfig < v3.2.7 segfaults if you use them :-(
- Rework FIG_set_color to distinguish between linetype and color.
- FIG_boxfill and FIG_filled_polygon sent incorrect object subtype and fill_style.
- Use shared routine for parsing terminal size options.
- Remove terminal options that are no longer needed
    dash/solid pointsmax depth inches/metric version
- Support only Fig output format version 3.2
- Replace Fig 3.2 file format documentation in fig.trm with
  updated copy from Xfig docs.
- rename ../term/object.h to ../term/fig_defs.h

Things that seem impossible or impractical to support:
- fractional linewidth
- utf8 (Xfig is limited by both X11 and PostScript fonts
- multi-component styles (e.g. pointinterval, zerror)
  (fig depth order is reverse of draw order)
- boxed text

I have attached a sample of the output

[Tatsuro MATSUOKA]

Ed Brambley
if you are using windows or Cygwin, you can download develepment snapshot binaries from

http://tmacchant3.starfree.jp/gnuplot/Eng/winbin/ (Windows)
http://tmacchant3.starfree.jp/gnuplot/Eng/cygbin/ (Cygwin)

On Ubuntu, you can bulld development version by following

sudo build-dep gnuplot # 18.04. if 16.04, gnuplot should be gnuplot5
git clone https://git.code.sf.net/p/gnuplot/gnuplot-main gnuplot-gnuplot-main
cd ()/gnuplot-gnuplot-main
./prepare
./configure
make
make check
make install

[Ed Brambley]

Thank you very much; this works very well in my own test cases, and is so much better than what was there before!

A small point, but this change breaks my old scripts in a few ways that would be simple to fix for backward compatibility:

1. "set terminal fig size 6 4 inches" now doesn't work, but used to. There now has to be a comma between the 6 and 4. This could be fixed by allowing both space and comma separation for the size option.
2. "set terminal fig dashed" doesn't work, but used to. Dashed is no longer an option. This could be fixed by having dashed be a recognized option that does nothing (except possibly printing an information message).

I don't know of any other incompatibilities at present. From my point of view, it would be nice if these changes could be made, as then I woudn't need to rewrite any of my script; I'm probably not the only one.

[Ethan Merritt]

As noted in the commit message, the "size" option is now handled by the same parsing routine used by other gnuplot terminals. But yeah "dashed" could be accepted and ignored.