gnuplot-beta — Developers and beta testers of new gnuplot code

Re: wxt mouse scrolling anomalies

[sfeam (E. Merritt) <eam...@gm...>]

On Saturday, 10 November 2012, pl...@pi... wrote:

> Actually,I'm getting some very odd things happening here.
> 
> Firstly , being still half asleep it typed in some data with commas and 
> did not get what I intended (so not bug yet)
> 
> gnuplot> f(x)=0
> gnuplot> plot "-" w l , f(x)
> input data ('e' ends) > -50,1
> input data ('e' ends) > 0, 1
> input data ('e' ends) > 50, -1
> input data ('e' ends) > e
> 
>          set xrange [ 0.00000 : 2.00000 ] noreverse nowriteback
>          set yrange [ -60.0000 : 60.0000 ] noreverse nowriteback

OK. So you have set explicit ranges.
 
> 
> Then I entered what I intended.
> 
> gnuplot> plot "-" w l , f(x)
> input data ('e' ends) > -50 -1
> input data ('e' ends) > 0 1
> input data ('e' ends) > 50 -1
> input data ('e' ends) > e
> 
> 
> However , this did not rescale, I got what appeared to be two flat lines 
> with the ranges still as shown above.

Yes.

> Then I hit the autoscale button (wxt) and got my triangular data line 

What's an "autoscale button"?  Do you mean the hot-key "a"?

> but the function f(x) was only plotted over its previous x extent of 0:2 
>   and was I tiny segment on the now correct ranges:

By default if you rescale/refresh a plot created from volatile
(cannot be re-read) data, then it uses the values already stored internally.
In your case this includes the function data sampled over the range [0:2].

> No amount of replot, rescale etc seems to correct this, so there is a 
> difference of behaviour here for in line data rather than a file. (Can't 
> re-read pipe I guess).

Yes, although actually an input pipe and a file act the same in this case.  
It is the difference between "refresh", which re-uses the existing data, 
and "replot" which goes back and reexecutes the previous plot command 
including reading from the data sources.  The program knows that it can't 
re-read from "-" so it automatically tries to avoid "replot".
You can force the same behaviour for a data file or input pipe using
   set datafile volatile
This can be useful for example if the data source is changing in realtime
but you want to zoom the currently visible display rather than
replacing it with new data.

Did you really mean to say that the "replot" command does not prompt you
to type in the data from "-" again?  That would be a bug.

> So it seems that  scolling , which implies setting an explicit range 
> instead of auto, changes the way the function limits are determined.

No, or at least I don't think of it that way.
The hot-keys, scrolling, and rotation in 3D all try to use "refresh"
rather than "replot" to avoid triggering a full re-read of all input on
each event. For large input files or for substantial computation made on
the input values, this can make a huge difference in responsiveness.
And of course for input from the command line via "-" it is not possible
to reread.

You can always force a full "replot" including reevaluation of the
input data if you want to.  That would also cause it to resample any
functions using the current active domain on X.

I suppose it might be reasonable to have "refresh" trigger resampling
of 2D functions.  It would be computationally intensive in 3D, however,
and hammer the responsiveness of mouse interaction.  

	Ethan

zoom or scroll can fail to render all or part of plot line

[<pl...@pi...>]

HI,

In trying to pin down the previous issue, I managed to reproduce another 
bug that has been bugging me for some time that I was never able to pin 
down.

some times, when scrolling or zooming, parts or all of the plotted line 
fail to render (on wxt, not tested elsewhere)

gnuplot> f(x)=0

gnuplot> !cat "fx_test.dat"
-50 -0.8
0 0.8
50 -0.7


  plot "fx_test.dat" w l , f(x)


one scroll up looses the left leg of the plot line. Scolling back it 
returns and two or more events is fine. Just the one position is 
affected and is repeatable once it happens.

On starting again the same plan did not reproduce it so another factor 
also must be involved. I mention it anyway because this is a long 
standing  problem.

I similar zoom problem is that certain zoom selections can result in 
nothing getting plotted.

I usually have to fiddle around to find a similar zoom selection a bit 
larger that does render correctly.

I suspect these two defects have the same root cause.

regards, Peter.

Re: wxt mouse scrolling anomalies

[<pl...@pi...>]

On 11/09/12 18:17, sfeam (Ethan Merritt) wrote:
> On Friday, 09 November 2012, pl...@pi... wrote:
>> Hi ,
>>
>> recent cvs gnuplot:
>>
>> I've just noticed that mouse scrolling window in wxt  causes functions
>> to be replotted full width , not with in the limits of data as in
>> initial plot.
>>
>> Also grid gets calculated to grossly different tic intervals on just one
>> scroll event in either direction. eg my plot auto scales to [-1:+1] and
>> gets 0.2 intervals. Or scroll it gets 0.5 intervals.
>>
>> Autoscale button fixes it , replot button/command, no.
>
> I am not seeing any of that here.
> So I really can't say what might be going wrong.

OK, I'll try to create a minimalistic test case that reproduces the problem.

>
>> I would have expected scroll to do no more that move what is already
>> there.
>
> Well no, that's not how it works.  If it did that it would show blank
> space in the area that enters your field of view.  Instead it adds some
> incremental amount to the upper/lower axis range and replots.
> I can see how there may be certain cases for which that could change
> the tic interval, but I haven't noticed it being a problem in practice.

Yes, obviously blank area has to be filled. I meant that the 
presentation of the data would remain the same ie the grid does not get 
recalibrated and funtions don't start getting rendered with different 
end points.

>
>> I don't recall seeing either of these before (though the function
>> range may not always be obvious).  Previously running prob 6m old CVS.
>
> I don't see anything much related to scrolling in the ChangeLog
> for the last 6 months.
> What "last modified" date is reported when you start the program?

         G N U P L O T
         Version 4.7 patchlevel 0    last modified 2012-10-16
         Build System: Linux i686

>
>> datafile has data in range [-50:50] which autoscales to [-60:60]
>>
>> f(x)=0
>> plot datafile, f(x)
>
> I tried that.  Nothing special happened with regard to scrolling.
>

Actually,I'm getting some very odd things happening here.

Firstly , being still half asleep it typed in some data with commas and 
did not get what I intended (so not bug yet)

gnuplot> f(x)=0
gnuplot> plot "-" w l , f(x)
input data ('e' ends) > -50,1
input data ('e' ends) > 0, 1
input data ('e' ends) > 50, -1
input data ('e' ends) > e

         set xrange [ 0.00000 : 2.00000 ] noreverse nowriteback
         set yrange [ -60.0000 : 60.0000 ] noreverse nowriteback


Then I entered what I intended.

gnuplot> plot "-" w l , f(x)
input data ('e' ends) > -50 -1
input data ('e' ends) > 0 1
input data ('e' ends) > 50 -1
input data ('e' ends) > e


However , this did not rescale, I got what appeared to be two flat lines 
with the ranges still as shown above.

Then I hit the autoscale button (wxt) and got my triangular data line 
but the function f(x) was only plotted over its previous x extent of 0:2 
  and was I tiny segment on the now correct ranges:


         set xrange [ * : * ] noreverse nowriteback  # (currently 
[-60.0000:60.0000] )
         set yrange [ * : * ] noreverse nowriteback  # (currently 
[-1.00000:1.00000] )

No amount of replot, rescale etc seems to correct this, so there is a 
difference of behaviour here for in line data rather than a file. (Can't 
re-read pipe I guess).

Now if I put the same data in a text file I can reproduce what I 
originally reported:

gnuplot> !cat "fx_test.dat"
-50 -1
0 1
50 -1

  plot "fx_test.dat" w l , f(x)


That test case plots the function to the width of the data on initial 
display and on autoscale button but any scroll event will cause the 
function to be plotted full with of x axis.

So it seems that  scolling , which implies setting an explicit range 
instead of auto, changes the way the function limits are determined.

regards, Peter.

Re: wxt mouse scrolling anomalies

[sfeam (E. Merritt) <eam...@gm...>]

On Friday, 09 November 2012, pl...@pi... wrote:
> Hi ,
> 
> recent cvs gnuplot:
> 
> I've just noticed that mouse scrolling window in wxt  causes functions 
> to be replotted full width , not with in the limits of data as in 
> initial plot.
> 
> Also grid gets calculated to grossly different tic intervals on just one 
> scroll event in either direction. eg my plot auto scales to [-1:+1] and 
> gets 0.2 intervals. Or scroll it gets 0.5 intervals.
> 
> Autoscale button fixes it , replot button/command, no.

I am not seeing any of that here.
So I really can't say what might be going wrong.

> I would have expected scroll to do no more that move what is already 
> there. 

Well no, that's not how it works.  If it did that it would show blank
space in the area that enters your field of view.  Instead it adds some
incremental amount to the upper/lower axis range and replots.
I can see how there may be certain cases for which that could change
the tic interval, but I haven't noticed it being a problem in practice.

> I don't recall seeing either of these before (though the function 
> range may not always be obvious).  Previously running prob 6m old CVS.

I don't see anything much related to scrolling in the ChangeLog
for the last 6 months.
What "last modified" date is reported when you start the program?

> datafile has data in range [-50:50] which autoscales to [-60:60]
> 
> f(x)=0
> plot datafile, f(x)

I tried that.  Nothing special happened with regard to scrolling.

Re: wxt mouse scrolling anomalies

[Mojca M. <moj...@gm...>]

On Fri, Nov 9, 2012 at 12:13 AM,  <pl...@pi...> wrote:
> Hi ,
>
> recent cvs gnuplot:
>
> I've just noticed that mouse scrolling window in wxt  causes functions
> to be replotted full width , not with in the limits of data as in
> initial plot.

Others will be able to answer better, but the functionality has indeed
been changed. If I used
    plot [0:] f(x)
in the old version and tried to zoom into some region to see more
details - say, between [3:4] - then gnuplot would zoom into [0:4].

If I then tried to shift, gnuplot would zoom in and out to keep the
condition [0:] valid instead of shifting the data.

I'm not sure if I understood what exactly is going on in your case.
The old functionality was weird, but it is possible that there are
cases when the new functionality also functions wrong.

Mojca

wxt mouse scrolling anomalies

[<pl...@pi...>]

Hi ,

recent cvs gnuplot:

I've just noticed that mouse scrolling window in wxt  causes functions 
to be replotted full width , not with in the limits of data as in 
initial plot.

Also grid gets calculated to grossly different tic intervals on just one 
scroll event in either direction. eg my plot auto scales to [-1:+1] and 
gets 0.2 intervals. Or scroll it gets 0.5 intervals.

Autoscale button fixes it , replot button/command, no.

I would have expected scroll to do no more that move what is already 
there. I don't recall seeing either of these before (though the function 
range may not always be obvious).  Previously running prob 6m old CVS.


datafile has data in range [-50:50] which autoscales to [-60:60]

f(x)=0
plot datafile, f(x)


Best regards, Peter.

Re: label format in 4.7

[Dima K. <gn...@di...>]

> On Tue, 30 Oct 2012 22:05:57 -0700
> "sfeam (Ethan Merritt)" <eam...@gm...> wrote:
>
> On Tuesday, 30 October 2012, Dmitri A. Sergatskov wrote:
> > The following code:
> > 
> > set format x "10^{%T}"
> > set format y "10^{%T}"
> > set logscale xy
> > set term x11 enh
> > plot [0.001:100] x
> > 
> > make plot with "bad" labels (see attached).
> > The result is what expected on 4.6.x (also attached)
> >  or on 4.7 with wxt term.
> 
> Thanks for the report.
> 
> This was a problem (fixed now) in gnuplot_x11.
> 
> A new more compact format for transfering coordinates
> between x11.trm and gnuplot_x11 was introduced recently,
> but the "T" case didn't quite match on the gnuplot_x11 end. 
> 
> 	Ethan

Hi Ethan. Sorry this is late, but the fixes you committed for this aren't 100%
right. The bug was that char_byte_offset is relative to the pointer that
sscanf() was passed. So in the 's' case, it's relative to buffer+2. I forgot the
+2 in my earlier code. Attaching a patch that redoes this fix simply by adding
in the +2.

dima

Re: strange parser error

[Ethan A M. <sf...@us...>]

On Wednesday, October 31, 2012 01:06:12 pm Dima Kogan wrote:
> >
> > On Wednesday, October 31, 2012 08:33:01 am Karl-Friedrich Ratzsch
> > wrote:
> > > 
> > > i observe a strange parser error, where any command containing + or
> > > - would no longer be accepted. It first occured to me after a
> > > mistype with the stats commmand, and boils down to this:
> > > 
> > > gp> a=0
> > > gp> stats file"
> > > 	undefined variable: file
> > > gp> a=a+1
> > >        ^
> > > 	';' expected

> > On Wed, 31 Oct 2012 10:17:06 -0700
> > Ethan A Merritt <sf...@us...> wrote:
> > Indeed quite strange.  I see the same thing with 4.4.4 and with
> > the current 4.6 CVS source.  
> > 
> > But it doesn't happen in 4.7,  so bisecting the changes from
> > 4.6 to 4.7 should identify the source of the problem and 
> > provide a fix.

Dima Kogan wrote> 
> Git bisect says the fix happened in
> 
> https://github.com/gnuplot/gnuplot/commit/9b944c2cd4c59545ce3cc903fdb88f01a8ca6e1e
> 

This turns out to be a good example of when automatic bisection yields only
a hint, not a fix.  The commit you found (I found it also) did not actually remove
the problem. It changed 2 call sites, one in stats.c and one in fit.c, 
such that try_to_get_string() was replaced with a call to string_or_express().
After this change the specific commands provided by Karl-Friedrich Ratzsch 
no longer trigger the error, but other calls to try_to_get_string() were 
still vulnerable.

The underlying problem was a failure to reset internal parsing state variables
after an error exit from command line parsing.  
Now fixed in CVS for 4.6 and 4.7.
	
	Ethan

Re: strange parser error

[Dima K. <gn...@di...>]

> On Wed, 31 Oct 2012 10:17:06 -0700
> Ethan A Merritt <sf...@us...> wrote:
>
> On Wednesday, October 31, 2012 08:33:01 am Karl-Friedrich Ratzsch
> wrote:
> > Hi,
> > 
> > i observe a strange parser error, where any command containing + or
> > - would no longer be accepted. It first occured to me after a
> > mistype with the stats commmand, and boils down to this:
> > 
> > 
> > gp> a=0
> > gp> stats file"
> > 	undefined variable: file
> > gp> a=a+1
> >        ^
> > 	';' expected
> > gp> stats "file"
> > 	warning: Skipping unreadable file "file"
> > 	Can't read data file
> > gp> a=a+1; print a
> > 1
> > gp>
> > 
> 
> Indeed quite strange.  I see the same thing with 4.4.4 and with
> the current 4.6 CVS source.  
> 
> But it doesn't happen in 4.7,  so bisecting the changes from
> 4.6 to 4.7 should identify the source of the problem and 
> provide a fix.
> 
> 	Ethan
> 
> 
> 
> 
> > Same happens after ommitting the leading " on giving the file name
> > in a fit command, eg.
> > 
> > 
> > gp> fit f(x) file"
> > 	undefinded variable: file
> > gp> a=a+1
> >     ';'expectd
> > 
> > 
> > 
> > So after "stats" or "fit" gives out the "undefined variable" error,
> > the parser will no longer understand any command containing "+" or
> > "-" ("*", "/" work, btw.). After a another stats command that works
> > (or gives another error message), the parsing will be OK again.
> > 
> > The error does not occur with "plot", strangely.
> > 
> > I'm using gp4.6.0, official windows build. Is anyone working on the
> > 4.6.1 build for windows, btw? Tatsuro used to do it, i think, but if
> > he's not around, i might volunteer if someone has a few pointers on
> > how it's done. I'd be starting from scratch, so it might be a few
> > days ...
> > 
> > Regards, Karl


Git bisect says the fix happened in

https://github.com/gnuplot/gnuplot/commit/9b944c2cd4c59545ce3cc903fdb88f01a8ca6e1e

Another interesting data point is that the issue only manifests when gnuplot
reads the input on STDIN, not from a file.

For reference, to find this commit, I did this:

$ git bisect start origin/master gnuplot-4-4-alpha

$ git bisect run zsh -c './prepare; ./configure --without-tutorial --without-cairo --without-lua; make -j7 -C src gnuplot || exit 125; HOME=/tmp src/gnuplot < /tmp/tst.gp |& grep -q expected; r=$?; git clean -ffdx; git reset --hard; exit $r'

with /tmp/tst.gp:

a=0
fit f(x) file"
a=a+1

Re: strange parser error

[Ethan A M. <sf...@us...>]

On Wednesday, October 31, 2012 08:33:01 am Karl-Friedrich Ratzsch wrote:
> Hi,
> 
> i observe a strange parser error, where any command containing + or
> - would no longer be accepted. It first occured to me after a
> mistype with the stats commmand, and boils down to this:
> 
> 
> gp> a=0
> gp> stats file"
> 	undefined variable: file
> gp> a=a+1
>        ^
> 	';' expected
> gp> stats "file"
> 	warning: Skipping unreadable file "file"
> 	Can�t read data file
> gp> a=a+1; print a
> 1
> gp>
> 

Indeed quite strange.  I see the same thing with 4.4.4 and with
the current 4.6 CVS source.  

But it doesn't happen in 4.7,  so bisecting the changes from
4.6 to 4.7 should identify the source of the problem and 
provide a fix.

	Ethan




> Same happens after ommitting the leading " on giving the file name
> in a fit command, eg.
> 
> 
> gp> fit f(x) file"
> 	undefinded variable: file
> gp> a=a+1
>     ';'expectd
> 
> 
> 
> So after "stats" or "fit" gives out the "undefined variable" error,
> the parser will no longer understand any command containing "+" or
> "-" ("*", "/" work, btw.). After a another stats command that works
> (or gives another error message), the parsing will be OK again.
> 
> The error does not occur with "plot", strangely.
> 
> I�m using gp4.6.0, official windows build. Is anyone working on the
> 4.6.1 build for windows, btw? Tatsuro used to do it, i think, but if
> he�s not around, i might volunteer if someone has a few pointers on
> how it�s done. I�d be starting from scratch, so it might be a few
> days ...
> 
> Regards, Karl

strange parser error

[Karl-Friedrich R. <mai...@gm...>]

Hi,

i observe a strange parser error, where any command containing + or
- would no longer be accepted. It first occured to me after a
mistype with the stats commmand, and boils down to this:


gp> a=0
gp> stats file"
	undefined variable: file
gp> a=a+1
       ^
	';' expected
gp> stats "file"
	warning: Skipping unreadable file "file"
	Can´t read data file
gp> a=a+1; print a
1
gp>


Same happens after ommitting the leading " on giving the file name
in a fit command, eg.


gp> fit f(x) file"
	undefinded variable: file
gp> a=a+1
    ';'expectd



So after "stats" or "fit" gives out the "undefined variable" error,
the parser will no longer understand any command containing "+" or
"-" ("*", "/" work, btw.). After a another stats command that works
(or gives another error message), the parsing will be OK again.

The error does not occur with "plot", strangely.

I´m using gp4.6.0, official windows build. Is anyone working on the
4.6.1 build for windows, btw? Tatsuro used to do it, i think, but if
he´s not around, i might volunteer if someone has a few pointers on
how it´s done. I´d be starting from scratch, so it might be a few
days ...

Regards, Karl

Re: label format in 4.7

[sfeam (E. Merritt) <eam...@gm...>]

On Tuesday, 30 October 2012, Dmitri A. Sergatskov wrote:
> The following code:
> 
> set format x "10^{%T}"
> set format y "10^{%T}"
> set logscale xy
> set term x11 enh
> plot [0.001:100] x
> 
> make plot with "bad" labels (see attached).
> The result is what expected on 4.6.x (also attached)
>  or on 4.7 with wxt term.

Thanks for the report.

This was a problem (fixed now) in gnuplot_x11.

A new more compact format for transfering coordinates
between x11.trm and gnuplot_x11 was introduced recently,
but the "T" case didn't quite match on the gnuplot_x11 end. 

	Ethan

> 
> Dmitri.
> --
> 

Re: filtering out redundant commands from x11

[<pl...@pi...>]

On 10/23/12 23:07, Mojca Miklavec wrote:
> the same is true for lua. Creating plots with 10^4  points is an
> overkill for both TeX and the PDF viewer.

I don't know the details how and when TeX is used but In relation to pdf 
I would be careful about assuming everyone reads a PDF at "fit to width" 
zoom level as you appear to do in making the statement.

I see no reason why someone would not want to create a pdf with 10^4 
points with the intention that the viewer may want to zoom in to any 
portion of the graph to examine the detail. I prefer svg for this sort 
of thing but pdf is a more ubiquitous format that will probably be used 
more often in this kind of case.

I often read papers in pdf and need to zoom in on the graphs. Sometimes 
the result is barely legible because the authors have included a poor 
resolution bitmap image which is very frustrating.

I see no, a priori reason why 10^4 points would be overkill for PDF.

regards, Peter.

Re: filtering out redundant commands from x11

[Mojca M. <moj...@gm...>]

On Tue, Oct 23, 2012 at 10:28 PM, Ethan A Merritt wrote:
>
> I suppose there remain other terminals that could benefit from
> filtering to reduce the size of the output.  lua?  aqua?

Aqua uses additional division of 20 "gnuplot points" per pixel. (That
probably means that it would see duplicates less often, but I might be
wrong. I'm not sure how X11 works. And that's not an excuse for not at
least doing a few benchmarks.)

The main question is: should that be handled in backend (AquaTerm
library itself ignoring new points when the new point is the same as
the last one) or inside gnuplot (detecting and removing duplicate
entries - but that could best be done already before vector() is
called)?


For ConTeXt I already filtered out all duplicates (repeated colour
settings, vector() leading to the same point as the previous one, ...
you can take a look at CONTEXT_vector(...)), but in cases where it
would make any difference (lots and lots of points) the terminal is
not able to handle the amount of data efficiently anyway. More or less
the same is true for lua. Creating plots with 10^4 points is an
overkill for both TeX and the PDF viewer.

(I'm not trying to say that it shouldn't be done for lua, I'm just
saying that it probably wouldn't benefit on the same front as it does
for X11. Lua/TikZ or ConTeXt could in principle also improve
efficiency by reducing precision and using shorter TeX/mp commands.)

Mojca

Re: filtering out redundant commands from x11 [was: x11 terminal respects the requested aspect ratio]

[Ethan A M. <sf...@us...>]

On Saturday, October 20, 2012 10:22:12 pm sfeam (Ethan Merritt) wrote:
> On Saturday, 20 October 2012, sfeam (Ethan Merritt) wrote:
> > On Saturday, 20 October 2012, Dima Kogan wrote:
> > > The optimization you're describing is more powerful. 
> > > Do you think there will be a noticeable difference
> > > between the two approaches with real-world data?
> > 
> > It is very hard to predict what data someone might
> > someday want to plot.  Here is a simple script that
> > generates highly redundant commands by drawing a
> > series of vectors touching head-to-tail.  
> > Suppression of the intervening M commands would reduce
> > the size of the output stream by 50%.
> 
> And here's another pathological case.
> This one is actually pretty common, or at least it 
> arises from a plot style I use often enough in real life.
> It's a scatter plot with variable color points,
> except that many (all in this example) of the points
> are the same color.  Again suppression of the redundant
> commands (g000000 in this example) gives a reduction in
> size approaching 50%.

I went ahead and modified x11.trm to filter out these two
common cases, using exactly the same logic already used by
the svg, canvas, and pdf terminals.  The emf and post terminals
do something similar but the details are different.

Filtering yields a 9% reduction in the number of bytes sent from
gnuplot to gnuplot_x11 during execution of all.dem.

Running uniq on the filtered output reports that < 0.1% of the 
remaining output lines are exact duplicates. That sets a rough
estimate on the expected benefit from more sophisticated filtering.

On the other hand, applying Dima's earlier patch on top of this, 
which switches to binary output for vectors, gains an additional 3%.  
I expect that using binary for moves as well as vectors would gain
a bit more.  So that may still be worth pursuing.

I suppose there remain other terminals that could benefit from
filtering to reduce the size of the output.  lua?  aqua?


	Ethan

Re: [PATCH] x11 terminal respects the requested aspect ratio

[sfeam (E. Merritt) <eam...@gm...>]

On Sunday, 21 October 2012, Daniel J Sebald wrote:
> On 10/21/2012 12:08 AM, Dima Kogan wrote:
> >> On Sat, 20 Oct 2012 23:23:35 -0500
> >> Daniel J Sebald<dan...@ie...>  wrote:
> >>
> >> Dima,
> >>
> >> Is this optimization something that should happen?  The duplicate moves
> >> is no issue.  However, I'm not completely comfortable with tossing out
> >> duplicate P and duplicate V, in a general sense.  I think in x11's
> >> current form throwing out duplicates won't cause an issue.  However,
> >> from the general viewpoint, it probably isn't good practice.
> >>
> >> Imagine I have some results with data points (0.123,0.321) (0,0) (0,0)
> >> (0.654,0.456).  The data appearing on the graph is then (0.123,0.321)
> >> (0,0) (0.654,0.456).  Say my terminal can create a file version of the
> >> graph, such as Qt currently does.  Then I have some secondary program
> >> that can import that graph and somehow pull out the data points for
> >> processing (e.g., statistics).  The resulting data would be missing one
> >> of the original samples, thereby throwing off statistics.  I have no
> >> specific example, but I'm imagining things like CAD programs with 2D
> >> splines interpolation that might have a point on top of another.  Or,
> >> think of the googlemaps where one can interactively drag points around.
> >
> > I'm only looking at the x11 terminal, and have no intentions of applying this
> > elsewhere, although this could certainly be done if one so desires. I would
> > argue that the main purpose of gnuplot is visualization, and we're allowed to
> > interpret the input in whichever way we like, as long as the visualization
> > result doesn't change. If somebody wants to manipulate the data, they should be
> > manipulating the input data, NOT the gnuplot output. At the very least this is
> > true of the x11 terminal. On top of that, the terminal already makes
> > modifications to the input: scaling and quantizing to the terminal coordinates.
> >
> >
> >> Is it correct that the scenario you have in mind is where the user plots
> >> a relatively low frequency function and extremely oversamples that
> >> function?  This optimization is then sort of correcting something the
> >> user should know better about.  How often will this situation arise?
> >
> > Yes, this is for heavily oversampled data. Most of the time this optimization
> > would do nothing, probably, but for particular data sets, the performance gains
> > are pretty big (see some earlier posts in this thread). This optimization is
> > pretty low-hanging fruit, so I think we should apply it. If I want to plot a
> > huge, smooth data set, it'd be great if I didn't have to manually downsample it
> > first, and if gnuplot was able to efficiently deal with it all by itself.
> >
> > dima
> 
> Mmm, not sure I agree with that.  Predominantly, the users of gnuplot 
> will be scientifically oriented folks, and I don't think that it is too 
> much of an expectation that the user know how to downsample data, and 
> properly.  Either that, or be content with a slow plot.
> 
> I've been an advocate of gnuplot support in Octave (vice-versa depending 
> upon one's viewpoint) for this very reason.  That is, if one is working 
> with large data sets and wants to do some processing before plotting, 
> say lowpass filter then decimate the data, Octave is just the tool to do 
> that quickly and simply.
> 
> I admit that the X11 plot is lacking in things like antialias filtering, 
> but if anything it would be better to add such features than to optimize 
> on the basis of the terminal's deficiencies.
> 
> Dan

Dan, I think you have misunderstood the context of this patch.
The idea is to tailor the output streamed from gnuplot to the 
separate display program gnuplot_x11.  I can think of no scenario
in which that stream would be captured for later analysis in a 
different tool.  We have plenty of other output modes where that
might make sense, but not the pipe to gnuplot_x11.  Downsampling
occurs when the user shrinks the x11 plot display window down 
to some small size.   Inside gnuplot the resolution is as high
as ever, but that cannot be displayed in the tiny plot window.
Conversely if the user maximizes the plot display window, it
makes sense to up the resolution of the plot commands sent to it.

	Ethan

Re: [PATCH] x11 terminal respects the requested aspect ratio

[Daniel J S. <dan...@ie...>]

On 10/21/2012 12:08 AM, Dima Kogan wrote:
>> On Sat, 20 Oct 2012 23:23:35 -0500
>> Daniel J Sebald<dan...@ie...>  wrote:
>>
>> Dima,
>>
>> Is this optimization something that should happen?  The duplicate moves
>> is no issue.  However, I'm not completely comfortable with tossing out
>> duplicate P and duplicate V, in a general sense.  I think in x11's
>> current form throwing out duplicates won't cause an issue.  However,
>> from the general viewpoint, it probably isn't good practice.
>>
>> Imagine I have some results with data points (0.123,0.321) (0,0) (0,0)
>> (0.654,0.456).  The data appearing on the graph is then (0.123,0.321)
>> (0,0) (0.654,0.456).  Say my terminal can create a file version of the
>> graph, such as Qt currently does.  Then I have some secondary program
>> that can import that graph and somehow pull out the data points for
>> processing (e.g., statistics).  The resulting data would be missing one
>> of the original samples, thereby throwing off statistics.  I have no
>> specific example, but I'm imagining things like CAD programs with 2D
>> splines interpolation that might have a point on top of another.  Or,
>> think of the googlemaps where one can interactively drag points around.
>
> I'm only looking at the x11 terminal, and have no intentions of applying this
> elsewhere, although this could certainly be done if one so desires. I would
> argue that the main purpose of gnuplot is visualization, and we're allowed to
> interpret the input in whichever way we like, as long as the visualization
> result doesn't change. If somebody wants to manipulate the data, they should be
> manipulating the input data, NOT the gnuplot output. At the very least this is
> true of the x11 terminal. On top of that, the terminal already makes
> modifications to the input: scaling and quantizing to the terminal coordinates.
>
>
>> Is it correct that the scenario you have in mind is where the user plots
>> a relatively low frequency function and extremely oversamples that
>> function?  This optimization is then sort of correcting something the
>> user should know better about.  How often will this situation arise?
>
> Yes, this is for heavily oversampled data. Most of the time this optimization
> would do nothing, probably, but for particular data sets, the performance gains
> are pretty big (see some earlier posts in this thread). This optimization is
> pretty low-hanging fruit, so I think we should apply it. If I want to plot a
> huge, smooth data set, it'd be great if I didn't have to manually downsample it
> first, and if gnuplot was able to efficiently deal with it all by itself.
>
> dima

Mmm, not sure I agree with that.  Predominantly, the users of gnuplot 
will be scientifically oriented folks, and I don't think that it is too 
much of an expectation that the user know how to downsample data, and 
properly.  Either that, or be content with a slow plot.

I've been an advocate of gnuplot support in Octave (vice-versa depending 
upon one's viewpoint) for this very reason.  That is, if one is working 
with large data sets and wants to do some processing before plotting, 
say lowpass filter then decimate the data, Octave is just the tool to do 
that quickly and simply.

I admit that the X11 plot is lacking in things like antialias filtering, 
but if anything it would be better to add such features than to optimize 
on the basis of the terminal's deficiencies.

Dan

Re: [PATCH] x11 terminal respects the requested aspect ratio

[sfeam (E. Merritt) <eam...@gm...>]

On Saturday, 20 October 2012, sfeam (Ethan Merritt) wrote:
> On Saturday, 20 October 2012, Dima Kogan wrote:
> > The optimization you're describing is more powerful. 
> > Do you think there will be a noticeable difference
> > between the two approaches with real-world data?
> 
> It is very hard to predict what data someone might
> someday want to plot.  Here is a simple script that
> generates highly redundant commands by drawing a
> series of vectors touching head-to-tail.  
> Suppression of the intervening M commands would reduce
> the size of the output stream by 50%.

And here's another pathological case.
This one is actually pretty common, or at least it 
arises from a plot style I use often enough in real life.
It's a scatter plot with variable color points,
except that many (all in this example) of the points
are the same color.  Again suppression of the redundant
commands (g000000 in this example) gives a reduction in
size approaching 50%.


set sample 1000
set term xlib
set output 'dots.x11'
plot '+' using (rand(0)):(rand(0)):(0) with dots lc rgb var

Re: [PATCH] x11 terminal respects the requested aspect ratio

[Dima K. <gn...@di...>]

> On Sat, 20 Oct 2012 23:23:35 -0500
> Daniel J Sebald <dan...@ie...> wrote:
>
> Dima,
> 
> Is this optimization something that should happen?  The duplicate moves 
> is no issue.  However, I'm not completely comfortable with tossing out 
> duplicate P and duplicate V, in a general sense.  I think in x11's 
> current form throwing out duplicates won't cause an issue.  However, 
> from the general viewpoint, it probably isn't good practice.
> 
> Imagine I have some results with data points (0.123,0.321) (0,0) (0,0) 
> (0.654,0.456).  The data appearing on the graph is then (0.123,0.321) 
> (0,0) (0.654,0.456).  Say my terminal can create a file version of the 
> graph, such as Qt currently does.  Then I have some secondary program 
> that can import that graph and somehow pull out the data points for 
> processing (e.g., statistics).  The resulting data would be missing one 
> of the original samples, thereby throwing off statistics.  I have no 
> specific example, but I'm imagining things like CAD programs with 2D 
> splines interpolation that might have a point on top of another.  Or, 
> think of the googlemaps where one can interactively drag points around.

I'm only looking at the x11 terminal, and have no intentions of applying this
elsewhere, although this could certainly be done if one so desires. I would
argue that the main purpose of gnuplot is visualization, and we're allowed to
interpret the input in whichever way we like, as long as the visualization
result doesn't change. If somebody wants to manipulate the data, they should be
manipulating the input data, NOT the gnuplot output. At the very least this is
true of the x11 terminal. On top of that, the terminal already makes
modifications to the input: scaling and quantizing to the terminal coordinates.


> Is it correct that the scenario you have in mind is where the user plots 
> a relatively low frequency function and extremely oversamples that 
> function?  This optimization is then sort of correcting something the 
> user should know better about.  How often will this situation arise?

Yes, this is for heavily oversampled data. Most of the time this optimization
would do nothing, probably, but for particular data sets, the performance gains
are pretty big (see some earlier posts in this thread). This optimization is
pretty low-hanging fruit, so I think we should apply it. If I want to plot a
huge, smooth data set, it'd be great if I didn't have to manually downsample it
first, and if gnuplot was able to efficiently deal with it all by itself.

dima

Re: [PATCH] x11 terminal respects the requested aspect ratio

[Daniel J S. <dan...@ie...>]

On 10/20/2012 07:46 PM, Dima Kogan wrote:
>>> OK. I revisited this (duplication suppression). Patch attached. I believe this
>>> optimization is equally applicable to P, M, and V. Note that this is all purely
>>> inboard, so there's no active position at all; that's an outboard concept. So
>>> for instance, a duplicated P command would normally draw the same point glyph
>>> multiple times in the same exact position, thus removing the duplication doesn't
>>> change the output. Tell me if I'm misunderstanding.
>>
>> I think you are not unstanding what I was trying to say.  The issue is not whether
>> repeated P commands are redundant, the question is whether or not a M command is
>> needed after the P.  Scenario: one might think that a series of points connected
>> by lines could be drawn as
>>    P(x1,y1) V(x2,y2) P(x2,y2) V(x3,y3) ...
>> But that doesn't work because P(x1,y1) doesn't leave the current position
>> at (x1,y1).  So instead one needs to do
>>    P(x1,y1) M(x1,y1) V(x2,y2) P(x2,y2) M(x2,y2) V(x3,y3) ...
>> My point is that all the M commands may seem redundant but they are not.
>> [NB: This is not the ordering produced by "with linespoints"]
>>
>>> Conclusions:
>>>
>>> 1. ftell() is way too slow
>>> 2. the code with the attached patch is significantly faster than before in the
>>>     best case, and about the same in the worst case
>>
>> I've been too busy to have a serious look at your non-redundancy patch,
>> but at first glance it looks more complicated than necessary.
>> Do you really need to set X11_IPC_LASTDATARUN_NONE for every single
>> command that doesn't change the current position?
>> Other terminal drivers manage just fine without this.
>> If the small set of commands that _do_ change the position (M, V, P, T, ??)
>> track the current position then no one else needs to care.
>
> You're right, I wasn't fully understanding what you meant. The optimization
> implemented in the patch I attached is a bit simpler than what you're
> describing. It only looks for identical, consecutive M, P or V commands, and
> suppresses any found duplicates.

Dima,

Is this optimization something that should happen?  The duplicate moves 
is no issue.  However, I'm not completely comfortable with tossing out 
duplicate P and duplicate V, in a general sense.  I think in x11's 
current form throwing out duplicates won't cause an issue.  However, 
from the general viewpoint, it probably isn't good practice.

Imagine I have some results with data points (0.123,0.321) (0,0) (0,0) 
(0.654,0.456).  The data appearing on the graph is then (0.123,0.321) 
(0,0) (0.654,0.456).  Say my terminal can create a file version of the 
graph, such as Qt currently does.  Then I have some secondary program 
that can import that graph and somehow pull out the data points for 
processing (e.g., statistics).  The resulting data would be missing one 
of the original samples, thereby throwing off statistics.  I have no 
specific example, but I'm imagining things like CAD programs with 2D 
splines interpolation that might have a point on top of another.  Or, 
think of the googlemaps where one can interactively drag points around.

Is it correct that the scenario you have in mind is where the user plots 
a relatively low frequency function and extremely oversamples that 
function?  This optimization is then sort of correcting something the 
user should know better about.  How often will this situation arise?

Dan

Re: [PATCH] x11 terminal respects the requested aspect ratio

[sfeam (E. Merritt) <eam...@gm...>]

On Saturday, 20 October 2012, Dima Kogan wrote:
> The optimization you're describing is more powerful. 
> Do you think there will be a noticeable difference
> between the two approaches with real-world data?

It is very hard to predict what data someone might
someday want to plot.  Here is a simple script that
generates highly redundant commands by drawing a
series of vectors touching head-to-tail.  
Suppression of the intervening M commands would reduce
the size of the output stream by 50%.

Plotting head-to-tail vectors sounds like a perfectly
reasonable thing to do, however rare it might be
compared to other types of plot.

	Ethan

lastx=0
lasty=0.5
set samples 1000

set term xlib
set output 'heads.x11'

plot '+' using (lastx):(lasty):(lastx = lastx+1, 1):\
         (dely=rand(0)-0.5,lasty=lasty+dely, dely)  \
         with vector nohead

Re: [PATCH] x11 terminal respects the requested aspect ratio

[Dima K. <gn...@di...>]

> > OK. I revisited this (duplication suppression). Patch attached. I believe this
> > optimization is equally applicable to P, M, and V. Note that this is all purely
> > inboard, so there's no active position at all; that's an outboard concept. So
> > for instance, a duplicated P command would normally draw the same point glyph
> > multiple times in the same exact position, thus removing the duplication doesn't
> > change the output. Tell me if I'm misunderstanding.
> 
> I think you are not unstanding what I was trying to say.  The issue is not whether
> repeated P commands are redundant, the question is whether or not a M command is
> needed after the P.  Scenario: one might think that a series of points connected
> by lines could be drawn as
>   P(x1,y1) V(x2,y2) P(x2,y2) V(x3,y3) ...
> But that doesn't work because P(x1,y1) doesn't leave the current position
> at (x1,y1).  So instead one needs to do
>   P(x1,y1) M(x1,y1) V(x2,y2) P(x2,y2) M(x2,y2) V(x3,y3) ...
> My point is that all the M commands may seem redundant but they are not.
> [NB: This is not the ordering produced by "with linespoints"]
>   
> > Conclusions:
> > 
> > 1. ftell() is way too slow
> > 2. the code with the attached patch is significantly faster than before in the
> >    best case, and about the same in the worst case
> 
> I've been too busy to have a serious look at your non-redundancy patch,
> but at first glance it looks more complicated than necessary.
> Do you really need to set X11_IPC_LASTDATARUN_NONE for every single
> command that doesn't change the current position?  
> Other terminal drivers manage just fine without this.  
> If the small set of commands that _do_ change the position (M, V, P, T, ??)
> track the current position then no one else needs to care.

You're right, I wasn't fully understanding what you meant. The optimization
implemented in the patch I attached is a bit simpler than what you're
describing. It only looks for identical, consecutive M, P or V commands, and
suppresses any found duplicates. If ANY command is seen between successive P
commands, say, nothing is suppressed. So for instance,

P 1 2 3
M 3 4 5
P 1 2 3

would NOT suppress anything because there's an M between the two Ps. To make
this optimization I need to remember what the last command was, and its
arguments. The enum is the command type.

To be clear, in the patch I attached, only duplicate commands are removed, thus
the M is always preserved after the P in your example. The optimization you're
describing is more powerful. Do you think there will be a noticeable difference
between the two approaches with real-world data?

The main thing I don't like about the patch is all the X11_IPC_LASTDATARUN_NONE
resets, but I can't think of a better way to eliminate these without resorting
to ftell() or redefining printf(). It's likely that I've gone way overboard on
the resets, and you can get away with them in just the few places you mention
(M, V, P, T, ...).

Let me know what you think. For the record, I did find a way to handle this
without requiring the manual X11_IPC_LASTDATARUN_NONE reset or suffering the
performance hits of doing an ftell. One can redefine the functions that interact
with the pipe:

static int X11_ipc_counter;
static int _fprintf_return, _fputs_return, _fputc_return;
#define fprintf(stream, format, ...)                            \
( _fprintf_return = fprintf(stream, format, ## __VA_ARGS__ ),   \
  (( stream == X11_ipc ) && X11_ipc_counter++),                 \
  _fprintf_return )

#define fputs(s, stream)                        \
( _fputs_return = fputs(s, stream),             \
  (( stream == X11_ipc ) && X11_ipc_counter++), \
  _fputs_return )

#define fputc(c, stream)                        \
( _fputc_return = fputc(c, stream),             \
  (( stream == X11_ipc ) && X11_ipc_counter++), \
  _fputc_return )


This is sorta weird, but serves to automatically keep track of the pipe
position. The extra code then is localized to the functions that need the
duplicate suppression (X11_vector, X11_point, etc). I.e. no resets are required
anywhere, like with ftell(), and the performance is as good as with the attached
patch. The downsides are that it

1. uses a C99-only construct (__VA_ARGS__ in the preprocessor)
2. uses a gcc-ism: , ## __VA_ARGS__
3. adds an unexpected surprise to anybody looking at the code, since a
   plain-looking fprintf() now does more than an uninitiated reader would
   expect.

Maybe these are acceptable, or somebody can see a way to work around these
downsides.

dima

Re: [PATCH] Unit tests don't use the local user's .gnuplot file

[Dima K. <gn...@di...>]

> On Sat, 20 Oct 2012 10:20:32 -0700
> "sfeam (Ethan Merritt)" <eam...@gm...> wrote:
>
> On Saturday, 20 October 2012, Dima Kogan wrote:
> > I'm attaching a patch to add a "-q" option to gnuplot to suppress parsing of
> > .gnuplot and gnuplotrc files. 
> 
> I like the idea of providing a way to suppress reading ~/.gnuplot.
> 
> Minor quibble: Isn't "-q" usually short for "--quiet" and interpreted
> as a request to suppress output rather than input?
> Not that I have a better letter of the alphabet to suggest....
> 

The options I chose (-q, --no-init-file) are what emacs uses. If some other
program uses other letters, I'd be ok with those as well. I do feel we should
match SOME precedent, instead of making up our own.


> > This is done primarily so that unit tests test the stock configuration
> 
> Hmm. I usually want the opposite.
> E.g. if I want to make sure that "set someoption obscure" doesn't
> trigger memory leaks, I put it in ~/.gnuplot and then run the
> unit tests under valgrind.
> For simple doesn't-crash sanity checks after applying a patch,
> I usually add any relevant options to ~/.gnuplot and run
> "make check".

OK. Sounds like there are two different use cases. One for tests during
development (use case you described) and another for deployment, i.e. when a
distribution package maintainer is running the unit tests as part of the
packaging process. For the latter, it makes sense to run without user-defined
.gnuplot.

dima

Re: [PATCH] Unit tests don't use the local user's .gnuplot file

[Petr M. <mi...@ph...>]

>> For simple doesn't-crash sanity checks after applying a patch,
>> I usually add any relevant options to ~/.gnuplot and run
>> "make check".
>
> I'm afraid that achieves rather a lot less than it's meant to.  Settings
> from ~/.gnuplot don't survive the first "reset" (at the end of simple.dem).

Unless there are "set term" options (it's easier to put them there instead 
of setting environmental GNUTERM).

---
PM

Re: [PATCH] Unit tests don't use the local user's .gnuplot file

[sfeam (E. Merritt) <eam...@gm...>]

On Saturday, 20 October 2012, Hans-Bernhard Bröker wrote:
> On 20.10.2012 19:20, sfeam (Ethan Merritt) wrote:
> 
> > I like the idea of providing a way to suppress reading ~/.gnuplot.
> 
> Well, one could always just brush the file under the carpet, e.g.:
> 
> 	env HOME=/some/where/else gnuplot
> 
> Look, Ma --- no need for a special option!

That's true for the case of running "make check" or 
gnuplot 'all.dem'.  It's not necessarily true for running one's own
scripts since they may explicitly load files relative to $HOME.
 
> >> This is done primarily so that unit tests test the stock configuration

 
> > For simple doesn't-crash sanity checks after applying a patch,
> > I usually add any relevant options to ~/.gnuplot and run
> > "make check".
> 
> I'm afraid that achieves rather a lot less than it's meant to.  Settings 
> from ~/.gnuplot don't survive the first "reset" (at the end of simple.dem).

Some do, some don't.  But I take your point.