Menu
▾
▴
gnuplot-beta
|
From: Dima K. <gn...@di...> - 2012-10-07 19:15:00
|
> On Sat, 29 Sep 2012 17:07:51 -0700 > Ethan Merritt <merritt@u.washington.edu> wrote: > > On Saturday, 29 September 2012, Dima Kogan wrote: > > > > 5. Adding default window size to the inboard x11 driver so that the 'terminal > > xlib' produces plots that have decent defaults when sent to the outboard > > driver manually. I'll do this. > > OK. It's not just xlib, however. I think (not 100% sure) that the same > problem arises whenever (ipc_back_fd == IPC_BACK_UNUSABLE), e.g. x11 output > from a script run non-interactively. Attached is a patch to address this. |
|
From: Dima K. <gn...@di...> - 2012-10-19 08:58:21
|
> On Sun, 7 Oct 2012 12:14:51 -0700 > Dima Kogan <gn...@di...> wrote: > > > On Sat, 29 Sep 2012 17:07:51 -0700 > > Ethan Merritt <merritt@u.washington.edu> wrote: > > > > On Saturday, 29 September 2012, Dima Kogan wrote: > > > > > > 5. Adding default window size to the inboard x11 driver so that the 'terminal > > > xlib' produces plots that have decent defaults when sent to the outboard > > > driver manually. I'll do this. > > > > OK. It's not just xlib, however. I think (not 100% sure) that the same > > problem arises whenever (ipc_back_fd == IPC_BACK_UNUSABLE), e.g. x11 output > > from a script run non-interactively. > > Attached is a patch to address this. This wasn't merged yet. Is there something wrong with this patch, or did it just fall through the cracks? Thanks dima |
|
From: Ethan M. <merritt@u.washington.edu> - 2012-10-19 15:24:12
|
On Friday, 19 October 2012, Dima Kogan wrote: > > On Sun, 7 Oct 2012 12:14:51 -0700 > > Dima Kogan <gn...@di...> wrote: > > > > > On Sat, 29 Sep 2012 17:07:51 -0700 > > > Ethan Merritt <merritt@u.washington.edu> wrote: > > > > > > On Saturday, 29 September 2012, Dima Kogan wrote: > > > > > > > > 5. Adding default window size to the inboard x11 driver so that the 'terminal > > > > xlib' produces plots that have decent defaults when sent to the outboard > > > > driver manually. I'll do this. > > > > > > OK. It's not just xlib, however. I think (not 100% sure) that the same > > > problem arises whenever (ipc_back_fd == IPC_BACK_UNUSABLE), e.g. x11 output > > > from a script run non-interactively. > > > > Attached is a patch to address this. > > This wasn't merged yet. Is there something wrong with this patch, or did it just > fall through the cracks? Sitting somewhere in my in-box. Sorry. Applied now. Ethan |
|
From: Daniel J S. <dan...@ie...> - 2012-10-20 17:54:49
|
On 10/20/2012 11:41 AM, sfeam (Ethan Merritt) wrote:
> On Friday, 19 October 2012, Dima Kogan wrote:
>>> On Wed, 3 Oct 2012 10:21:19 -0700
>>> Ethan A Merritt<sf...@us...> wrote:
>>>
>>> On Wednesday, October 03, 2012 01:39:08 am Dima Kogan wrote:
>>>>> On Sat, 29 Sep 2012 17:07:51 -0700
>>>>> Ethan Merritt<merritt@u.washington.edu> wrote:
>>>>>
>>>>>> 6. Removing duplicate messages (such as duplicate consecutive V commands) sounds
>>>>>> great. We should do it
>>>>> OK. Low priority because it's relatively rare for normal plots.
>>>>
>>>> I did a first pass at this. Patch attached. Good news is that as expected, the
>>>> traffic drops dramatically. Inboard timing drops from about 0.9s to about 0.45s.
>>>> The outboard, however, drops from 0.85s to 0.01s! The reason the inboard didn't
>>>> drop as much is that it still has to parse the original huge data file. I have
>>>> some lingering concerns about the patch I'm attaching.
>>>
>>>> I use ftell() to check to see that the V commands are indeed consecutive.
>>>> This might have a non-negligible cost, so I'd check before committing this.
>>>
>>> That's clever, but I agree that there is potential cost.
>>> Other terminal drivers do the same job without resorting to ftell().
>>> The trick is that any command that potentially affects the current
>>> active position must either update or invalidate the inboard copy
>>> of x_last and y_last. For example, term->put_text() would set
>>> x_last = y_last = INVALID; /* #define INVALID -1 */
>>> before leaving.
>>>
>>> The down side is that unlike your ftell() version this approach requires
>>> finding all the places that might affect current position. I've attached a
>>> first-pass patch that catches most of them, but I probably missed some.
>>> Other terminal drivers can serve as a model.
>>>
>>>
>>>> At this point, my test case is clearly broken since we've been able to optimize
>>>> away all its complexity.
>>>
>>> Right. I modified your original data generation script to produce longer
>>> vectors and some gaps, so the the plot would contain move commands as well
>>> as vector commands. This gives a more realistic mix of commands:
>>>
>>> perl -e 'for(0..2000000) \
>>> { print "$_ " . sin($_/1000) . "\n"; print "\n" if $_ % 100 == 0; }' \
>>> > breaks.ascii
>>>
>>> With this test data the reduction in size from removing redundant commands
>>> is less than 10%. I tested using the "uniq" command rather than patching
>>> the driver source code. 10% max didn't seem very significant to me,
>>> which was why I said it was low priority.
>>>
>>>> Is the same optimization valid for P commands?
>>>
>>> I don't think so. But it does apply to M commands.
>>>
>>>> I'm thinking of just generating a bunch of discrete points, and sending
>>>> them over as P commands. That sounds good, right?
>>>
>>> I don't think that the active position after drawing a point symbol is
>>> guaranteed to be at the center of the point. So in the sequence
>>> Move(x,y); Point(x,y); Move(x,y); Vector(x1,y1);
>>> the second Move is not redundant.
>>>
>>> Of course, we could change the code so that Point(x,y) it _is_ guaranteed
>>> to leave the active position at (x,y).
>>
>>
>> OK. I revisited this (duplication suppression). Patch attached. I believe this
>> optimization is equally applicable to P, M, and V. Note that this is all purely
>> inboard, so there's no active position at all; that's an outboard concept. So
>> for instance, a duplicated P command would normally draw the same point glyph
>> multiple times in the same exact position, thus removing the duplication doesn't
>> change the output. Tell me if I'm misunderstanding.
>
> I think you are not unstanding what I was trying to say. The issue is not whether
> repeated P commands are redundant, the question is whether or not a M command is
> needed after the P. Scenario: one might think that a series of points connected
> by lines could be drawn as
> P(x1,y1) V(x2,y2) P(x2,y2) V(x3,y3) ...
> But that doesn't work because P(x1,y1) doesn't leave the current position
> at (x1,y1). So instead one needs to do
> P(x1,y1) M(x1,y1) V(x2,y2) P(x2,y2) M(x2,y2) V(x3,y3) ...
> My point is that all the M commands may seem redundant but they are not.
> [NB: This is not the ordering produced by "with linespoints"]
Might it pay to add another symbol that means both P (draw point) and M
(move)? I'd guess for some plots with many points, the data stream
consists mainly of P/M commands. Then again, it has to be encoded so
that point/vector are done at the same time. If gnuplot first draws the
line connecting points, then draws the series of points, a combination
P/M command doesn't help any.
Dan
|
|
From: sfeam (E. Merritt) <eam...@gm...> - 2012-10-20 18:37:56
|
On Saturday, 20 October 2012, Daniel J Sebald wrote:
> > Scenario: one might think that a series of points connected
> > by lines could be drawn as
> > P(x1,y1) V(x2,y2) P(x2,y2) V(x3,y3) ...
> > But that doesn't work because P(x1,y1) doesn't leave the current
> > position at (x1,y1). So instead one needs to do
> > P(x1,y1) M(x1,y1) V(x2,y2) P(x2,y2) M(x2,y2) V(x3,y3) ...
> > My point is that all the M commands may seem redundant but they are not.
> > [NB: This is not the ordering produced by "with linespoints"]
>
> Might it pay to add another symbol that means both P (draw point) and M
> (move)?
No. If you wanted to make P(x,y) leave the current position at (x,y)
you'd modify the point drawing code in gnuplot_x11. But then you pay
the price, admittedly a very small price, of executing an extra move
that is unnecessary in virtually all cases.
> I'd guess for some plots with many points, the data stream
> consists mainly of P/M commands.
If the plot contains only points then no move commands are necessary,
since the term->point(x, y, pointtype) already contains the coordinates.
From what I've seen in working on other terminal types, the most
common types of redundancy in order of frequency are
(1) successive set_color or linetype commands that do nothing.
This happens when the core routines loop over some set of
objects expecting each one to require a new lt or color, but then
some of them are not actually drawn (out of range, blank text label,
both linetype and color are given but the latter supercedes the
former, ...) So you get
set color for point 1
(oops, no point 1)
set color for point 2
(oops, no point 2)
set color for point 3
(oops, point 3 has its own color spec)
set _real_ color for point 3
draw point 3
(2) a sequence of vector() commands that are effectively duplicates
due to the coarse screen resolution.
(3) a sequence of line segments drawn independently but nevertheless
arranged head-to-tail: M(x1,y1) V(x2,y2), M(x2,y2) V(x3,y3), ...
(4) a sequence of move() commands, of which only the last one is needed
I suspect only (1) and (2) are worth worrying about in practice,
although you could probably construct some pathological examples
of other redundancy.
|
|
From: sfeam (E. Merritt) <eam...@gm...> - 2012-10-21 04:20:38
|
On Saturday, 20 October 2012, Dima Kogan wrote:
> The optimization you're describing is more powerful.
> Do you think there will be a noticeable difference
> between the two approaches with real-world data?
It is very hard to predict what data someone might
someday want to plot. Here is a simple script that
generates highly redundant commands by drawing a
series of vectors touching head-to-tail.
Suppression of the intervening M commands would reduce
the size of the output stream by 50%.
Plotting head-to-tail vectors sounds like a perfectly
reasonable thing to do, however rare it might be
compared to other types of plot.
Ethan
lastx=0
lasty=0.5
set samples 1000
set term xlib
set output 'heads.x11'
plot '+' using (lastx):(lasty):(lastx = lastx+1, 1):\
(dely=rand(0)-0.5,lasty=lasty+dely, dely) \
with vector nohead
|
|
From: Daniel J S. <dan...@ie...> - 2012-10-21 04:23:46
|
On 10/20/2012 07:46 PM, Dima Kogan wrote: >>> OK. I revisited this (duplication suppression). Patch attached. I believe this >>> optimization is equally applicable to P, M, and V. Note that this is all purely >>> inboard, so there's no active position at all; that's an outboard concept. So >>> for instance, a duplicated P command would normally draw the same point glyph >>> multiple times in the same exact position, thus removing the duplication doesn't >>> change the output. Tell me if I'm misunderstanding. >> >> I think you are not unstanding what I was trying to say. The issue is not whether >> repeated P commands are redundant, the question is whether or not a M command is >> needed after the P. Scenario: one might think that a series of points connected >> by lines could be drawn as >> P(x1,y1) V(x2,y2) P(x2,y2) V(x3,y3) ... >> But that doesn't work because P(x1,y1) doesn't leave the current position >> at (x1,y1). So instead one needs to do >> P(x1,y1) M(x1,y1) V(x2,y2) P(x2,y2) M(x2,y2) V(x3,y3) ... >> My point is that all the M commands may seem redundant but they are not. >> [NB: This is not the ordering produced by "with linespoints"] >> >>> Conclusions: >>> >>> 1. ftell() is way too slow >>> 2. the code with the attached patch is significantly faster than before in the >>> best case, and about the same in the worst case >> >> I've been too busy to have a serious look at your non-redundancy patch, >> but at first glance it looks more complicated than necessary. >> Do you really need to set X11_IPC_LASTDATARUN_NONE for every single >> command that doesn't change the current position? >> Other terminal drivers manage just fine without this. >> If the small set of commands that _do_ change the position (M, V, P, T, ??) >> track the current position then no one else needs to care. > > You're right, I wasn't fully understanding what you meant. The optimization > implemented in the patch I attached is a bit simpler than what you're > describing. It only looks for identical, consecutive M, P or V commands, and > suppresses any found duplicates. Dima, Is this optimization something that should happen? The duplicate moves is no issue. However, I'm not completely comfortable with tossing out duplicate P and duplicate V, in a general sense. I think in x11's current form throwing out duplicates won't cause an issue. However, from the general viewpoint, it probably isn't good practice. Imagine I have some results with data points (0.123,0.321) (0,0) (0,0) (0.654,0.456). The data appearing on the graph is then (0.123,0.321) (0,0) (0.654,0.456). Say my terminal can create a file version of the graph, such as Qt currently does. Then I have some secondary program that can import that graph and somehow pull out the data points for processing (e.g., statistics). The resulting data would be missing one of the original samples, thereby throwing off statistics. I have no specific example, but I'm imagining things like CAD programs with 2D splines interpolation that might have a point on top of another. Or, think of the googlemaps where one can interactively drag points around. Is it correct that the scenario you have in mind is where the user plots a relatively low frequency function and extremely oversamples that function? This optimization is then sort of correcting something the user should know better about. How often will this situation arise? Dan |
|
From: sfeam (E. Merritt) <eam...@gm...> - 2012-10-21 05:22:22
|
On Saturday, 20 October 2012, sfeam (Ethan Merritt) wrote: > On Saturday, 20 October 2012, Dima Kogan wrote: > > The optimization you're describing is more powerful. > > Do you think there will be a noticeable difference > > between the two approaches with real-world data? > > It is very hard to predict what data someone might > someday want to plot. Here is a simple script that > generates highly redundant commands by drawing a > series of vectors touching head-to-tail. > Suppression of the intervening M commands would reduce > the size of the output stream by 50%. And here's another pathological case. This one is actually pretty common, or at least it arises from a plot style I use often enough in real life. It's a scatter plot with variable color points, except that many (all in this example) of the points are the same color. Again suppression of the redundant commands (g000000 in this example) gives a reduction in size approaching 50%. set sample 1000 set term xlib set output 'dots.x11' plot '+' using (rand(0)):(rand(0)):(0) with dots lc rgb var |
|
From: Ethan A M. <sf...@us...> - 2012-10-23 20:31:28
|
On Saturday, October 20, 2012 10:22:12 pm sfeam (Ethan Merritt) wrote: > On Saturday, 20 October 2012, sfeam (Ethan Merritt) wrote: > > On Saturday, 20 October 2012, Dima Kogan wrote: > > > The optimization you're describing is more powerful. > > > Do you think there will be a noticeable difference > > > between the two approaches with real-world data? > > > > It is very hard to predict what data someone might > > someday want to plot. Here is a simple script that > > generates highly redundant commands by drawing a > > series of vectors touching head-to-tail. > > Suppression of the intervening M commands would reduce > > the size of the output stream by 50%. > > And here's another pathological case. > This one is actually pretty common, or at least it > arises from a plot style I use often enough in real life. > It's a scatter plot with variable color points, > except that many (all in this example) of the points > are the same color. Again suppression of the redundant > commands (g000000 in this example) gives a reduction in > size approaching 50%. I went ahead and modified x11.trm to filter out these two common cases, using exactly the same logic already used by the svg, canvas, and pdf terminals. The emf and post terminals do something similar but the details are different. Filtering yields a 9% reduction in the number of bytes sent from gnuplot to gnuplot_x11 during execution of all.dem. Running uniq on the filtered output reports that < 0.1% of the remaining output lines are exact duplicates. That sets a rough estimate on the expected benefit from more sophisticated filtering. On the other hand, applying Dima's earlier patch on top of this, which switches to binary output for vectors, gains an additional 3%. I expect that using binary for moves as well as vectors would gain a bit more. So that may still be worth pursuing. I suppose there remain other terminals that could benefit from filtering to reduce the size of the output. lua? aqua? Ethan |
|
From: Daniel J S. <dan...@ie...> - 2012-10-22 02:51:54
|
On 10/21/2012 12:08 AM, Dima Kogan wrote: >> On Sat, 20 Oct 2012 23:23:35 -0500 >> Daniel J Sebald<dan...@ie...> wrote: >> >> Dima, >> >> Is this optimization something that should happen? The duplicate moves >> is no issue. However, I'm not completely comfortable with tossing out >> duplicate P and duplicate V, in a general sense. I think in x11's >> current form throwing out duplicates won't cause an issue. However, >> from the general viewpoint, it probably isn't good practice. >> >> Imagine I have some results with data points (0.123,0.321) (0,0) (0,0) >> (0.654,0.456). The data appearing on the graph is then (0.123,0.321) >> (0,0) (0.654,0.456). Say my terminal can create a file version of the >> graph, such as Qt currently does. Then I have some secondary program >> that can import that graph and somehow pull out the data points for >> processing (e.g., statistics). The resulting data would be missing one >> of the original samples, thereby throwing off statistics. I have no >> specific example, but I'm imagining things like CAD programs with 2D >> splines interpolation that might have a point on top of another. Or, >> think of the googlemaps where one can interactively drag points around. > > I'm only looking at the x11 terminal, and have no intentions of applying this > elsewhere, although this could certainly be done if one so desires. I would > argue that the main purpose of gnuplot is visualization, and we're allowed to > interpret the input in whichever way we like, as long as the visualization > result doesn't change. If somebody wants to manipulate the data, they should be > manipulating the input data, NOT the gnuplot output. At the very least this is > true of the x11 terminal. On top of that, the terminal already makes > modifications to the input: scaling and quantizing to the terminal coordinates. > > >> Is it correct that the scenario you have in mind is where the user plots >> a relatively low frequency function and extremely oversamples that >> function? This optimization is then sort of correcting something the >> user should know better about. How often will this situation arise? > > Yes, this is for heavily oversampled data. Most of the time this optimization > would do nothing, probably, but for particular data sets, the performance gains > are pretty big (see some earlier posts in this thread). This optimization is > pretty low-hanging fruit, so I think we should apply it. If I want to plot a > huge, smooth data set, it'd be great if I didn't have to manually downsample it > first, and if gnuplot was able to efficiently deal with it all by itself. > > dima Mmm, not sure I agree with that. Predominantly, the users of gnuplot will be scientifically oriented folks, and I don't think that it is too much of an expectation that the user know how to downsample data, and properly. Either that, or be content with a slow plot. I've been an advocate of gnuplot support in Octave (vice-versa depending upon one's viewpoint) for this very reason. That is, if one is working with large data sets and wants to do some processing before plotting, say lowpass filter then decimate the data, Octave is just the tool to do that quickly and simply. I admit that the X11 plot is lacking in things like antialias filtering, but if anything it would be better to add such features than to optimize on the basis of the terminal's deficiencies. Dan |
|
From: sfeam (E. Merritt) <eam...@gm...> - 2012-10-22 05:24:49
|
On Sunday, 21 October 2012, Daniel J Sebald wrote: > On 10/21/2012 12:08 AM, Dima Kogan wrote: > >> On Sat, 20 Oct 2012 23:23:35 -0500 > >> Daniel J Sebald<dan...@ie...> wrote: > >> > >> Dima, > >> > >> Is this optimization something that should happen? The duplicate moves > >> is no issue. However, I'm not completely comfortable with tossing out > >> duplicate P and duplicate V, in a general sense. I think in x11's > >> current form throwing out duplicates won't cause an issue. However, > >> from the general viewpoint, it probably isn't good practice. > >> > >> Imagine I have some results with data points (0.123,0.321) (0,0) (0,0) > >> (0.654,0.456). The data appearing on the graph is then (0.123,0.321) > >> (0,0) (0.654,0.456). Say my terminal can create a file version of the > >> graph, such as Qt currently does. Then I have some secondary program > >> that can import that graph and somehow pull out the data points for > >> processing (e.g., statistics). The resulting data would be missing one > >> of the original samples, thereby throwing off statistics. I have no > >> specific example, but I'm imagining things like CAD programs with 2D > >> splines interpolation that might have a point on top of another. Or, > >> think of the googlemaps where one can interactively drag points around. > > > > I'm only looking at the x11 terminal, and have no intentions of applying this > > elsewhere, although this could certainly be done if one so desires. I would > > argue that the main purpose of gnuplot is visualization, and we're allowed to > > interpret the input in whichever way we like, as long as the visualization > > result doesn't change. If somebody wants to manipulate the data, they should be > > manipulating the input data, NOT the gnuplot output. At the very least this is > > true of the x11 terminal. On top of that, the terminal already makes > > modifications to the input: scaling and quantizing to the terminal coordinates. > > > > > >> Is it correct that the scenario you have in mind is where the user plots > >> a relatively low frequency function and extremely oversamples that > >> function? This optimization is then sort of correcting something the > >> user should know better about. How often will this situation arise? > > > > Yes, this is for heavily oversampled data. Most of the time this optimization > > would do nothing, probably, but for particular data sets, the performance gains > > are pretty big (see some earlier posts in this thread). This optimization is > > pretty low-hanging fruit, so I think we should apply it. If I want to plot a > > huge, smooth data set, it'd be great if I didn't have to manually downsample it > > first, and if gnuplot was able to efficiently deal with it all by itself. > > > > dima > > Mmm, not sure I agree with that. Predominantly, the users of gnuplot > will be scientifically oriented folks, and I don't think that it is too > much of an expectation that the user know how to downsample data, and > properly. Either that, or be content with a slow plot. > > I've been an advocate of gnuplot support in Octave (vice-versa depending > upon one's viewpoint) for this very reason. That is, if one is working > with large data sets and wants to do some processing before plotting, > say lowpass filter then decimate the data, Octave is just the tool to do > that quickly and simply. > > I admit that the X11 plot is lacking in things like antialias filtering, > but if anything it would be better to add such features than to optimize > on the basis of the terminal's deficiencies. > > Dan Dan, I think you have misunderstood the context of this patch. The idea is to tailor the output streamed from gnuplot to the separate display program gnuplot_x11. I can think of no scenario in which that stream would be captured for later analysis in a different tool. We have plenty of other output modes where that might make sense, but not the pipe to gnuplot_x11. Downsampling occurs when the user shrinks the x11 plot display window down to some small size. Inside gnuplot the resolution is as high as ever, but that cannot be displayed in the tiny plot window. Conversely if the user maximizes the plot display window, it makes sense to up the resolution of the plot commands sent to it. Ethan |
