Menu
▾
▴
gnuplot-beta
|
From: Juhász P. <pet...@gm...> - 2010-10-07 20:07:39
|
Dear gnuplot developers, while doing some work on a large dataset, I've noticed that gnuplot uses what could be construed as an excessive amount of memory if the terminal is set to pdfcairo. (My machine actually ran out of physical memory and began to thrash the swap, that's why I noticed). I've done some simple benchmarking with random data (k*1e5 pairs of random numbers generated by perl, where k went from 1 to 10; then this was plotted by: gnuplot -e 'set term pdf;set out "r.pdf";plot "r.txt" w d'). >From what I can see from the benchmark, the dependence of the memory consumption on the dataset size is completely linear - this is expected and understandable. What is not expected and hardly understandable is the ratio: it needs about 872 bytes per point. The pdfcairo terminal also takes a long time: about 27.9 s to plot 1 million points. (The time dependence is also linear.) For comparison, the same data for the postscript terminal: little more than 60 bytes per point (which is just what the plot->points array needs) and just above one second of execution time for the same 1 million points. Now I haven't delved into the innards of the pdfcairo terminal driver, and if you tell me that this is normal and expected, I'll believe you. But I can't imagine why it needs over 800 bytes to store a single point. A not entirely unrelated question: how do I compile gnuplot with profiling? I tried giving CFLAGS="-g -pg" to the configure script, but the gmon.out file was not there after running gnuplot. Péter Juhász |
|
From: Hans-Bernhard B. <HBB...@t-...> - 2010-10-07 20:55:44
|
On 07.10.2010 22:07, Juhász Péter wrote: > Now I haven't delved into the innards of the pdfcairo terminal driver, ... not to mention the innards of the cairo library, and the libraries used by that, etc. This Cairo stuff carries around _way_ more code than traditional gnuplot terminal drivers did. > A not entirely unrelated question: how do I compile gnuplot with > profiling? I tried giving CFLAGS="-g -pg" to the configure script, but > the gmon.out file was not there after running gnuplot. You need to get the -pg into the LDFLAGS, too. (And profiling without the usual -O2 would be a futile exercise...): make CFLAGS='-g -O2 -pg' CXXFLAGS='-g -pg' (Ordinarily, the CFLAGS would be used in the link, too, but we use g++ to link, thus the CXXFLAGS). |
|
From: Ethan M. <merritt@u.washington.edu> - 2010-10-08 00:34:34
|
On Thursday 07 October 2010 01:07:28 pm Juhász Péter wrote:
> Dear gnuplot developers,
>
> while doing some work on a large dataset, I've noticed that gnuplot uses
> what could be construed as an excessive amount of memory if the terminal
> is set to pdfcairo. (My machine actually ran out of physical memory and
> began to thrash the swap, that's why I noticed).
>
> I've done some simple benchmarking with random data (k*1e5 pairs of
> random numbers generated by perl, where k went from 1 to 10; then this
> was plotted by:
> gnuplot -e 'set term pdf;set out "r.pdf";plot "r.txt" w d').
>
> >From what I can see from the benchmark, the dependence of the memory
> consumption on the dataset size is completely linear - this is expected
> and understandable. What is not expected and hardly understandable is
> the ratio: it needs about 872 bytes per point.
How are you tracking memory usage?
> The pdfcairo terminal also takes a long time: about 27.9 s to plot 1
> million points. (The time dependence is also linear.)
One interesting thing is that less than half of this time is spent
creating the cairo image internally; more of the time is spent
converting it to pdf afterward.
test file bigdata.dat was created by
for (i=0; i<1000000; i++) printf("%g %g\n",drand48(), drand48());
[1] time gnuplot -e 'set term png; plot "bigdata.dat" using 1:2 with dots' > bigdata.png
0.883u 0.060s 0:01.00 94.0% 0+0k 0+24io 0pf+0w
[2] time gnuplot -e 'set term pngcairo; plot "bigdata.dat" using 1:2 with dots' > bigdata.pngcairo
15.339u 0.044s 0:17.03 90.2% 0+0k 0+384io 0pf+0w
[3] time gnuplot -e 'set term pdfcairo; plot "bigdata.dat" using 1:2 with dots' > bigdata.pdf
37.023u 0.533s 0:38.02 98.7% 0+0k 0+51520io 0pf+0w
The long-standing admonition still holds. Vector formats like PostScript,
PDF, SVG are bad choices for creating a plot with a million points.
A bitmap format like PNG does much better. "Ah", you say,
"but I asked cairo to produce a PNG file." Well yes, but that's not the way
cairo works. First it creates the requested graphic in a standard internal
representation that maintains all the necessary information for vector output.
Then when you are ready to dump it to a file, it converts the internal
representation to the final output representation.
Or at least that's how I understand it.
Time to revisit the idea of a pdflatex terminal?
Ethan
> For comparison, the same data for the postscript terminal: little more
> than 60 bytes per point (which is just what the plot->points array
> needs) and just above one second of execution time for the same 1
> million points.
>
> Now I haven't delved into the innards of the pdfcairo terminal driver,
> and if you tell me that this is normal and expected, I'll believe you.
> But I can't imagine why it needs over 800 bytes to store a single point.
>
> A not entirely unrelated question: how do I compile gnuplot with
> profiling? I tried giving CFLAGS="-g -pg" to the configure script, but
> the gmon.out file was not there after running gnuplot.
>
> Péter Juhász
|
|
From: Peter J. <pet...@gm...> - 2010-10-08 09:08:18
|
On Fri, Oct 8, 2010 at 2:32 AM, Ethan Merritt <merritt@u.washington.edu> wrote:
> On Thursday 07 October 2010 01:07:28 pm Juhász Péter wrote:
>> Dear gnuplot developers,
>>
>> while doing some work on a large dataset, I've noticed that gnuplot uses
>> what could be construed as an excessive amount of memory if the terminal
>> is set to pdfcairo. (My machine actually ran out of physical memory and
>> began to thrash the swap, that's why I noticed).
>>
>> I've done some simple benchmarking with random data (k*1e5 pairs of
>> random numbers generated by perl, where k went from 1 to 10; then this
>> was plotted by:
>> gnuplot -e 'set term pdf;set out "r.pdf";plot "r.txt" w d').
>>
>> >From what I can see from the benchmark, the dependence of the memory
>> consumption on the dataset size is completely linear - this is expected
>> and understandable. What is not expected and hardly understandable is
>> the ratio: it needs about 872 bytes per point.
>
> How are you tracking memory usage?
Simply: ps ux | grep gnuplot. I generated ten datasets with k*1e5
points (k=1:10) and I saved the values of the VSZ and RSS columns of
the ps output for each dataset, then I fitted a line on the memory
data. So the precise statement is that memory usage grows by about 872
bytes per point.
Glancing at the source code, this may not be that mysterious at all.
It does a lot more that just drawing a dot, even if the plot style is
dots.
>
>> The pdfcairo terminal also takes a long time: about 27.9 s to plot 1
>> million points. (The time dependence is also linear.)
>
> One interesting thing is that less than half of this time is spent
> creating the cairo image internally; more of the time is spent
> converting it to pdf afterward.
>
> test file bigdata.dat was created by
> for (i=0; i<1000000; i++) printf("%g %g\n",drand48(), drand48());
>
> [1] time gnuplot -e 'set term png; plot "bigdata.dat" using 1:2 with dots' > bigdata.png
> 0.883u 0.060s 0:01.00 94.0% 0+0k 0+24io 0pf+0w
> [2] time gnuplot -e 'set term pngcairo; plot "bigdata.dat" using 1:2 with dots' > bigdata.pngcairo
> 15.339u 0.044s 0:17.03 90.2% 0+0k 0+384io 0pf+0w
> [3] time gnuplot -e 'set term pdfcairo; plot "bigdata.dat" using 1:2 with dots' > bigdata.pdf
> 37.023u 0.533s 0:38.02 98.7% 0+0k 0+51520io 0pf+0w
>
> The long-standing admonition still holds. Vector formats like PostScript,
> PDF, SVG are bad choices for creating a plot with a million points.
> A bitmap format like PNG does much better.
I'm aware of this, the above example of one million points was only
for testing purposes.
"Ah", you say,
> "but I asked cairo to produce a PNG file." Well yes, but that's not the way
> cairo works. First it creates the requested graphic in a standard internal
> representation that maintains all the necessary information for vector output.
> Then when you are ready to dump it to a file, it converts the internal
> representation to the final output representation.
> Or at least that's how I understand it.
>
> Time to revisit the idea of a pdflatex terminal?
>
> Ethan
>
>> For comparison, the same data for the postscript terminal: little more
>> than 60 bytes per point (which is just what the plot->points array
>> needs) and just above one second of execution time for the same 1
>> million points.
>>
>> Now I haven't delved into the innards of the pdfcairo terminal driver,
>> and if you tell me that this is normal and expected, I'll believe you.
>> But I can't imagine why it needs over 800 bytes to store a single point.
>>
>> A not entirely unrelated question: how do I compile gnuplot with
>> profiling? I tried giving CFLAGS="-g -pg" to the configure script, but
>> the gmon.out file was not there after running gnuplot.
>>
>> Péter Juhász
>
|
|
From: Christoph B. <us...@be...> - 2010-10-08 14:39:21
|
Ethan Merritt schrieb: > > The long-standing admonition still holds. Vector formats like PostScript, > PDF, SVG are bad choices for creating a plot with a million points. > A bitmap format like PNG does much better. "Ah", you say, > "but I asked cairo to produce a PNG file." Well yes, but that's not the way > cairo works. First it creates the requested graphic in a standard internal > representation that maintains all the necessary information for vector output. > Then when you are ready to dump it to a file, it converts the internal > representation to the final output representation. > Or at least that's how I understand it. > > Time to revisit the idea of a pdflatex terminal? How do you think would a pdflatex terminal solve this problem and what configuration do you have in mind for a pdflatex terminal? In my eyes there are many different possibilities to realize a pdflatex terminal, but I don't see any advantages of this compared to the cairo terminals (apart from the mathematical typesetting). Christoph |
|
From: Ethan M. <merritt@u.washington.edu> - 2010-10-08 16:56:34
|
On Friday 08 October 2010 02:07:42 am Christoph Bersch wrote: > Ethan Merritt schrieb: > > > > The long-standing admonition still holds. Vector formats like PostScript, > > PDF, SVG are bad choices for creating a plot with a million points. > > A bitmap format like PNG does much better. "Ah", you say, > > "but I asked cairo to produce a PNG file." Well yes, but that's not the way > > cairo works. First it creates the requested graphic in a standard internal > > representation that maintains all the necessary information for vector output. > > Then when you are ready to dump it to a file, it converts the internal > > representation to the final output representation. > > Or at least that's how I understand it. > > > > Time to revisit the idea of a pdflatex terminal? > > How do you think would a pdflatex terminal solve this problem and what > configuration do you have in mind for a pdflatex terminal? I assume that the goal is to end up with a PDF file that contains a plot constructed from an unreasonably large number of points. This can be addressed by wrapping a bitmap image in a PDF file so that the text is handled at full resolution while the bitmap image is handled at a fixed maximum resolution that caps the total file size. We already have a model for doing exactly this in gnuplot - the epslatex terminal. It splits the task into a text part and a non-text part, letting latex handle the text and the postscript terminal handle the rest. I think it would be relatively easy to modify this so that the non-text part is handled by the existing png driver and the latex part is tweaked slightly for use with pdflatex rather than latex. Obviously it is not strictly necessary to involve latex in this process. I think the same split-and-recombine approach could be implemented by importing a bitmap image generated by gd.trm into a complete document created by pdfcairo. But this would involve more work, as the code would need to be written from scratch rather than being trivially modified from the existing epslatex terminal driver. I can see arguments in favor of either path. The argument in favor of png+pdflatex is that it could be supported without requiring that gnuplot be built with the cairo libraries, and the amount of new coding and debugging is expected to be small. The arguments in favor of png+pdfcairo are that gnuplot itself would emit the final pdf document rather than requiring a separate run of pdflatex. > In my eyes there are many different possibilities to realize a pdflatex > terminal, but I don't see any advantages of this compared to the cairo > terminals (apart from the mathematical typesetting). Either way, someone has to code up the new terminal. My estimate is that basing it on the existing epslatex terminal would be relatively simple. But if the hypothetical "someone" prefers to work with cairo and is willing to take on the more complicated job, that's great also. Either way there's already a patch on SourceForge that modifies gd.trm so that it produces only the non-text portion of a plot. Ethan |
|
From: Allin C. <cot...@wf...> - 2010-10-08 19:51:46
|
On Fri, 8 Oct 2010, Ethan Merritt wrote: > On Friday 08 October 2010 02:07:42 am Christoph Bersch wrote: > > Ethan Merritt schrieb: > > > > > > The long-standing admonition still holds. Vector formats like PostScript, > > > PDF, SVG are bad choices for creating a plot with a million points. > > > A bitmap format like PNG does much better. "Ah", you say, > > > "but I asked cairo to produce a PNG file." Well yes, but that's not the way > > > cairo works. First it creates the requested graphic in a standard internal > > > representation that maintains all the necessary information for vector output. > > > Then when you are ready to dump it to a file, it converts the internal > > > representation to the final output representation. > > > Or at least that's how I understand it. > > > > > > Time to revisit the idea of a pdflatex terminal? > > > > How do you think would a pdflatex terminal solve this problem and what > > configuration do you have in mind for a pdflatex terminal? > > I assume that the goal is to end up with a PDF file that contains > a plot constructed from an unreasonably large number of > points... Isn't the best solution to down-sample the plot? Embedding a bitmap in PDF output seems retrograde to me (although I do see how it addresses the problem at hand). People often read PDFs on-screen and then it's nice to have the non-text material in vector form (resize at will). Allin Cottrell |
|
From: Hans-Bernhard B. <HBB...@t-...> - 2010-10-08 21:03:35
|
On 08.10.2010 21:51, Allin Cottrell wrote: > Isn't the best solution to down-sample the plot? Not really. Down-sampling a scatter plot defeats the purpose of the plot. The best solution is "Don't do that, then". Don't make million-point scatter plots in vector format if you're not prepared to accept the consequences (huge files, slow rendering times, possible rejection in print). What one can do instead is down-sample and histogrammize the _data_, then make a different kind of plot of those. E.g. a color-coded density plot. |
