Re: Excessive memory usage with the pdfcairo terminal?

[Peter J. <pet...@gm...>]

On Fri, Oct 8, 2010 at 2:32 AM, Ethan Merritt <merritt@u.washington.edu> wrote:
> On Thursday 07 October 2010 01:07:28 pm Juhász Péter wrote:
>> Dear gnuplot developers,
>>
>> while doing some work on a large dataset, I've noticed that gnuplot uses
>> what could be construed as an excessive amount of memory if the terminal
>> is set to pdfcairo. (My machine actually ran out of physical memory and
>> began to thrash the swap, that's why I noticed).
>>
>> I've done some simple benchmarking with random data (k*1e5 pairs of
>> random numbers generated by perl, where k went from 1 to 10; then this
>> was plotted by:
>> gnuplot -e 'set term pdf;set out "r.pdf";plot "r.txt" w d').
>>
>> >From what I can see from the benchmark, the dependence of the memory
>> consumption on the dataset size is completely linear - this is expected
>> and understandable. What is not expected and hardly understandable is
>> the ratio: it needs about 872 bytes per point.
>
> How are you tracking memory usage?

Simply: ps ux | grep gnuplot. I generated ten datasets with k*1e5
points (k=1:10) and I saved the values of the VSZ and RSS columns of
the ps output for each dataset, then I fitted a line on the memory
data. So the precise statement is that memory usage grows by about 872
bytes per point.

Glancing at the source code, this may not be that mysterious at all.
It does a lot more that just drawing a dot, even if the plot style is
dots.

>
>> The pdfcairo terminal also takes a long time: about 27.9 s to plot 1
>> million points. (The time dependence is also linear.)
>
> One interesting thing is that less than half of this time is spent
> creating the cairo image internally; more of the time is spent
> converting it to pdf afterward.
>
> test file bigdata.dat was created by
>   for (i=0; i<1000000; i++) printf("%g %g\n",drand48(), drand48());
>
> [1] time gnuplot -e 'set term png; plot "bigdata.dat" using 1:2 with dots' > bigdata.png
> 0.883u 0.060s 0:01.00 94.0%     0+0k 0+24io 0pf+0w
> [2] time gnuplot -e 'set term pngcairo; plot "bigdata.dat" using 1:2 with dots' > bigdata.pngcairo
> 15.339u 0.044s 0:17.03 90.2%    0+0k 0+384io 0pf+0w
> [3] time gnuplot -e 'set term pdfcairo; plot "bigdata.dat" using 1:2 with dots' > bigdata.pdf
> 37.023u 0.533s 0:38.02 98.7%    0+0k 0+51520io 0pf+0w
>
> The long-standing admonition still holds. Vector formats like PostScript,
> PDF, SVG are bad choices for creating a plot with a million points.
> A bitmap format like PNG does much better.

I'm aware of this, the above example of one million points was only
for testing purposes.

"Ah", you say,
> "but I asked cairo to produce a PNG file." Well yes, but that's not the way
> cairo works.  First it creates the requested graphic in a standard internal
> representation that maintains all the necessary information for vector output.
> Then when you are ready to dump it to a file, it converts the internal
> representation to the final output representation.
> Or at least that's how I understand it.
>
> Time to revisit the idea of a pdflatex terminal?
>
>        Ethan
>
>> For comparison, the same data for the postscript terminal: little more
>> than 60 bytes per point (which is just what the plot->points array
>> needs) and just above one second of execution time for the same 1
>> million points.
>>
>> Now I haven't delved into the innards of the pdfcairo terminal driver,
>> and if you tell me that this is normal and expected, I'll believe you.
>> But I can't imagine why it needs over 800 bytes to store a single point.
>>
>> A not entirely unrelated question: how do I compile gnuplot with
>> profiling? I tried giving CFLAGS="-g -pg" to the configure script, but
>> the gmon.out file was not there after running gnuplot.
>>
>> Péter Juhász
>