Re: Excessive memory usage with the pdfcairo terminal?

[Ethan M. <merritt@u.washington.edu>]

On Thursday 07 October 2010 01:07:28 pm Juhász Péter wrote:
> Dear gnuplot developers,
> 
> while doing some work on a large dataset, I've noticed that gnuplot uses
> what could be construed as an excessive amount of memory if the terminal
> is set to pdfcairo. (My machine actually ran out of physical memory and
> began to thrash the swap, that's why I noticed).
> 
> I've done some simple benchmarking with random data (k*1e5 pairs of
> random numbers generated by perl, where k went from 1 to 10; then this
> was plotted by:
> gnuplot -e 'set term pdf;set out "r.pdf";plot "r.txt" w d').
> 
> >From what I can see from the benchmark, the dependence of the memory
> consumption on the dataset size is completely linear - this is expected
> and understandable. What is not expected and hardly understandable is
> the ratio: it needs about 872 bytes per point.

How are you tracking memory usage?

> The pdfcairo terminal also takes a long time: about 27.9 s to plot 1
> million points. (The time dependence is also linear.)

One interesting thing is that less than half of this time is spent
creating the cairo image internally; more of the time is spent
converting it to pdf afterward.

test file bigdata.dat was created by
   for (i=0; i<1000000; i++) printf("%g %g\n",drand48(), drand48());

[1] time gnuplot -e 'set term png; plot "bigdata.dat" using 1:2 with dots' > bigdata.png
0.883u 0.060s 0:01.00 94.0%     0+0k 0+24io 0pf+0w
[2] time gnuplot -e 'set term pngcairo; plot "bigdata.dat" using 1:2 with dots' > bigdata.pngcairo
15.339u 0.044s 0:17.03 90.2%    0+0k 0+384io 0pf+0w
[3] time gnuplot -e 'set term pdfcairo; plot "bigdata.dat" using 1:2 with dots' > bigdata.pdf
37.023u 0.533s 0:38.02 98.7%    0+0k 0+51520io 0pf+0w

The long-standing admonition still holds. Vector formats like PostScript,
PDF, SVG are bad choices for creating a plot with a million points.
A bitmap format like PNG does much better.  "Ah", you say, 
"but I asked cairo to produce a PNG file." Well yes, but that's not the way
cairo works.  First it creates the requested graphic in a standard internal
representation that maintains all the necessary information for vector output.
Then when you are ready to dump it to a file, it converts the internal
representation to the final output representation.
Or at least that's how I understand it.

Time to revisit the idea of a pdflatex terminal?

	Ethan
 
> For comparison, the same data for the postscript terminal: little more
> than 60 bytes per point (which is just what the plot->points array
> needs) and just above one second of execution time for the same 1
> million points.
> 
> Now I haven't delved into the innards of the pdfcairo terminal driver,
> and if you tell me that this is normal and expected, I'll believe you.
> But I can't imagine why it needs over 800 bytes to store a single point.
> 
> A not entirely unrelated question: how do I compile gnuplot with
> profiling? I tried giving CFLAGS="-g -pg" to the configure script, but
> the gmon.out file was not there after running gnuplot.
> 
> Péter Juhász