[Gnuplot-info] pm3d with sparse data

[Lawrence M. <law...@ed...>]

Hello,

Firstly, thank you very much to the developers of gnuplot, I've been 
using it for some time, fantastic piece of software. I'm having a bit of 
trouble at the moment producing a surface plot with pm3d, however, so 
thought I'd ask for some help.

For interest sake, this is a plot of the results of a particle filter, a 
sequential MCMC (Markov Chain Monte Carlo) sampling technique. On the x 
axis is time, the y axis value, and the z axis the density of the 
probability distribution. Simply put, I'm representing a two-dimensional 
probability distribution as a 3d histogram.

The data I'm working with consists of individual samples from the 
probability distribution. This is "binned" into grid data in a 
preprocessing step by dividing the x and y axes into regular intervals 
to form a grid of squares, and counting the number of samples that fall 
into each square, this count becomes the density for the z axis.

Now, this is all well and good, and I can produce a pm3d surface plot, 
fantastic.

However, the data I'm working with is sparse. That is to say, many of 
those bins have no samples falling in them, and thus have a value of 
zero. To produce plots I've had to pad out my data files with all the 
zero size bins, so that all the isolines forming the surface have the 
same number of data points and the surface forms up nicely. For some 
plots this is fine, although they still take a long time to produce, but 
for most it makes the data files enormous, 250 MB some of them. Gnuplot 
nobly tackles the task regardless, but it runs out of memory very 
quickly trying to plot all those points! The plot is also too 
complicated to feasibly rotate and zoom in on etc on the screen, which 
would be useful for exploring these distributions.

If I don't pad out the data files with zero size bins, the isolines 
don't have the same number of points, and the plot has lots of artifacts 
from trying to join points between isolines. On the other hand, it is 
quickly drawn and I can rotate it etc no problem.

So what I'm wondering is if there's some way of doing this that:

a) Maintains the sparse representation of my data files, that is, only 
have non-zero bins in there.
b) Efficiently produces a plot, as when I don't pad the data files with 
zero size bins.
c) Produces a "correct" plot, as when I do pad the data files with zero 
size bins.

I've tried a few things:

a) dgrid3d: unfortunately the way the weighting works in this confounds 
what I'm trying to represent. If it could do some sort of nearest 
neighbour thing instead that could work.
b) Making sure every non-zero bin has its adjacent bins explicitly 
defined in the file, even if these are zero bins. This makes the 
isolines correct, but doesn't fix the problem of how to join them to 
form the surface.
c) I've been looking at the interpolation features in the docs, but I'm 
not sure if and how they can be applied to this situation. If I could 
linearly interpolate between the defined points in each isoline 
(defining adjacent points as in (b)), then resample at regular intervals 
so that each isoline had the same number of points, that would be perfect.

Would anyone have some ideas on how to tackle this?

Cheers,

Lawrence