gnuplot-info

[Gnuplot-info] pm3d with sparse data

[Lawrence M. <law...@ed...>]

Hello,

Firstly, thank you very much to the developers of gnuplot, I've been 
using it for some time, fantastic piece of software. I'm having a bit of 
trouble at the moment producing a surface plot with pm3d, however, so 
thought I'd ask for some help.

For interest sake, this is a plot of the results of a particle filter, a 
sequential MCMC (Markov Chain Monte Carlo) sampling technique. On the x 
axis is time, the y axis value, and the z axis the density of the 
probability distribution. Simply put, I'm representing a two-dimensional 
probability distribution as a 3d histogram.

The data I'm working with consists of individual samples from the 
probability distribution. This is "binned" into grid data in a 
preprocessing step by dividing the x and y axes into regular intervals 
to form a grid of squares, and counting the number of samples that fall 
into each square, this count becomes the density for the z axis.

Now, this is all well and good, and I can produce a pm3d surface plot, 
fantastic.

However, the data I'm working with is sparse. That is to say, many of 
those bins have no samples falling in them, and thus have a value of 
zero. To produce plots I've had to pad out my data files with all the 
zero size bins, so that all the isolines forming the surface have the 
same number of data points and the surface forms up nicely. For some 
plots this is fine, although they still take a long time to produce, but 
for most it makes the data files enormous, 250 MB some of them. Gnuplot 
nobly tackles the task regardless, but it runs out of memory very 
quickly trying to plot all those points! The plot is also too 
complicated to feasibly rotate and zoom in on etc on the screen, which 
would be useful for exploring these distributions.

If I don't pad out the data files with zero size bins, the isolines 
don't have the same number of points, and the plot has lots of artifacts 
from trying to join points between isolines. On the other hand, it is 
quickly drawn and I can rotate it etc no problem.

So what I'm wondering is if there's some way of doing this that:

a) Maintains the sparse representation of my data files, that is, only 
have non-zero bins in there.
b) Efficiently produces a plot, as when I don't pad the data files with 
zero size bins.
c) Produces a "correct" plot, as when I do pad the data files with zero 
size bins.

I've tried a few things:

a) dgrid3d: unfortunately the way the weighting works in this confounds 
what I'm trying to represent. If it could do some sort of nearest 
neighbour thing instead that could work.
b) Making sure every non-zero bin has its adjacent bins explicitly 
defined in the file, even if these are zero bins. This makes the 
isolines correct, but doesn't fix the problem of how to join them to 
form the surface.
c) I've been looking at the interpolation features in the docs, but I'm 
not sure if and how they can be applied to this situation. If I could 
linearly interpolate between the defined points in each isoline 
(defining adjacent points as in (b)), then resample at regular intervals 
so that each isoline had the same number of points, that would be perfect.

Would anyone have some ideas on how to tackle this?

Cheers,

Lawrence

Re: [Gnuplot-info] pm3d with sparse data

[<HBB...@t-...>]

Lawrence Murray wrote:

> For interest sake, this is a plot of the results of a particle filter, a 
> sequential MCMC (Markov Chain Monte Carlo) sampling technique. On the x 
> axis is time, the y axis value, and the z axis the density of the 
> probability distribution. Simply put, I'm representing a two-dimensional 
> probability distribution as a 3d histogram.

But I'm afraid you didn't follow the first rule of histogram generation: 
think before you bin.  You appear to have binned way finer than your 
data supports.  As a rule of thumb histogram bins with zero entries 
should be the exception, not the rule.  Statistically the average count 
per bin should be at least roughly 5, because smaller counts are 
dominated by randomness.

Put another way: there's no way the display of your histogram can 
faithfully reproduce 256 MBytes' worth of data --- even a rather 
high-res screen image stores nowhere near that much information.  I.e. 
even if gnuplot managed to plot that file, the result would be 
completely useless.

Re: [Gnuplot-info] pm3d with sparse data

[Lawrence M. <law...@ed...>]

Hans-Bernhard Bröker wrote:
> But I'm afraid you didn't follow the first rule of histogram 
> generation: think before you bin.  You appear to have binned way finer 
> than your data supports.  As a rule of thumb histogram bins with zero 
> entries should be the exception, not the rule.  Statistically the 
> average count per bin should be at least roughly 5, because smaller 
> counts are dominated by randomness.
>
> Put another way: there's no way the display of your histogram can 
> faithfully reproduce 256 MBytes' worth of data --- even a rather 
> high-res screen image stores nowhere near that much information.  I.e. 
> even if gnuplot managed to plot that file, the result would be 
> completely useless.
Thanks for your response. I appreciate what you're saying, but the 
distributions I'm dealing with by nature consist of finely separated 
sharp peaks, such that the high-res binning is required to discriminate 
them. For example, out of 10 000 points, some bins may have over 1000 
points even at this resolution. This is a characteristic of the data, it 
is not due to noise. Yes, you can adjust the xrange and yrange to zoom 
in on clusters of peaks and get a more typical histogram for local 
regions, but it's nice to get an overall picture as well, which is what 
I'm trying to achieve here.

Perhaps I wasn't clear also, the data files I'm working with are 150 KB 
at most, it's only when padding them with zeros to give a dense grid 
structure for gnuplot that they become unwieldy, up to 250 MB. It's 
rather like sparse vs dense matrix representations.

Plotting with impulses or just the isolines, without forming a surface, 
seems to achieve reasonable results, although I think the pm3d surface 
plot more clearly presents the data. Is there some way of interpolating 
and resampling between points of the isolines to join them up into a 
surface, without explicitly entering all the points into the data file?

Cheers,

Lawrence

Re: [Gnuplot-info] pm3d with sparse data

[<HBB...@t-...>]

Lawrence Murray wrote:

> Thanks for your response. I appreciate what you're saying, but the 
> distributions I'm dealing with by nature consist of finely separated 
> sharp peaks, such that the high-res binning is required to discriminate 
> them. 

I have to disagree.  The way you describe your data, what is required to 
display them is peak finding, not binning.  You obviously want to 
display the position of those peaks, not their height.  A colour map is 
rather certainly not the best way of displaying such data.

> Perhaps I wasn't clear also, the data files I'm working with are 150 KB 
> at most, it's only when padding them with zeros to give a dense grid 
> structure for gnuplot that they become unwieldy, up to 250 MB. It's 
> rather like sparse vs dense matrix representations.

To arrive a 250 MB, you would have to blow up the x and y resolution of 
those meshes to upward of 2000x2000 points.  That's plainly unplottable 
--- your entire screen doesn't have that many pixels, not even to begin 
with the actual reslution of human eyesight.  You're on an impossible 
mission there, trying to display data with more x and y resolution than 
can possibly fit on your screen.

Colour-coded impulses or points would make a lot more sense for this 
data set.  There's nothing smooth or continuous about it.  We're talking 
about a bed-of-nails here.