Re: Smooth histograms

[<pl...@pi...>]

On Sat, 24 May 2008 01:15:37 +0200, Philipp K. Janert <ja...@ie...>  
wrote:

> I just submitted a patch (1970923) which
> draws a smooth histogram-like curve
> for a random collection of points, using
> a Gaussian kernel density estimation
> algorithm.
> Demos are found here:
> 	www.philipp-janert.com/kdensity


very interesting.

My initial impression on looking at your top left example is that there is  
a phase shift of +half a box in x most visible in the 0.01 and 0.05 plots.

Considering that x=0 is in the middle of the first box it appears that the  
fits are responding in a way that aligns with the right of each box.

It's a bit subjective due to the nature of the data but this is my  
impression for the peaks at 0.2 0.4 and 0.5

Maybe you could test this effect with a rapid change in the data.


I also think there is an egde effect at the begining and end of the data.  
This is a common problem when applying this sort of technique to image  
data. How to deal with edges when the kernel goes outside the data. There  
are several "solutions" which involve falsely extening the data but  
applying a kernel to an incompete sample range is effectively filling it  
with zeros and is equally false.

It's like a running mean cannot be meaningful upto the edges of the sample  
range since there are not enough samples to take the mean over. This also  
gives an artificial drop off at the edges. This is also rather marked near  
the origin in your lognormal example.

In image processing it's just a case of prettying up the edges but in a  
scientific context this is clearly not appropriate.

I think the only rigourous way to deal with this is not to plot the part  
where the data is incomplete.

I hope the comments are useful.

best regards, Peter.