Re: [Matplotlib-users] cross correlation

[Sudheer J. <sud...@ya...>]

Thank you Pierre,
                              I will test the other options. I did not know the number limitation in case of plt.xcorr.
Thanks a lot
with best regards,
Sudheer
***************************************************************
Sudheer Joseph 
Indian National Centre for Ocean Information Services
Ministry of Earth Sciences, Govt. of India
POST BOX NO: 21, IDA Jeedeemetla P.O.
Via Pragathi Nagar,Kukatpally, Hyderabad; Pin:5000 55
Tel:+91-40-23886047(O),Fax:+91-40-23895011(O),
Tel:+91-40-23044600(R),Tel:+91-40-9440832534(Mobile)
E-mail:sjo...@gm...;sud...@ya...
Web- http://oppamthadathil.tripod.com
***************************************************************


________________________________
 From: Pierre Haessig <pie...@cr...>
To: mat...@li... 
Sent: Thursday, 21 February 2013 9:52 PM
Subject: Re: [Matplotlib-users] cross correlation
 

Hi Sudheer,

Le 21/02/2013 02:22, Sudheer Joseph a écrit : 
Thank you  very much Smith and Paul,
>                                          I was away from office due to a medical situation. So could not respond and thank you regarding the help. I have got the results now and the tips from both of you were extremely useful. I am facing an issue with the code when I call plt.xcorr,  in a loop. it builds up usage of memory by python and reaches to the RAM what ever available ( in my 4 GB laptop it reaches almost full and in my 24 GB desktop it reaches the available. I suspected the plot not being closed during each iteration so have given a plt.close('all') in the loop. after which it is taking a good time to run the code which was otherwise faster until ram usage reaches its maximum.
>Is there a way to get out of this situation?. I am attaching the code here and also the link to the data I am using. If possible kindly help.
>
Thanks for sharing the code. By a quick look at gen_xcorr_wnd.py,
    you are generating a quite high number (about len(lons)*len(lats))
    of xcorr series over 365 lags. Here are two thoughts about why I
    would not recommend using xcorr from matplotlib for this job :

1) There is an overhead in creating a plot object which is
    unnecessary since you're only interested in correlation values

2) internally, plt.xcorr uses numpy.correlate
    (https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/axes.py#L4319 and https://github.com/numpy/numpy/blob/master/numpy/core/numeric.py#L731) which is quite fast but unfortunately cannot be well tuned in terms of the output length (only three modes : 'valid', 'same' or 'full'. Matplotlib uses 'full' )
All this to say that when you're interested in 365 correlation
    values, the internal computations takes place on (N+M-1) points
    (where N, M are the length of the input vectors, i.e. 2189 if I'm
    right) and so about 90 % of the output is thrown away.



This being said, there is a tiny issue : I don't know a good module
    which has the (x)correlation function. statsmodel has acf (aka
    correlation) but I don't remember if there is crosscorrelation. For
    acf has two computation modes : one based on fft, one based on
    numpy.correlate which suffer from the same problem as matplotlib's
    xcorr ( https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/stattools.py#L347)

best,
Pierre

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________
Matplotlib-users mailing list
Mat...@li...
https://lists.sourceforge.net/lists/listinfo/matplotlib-users