ccplot-general

ccplot errors

[yinjinfang88 <yin...@16...>]

Dear all,
      I  run ccplot  for visualising CloudSat data. However, I got a error at line 1504 in ccplot file. That is:
        data = cctk.interpolate2d(data, X, Y, (0, e2-e1, e2-e1),
                             (ve1, ve2, resolution), float("nan"), 0, radius)
and the error message is as follows:

Traceback (most recent call last):
  File "/usr/local/bin/ccplot", line 2265, in <module>
    main(sys.argv)
  File "/usr/local/bin/ccplot", line 2183, in main
    opts=opts.plot_opts)
  File "/usr/local/bin/ccplot", line 1505, in plot_profile
    (ve1, ve2, resolution), float("nan"), 0, radius)
TypeError: Cannot cast array data from dtype('float64') to dtype('float32') according to the rule 'safe'

 Did something wrong？ 
Thinks!

     Yin

Re: ccplot errors

[Peter K. <pet...@wa...>]

On 05/26/2013 01:31 PM, yinjinfang88 wrote:
> Dear all,
>       I  run ccplot  for visualising CloudSat data. However, I got a
> error at line 1504 in ccplot file. That is:
>         data = cctk.interpolate2d(data, X, Y, (0, e2-e1, e2-e1),
>                              (ve1, ve2, resolution), float("nan"), 0, radius)
> and the error message is as follows:
>  
> Traceback (most recent call last):
>   File "/usr/local/bin/ccplot", line 2265, in <module>
>     main(sys.argv)
>   File "/usr/local/bin/ccplot", line 2183, in main
>     opts=opts.plot_opts)
>   File "/usr/local/bin/ccplot", line 1505, in plot_profile
>     (ve1, ve2, resolution), float("nan"), 0, radius)
> TypeError: Cannot cast array data from dtype('float64') to dtype('float32') according to the rule 'safe'
>  
>  Did something wrong？
> Thinks!
>  
>      Yin

Dear Yin,

Thank you very much for the report. This is a bug in ccplot. Could you
please let me know what CloudSat granule it was and the command you
used? It'll be helpful in pinpointing and fixing the problem.

Regards,

Peter

Re: ccplot errors

[Peter K. <pet...@wa...>]

Hi Yin,

Thank you for doing additional testing. That is a rather perplexing bug.
It is actually caused by a change in behaviour between numpy 1.6.2 and
1.7.1.

python
Python 2.7.3 (default, Jan  2 2013, 13:56:14)
[GCC 4.7.2] on linux2
>>> import numpy as np
>>> np.__version__
'1.6.2'
>>> x, y = np.meshgrid(np.arange(0, 2, dtype=np.int16), np.arange(3, 5,
dtype=np.float32))
>>> print x.dtype, y.dtype
int16 float32

vs.

python
Python 2.7.3 (default, Jan  2 2013, 13:56:14)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.__version__
'1.7.1'
>>> x, y = np.meshgrid(np.arange(0, 2, dtype=np.int16), np.arange(3, 5,
dtype=np.float32))
>>> print x.dtype, y.dtype
int64 float64

The newer version does upcasting to 64 bits. I'm not sure if this is the
intended behaviour, perhaps a bug report should be filed against numpy.

In any case, this can be worked around by casting X, Y explicitly. I
think your fix is essentially good, although I'd prefer casting with
astype, as it creates contiguous arrays.

Attached is a patch. Please let me know if it works for you.

You are most welcome to modify the source code of ccplot, it is open
source software, it makes me happy to see contributions and bug reports
from other people.

I'll answer your other question about cloud mask soon.

Btw. there is a new program in the making that is supposed to complement
ccplot in browsing larger amount of granules. It is a web app, with
server-side importing of granules.

https://github.com/peterkuma/ccbrowse
browse.ccplot.org (demo example, please open in recent firefox or chrome)

Support for CloudSat is kind of secondary at the moment, as it requires
CALIPSO granule to be imported first, so that relative time shift can be
calculated correctly. It can be installed and used locally as well.

Regards,

Peter

On 05/27/2013 05:19 AM, yinjinfang88 wrote:
> Hi Peter,
>        Thanks very much for your fast reply. The CloudSat data (granule
> number 14779) is 2009037050924_14779_CS_2B-GEOPROF_GRANULE_P_R04_E02.hdf
> , which is a example data for ccplot.
> You can download it at the web site
> http://sourceforge.net/projects/ccplot/files/products/. I printed the
> data type of  the variables of data, X, Y, ve1, ve2, resolution, and
> radius. There are shown as
>         print(data.dtype)   ----> float32
>         print(X.dtype)      ----> float64
>         print(Y.dtype)      ----> int16   
>         print(ve1.dtype)    ----> int16
>         print(ve2.dtype)    ----> int16.
> It can be seen that the error message resulted from the data type of
> variable X. I tried to add two key words of sparse=False, copy=False for
> np.meshgrid function at lines 1498 and 1501, respectively.
> Then the data type of  variable X is shown as float32. Although the
> ccplot works now, I don't know whether it is right or not. Because I
> am a newer of Python. 
>  
> I have a another question. There is a variable named cloud_mask
>  in CloudSat data,  which demonstrates the reliability of the Rader
> reflectivity. I want to get rid of the Rader reflectivity when the value
> of cloud_mask is less than 20. However, I don't know where I can add
> some condition select and judgement sentences in ccplot. Could you
> please show me a example?
>  
> It is impolite to modify your source codes without any permission. I ask
> for your forgiveness. 
>  
> Thanks again.
>  
>                                           Yin
>  
>  
> ------------------------------------------------------------------------
> yinjinfang88
>  
> *From:* Peter Kuma <mailto:pet...@wa...>
> *Date:* 2013-05-27 02:53
> *To:* yinjinfang88 <mailto:yin...@16...>
> *CC:* ccplot-general <mailto:ccp...@li...>
> *Subject:* Re: ccplot errors
> On 05/26/2013 01:31 PM, yinjinfang88 wrote:
>> Dear all,
>>       I  run ccplot  for visualising CloudSat data. However, I got a
>> error at line 1504 in ccplot file. That is:
>>         data = cctk.interpolate2d(data, X, Y, (0, e2-e1, e2-e1),
>>                              (ve1, ve2, resolution), float("nan"), 0, radius)
>> and the error message is as follows:
>>  
>> Traceback (most recent call last):
>>   File "/usr/local/bin/ccplot", line 2265, in <module>
>>     main(sys.argv)
>>   File "/usr/local/bin/ccplot", line 2183, in main
>>     opts=opts.plot_opts)
>>   File "/usr/local/bin/ccplot", line 1505, in plot_profile
>>     (ve1, ve2, resolution), float("nan"), 0, radius)
>> TypeError: Cannot cast array data from dtype('float64') to dtype('float32') according to the rule 'safe'
>>  
>>  Did something wrong？
>> Thinks!
>>  
>>      Yin
>  
> Dear Yin,
>  
> Thank you very much for the report. This is a bug in ccplot. Could you
> please let me know what CloudSat granule it was and the command you
> used? It'll be helpful in pinpointing and fixing the problem.
>  
> Regards,
>  
> Peter

Re: ccplot errors

[Peter K. <pet...@wa...>]

Hi Yin,

If it does not cause any problems, then I believe it is not necessary.
The code is mostly type agnostic, with the exception of certain
arguments to functions implemented in C (layermap & interpolate2d). The
arguments specifying extent in interpolate2d are parsed by
PyArg_ParseTuple, which does the conversion from double to float without
complaining. It is silently reducing precision, but this should not be a
problem (as the precision in products is 32-bit anyway).

ccplot is in need of more attention. In the next version 1.5, PyNIO is
to be ditched in favour of a cython module interfacing directly with
libhdf and libhdfeos. The benefit is that all other dependencies are
easily installed and available on windows. I'm currently working on a
set of unit tests, as there are many different use cases, and any
slightly more extensive changes to the source code are likely to break
some of them.

Regards,

Peter

On 05/28/2013 03:56 PM, yinjinfang88 wrote:
> Hi Peter,
>         Thank you very much for your help. It works now. However, there
> is a different data type of  ve1 and ve2.
> ccplot -x 500..1000 -c cloudsat-reflectivity.cmap  -o cloudsat-reflec.png cloudsat-reflec  2009037050924_14779_CS_2B-GEOPROF_GRANULE_P_R04_E02.hdf 
> print(ve1.dtype)    ----> int16
> print(ve2.dtype)    ----> int16
>  
> If the hight is limited by the parameter "-y 0..10000'', the data type
> of vel and ve2 becomes as float64. 
> ccplot -x 500..1000 -y 0..10000 -c cloudsat-reflectivity.cmap  -o cloudsat-reflec.png cloudsat-reflec  2009037050924_14779_CS_2B-GEOPROF_GRANULE_P_R04_E02.hdf  
> 
> print(ve1.dtype)    ----> float64
> print(ve2.dtype)    ----> float64
>  
>           As mentioned above, the data type of vel and ve2 changes with
> the ccplot command with different parameters. Can we fix the data type
> of vel and ve2 at the beginning of the ccplot file.
>  
>                    Yin 
>  
>  
> ------------------------------------------------------------------------
> yinjinfang88
>  
> *From:* Peter Kuma <mailto:pet...@wa...>
> *Date:* 2013-05-28 04:29
> *To:* yinjinfang88 <mailto:yin...@16...>
> *CC:* ccplot-general <mailto:ccp...@li...>
> *Subject:* Re: ccplot errors
> Hi Yin,
>  
> Thank you for doing additional testing. That is a rather perplexing bug.
> It is actually caused by a change in behaviour between numpy 1.6.2 and
> 1.7.1.
>  
> python
> Python 2.7.3 (default, Jan  2 2013, 13:56:14)
> [GCC 4.7.2] on linux2
>>>> import numpy as np
>>>> np.__version__
> '1.6.2'
>>>> x, y = np.meshgrid(np.arange(0, 2, dtype=np.int16), np.arange(3, 5,
> dtype=np.float32))
>>>> print x.dtype, y.dtype
> int16 float32
>  
> vs.
>  
> python
> Python 2.7.3 (default, Jan  2 2013, 13:56:14)
> [GCC 4.7.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import numpy as np
>>>> np.__version__
> '1.7.1'
>>>> x, y = np.meshgrid(np.arange(0, 2, dtype=np.int16), np.arange(3, 5,
> dtype=np.float32))
>>>> print x.dtype, y.dtype
> int64 float64
>  
> The newer version does upcasting to 64 bits. I'm not sure if this is the
> intended behaviour, perhaps a bug report should be filed against numpy.
>  
> In any case, this can be worked around by casting X, Y explicitly. I
> think your fix is essentially good, although I'd prefer casting with
> astype, as it creates contiguous arrays.
>  
> Attached is a patch. Please let me know if it works for you.
>  
> You are most welcome to modify the source code of ccplot, it is open
> source software, it makes me happy to see contributions and bug reports
> from other people.
>  
> I'll answer your other question about cloud mask soon.
>  
> Btw. there is a new program in the making that is supposed to complement
> ccplot in browsing larger amount of granules. It is a web app, with
> server-side importing of granules.
>  
> https://github.com/peterkuma/ccbrowse
> browse.ccplot.org (demo example, please open in recent firefox or chrome)
>  
> Support for CloudSat is kind of secondary at the moment, as it requires
> CALIPSO granule to be imported first, so that relative time shift can be
> calculated correctly. It can be installed and used locally as well.
>  
> Regards,
>  
> Peter

Re: ccplot errors

[Peter K. <pet...@wa...>]

On 05/27/2013 05:19 AM, yinjinfang88 wrote:
[...]
> I have a another question. There is a variable named cloud_mask
>  in CloudSat data,  which demonstrates the reliability of the Rader
> reflectivity. I want to get rid of the Rader reflectivity when the value
> of cloud_mask is less than 20. However, I don't know where I can add
> some condition select and judgement sentences in ccplot. Could you
> please show me a example?

Hi Yin,

That should go somewhere below:

1408            np.place(data, data == -88.88, float("nan"))

Reading the dataset would go under:

1307    try: # Try CloudSat names.

A command-line flag would be nice. E.g. -f filter, where filter is none
or cloudmask=n, and n is integer specifying low threshold of CPR Cloud Mask.

Patch attached. Do let me know if you spot any bugs in it. The feature
is documented in the man page.

Regards,

Peter