Menu
▾
▴
opengrads-devel
|
From: Arlindo da S. <da...@al...> - 2008-02-02 20:21:14
|
Jennifer/Brian et al, 1) I have patched the supplibs sources to build with HDF v4.2r3 (just released). I'll be rebuilding and posting sources/binaries to sf.net under version supplibs-2.0.1. For documentation on checking out/building the supplibs check the wiki: http://opengrads.org/wiki/index.php?title=Supplemental_Libraries_%28Supplibs%29 2) I have obtained your original grads-2.0.a0 tar ball and updated the build mechanism to work with the new supplibs. Please consult the ChangeLog for a descriptions of the mods I made. (see below). Please check it out directly from our CVS repository: gacvs co -P -r GRADS2_DEV_BRANCH grads cd grads ./configure make Notice that gacvs is an alias for cvs -z3 -d:ext:da...@op...:/cvsroot/opengrads (replace "dasilva" with your own user id at sf.net). You also checkoput anonymously from sf.net: http://sourceforge.net/cvs/?group_id=161773 3) I also updated the test suite for working with v2; just type make check I also updated lats4d to work with grads v2. Of course, you can only specify -format stream, sequential, stats or gs scripts to be executed. Formats grib and coards cannot be specified because grads v2 does not have LATS. I have some documentation about the test suite here: http://opengrads.org/wiki/index.php?title=PyTests:_GrADS_Test_Suite 4) Of the 70 tests, gradsv2 is failing 16. They are related to 2 issues: a) The "sdfopen model.hdf" is not quite working: it reports the file as having 1 timestep instead of 5. I verified that the RH binary Jennifer posted has exactly the same problem, so it does not appear to be my build. b) When using "set gxout fvwrite" to write binary files the missing value is not being written correctly, in either little-endian or big-endian mode. Is this a bug or a feature (I haven't verified whether v1.9 does the same)? Brian/Jennifer: I am submitting this to you as patches to v2 for your review and adoption in v2 as you see fit. If you do adopt these or your own mods for building with the new supplibs and put out a new source tarball, I can help with "out of the box" builds for distribution by COLA. I am not planning to distributed this tag at the opengrads sf.net download area as it does not yet have the hooks for developing/running opengrads extensions. If you are interested, I can stop at COLA one of these mornings on my way to work and do a quick walkthru of the test suite. Although it is written in Python using PyUnit (a rip off of JUnit which you may be familiar with from Java), you can write all of your tests as gs scripts. (However, I chose to write my tests in python). It has been extremely useful for testing the binaries on the different platforms. It only requires python v2.3 or later which is available everywhere except for old servers. Pat: Let me know if this works with Fedora. Cheers! Arlindo --------------------------------------------------------------------------------------------- GrADS ChangeLog 2008-01-31 <da...@op...>, Version GrADS 2.0.a0 * Imported Version 2.0.a0 sources from COLA into COLA vendor branch. 2008-01-31 <da...@op...>, Version GrADS 2.0.a0-1 * This version temporarily placed under GRADS2_DEV_BRANCH * Adapted build mechanism from v1.9.0-rc1 for v2.0.a0 * grads.h: no longer includes gasdf.h, and gasdf prototypes have been moved into gasdf.h. This is important to avoid including "netcdf.h" from NetCDF when building with NC-DAP (OPeNDAP implementation of NetCDF). * gagui.c, gsgui.c: replaced with v1.9.0-rc1 to cope with new supplibs, also revised license terms to GPL 2 * gatypes.h: renamed Pixel->gaPixel to avoid conflict with new supplibs. * grads.c: changed main() to Main() for Win32; added main.c, main_win32.c * GNUmakefile: added for Win32 * w32_dist.csh: added for Win32 * pcx11e.[ch]: updated license to GPL * gacfg.c: removed (c) 1997 Arlindo da Silva + old, unclear, license terms -- Arlindo da Silva da...@al... |
|
From: Jennifer A. <jm...@co...> - 2008-02-03 00:22:17
|
On Feb 2, 2008, at 3:21 PM, Arlindo da Silva wrote: > Jennifer/Brian et al, > > 1) I have patched the supplibs sources to build with HDF v4.2r3 > (just released). I'll be rebuilding and posting sources/binaries to > sf.net under version supplibs-2.0.1. For documentation on checking > out/building the supplibs check the wiki: > > http://opengrads.org/wiki/index.php?title=Supplemental_Libraries_% > 28Supplibs%29 > 2) I have obtained your original grads-2.0.a0 tar ball and updated > the build mechanism to work with the new supplibs. Please consult > the ChangeLog for a descriptions of the mods I made. (see below). > Please check it out directly from our CVS repository: Arlindo -- There is a bug in seekgb.c of the g2clib that causes it to fail to detect the end of a grib2 record if sizeof(g2int) != 4 (i.e. on 64- bit systems). I guess we should put the patch in the subblib src files we are redistributing (on your site and COLA's). This bug causes gribmap to fail to find any records -- pretty bad. # diff ./seekgb.c ~/Desktop/seekgb.c 39a40 > int hdr; 68,69c69,70 < k4=fread(&end,sizeof(g2int),1,lugb); < if (k4 == 1 && end == 926365495) { //GRIB message found --- > k4=fread(&hdr,4,1,lugb); > if (k4 == 1 && hdr == 926365495) { //GRIB message found > > > gacvs co -P -r GRADS2_DEV_BRANCH grads > > cd grads > ./configure > make > > Notice that gacvs is an alias for > > cvs -z3 -d:ext:da...@op...:/cvsroot/ > opengrads > > (replace "dasilva" with your own user id at sf.net). You also > checkoput anonymously from sf.net: > > http://sourceforge.net/cvs/?group_id=161773 > > 3) I also updated the test suite for working with v2; just type > > make check > > I also updated lats4d to work with grads v2. Of course, you can > only specify -format stream, sequential, stats or gs scripts to be > executed. Formats grib and coards cannot be specified because grads > v2 does not have LATS. > > I have some documentation about the test suite here: > > http://opengrads.org/wiki/index.php?title=PyTests:_GrADS_Test_Suite > > > 4) Of the 70 tests, gradsv2 is failing 16. They are related to 2 > issues: > > a) The "sdfopen model.hdf" is not quite working: it reports the > file as having 1 timestep instead of 5. I verified that the RH > binary Jennifer posted has exactly the same problem, so it does not > appear to be my build. The code path for hdf in sdfopen is new. I will look at it and see what's going wrong. Where is model.hdf? > b) When using "set gxout fvwrite" to write binary files the missing > value is not being written correctly, in either little-endian or > big-endian mode. Is this a bug or a feature (I haven't verified > whether v1.9 does the same)? Can you be more specific about what 'not being written correctly' means? Can you give me a script segment that shows what it is doing wrong? Undef handling in v2 is entirely different than 1.9 -- it won't help to look back. > Brian/Jennifer: I am submitting this to you as patches to v2 for > your review and adoption in v2 as you see fit. I will put a priority on importing the build stuff. --Jennifer > If you do adopt these or your own mods for building with the new > supplibs and put out a new source tarball, I can help with "out of > the box" builds for distribution by COLA. I am not planning to > distributed this tag at the opengrads sf.net download area as it > does not yet have the hooks for developing/running opengrads > extensions. > If you are interested, I can stop at COLA one of these mornings on > my way to work and do a quick walkthru of the test suite. Although > it is written in Python using PyUnit (a rip off of JUnit which you > may be familiar with from Java), you can write all of your tests > as gs scripts. (However, I chose to write my tests in python). It > has been extremely useful for testing the binaries on the different > platforms. It only requires python v2.3 or later which is available > everywhere except for old servers. > > Pat: Let me know if this works with Fedora. > > Cheers! > > Arlindo > > > > ---------------------------------------------------------------------- > ----------------------- > GrADS ChangeLog > > 2008-01-31 <da...@op...>, Version GrADS 2.0.a0 > * Imported Version 2.0.a0 sources from COLA into COLA vendor > branch. > > 2008-01-31 <da...@op...>, Version GrADS 2.0.a0-1 > * This version temporarily placed under GRADS2_DEV_BRANCH > * Adapted build mechanism from v1.9.0-rc1 for v2.0.a0 > * grads.h: no longer includes gasdf.h, and gasdf prototypes have > been moved into gasdf.h. This is important to avoid including > "netcdf.h" from NetCDF when building with NC-DAP (OPeNDAP > implementation of NetCDF). > * gagui.c, gsgui.c: replaced with v1.9.0-rc1 to cope with new > supplibs, also revised license terms to GPL 2 > * gatypes.h: renamed Pixel->gaPixel to avoid conflict with new > supplibs. > * grads.c: changed main() to Main() for Win32; added main.c, > main_win32.c > * GNUmakefile: added for Win32 > * w32_dist.csh: added for Win32 > * pcx11e.[ch]: updated license to GPL > * gacfg.c: removed (c) 1997 Arlindo da Silva + old, unclear, > license terms > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Arlindo da Silva > da...@al... > ---------------------------------------------------------------------- > --- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Opengrads-devel mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/opengrads-devel > > --===============1092192969==-- -- Jennifer M. Adams IGES/COLA 4041 Powder Mill Road, Suite 302 Calverton, MD 20705 jm...@co... |
|
From: Arlindo da S. <da...@al...> - 2008-02-03 03:02:24
|
On Feb 2, 2008 7:21 PM, Jennifer Adams <jm...@co...> wrote:
>
>
> Arlindo --
> There is a bug in seekgb.c of the g2clib that causes it to fail to detect
> the end of a grib2 record if sizeof(g2int) != 4 (i.e. on 64-bit systems).
> I guess we should put the patch in the subblib src files we are
> redistributing (on your site and COLA's). This bug causes gribmap to fail to
> find any records -- pretty bad.
>
OK, I'll patch it and check it in. Fell free to to checkout the supplib
sources and commit but fixes like this one.
BTW, we autoconfiscated grib2c for easier integration in the supplibs; other
users of the library may appreciate this as well. We should contribute this,
along with your bug fix, to the 'upstream' maintainers. Are you in contact
with them?
Another important point. I need a small grib-2 file to add to the test
suite. Right now I have model.grb, model.nc and model.hdf - all the same
file, just in different formats.
(I test binary files by writing them out with 'fwrite' and then reading them
back.)
This allows me to apply the same test code to all of them. Ideally, one
should have a similar model.grb2 file. Wes attempted to convert the
existing model.grb to grb 2 but wasn't able to because model.grb does not
conform to NCEP's tables, or some other obscure grib reason.
> The code path for hdf in sdfopen is new. I will look at it and see what's
> going wrong.
>
It would be good to have model_nc.ctl and model_hdf.ctl so I can test the
ctl interface for netcdf/hdf as well. If you contribute those 2 ctls I can
add the additional tests.
>
Where is model.hdf?
>
When you checkout the sources look under pytests/data, all test data is
there.
> b) When using "set gxout fvwrite" to write binary files the missing value
> is not being written correctly, in either little-endian or big-endian mode.
> Is this a bug or a feature (I haven't verified whether v1.9 does the
> same)?
>
> Can you be more specific about what 'not being written correctly' means?
> Can you give me a script segment that shows what it is doing wrong? Undef
> handling in v2 is entirely different than 1.9 -- it won't help to look
> back.
>
The undef value is 1E+20 for the opened file (model.nc). I then use fwrite
to write all vars, levels, times, to disk. When I read it back a different
value is used for UNDEF:
2.2616e+06. Here is how to reproduce the error outside the testing
framework. From the Unix command line:
- check out grads sources and go to the testing directory:
% cd grads/pytests
- Convince yourself that undef of input file is 1e+20
% ncdump data/model.nc
- Now convert model.nc to binary using 'fwrite'
% mkdir output
% ./lats4d -i model/model.nc -o output/stream -format stream -be -v
(or write a simple script to do just that; choose a variable with undefs,
like "ua").
- use stream.ctl to read file output/stream.bin that you just created. You
can either open this file in GrADS or use lats4d again for a quick summary:
% ./lats4d -i stream.ctl -format stats
...
Name Lev Min Max MEAN STDV RMS
--------- ---- ----------- ----------- ----------- ----------- -----------
ua 1000 -12.9719 2.2616e+06 1.1868e+06 1.1296e+06 1.6383e+06
ua 850 -22.5107 2.2616e+06 3.0661e+05 7.7434e+05 8.3272e+05
ua 700 -19.0453 2.2616e+06 1.4750e+05 5.5850e+05 5.7757e+05
ua 500 -26.0697 2.2616e+06 2055.4200 6.8046e+04 6.8067e+04
ua 300 -30.2449 73.7551 12.3418 16.0524 20.2466
ua 200 -18.9862 81.0138 14.3248 16.6574 21.9678
ua 100 -19.2900 53.7100 11.3787 12.8745 17.1807
--------- ---- ----------- ----------- ----------- ----------- -----------
+ ua 7 -30.2449 2.2616e+06 2.3472e+05 3.6177e+05 7.2858e+05
The maximum value is what was mistakenly written for UNDEF. Incidentally,
this problem does not happens with model.grb:
% ./lats4d -i model/model.ctl -o output/stream -format stream -be -v
% ./lats4d -i stream.ctl -format stats
...
Name Lev Min Max MEAN STDV RMS
--------- ---- ----------- ----------- ----------- ----------- -----------
ua 1000 -12.9719 20.0281 -0.5000 6.5431 6.5601
ua 850 -22.5107 29.7393 1.6346 8.3796 8.5361
ua 700 -19.0453 33.9547 3.8051 9.2721 10.0211
ua 500 -26.0697 50.9303 6.8472 11.9436 13.7655
ua 300 -30.2449 73.7551 12.3418 16.0524 20.2466
ua 200 -18.9862 81.0138 14.3248 16.6574 21.9678
ua 100 -19.2900 53.7100 11.3787 12.8745 17.1807
--------- ---- ----------- ----------- ----------- ----------- -----------
+ ua 7 -30.2449 81.0138 7.1189 11.9898 15.0878
The test that is failing writes each input file to disk, and reads it
back checking each variable/level/time.
Let me know if you have any questions.
Arlindo
--
Arlindo da Silva
da...@al...
|
|
From: Arlindo da S. <da...@al...> - 2008-02-03 05:09:22
|
On Feb 2, 2008 7:21 PM, Jennifer Adams <jm...@co...> wrote:
>
>
> Arlindo --
> There is a bug in seekgb.c of the g2clib that causes it to fail to detect
> the end of a grib2 record if sizeof(g2int) != 4 (i.e. on 64-bit systems).
> I guess we should put the patch in the subblib src files we are
> redistributing (on your site and COLA's). This bug causes gribmap to fail to
> find any records -- pretty bad.
>
> # diff ./seekgb.c ~/Desktop/seekgb.c
> 39a40
> > int hdr;
> 68,69c69,70
> < k4=fread(&end,sizeof(g2int),1,lugb);
> < if (k4 == 1 && end == 926365495) { //GRIB message found
> ---
> > k4=fread(&hdr,4,1,lugb);
> > if (k4 == 1 && hdr == 926365495) { //GRIB message found
>
>
If it is OK with you I'll replace int with g2int for consistency with rest
of grib2c. They do assume that g2int = int are 32 bits, but keeping
everything in terms of g2int will make it easier to update the code if we
ran into ints that are > 32 bits.
g2int hdr;
...
k4=fread(&hdr,sizeof(hdr),1,lugb); if (k4 == 1 && hdr
== 926365495) { //GRIB message found
--
Arlindo da Silva
da...@al...
|
|
From: Brian D. <do...@co...> - 2008-02-03 14:57:37
|
lengths in a GRIB msg are defined by the GRIB specification and are
independent of systems definitions like g2int. It is appropriate to
hardcode the length. Or to use a typedef that is dependent on the
GRIB definitions (ie, hardcoded). It probably doesn't matter how
long hdr is (as long as it is at least 4 bytes ) but you want to read
exactly 4 bytes no more no less.. Brian
On Feb 3, 2008, at 12:09 AM, Arlindo da Silva wrote:
> On Feb 2, 2008 7:21 PM, Jennifer Adams <jm...@co...> wrote:
>
>
> Arlindo --
> There is a bug in seekgb.c of the g2clib that causes it to fail to
> detect the end of a grib2 record if sizeof(g2int) != 4 (i.e. on 64-
> bit systems). I guess we should put the patch in the subblib src
> files we are redistributing (on your site and COLA's). This bug
> causes gribmap to fail to find any records -- pretty bad.
>
> # diff ./seekgb.c ~/Desktop/seekgb.c
> 39a40
> > int hdr;
> 68,69c69,70
> < k4=fread(&end,sizeof(g2int),1,lugb);
> < if (k4 == 1 && end == 926365495) { //GRIB
> message found
> ---
> > k4=fread(&hdr,4,1,lugb);
> > if (k4 == 1 && hdr == 926365495) { //GRIB
> message found
>
>
> If it is OK with you I'll replace int with g2int for consistency
> with rest of grib2c. They do assume that g2int = int are 32 bits,
> but keeping everything in terms of g2int will make it easier to
> update the code if we ran into ints that are > 32 bits.
>
> g2int hdr;
> ...
> k4=fread(&hdr,sizeof(hdr),1,lugb);
> if (k4 == 1 && hdr == 926365495) { //GRIB message
> found
>
>
>
> --
> Arlindo da Silva
> da...@al...
|
|
From: Arlindo da S. <da...@al...> - 2008-02-03 15:36:49
|
On Feb 3, 2008 9:57 AM, Brian Doty <do...@co...> wrote:
> lengths in a GRIB msg are defined by the GRIB specification and are
> independent of systems definitions like g2int. It is appropriate to
> hardcode the length. Or to use a typedef that is dependent on the
> GRIB definitions (ie, hardcoded). It probably doesn't matter how
> long hdr is (as long as it is at least 4 bytes ) but you want to read
> exactly 4 bytes no more no less.. Brian
>
You and Jennifer are right. Do you think grib2c is robust enough to work on
machines where sizeof(int)!=4? If not, it may pay to add something to
configure.ac to detect such cases and stop with an error.
Pat: do you know how to do this?
Arlindo
>
> On Feb 3, 2008, at 12:09 AM, Arlindo da Silva wrote:
>
> > On Feb 2, 2008 7:21 PM, Jennifer Adams <jm...@co...> wrote:
> >
> >
> > Arlindo --
> > There is a bug in seekgb.c of the g2clib that causes it to fail to
> > detect the end of a grib2 record if sizeof(g2int) != 4 (i.e. on 64-
> > bit systems). I guess we should put the patch in the subblib src
> > files we are redistributing (on your site and COLA's). This bug
> > causes gribmap to fail to find any records -- pretty bad.
> >
> > # diff ./seekgb.c ~/Desktop/seekgb.c
> > 39a40
> > > int hdr;
> > 68,69c69,70
> > < k4=fread(&end,sizeof(g2int),1,lugb);
> > < if (k4 == 1 && end == 926365495) { //GRIB
> > message found
> > ---
> > > k4=fread(&hdr,4,1,lugb);
> > > if (k4 == 1 && hdr == 926365495) { //GRIB
> > message found
> >
> >
> > If it is OK with you I'll replace int with g2int for consistency
> > with rest of grib2c. They do assume that g2int = int are 32 bits,
> > but keeping everything in terms of g2int will make it easier to
> > update the code if we ran into ints that are > 32 bits.
> >
> > g2int hdr;
> > ...
> > k4=fread(&hdr,sizeof(hdr),1,lugb);
> > if (k4 == 1 && hdr == 926365495) { //GRIB message
> > found
> >
> >
> >
> > --
> > Arlindo da Silva
> > da...@al...
>
>
--
Arlindo da Silva
da...@al...
|
|
From: Patrice D. <per...@fr...> - 2008-02-03 17:16:46
|
On Sun, Feb 03, 2008 at 10:36:51AM -0500, Arlindo da Silva wrote: > On Feb 3, 2008 9:57 AM, Brian Doty <do...@co...> wrote: > > > lengths in a GRIB msg are defined by the GRIB specification and are > > independent of systems definitions like g2int. It is appropriate to > > hardcode the length. Or to use a typedef that is dependent on the > > GRIB definitions (ie, hardcoded). It probably doesn't matter how > > long hdr is (as long as it is at least 4 bytes ) but you want to read > > exactly 4 bytes no more no less.. Brian > > > > > You and Jennifer are right. Do you think grib2c is robust enough to work on > machines where sizeof(int)!=4? If not, it may pay to add something to > configure.ac to detect such cases and stop with an error. > > Pat: do you know how to do this? There are indeed autoconf macros to test for a type length, AC_CHECK_SIZEOF, for example AC_CHECK_SIZEOF([uint8_t]) if test $ac_cv_sizeof_uint8_t -eq 1; then ... fi There are types of given length in #include <inttypes.h> like int32_t uint32_t int16_t and so on and so forth (it is in POSIX 99). In libdap, if inttypes.h is found and the size of the types are right, an include file based on inttypes.h is used. Otherwise types with the correct size are searched for and corresponding symbols are set in config.h. The main configure macro is in conf/acinclude.m4 in DODS_CHECK_SIZES. -- Pat |
|
From: Brian D. <do...@co...> - 2008-02-03 16:14:40
|
sorry about our email ssytem tagging this stuff as junk. it is very weird. I don't think grib2c would work on systems where sizeof(int) is 2. Are there any such systems left? I am remembering that the size is 8 on our 64 bit linux, in which case it works ok with this small patch. But my memory is vague on this; Jennifer may remember better... Brian On Feb 3, 2008, at 10:36 AM, Arlindo da Silva wrote: > Do you think grib2c is robust enough to work on machines where > sizeof(int)!=4? |
|
From: Patrice D. <per...@fr...> - 2008-02-03 17:18:57
|
On Sun, Feb 03, 2008 at 11:14:39AM -0500, Brian Doty wrote: > sorry about our email ssytem tagging this stuff as junk. it is very > weird. > > I don't think grib2c would work on systems where sizeof(int) is 2. > Are there any such systems left? I am remembering that the size is 8 > on our 64 bit linux, in which case it works ok with this small > patch. But my memory is vague on this; Jennifer may remember > better... Brian My recall is that the size of int is 4 on 64 bit linux, it is long that has a different length. But in any case one should not rely on a specific sizeof(int) since (unless I am wrong) it is not in the standard. -- Pat |
|
From: Arlindo da S. <da...@al...> - 2008-02-03 18:27:08
|
On Feb 3, 2008 11:14 AM, Brian Doty <do...@co...> wrote: > sorry about our email ssytem tagging this stuff as junk. it is very > weird. > > I don't think grib2c would work on systems where sizeof(int) is 2. > Are there any such systems left? I am remembering that the size is 8 > on our 64 bit linux, in which case it works ok with this small > patch. But my memory is vague on this; Jennifer may remember > better... Brian > I think that compilers on x86_64 still use 32-bits for int (I just checked gcc on my x86_64 linux box), so we don't have much to worry about right now. Adding a test to configure would alert us if we are trying to build on anything that does not have sizeof(int)=4 so that we can have it addressed on the spot, rather than building grads and finding out that grads does not work. Arlindo -- Arlindo da Silva da...@al... |
|
From: Jennifer A. <jm...@co...> - 2008-02-06 15:44:10
|
On Feb 2, 2008, at 3:21 PM, Arlindo da Silva wrote:
>
> 4) Of the 70 tests, gradsv2 is failing 16. They are related to 2
> issues:
>
> a) The "sdfopen model.hdf" is not quite working: it reports the
> file as having 1 timestep instead of 5. I verified that the RH
> binary Jennifer posted has exactly the same problem, so it does not
> appear to be my build.
Arlindo,
I have put some print statments in gasdf.c to see why this is happening.
The HDF routine SDdiminfo returns the following information on
model.hdf:
dim_id,name,dimsize,dtype,ndatts = 327680 time 0 6 1
dim_id,name,dimsize,dtype,ndatts = 327681 longitude 72 6 2
dim_id,name,dimsize,dtype,ndatts = 327682 latitude 46 6 2
dim_id,name,dimsize,dtype,ndatts = 327683 levels 7 6 5
Looks like the file has a time dimension of size 0. I remember now
that when I discovered this during development, I put in the
following line:
if (dimsize==0) dimsize=1; /* would you have a coordinate axis with
size==0 */
The interesting thing is that when I run ncdump (from the HDF4.2r2
library build) it gives me this:
dimensions:
time = UNLIMITED ; // (5 currently)
longitude = 72 ;
latitude = 46 ;
levels = 7 ;
--Jennifer
|
|
From: Arlindo da S. <da...@al...> - 2008-02-06 18:09:46
|
On Feb 6, 2008 10:44 AM, Jennifer Adams <jm...@co...> wrote: > > Arlindo, > I have put some print statments in gasdf.c to see why this is happening. > The HDF routine SDdiminfo returns the following information on > model.hdf: > > dim_id,name,dimsize,dtype,ndatts = 327680 time 0 6 1 > dim_id,name,dimsize,dtype,ndatts = 327681 longitude 72 6 2 > dim_id,name,dimsize,dtype,ndatts = 327682 latitude 46 6 2 > dim_id,name,dimsize,dtype,ndatts = 327683 levels 7 6 5 > > Looks like the file has a time dimension of size 0. I remember now > that when I discovered this during development, I put in the > following line: > if (dimsize==0) dimsize=1; /* would you have a coordinate axis with > size==0 */ > > The interesting thing is that when I run ncdump (from the HDF4.2r2 > library build) it gives me this: > > dimensions: > time = UNLIMITED ; // (5 currently) > longitude = 72 ; > latitude = 46 ; > levels = 7 ; > It is possible that this is a bug in the SD interface. Ncdump uses the "NC" interface which is sometimes a more basic, low level interface than SD. (I say "sometimes" because the hdf-4 is not so cleanly layered). I suspect that your ctl/"DTYPE hdf" interface will have similar problems, correct? The file "model.hdf" was written with the "NC" interface through LATS. Leading to this version of hdf-4, I reported a couple of bugs to THG in the hrepack utility which did not work properly with HDF files written by the "NC" interface (all GMAO files are written that way, although LATS is not involved). It is very possible that this is another one of those bugs. We have 2 options, report the bug to the THG (we will need a small example demonstrating the problem), or try to dig in the hdf-4 sources and contribute a patch to them. They have usually being very responsive to my bug reports. What do you prefer? Arlindo -- Arlindo da Silva da...@al... |
|
From: Jennifer A. <jm...@co...> - 2008-02-06 19:34:54
|
On Feb 6, 2008, at 1:09 PM, Arlindo da Silva wrote: > On Feb 6, 2008 10:44 AM, Jennifer Adams <jm...@co...> wrote: > > Arlindo, > I have put some print statments in gasdf.c to see why this is > happening. > The HDF routine SDdiminfo returns the following information on > model.hdf: > > dim_id,name,dimsize,dtype,ndatts = 327680 time 0 6 1 > dim_id,name,dimsize,dtype,ndatts = 327681 longitude 72 6 2 > dim_id,name,dimsize,dtype,ndatts = 327682 latitude 46 6 2 > dim_id,name,dimsize,dtype,ndatts = 327683 levels 7 6 5 > > Looks like the file has a time dimension of size 0. I remember now > that when I discovered this during development, I put in the > following line: > if (dimsize==0) dimsize=1; /* would you have a coordinate > axis with > size==0 */ > > The interesting thing is that when I run ncdump (from the HDF4.2r2 > library build) it gives me this: > > dimensions: > time = UNLIMITED ; // (5 currently) > longitude = 72 ; > latitude = 46 ; > levels = 7 ; > > It is possible that this is a bug in the SD interface. Ncdump uses > the "NC" interface which is sometimes a more basic, low level > interface than SD. (I say "sometimes" because the hdf-4 is not so > cleanly layered). I suspect that your ctl/"DTYPE hdf" interface > will have similar problems, correct? The 'dtype hdfsds' works fine because the required TDEF metadata is provided in the descriptor file -- is doesn't need to use SDdiminfo to discover the size of the time axis. > The file "model.hdf" was written with the "NC" interface through > LATS. Leading to this version of hdf-4, I reported a couple of bugs > to THG in the hrepack utility which did not work properly with HDF > files written by the "NC" interface (all GMAO files are written > that way, although LATS is not involved). It is very possible that > this is another one of those bugs. > > We have 2 options, report the bug to the THG (we will need a small > example demonstrating the problem), or try to dig in the hdf-4 > sources and contribute a patch to them. They have usually being > very responsive to my bug reports. What do you prefer? As I see it, there is a bug in the lats4d pathway that creates hdf files, not in the v2 code that reads them. Jennifer |
|
From: Arlindo da S. <da...@al...> - 2008-02-06 20:36:08
|
On Feb 6, 2008 2:27 PM, Jennifer Adams <jm...@co...> wrote:
>
> On Feb 6, 2008, at 1:09 PM, Arlindo da Silva wrote:
>
> On Feb 6, 2008 10:44 AM, Jennifer Adams <jm...@co...> wrote:
>
> >
> > Arlindo,
> > I have put some print statments in gasdf.c to see why this is happening.
> > The HDF routine SDdiminfo returns the following information on
> > model.hdf:
> >
> > dim_id,name,dimsize,dtype,ndatts = 327680 time 0 6 1
> > dim_id,name,dimsize,dtype,ndatts = 327681 longitude 72 6 2
> > dim_id,name,dimsize,dtype,ndatts = 327682 latitude 46 6 2
> > dim_id,name,dimsize,dtype,ndatts = 327683 levels 7 6 5
> >
> > Looks like the file has a time dimension of size 0. I remember now
> > that when I discovered this during development, I put in the
> > following line:
> > if (dimsize==0) dimsize=1; /* would you have a coordinate axis
> > with
> > size==0 */
> >
> > The interesting thing is that when I run ncdump (from the HDF4.2r2
> > library build) it gives me this:
> >
> > dimensions:
> > time = UNLIMITED ; // (5 currently)
> > longitude = 72 ;
> > latitude = 46 ;
> > levels = 7 ;
> >
>
> It is possible that this is a bug in the SD interface. Ncdump uses the
> "NC" interface which is sometimes a more basic, low level interface than
> SD. (I say "sometimes" because the hdf-4 is not so cleanly layered). I
> suspect that your ctl/"DTYPE hdf" interface will have similar problems,
> correct?
>
> The 'dtype hdfsds' works fine because the required TDEF metadata is
> provided in the descriptor file -- is doesn't need to use SDdiminfo to
> discover the size of the time axis.
>
However, can it access variables for t>1? It is possible that the library
has checks for not allowing access beyond the dimension size.
>
> The file "model.hdf" was written with the "NC" interface through LATS. Leading
> to this version of hdf-4, I reported a couple of bugs to THG in the hrepack
> utility which did not work properly with HDF files written by the "NC"
> interface (all GMAO files are written that way, although LATS is not
> involved). It is very possible that this is another one of those bugs.
>
>
> We have 2 options, report the bug to the THG (we will need a small example
> demonstrating the problem), or try to dig in the hdf-4 sources and
> contribute a patch to them. They have usually being very responsive to my
> bug reports. What do you prefer?
>
>
> As I see it, there is a bug in the lats4d pathway that creates hdf files,
> not in the v2 code that reads them.
>
It is always possible, but I find it very unlikely. I am in the middle of a
day long meeting. I'll confirm later in day whether the same problem occurs
with the official GMAO files which do not involves LATS at all. I expect it
does.
Arlindo.
--
Arlindo da Silva
da...@al...
|
|
From: Jennifer A. <jm...@co...> - 2008-02-06 20:53:42
|
On Feb 6, 2008, at 3:36 PM, Arlindo da Silva wrote: > > > On Feb 6, 2008 2:27 PM, Jennifer Adams <jm...@co...> wrote: > > On Feb 6, 2008, at 1:09 PM, Arlindo da Silva wrote: > >> On Feb 6, 2008 10:44 AM, Jennifer Adams <jm...@co...> wrote: >> >> Arlindo, >> I have put some print statments in gasdf.c to see why this is >> happening. >> The HDF routine SDdiminfo returns the following information on >> model.hdf: >> >> dim_id,name,dimsize,dtype,ndatts = 327680 time 0 6 1 >> dim_id,name,dimsize,dtype,ndatts = 327681 longitude 72 6 2 >> dim_id,name,dimsize,dtype,ndatts = 327682 latitude 46 6 2 >> dim_id,name,dimsize,dtype,ndatts = 327683 levels 7 6 5 >> >> Looks like the file has a time dimension of size 0. I remember now >> that when I discovered this during development, I put in the >> following line: >> if (dimsize==0) dimsize=1; /* would you have a coordinate >> axis with >> size==0 */ >> >> The interesting thing is that when I run ncdump (from the HDF4.2r2 >> library build) it gives me this: >> >> dimensions: >> time = UNLIMITED ; // (5 currently) >> longitude = 72 ; >> latitude = 46 ; >> levels = 7 ; >> >> It is possible that this is a bug in the SD interface. Ncdump uses >> the "NC" interface which is sometimes a more basic, low level >> interface than SD. (I say "sometimes" because the hdf-4 is not so >> cleanly layered). I suspect that your ctl/"DTYPE hdf" interface >> will have similar problems, correct? > The 'dtype hdfsds' works fine because the required TDEF metadata is > provided in the descriptor file -- is doesn't need to use SDdiminfo > to discover the size of the time axis. > > However, can it access variables for t>1? It is possible that the > library has checks for not allowing access beyond the dimension size. Here's the descriptor: dset ../../opengrads/pytests/data/model.hdf dtype hdfsds title test undef -8888 missing_value xdef 72 linear 0 5 ydef 46 linear -90 4 zdef 7 levels 1000 850 700 500 300 200 100 tdef 5 linear 00Z01JAN1987 1dy vars 1 ua 7 t,z,y,x Eastward wind [m/s] endvars All 5 time steps are displayed. > > > >> The file "model.hdf" was written with the "NC" interface through >> LATS. Leading to this version of hdf-4, I reported a couple of >> bugs to THG in the hrepack utility which did not work properly >> with HDF files written by the "NC" interface (all GMAO files are >> written that way, although LATS is not involved). It is very >> possible that this is another one of those bugs. >> >> We have 2 options, report the bug to the THG (we will need a small >> example demonstrating the problem), or try to dig in the hdf-4 >> sources and contribute a patch to them. They have usually being >> very responsive to my bug reports. What do you prefer? > > As I see it, there is a bug in the lats4d pathway that creates hdf > files, not in the v2 code that reads them. > > It is always possible, but I find it very unlikely. I am in the > middle of a day long meeting. I'll confirm later in day whether the > same problem occurs with the official GMAO files which do not > involves LATS at all. I expect it does. By "lats4d pathway" I meant include the use of the HDF library to write out the files. It may not be in the LATS code proper, but it's somewhere along the line between the lats4d command at the GrADS command prompt and the output file model.hdf. Jennifer > > > Arlindo. > > -- > Arlindo da Silva > da...@al... -- Jennifer M. Adams IGES/COLA 4041 Powder Mill Road, Suite 302 Calverton, MD 20705 jm...@co... |
|
Re: [Opengrads-devel] [Junk released by Allow List] Building GrADS
v2 with new supplibs: first crack
From: Jennifer A. <jm...@co...> - 2008-02-06 21:06:00
|
On Feb 2, 2008, at 3:21 PM, Arlindo da Silva wrote:
> b) When using "set gxout fvwrite" to write binary files the missing
> value is not being written correctly, in either little-endian or
> big-endian mode. Is this a bug or a feature (I haven't verified
> whether v1.9 does the same)?
I found a bug in the HDF pathway in gasdf.c :
Near the end of the gadsdf() subroutine, the varids in the gavar
structure for each variable are reset to -999 so that the undef
values and scale factor/add offset values will be retrieved in the I/
O layer. I did it for ncvids, but not sdvids. Add the 2nd line below
and see if that isn't a workable short term fix.
newpvar->ncvid = -999; /* reset the varid so undefs will be
read in gaio.c */
newpvar->sdvid = -999; /* reset the varid so undefs will be
read in gaio.c */
There have been other patches to gasdf.c in the past week, as well as
other unrelated bug fixes, so that one change may not fix everything
for you. I will be putting out 2.0.a1 pretty soon.
Jennifer
|
|
From: Jennifer A. <jm...@co...> - 2008-02-07 12:47:23
|
Hoop pointed out that if SDdiminfo returns 0, then it's an unlimited dimension and requires special treatment. I will make some changes and that will likely solve the problem. Jennifer |
|
From: Jennifer A. <jm...@co...> - 2008-02-07 13:08:35
|
GrADS has got it right now: dim_id,name,dimsize,dtype,ndatts = 327680 time 5 6 1 dim_id,name,dimsize,dtype,ndatts = 327681 longitude 72 6 2 dim_id,name,dimsize,dtype,ndatts = 327682 latitude 46 6 2 dim_id,name,dimsize,dtype,ndatts = 327683 levels 7 6 5 Thanks, Hoop! Jennifer |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X