Hi Charlie,
have added nco_netcdf.h and nco_netcdf.c ( the wrapper
functions) to the repository.
Have also changed and commited nc.h and ncks.c
Could you try compiling on SGI, alphas etc.
( i hope i used the right CVS commands ! )
And see if ncks works as normal
regards Henry
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
OK, I think I misunderstood your previous message about where the
size_t pointers were causing warnings. As I now understand it,
the netCDF3 library uses type size_t variables to hold array sizes,
whereas NCO uses type long. Array values, however, are never
stored in type size_t variables. Before we decide how best to get
rid of the warning messages I just want to make sure that this is
in fact what the problem is. There is something on p. 43 of the
netcdf manual rationalizing the use of size_t. If this is the case
then I think I was mistaken about potential portability problems
to 64 bit platforms.
One problem is that size_t itself is not exactly representable as
a netCDF type because it is usually unsigned int. I am now
hesitant to change NCO to use size_t instead of long for
all dimension sizes and strides. I do not care about portability
to 16-bit platforms like msdos. I am beginning to think that
casting the longs to size_t in the wrappers is a simpler,
cleaner solution than changing the existing NCO. What
do you think?
Charlie
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I agree with you about the re-casting of long * to
size_t * in the wrappers. It saves having to change all
the nco code. Shall give it a go
regards Henry
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Charlie,
Have done some more mods to the wrapper
functions ( not commited them yet) . I'm off on
holiday tomorrow for a week in the big apple (
my first time in the states ). Shall be back
on sunday 26th nov . see you soon
Henry
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, I had a nice break. Winter seems to have
arrived here in So. Cal. I'd forgotten what
a nice storm can feel like!
>Just a brief question - With the NC_LONG aliased
>to NC_INT for netCDF3 how are we supposed to >detect type longs in a NetCDF file
Not sure I understand the question but I think
you may not understand the storage format, so let
me try to explain. netCDF files do not know anything about the type of machine they will be
read on. So NC_INT = NC_LONG variables are always
32 bit integers as stored in the file. netCDF3
automatically converts whatever is in the file
to the native type you specify with the _get()
command. So if we want to use native type
long int to hold NC_INT = NC_LONG variables,
netcdf3 automatically does any necessary conversion (not true with netCDF2). Thus it
is up to us to decide how we wish to store
NC_INT = NC_LONG types. For now I think we
should continue to use long int's to hold
this type. Thus the netcdf3 NCO code should
everywhere begin to use sizeof(long) instead
of nctypelen(long) whenever allocating buffers
to hold NC_INT = NC_LONG. If you want, you
can typedef nco_long long and use
sizeof(nco_long) etc. in the NCO code so we
can then fiddle with the nco_long type more
easily in the future. Does this make things
more clear or did I miss the point of your question?
Thanks,
Charlie
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Charlie,
Shall re-phrase my earlier question.
Given the enum nctype in netcdf.h has no type NC_LONG
How can we detect when we come across a varible of type
long -- for example when using nc_inq_var ?
P.S . maybe I'm being really thick and missing something !
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
> Given the enum nctype in netcdf.h has no type NC_LONG
I assume you mean the enum nc_type in the netcdf3
header file netcdf.h
> How can we detect when we come across a varible of type long?
We only know the on-disk type of variable will
say that it is an NC_INT. We will store it in
RAM as a native type long. The netcdf library
will do any necessary conversion for us within
the nc_get_var_long() routine.
> for example when using nc_inq_var ?
Whenever nc_inq_var() returns NC_INT then we
store it in a variable of native type long.
native type long = native type int on 32 bit
machines, but not necessarily on 64 bit machines.
But this should not matter. We will always
store type NC_INT in native type long.
Or, as I suggested, store NC_INT in native
type nco_int where we have previously
typedef nco_int int
This would make it easy to change internal
storage type from int to long just by changing
a single typedef.
When netCDF begins to support an type NC_LONG
(sometime in the future), this capability
might be useful.
Does this help?
Charlie
P.S. I just got "permission" to be using the
64 bit itanium machines on the compile farm.
But the login is not working. I will let you
know when/if I get this sorted out.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Charlie,
Have changed the wrapper functions to include casting from
from type long to size_t. This avoids compiler warnings.
Would appreciate it if you could try out the new ncks containing the netcdf3 calls.
Looks like there is a problem with test ncea 4 (packing and
unpacking)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hope you enjoyed your first trip to the states.
Pretty crazy over here with the election.
A sizeable minority is asking that HRH Queen Elizabeth
be instated as head of our country.
I tested the netCDF3 branch on Linux/Intel and it looks good.
Yes, the packing test should still fail until further notice.
I put the test in to fail to remind me that packing is not finished.
nco_netcdf.c causes quite a few warnings on the SGI IRIX
platform. SGI has great compilers so I take these seriously,
and have appended the warnings so you can study them
and see if there are any problems indicated. Apparently
SGI emits warnings when implicitly converting
unsigned char * to const char *, etc. We need to get rid of those
warnings somehow. If I'm reading the warnings
correctly then the netCDF3 interface prototypes in netcdf.h
are inconsistent with the declared types of NC_CHAR and
NC_BYTE.
It would be nice if you had an account on
an SGI so you could work on this interactively yourself.
I think sourceforge has a "compile farm" but it does not
include an SGI machine, bummer.
Anyway, look at the SGI warning appended below.
See if they reveal any real problems.
Let me know one way or the other.
If you deem it safe then I can change the casting
on a case by case basis until all the warnings
disappear on the SGI. Before proceeding, however,
I need you to tell me that will not break anything.
Thanks,
Charlie
cc -DSGIMP64 -I/home/ess/gudrun/didi/netcdf-3.4/include -64 -mips4 -O2 -mp -mpio
-c ../src/nco/nco_netcdf.c -o /home/ess/zender/zender/obj/SGIMP64/nco_netcdf.o
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 371
Argument of type "unsigned char *" is incompatible with parameter of type
"const char *".
rcd = nc_put_att_text( nc_id,var_id,att_nm,(size_t)att_len,(unsigned cha
r *)vp);
^
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 402
Argument of type "unsigned char *" is incompatible with parameter of type
"char *".
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 569
Argument of type "unsigned char *" is incompatible with parameter of type
"const char *".
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 641
Argument of type "unsigned char *" is incompatible with parameter of type
"const char *".
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 644
Argument of type "signed char *" is incompatible with parameter of type
"const unsigned char *".
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 705
Argument of type "unsigned char *" is incompatible with parameter of type
"const char *".
Hi Charlie,
Wow, the SG compiler is really sensitive. Its OK to change
the castings of NC_CHAR. I assume that char is by default
unsigned.
I think we have been a bit over-optimistic in the simplicity
of the wrapper functions. We shall have to watch very carefully when converting netCDF 2 code which uses
the -1 return value. ( eg ncvarid ) . The netCDF3 code uses
the error codes if for example a requested variable or attribute
is not present in the file. I have already made ammends to
nco_inq_varid but more are bound to come up ! Its a case of
deciding to change the old code or adding a shadow -1 return value within the wrapper function.
Had a good holiday -- New York was awesome
Those deli's sure know how to fill a sandwich
regards Henry
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
A subtle problem like the one you point out deserves a thoughtful response:
> I think we have been a bit over-optimistic in the simplicity
> of the wrapper functions. We shall have to watch very carefully when
> converting netCDF 2 code which uses
> the -1 return value. ( eg ncvarid ) . The netCDF3 code uses
> the error codes if for example a requested variable or attribute
> is not present in the file. I have already made ammends to
> nco_inq_varid but more are bound to come up ! Its a case of
> deciding to change the old code or adding a shadow -1 return value within the
> wrapper function.
NCO has some routines where receiving an error code from, e.g.,
ncvarid() is expected, and some where receiving an error code causes
an exit() (perhaps preceded by specific diagnostics of what went
wrong). So we do not want the NCO netcdf3 code to automatically exit
when *_inq_* calls receive an error value. However, adding exceptional
code to nco_inq_varid(), i.e., mimicing the netcdf2() interface with
netcdf3() calls, is not, I think, the right way to handle this
complexity because it violates the consistency between nco_* wrapper
functions and the underlying netcdf3() calls. Let us keep to the
paradigm that the nco_foo wrapper functions simply perform the extra
step of printing an error message before exiting whenever rcd !=
NC_NOERR after calling nc_foo.
To handle the situation where rcd != NO_NOERR I propose one of two
solutions. #1. We add new wrappers, e.g.,
int nco_inq_varid_flg(int nc_id, char *var_nm, int *rcd)
where rcd is set by the wrapped netcdf3 routine to the return code
value so that the calling routine may handle the error. Thus, regions
where old NCO allowed -1 return values (by setting ncopts=0 before
call and ncopts=NC_VERBOSE | NC_FATAL after call would use these
special nco_inq_varid_flg() and nco_inq_dimid_flg() routines as
follows
var_id=nco_inq_varid_rcd(nc_id,var_nm,&rcd)
if(rcd == NC_ENOTVAR){...}else{possibly print error and die}
#2. Simply put the netcdf3() call directly in the NCO code and not to
use a wrapper for netcdf3() calls that are allowed to return an error
message.
The advantage of #1 is that it helps encapsulate all the netcdf3()
calls in one file, nco_netcdf.c. These wrappers could someday be
expanded to handle, say, hdf5 or some new data I/O and the NCO code
itself would not need to change. Granted, that day is a long day off
but there is something cleaner about working to the point where ncks.c
and nc_utl.c can be compiled without the netcdf library because all
netcdf-specific calls are isolated in nco_netcdf.c. The advantage of
#2 is that it's simpler and no dirtier than what's already in NCO. And
since you're coding this, you get to decide which solution to
implement! Of course you can also suggest another
approach. Just try to keep any existing error diagnostics intact, e.g., if
NCO currently takes the effort to print the file name and variable
name where error occurred, so should the new NCO netcdf3() branch.
Thanks,
Charlie
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Charlie,
What about treating netCDF3 errors like netCDF2 errors.
i.e. we decide what errors are fatal , and if the error number
is required they set a flag prior to the call, so that non-fatal errors filter through ?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Can you send an example of what this would look
like for, say, the nc_inq_varid() -type of
situation that has raised this as a question?
Thanks,
Charlie
P.S. I switched to mozilla for my browser and
it still does not have SSL login capability
so many of my responses will have author=nobody
until mozilla fixes this in the nightly builds.
I apologize for any confusion this may cause
but my switch to mozilla was dictated by other reasons.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
OK I just checked in the changes to the 1.3 branch
that are required for clean compilation on SGI.
Instead of casting to type char * I changed the
netcdf3 functions to call the *_uchar functions
instead of the the *_text functions. I thought
this was clearer.
Unfortunately, I just ran nco_tst.sh on the
1.3 branch and there are significant errors
in the handling of NC_LONG variables on the
64-bit SGI machine. This is not terribly
surprising since NC_LONG == NC_INT = 32 bit
integers on all machines but internal representation of type long on SGI is 64 bits.
It took some time to get this right on the
main NCO branch a few years back when 64 bit
machines came along. You need to audit the
1.3 branch code to see where the mismatch
is occuring. Please read netcdf3 manual p. 130.
Perhaps we should change NC_LONG to NC_INT in
the 1.3 branch code?
Presumably this problem will be triggered on
any 64 bit machine, not just SGI. Do you have
access to such a machine? Do you feel like
working on an itanium machine? I know sourceforge
has itanium's in its compile farm and it might
be a fun project to try to work out these problems
on an itanium. If you are unable to debug the
NC_LONG problem let me know and I will try to
give it a crack over the holidays. But I think
this would be a good learning experience for you,
and it would be cool to have an itanium port of
NCO working. Of course this presume that netCDF
will work out of the box on itanium, too.
Charlie
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Charlie,
This conversion to netCDF3 is a MAJOR TASK -- I don't
mind while I'm not working I have the time. I don't mind
having a crack on the "itanium" machine.
You changed the _text calls to _uchar . Is this working ok
on 32 bit / LINUX machines ?
Regards Henry
P.S Maybe we should have a conflab on the phone about
all the changes going down -- My number is
UK --- 01727 846029 ( I think you are 8 hours behind)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think you're done with 90% of the coding but
getting the last 10% is always the hardest.
Remember, it's possible the 64 bit issue is
unrelated to the netcdf3 changes, and that
they have just exposed a weakness in another
part of the code.
> You changed the _text calls to _uchar . Is this > working ok on 32 bit / LINUX machines ?
I think so. After this experience it's become
clear that some user programs should use the
_text() calls but that NCO, which performs
low level type conversions, should use the
_uchar() calls.
I assume you are using cvs update -kk
to track the changes I make. On Linux the 1.3
branch currently performs as expected by nco_tst.sh except for ncwa test 33 where I get
ncwa 33: ttl would overflow without dbl_prc patch, wraps anyway: -31536S =?= -32768
In other words the expected wrapping that occurs
with overflow has changed. However, I have upgraded to glibc 2.2 and it's possible that the
behavior of integer wrapping (which is presumably
ill-defined by nature) has changed and I'm not
worried about this too much, just curious whether
you can duplicate it.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Charlie,
Have just updated my netCDF 3 branch and everything is
broken. I can't print out any variables with ncks. Maybe
its my setup. Also getting an error message " Attempt to convert between text and numbers" when trying to print out
text ( ncks -H -v fl_nm_arr in.nc ). This was why I orginally
decided to use _text instead of _uchar. Whats working on your setup ?
Henry
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
On my linux box nco_tst.sh seems to works fine BUT
you're right, when I use simple ncks commands
everything goes to hell. On the IRIX machine,
everything appears to work with nco_tst.sh except
for NC_LONG stuff like we discussed.
I think you should go back to using the _text
functions for now. In other words back out my
changes. CVS allows you to retrieve previous
versions of files, do that. The first goal is
to get stuff working on your machine, Linux.
Hopefully backing out my changes to _uchar will
help the Linux performance, although I'm not
sure why the "converting between text and numbers"
error has begun to pop up. It's disconcerting.
I actually had the netcdf 1.3 branch hang my
Linux machine yesterday after making one small
change. That's never happened before.
One priority now is to add a test to nco_tst.sh
that will fail right now so that if the code
gets into this state again we have an easy way
to know. All I do when I make a change is make
sure nco_tst.sh works as expected. So I had no
easy way to know that things had broken on Linux.
The NC_LONG issue may be playing a role with
the Linux problems. The problem with NC_LONGs
on IRIX is clear to me now. If you look at
the memory allocation, NCO always allocates
using nctypelen(NC_LONG)=4 whereas it reads
into native type longs where sizeof(long) = 8
on IRIX. So changine get_var_long to get_var_int
might be a quick fix for IRIX, but it leaves
other problems remaining. netcdf3 always reads
into native types so we essentially are forced
to pick a native type to handle NC_LONG.
With netcdf2 we did not have to do that because
we had the nclong alias to use. It seems to me
the only logical choice to handle NC_LONG is
to use native type int (= 4 bytes on all machines
I know of) and, to lessen confusion, change
all occurences of NC_LONG to NC_INT. Does this
make sense? Again, there is no reason I can
think of that NC_LONG issues are causing current
problems with Linux, but you never know.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Charlie,
Have commited the following
nco_netcdf.c -- converted _uchar back to_text
nco_netcdf.h -- added nco_inq_varid_flg and nco_inq_dimid_flg
ncks -- added the above calls and corrected some errors
nco_tst.sh -- more tests for ncks
looks like ncks is working ok now -- check it out !
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
This sounds great. I'll update my copy and test while on the road.
Once things appear to be working on Linux, please make a
CVS tag, e.g., nco-1_3_3 or whatever so we have a clean
branch. After that we'll have to worry about 64 bit machines
and getting rid of nctypelen(). It occurred to me that since the
netcdf3 paradigm is to let the user put any netcdf type, e.g.,
NC_LONG, into any machine internal type, e.g., int, long,
that we maybe we should make our choice of representation
of NC_LONG generic. For example, we define a new type
typedef NCO_INT int
typedef NCO_FLOAT float
etc.
which tells NCO that it should internally represent the
netcdf3 type NC_LONG as a long, and type NC_FLOAT
as a float. Then we can do a machine dependent
definition so that
SGI 64 bits: typedef NCO_INT int
Intel 32 bits: typedef NCO_INT int
and on both typedef NCO_LONG NCO_INT
Anyway, we must drop all references to nclong
(a netCDF2 thingy), and we should, in my opinion,
start using NC_INT instead of NC_LONG just to avoid
confusion as the manual recommends. Thoughts on this?
More after Xmas,
Cheers,
Charlie
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Charlie,
Sorry I've not been contributing for a while
but I have some time now. I'm still a little unsure about how the longs variables are delt with.
According to the netCDF manual " long is a depreciated synonom of int !!" -- Still tell us what do with nctypelen call and I should be able
to continue with the upgrade to netCDF3
regards Henry
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Last night I finally put out 1.2.2. Thanks for your help on that, by the way.
For 1.3, for now, nctypelen(NC_FOO) should everywhere be replaced by nco_typ_lng(NC_FOO)
where that function is defined below:
nco_typ_lng(nc_type nco_typ)
{
/* Purpose: return native size of specified netCDF type */
switch(type){
case NC_FLOAT:
return sizeof(float);
case NC_DOUBLE:
return sizeof(double);
case NC_INT:
return sizeof(int);
case NC_LONG:
return sizeof(long);
case NC_SHORT:
return sizeof(short);
case NC_CHAR:
return sizeof(unsigned char);
case NC_BYTE:
return sizeof(signed char);
default: dfl_case_nctype_err(); break;
} /* end switch */
} /* end nco_typ_lng() */
and dfl_case_nctype_err() is something I added
to the 1.2.x branch recently.
void
dfl_case_nctype_err(void)
{
/* Purpose: Convenience routine for printing error and exiting when
switch(nctype) statement receives an illegal default case
NCO emits warnings when compiled by GCC with -DNETCDF2_ONLY since,
apparently, there are a whole bunch of things besides numeric
types in the old nctype enum and gcc warns about enums that are
not exhaustively considered in switch() statements.
All these default statements can be removed with netCDF3 interface
so perhaps these should be surrounded with #ifdef NETCDF2_ONLY
constructs, but they actually do make sense for netCDF3 as well
so I have implemented a uniform error function, dfl_case_nctype_err(),
to be called by all routines which emit errors only when compiled with
NETCDF2_ONLY.
This makes the behavior easy to modify or remove in the future.
Placing this in its own routine also has the virtue of saving many lines
of code since this function is used in many many switch() statements. */
(void)fprintf(stdout,"%s: ERROR switch(nctype) statement fell through to default case, which is illegal.\nNot handling the default case causes gcc to emit warnings when compiling NCO with the NETCDF2_ONLY token (because nctype defintion is braindead in netCDF2). Exiting...",prg_nm_get());
exit(EXIT_FAILURE);
} /* end dfl_case_nctype_err() */
I'd like to begin to help with the 1.3
branch soon, as the 1.2.x stuff is basically
done and I want to merge the last changes into
1.3. So when you get things in 1.3 to a point
where they are either working or you are at a
good stopping point where I can muck around
for a bit, let me know.
Thanks,
Charlie
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Charlie,
have added nco_netcdf.h and nco_netcdf.c ( the wrapper
functions) to the repository.
Have also changed and commited nc.h and ncks.c
Could you try compiling on SGI, alphas etc.
( i hope i used the right CVS commands ! )
And see if ncks works as normal
regards Henry
Henry,
OK, I think I misunderstood your previous message about where the
size_t pointers were causing warnings. As I now understand it,
the netCDF3 library uses type size_t variables to hold array sizes,
whereas NCO uses type long. Array values, however, are never
stored in type size_t variables. Before we decide how best to get
rid of the warning messages I just want to make sure that this is
in fact what the problem is. There is something on p. 43 of the
netcdf manual rationalizing the use of size_t. If this is the case
then I think I was mistaken about potential portability problems
to 64 bit platforms.
One problem is that size_t itself is not exactly representable as
a netCDF type because it is usually unsigned int. I am now
hesitant to change NCO to use size_t instead of long for
all dimension sizes and strides. I do not care about portability
to 16-bit platforms like msdos. I am beginning to think that
casting the longs to size_t in the wrappers is a simpler,
cleaner solution than changing the existing NCO. What
do you think?
Charlie
Hi Charlie,
I agree with you about the re-casting of long * to
size_t * in the wrappers. It saves having to change all
the nco code. Shall give it a go
regards Henry
Hi Charlie,
Have done some more mods to the wrapper
functions ( not commited them yet) . I'm off on
holiday tomorrow for a week in the big apple (
my first time in the states ). Shall be back
on sunday 26th nov . see you soon
Henry
Hi Charlie,
I agree with you about the re-casting of long *
Hi Charlie, Hope you had a good Xmas break.
Just a brief question - With the NC_LONG aliased
to NC_INT for netCDF3 how are we supposed to detect type longs in a NetCDF file ?
Yes, I had a nice break. Winter seems to have
arrived here in So. Cal. I'd forgotten what
a nice storm can feel like!
>Just a brief question - With the NC_LONG aliased
>to NC_INT for netCDF3 how are we supposed to >detect type longs in a NetCDF file
Not sure I understand the question but I think
you may not understand the storage format, so let
me try to explain. netCDF files do not know anything about the type of machine they will be
read on. So NC_INT = NC_LONG variables are always
32 bit integers as stored in the file. netCDF3
automatically converts whatever is in the file
to the native type you specify with the _get()
command. So if we want to use native type
long int to hold NC_INT = NC_LONG variables,
netcdf3 automatically does any necessary conversion (not true with netCDF2). Thus it
is up to us to decide how we wish to store
NC_INT = NC_LONG types. For now I think we
should continue to use long int's to hold
this type. Thus the netcdf3 NCO code should
everywhere begin to use sizeof(long) instead
of nctypelen(long) whenever allocating buffers
to hold NC_INT = NC_LONG. If you want, you
can typedef nco_long long and use
sizeof(nco_long) etc. in the NCO code so we
can then fiddle with the nco_long type more
easily in the future. Does this make things
more clear or did I miss the point of your question?
Thanks,
Charlie
Hi Charlie,
Shall re-phrase my earlier question.
Given the enum nctype in netcdf.h has no type NC_LONG
How can we detect when we come across a varible of type
long -- for example when using nc_inq_var ?
P.S . maybe I'm being really thick and missing something !
> Given the enum nctype in netcdf.h has no type NC_LONG
I assume you mean the enum nc_type in the netcdf3
header file netcdf.h
> How can we detect when we come across a varible of type long?
We only know the on-disk type of variable will
say that it is an NC_INT. We will store it in
RAM as a native type long. The netcdf library
will do any necessary conversion for us within
the nc_get_var_long() routine.
> for example when using nc_inq_var ?
Whenever nc_inq_var() returns NC_INT then we
store it in a variable of native type long.
native type long = native type int on 32 bit
machines, but not necessarily on 64 bit machines.
But this should not matter. We will always
store type NC_INT in native type long.
Or, as I suggested, store NC_INT in native
type nco_int where we have previously
typedef nco_int int
This would make it easy to change internal
storage type from int to long just by changing
a single typedef.
When netCDF begins to support an type NC_LONG
(sometime in the future), this capability
might be useful.
Does this help?
Charlie
P.S. I just got "permission" to be using the
64 bit itanium machines on the compile farm.
But the login is not working. I will let you
know when/if I get this sorted out.
Hi Charlie,
Have changed the wrapper functions to include casting from
from type long to size_t. This avoids compiler warnings.
Would appreciate it if you could try out the new ncks containing the netcdf3 calls.
Looks like there is a problem with test ncea 4 (packing and
unpacking)
Hi Henry,
Hope you enjoyed your first trip to the states.
Pretty crazy over here with the election.
A sizeable minority is asking that HRH Queen Elizabeth
be instated as head of our country.
I tested the netCDF3 branch on Linux/Intel and it looks good.
Yes, the packing test should still fail until further notice.
I put the test in to fail to remind me that packing is not finished.
nco_netcdf.c causes quite a few warnings on the SGI IRIX
platform. SGI has great compilers so I take these seriously,
and have appended the warnings so you can study them
and see if there are any problems indicated. Apparently
SGI emits warnings when implicitly converting
unsigned char * to const char *, etc. We need to get rid of those
warnings somehow. If I'm reading the warnings
correctly then the netCDF3 interface prototypes in netcdf.h
are inconsistent with the declared types of NC_CHAR and
NC_BYTE.
It would be nice if you had an account on
an SGI so you could work on this interactively yourself.
I think sourceforge has a "compile farm" but it does not
include an SGI machine, bummer.
Anyway, look at the SGI warning appended below.
See if they reveal any real problems.
Let me know one way or the other.
If you deem it safe then I can change the casting
on a case by case basis until all the warnings
disappear on the SGI. Before proceeding, however,
I need you to tell me that will not break anything.
Thanks,
Charlie
cc -DSGIMP64 -I/home/ess/gudrun/didi/netcdf-3.4/include -64 -mips4 -O2 -mp -mpio
-c ../src/nco/nco_netcdf.c -o /home/ess/zender/zender/obj/SGIMP64/nco_netcdf.o
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 371
Argument of type "unsigned char *" is incompatible with parameter of type
"const char *".
rcd = nc_put_att_text( nc_id,var_id,att_nm,(size_t)att_len,(unsigned cha
r *)vp);
^
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 402
Argument of type "unsigned char *" is incompatible with parameter of type
"char *".
rcd = nc_get_att_text( nc_id,var_id,att_nm,(unsigned char *)vp);
^
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 536
Argument of type "unsigned char *" is incompatible with parameter of type
"char *".
rcd = nc_get_var1_text( nc_id,var_id,(size_t*)indx,(unsigned char *)vp);
^
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 569
Argument of type "unsigned char *" is incompatible with parameter of type
"const char *".
rcd = nc_put_var1_text( nc_id,var_id,(size_t*)indx,(unsigned char *) vp)
;
^
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 607
Argument of type "unsigned char *" is incompatible with parameter of type
"char *".
rcd = nc_get_vara_text( nc_id,var_id,(size_t *)srt, (size_t *)cnt,(unsig
ned char *) vp);
^
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 641
Argument of type "unsigned char *" is incompatible with parameter of type
"const char *".
rcd = nc_put_vara_text( nc_id,var_id,(size_t *)srt, (size_t *)cnt,(unsig
ned char *) vp);
^
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 644
Argument of type "signed char *" is incompatible with parameter of type
"const unsigned char *".
rcd = nc_put_vara_uchar( nc_id,var_id,(size_t *)srt, (size_t *)cnt,(signed
char *)vp);
^
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 673
Argument of type "unsigned char *" is incompatible with parameter of type
"char *".
rcd = nc_get_varm_text( nc_id,var_id,(size_t *)srt, (size_t *)cnt,(ptrdi
ff_t *)srd,(ptrdiff_t *)map,(unsigned char *) vp);
^
cc-1164 cc: WARNING File = ../src/nco/nco_netcdf.c, Line = 705
Argument of type "unsigned char *" is incompatible with parameter of type
"const char *".
rcd = nc_put_varm_text( nc_id,var_id,(size_t *)srt, (size_t *)cnt,(ptrdi
ff_t *)srd,(ptrdiff_t *)map,(unsigned char *) vp);
Hi Charlie,
Wow, the SG compiler is really sensitive. Its OK to change
the castings of NC_CHAR. I assume that char is by default
unsigned.
I think we have been a bit over-optimistic in the simplicity
of the wrapper functions. We shall have to watch very carefully when converting netCDF 2 code which uses
the -1 return value. ( eg ncvarid ) . The netCDF3 code uses
the error codes if for example a requested variable or attribute
is not present in the file. I have already made ammends to
nco_inq_varid but more are bound to come up ! Its a case of
deciding to change the old code or adding a shadow -1 return value within the wrapper function.
Had a good holiday -- New York was awesome
Those deli's sure know how to fill a sandwich
regards Henry
Hi Henry,
A subtle problem like the one you point out deserves a thoughtful response:
> I think we have been a bit over-optimistic in the simplicity
> of the wrapper functions. We shall have to watch very carefully when
> converting netCDF 2 code which uses
> the -1 return value. ( eg ncvarid ) . The netCDF3 code uses
> the error codes if for example a requested variable or attribute
> is not present in the file. I have already made ammends to
> nco_inq_varid but more are bound to come up ! Its a case of
> deciding to change the old code or adding a shadow -1 return value within the
> wrapper function.
NCO has some routines where receiving an error code from, e.g.,
ncvarid() is expected, and some where receiving an error code causes
an exit() (perhaps preceded by specific diagnostics of what went
wrong). So we do not want the NCO netcdf3 code to automatically exit
when *_inq_* calls receive an error value. However, adding exceptional
code to nco_inq_varid(), i.e., mimicing the netcdf2() interface with
netcdf3() calls, is not, I think, the right way to handle this
complexity because it violates the consistency between nco_* wrapper
functions and the underlying netcdf3() calls. Let us keep to the
paradigm that the nco_foo wrapper functions simply perform the extra
step of printing an error message before exiting whenever rcd !=
NC_NOERR after calling nc_foo.
To handle the situation where rcd != NO_NOERR I propose one of two
solutions. #1. We add new wrappers, e.g.,
int nco_inq_varid_flg(int nc_id, char *var_nm, int *rcd)
where rcd is set by the wrapped netcdf3 routine to the return code
value so that the calling routine may handle the error. Thus, regions
where old NCO allowed -1 return values (by setting ncopts=0 before
call and ncopts=NC_VERBOSE | NC_FATAL after call would use these
special nco_inq_varid_flg() and nco_inq_dimid_flg() routines as
follows
var_id=nco_inq_varid_rcd(nc_id,var_nm,&rcd)
if(rcd == NC_ENOTVAR){...}else{possibly print error and die}
#2. Simply put the netcdf3() call directly in the NCO code and not to
use a wrapper for netcdf3() calls that are allowed to return an error
message.
The advantage of #1 is that it helps encapsulate all the netcdf3()
calls in one file, nco_netcdf.c. These wrappers could someday be
expanded to handle, say, hdf5 or some new data I/O and the NCO code
itself would not need to change. Granted, that day is a long day off
but there is something cleaner about working to the point where ncks.c
and nc_utl.c can be compiled without the netcdf library because all
netcdf-specific calls are isolated in nco_netcdf.c. The advantage of
#2 is that it's simpler and no dirtier than what's already in NCO. And
since you're coding this, you get to decide which solution to
implement! Of course you can also suggest another
approach. Just try to keep any existing error diagnostics intact, e.g., if
NCO currently takes the effort to print the file name and variable
name where error occurred, so should the new NCO netcdf3() branch.
Thanks,
Charlie
Hi Charlie,
What about treating netCDF3 errors like netCDF2 errors.
i.e. we decide what errors are fatal , and if the error number
is required they set a flag prior to the call, so that non-fatal errors filter through ?
Henry,
Can you send an example of what this would look
like for, say, the nc_inq_varid() -type of
situation that has raised this as a question?
Thanks,
Charlie
P.S. I switched to mozilla for my browser and
it still does not have SSL login capability
so many of my responses will have author=nobody
until mozilla fixes this in the nightly builds.
I apologize for any confusion this may cause
but my switch to mozilla was dictated by other reasons.
OK I just checked in the changes to the 1.3 branch
that are required for clean compilation on SGI.
Instead of casting to type char * I changed the
netcdf3 functions to call the *_uchar functions
instead of the the *_text functions. I thought
this was clearer.
Unfortunately, I just ran nco_tst.sh on the
1.3 branch and there are significant errors
in the handling of NC_LONG variables on the
64-bit SGI machine. This is not terribly
surprising since NC_LONG == NC_INT = 32 bit
integers on all machines but internal representation of type long on SGI is 64 bits.
It took some time to get this right on the
main NCO branch a few years back when 64 bit
machines came along. You need to audit the
1.3 branch code to see where the mismatch
is occuring. Please read netcdf3 manual p. 130.
Perhaps we should change NC_LONG to NC_INT in
the 1.3 branch code?
Presumably this problem will be triggered on
any 64 bit machine, not just SGI. Do you have
access to such a machine? Do you feel like
working on an itanium machine? I know sourceforge
has itanium's in its compile farm and it might
be a fun project to try to work out these problems
on an itanium. If you are unable to debug the
NC_LONG problem let me know and I will try to
give it a crack over the holidays. But I think
this would be a good learning experience for you,
and it would be cool to have an itanium port of
NCO working. Of course this presume that netCDF
will work out of the box on itanium, too.
Charlie
Hi Charlie,
This conversion to netCDF3 is a MAJOR TASK -- I don't
mind while I'm not working I have the time. I don't mind
having a crack on the "itanium" machine.
You changed the _text calls to _uchar . Is this working ok
on 32 bit / LINUX machines ?
Regards Henry
P.S Maybe we should have a conflab on the phone about
all the changes going down -- My number is
UK --- 01727 846029 ( I think you are 8 hours behind)
> This conversion to netCDF3 is a MAJOR TASK
I think you're done with 90% of the coding but
getting the last 10% is always the hardest.
Remember, it's possible the 64 bit issue is
unrelated to the netcdf3 changes, and that
they have just exposed a weakness in another
part of the code.
> You changed the _text calls to _uchar . Is this > working ok on 32 bit / LINUX machines ?
I think so. After this experience it's become
clear that some user programs should use the
_text() calls but that NCO, which performs
low level type conversions, should use the
_uchar() calls.
I assume you are using cvs update -kk
to track the changes I make. On Linux the 1.3
branch currently performs as expected by nco_tst.sh except for ncwa test 33 where I get
ncwa 33: ttl would overflow without dbl_prc patch, wraps anyway: -31536S =?= -32768
In other words the expected wrapping that occurs
with overflow has changed. However, I have upgraded to glibc 2.2 and it's possible that the
behavior of integer wrapping (which is presumably
ill-defined by nature) has changed and I'm not
worried about this too much, just curious whether
you can duplicate it.
Hi Charlie,
Have just updated my netCDF 3 branch and everything is
broken. I can't print out any variables with ncks. Maybe
its my setup. Also getting an error message " Attempt to convert between text and numbers" when trying to print out
text ( ncks -H -v fl_nm_arr in.nc ). This was why I orginally
decided to use _text instead of _uchar. Whats working on your setup ?
Henry
On my linux box nco_tst.sh seems to works fine BUT
you're right, when I use simple ncks commands
everything goes to hell. On the IRIX machine,
everything appears to work with nco_tst.sh except
for NC_LONG stuff like we discussed.
I think you should go back to using the _text
functions for now. In other words back out my
changes. CVS allows you to retrieve previous
versions of files, do that. The first goal is
to get stuff working on your machine, Linux.
Hopefully backing out my changes to _uchar will
help the Linux performance, although I'm not
sure why the "converting between text and numbers"
error has begun to pop up. It's disconcerting.
I actually had the netcdf 1.3 branch hang my
Linux machine yesterday after making one small
change. That's never happened before.
One priority now is to add a test to nco_tst.sh
that will fail right now so that if the code
gets into this state again we have an easy way
to know. All I do when I make a change is make
sure nco_tst.sh works as expected. So I had no
easy way to know that things had broken on Linux.
The NC_LONG issue may be playing a role with
the Linux problems. The problem with NC_LONGs
on IRIX is clear to me now. If you look at
the memory allocation, NCO always allocates
using nctypelen(NC_LONG)=4 whereas it reads
into native type longs where sizeof(long) = 8
on IRIX. So changine get_var_long to get_var_int
might be a quick fix for IRIX, but it leaves
other problems remaining. netcdf3 always reads
into native types so we essentially are forced
to pick a native type to handle NC_LONG.
With netcdf2 we did not have to do that because
we had the nclong alias to use. It seems to me
the only logical choice to handle NC_LONG is
to use native type int (= 4 bytes on all machines
I know of) and, to lessen confusion, change
all occurences of NC_LONG to NC_INT. Does this
make sense? Again, there is no reason I can
think of that NC_LONG issues are causing current
problems with Linux, but you never know.
Hi Charlie,
Shall add some tests to nco_tst.sh for ncks
The current tests clearly are not sufficient
Regards Henry
Hi Charlie,
Have commited the following
nco_netcdf.c -- converted _uchar back to_text
nco_netcdf.h -- added nco_inq_varid_flg and nco_inq_dimid_flg
ncks -- added the above calls and corrected some errors
nco_tst.sh -- more tests for ncks
looks like ncks is working ok now -- check it out !
Henry,
This sounds great. I'll update my copy and test while on the road.
Once things appear to be working on Linux, please make a
CVS tag, e.g., nco-1_3_3 or whatever so we have a clean
branch. After that we'll have to worry about 64 bit machines
and getting rid of nctypelen(). It occurred to me that since the
netcdf3 paradigm is to let the user put any netcdf type, e.g.,
NC_LONG, into any machine internal type, e.g., int, long,
that we maybe we should make our choice of representation
of NC_LONG generic. For example, we define a new type
typedef NCO_INT int
typedef NCO_FLOAT float
etc.
which tells NCO that it should internally represent the
netcdf3 type NC_LONG as a long, and type NC_FLOAT
as a float. Then we can do a machine dependent
definition so that
SGI 64 bits: typedef NCO_INT int
Intel 32 bits: typedef NCO_INT int
and on both typedef NCO_LONG NCO_INT
Anyway, we must drop all references to nclong
(a netCDF2 thingy), and we should, in my opinion,
start using NC_INT instead of NC_LONG just to avoid
confusion as the manual recommends. Thoughts on this?
More after Xmas,
Cheers,
Charlie
Hi Charlie,
Sorry I've not been contributing for a while
but I have some time now. I'm still a little unsure about how the longs variables are delt with.
According to the netCDF manual " long is a depreciated synonom of int !!" -- Still tell us what do with nctypelen call and I should be able
to continue with the upgrade to netCDF3
regards Henry
Hi Henry,
Last night I finally put out 1.2.2. Thanks for your help on that, by the way.
For 1.3, for now, nctypelen(NC_FOO) should everywhere be replaced by nco_typ_lng(NC_FOO)
where that function is defined below:
nco_typ_lng(nc_type nco_typ)
{
/* Purpose: return native size of specified netCDF type */
switch(type){
case NC_FLOAT:
return sizeof(float);
case NC_DOUBLE:
return sizeof(double);
case NC_INT:
return sizeof(int);
case NC_LONG:
return sizeof(long);
case NC_SHORT:
return sizeof(short);
case NC_CHAR:
return sizeof(unsigned char);
case NC_BYTE:
return sizeof(signed char);
default: dfl_case_nctype_err(); break;
} /* end switch */
} /* end nco_typ_lng() */
and dfl_case_nctype_err() is something I added
to the 1.2.x branch recently.
void
dfl_case_nctype_err(void)
{
/* Purpose: Convenience routine for printing error and exiting when
switch(nctype) statement receives an illegal default case
NCO emits warnings when compiled by GCC with -DNETCDF2_ONLY since,
apparently, there are a whole bunch of things besides numeric
types in the old nctype enum and gcc warns about enums that are
not exhaustively considered in switch() statements.
All these default statements can be removed with netCDF3 interface
so perhaps these should be surrounded with #ifdef NETCDF2_ONLY
constructs, but they actually do make sense for netCDF3 as well
so I have implemented a uniform error function, dfl_case_nctype_err(),
to be called by all routines which emit errors only when compiled with
NETCDF2_ONLY.
This makes the behavior easy to modify or remove in the future.
Placing this in its own routine also has the virtue of saving many lines
of code since this function is used in many many switch() statements. */
(void)fprintf(stdout,"%s: ERROR switch(nctype) statement fell through to default case, which is illegal.\nNot handling the default case causes gcc to emit warnings when compiling NCO with the NETCDF2_ONLY token (because nctype defintion is braindead in netCDF2). Exiting...",prg_nm_get());
exit(EXIT_FAILURE);
} /* end dfl_case_nctype_err() */
I'd like to begin to help with the 1.3
branch soon, as the 1.2.x stuff is basically
done and I want to merge the last changes into
1.3. So when you get things in 1.3 to a point
where they are either working or you are at a
good stopping point where I can muck around
for a bit, let me know.
Thanks,
Charlie