Sorry for the generic Subject, but I'm not sure what I want to do...if I can do it. Namely, I decided to fiddle around with ncremap because, well, it's new! So, I took a lat-lon file of mine along with a grid SCRIP file (of a cubed-sphere grid) and passed it through ncremap...and it worked. Well, it didn't crash. As I have very little clue what I'm doing, I took this as a good first step. Now about the only way I have to explore cubed-sphere files is a GrADS script that can sort of display variables not on a lat-lon grid through magic (to me). But, it needs files where the variables are like:
You successfully mapped the lat-long grid file to the cubed-sphere. Now remap the original (or the cubed sphere data) to any other grid already embedded in a data file (not a SCRIP grid file) dst.nc with ncremap -i in.nc -d dst.nc -o out.nc
Notice how you did not need a grid/SCRIP file for the remapping? That is ncremap's signature feature.
cz
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hmm. Okay. That did seem to do something, but at first glance, I'm thinking it did something weird. Faces of the cube seem rotated.
Maybe I should ask the reverse question. I can always view a lat-lon file. I really want to use ncremap so I can convert a native cubed-sphere output to lat-lon so it can be opened in GrADS/panoply.
Let's say I wanted to go from a cubed-sphere file (which I have a SCRIP file, PE48x288-CF.nc4), into a lat-lon grid. How would I do that? I tried:
(315) $ ncremap -i moist_c48.nc4 -s PE48x288-CF.nc4 -d moist_144.nc4 -o moist_c48_to_144.test.nc4
Started processing at Fri May 27 07:40:25 EDT 2016.
NCO ncremap version is "4.6.0"
Destination grid will be inferred from data-file
Weight-generation type: esmf
ESMF's ESMF_RegridWeightGen will generate mapping weights and map-file
Algorithm used to generate weights in map-file is: bilinear
/ford1/share/gmao_SIteam/Baselibs/TmpBaselibs/GMAO-Baselibs-5_0_1_with_NCO460/x86_64-unknown-linux-gnu/gfortran_6.1.0-openmpi_1.10.2/Linux/bin/ESMF_RegridWeightGen
Source grid supplied by user as PE48x288-CF.nc4
Input #00: /home/mathomp4/ncremap/Cube-to-Latlon/moist_c48.nc4
Map/Wgt : /tmp/ncremap_tmp_map_esmf_bilinear.nc.pid1934
Regridded: moist_c48_to_144.test.nc4
ncks: INFO nco_fl_open() reports current extended filetype = NC_FORMATX_NC3 does not equal previous extended filetype = NC_FORMATX_HDF5. This is expected when NCO is instructed to convert filetypes, i.e., to read from one type and write to another. And when NCO generates grids or templates (which are always netCDF3) when the input file is netCDF4. It is also expected when multi-file operators receive files known to be of different types. However, it could also indicate an unexpected change in input dataset type of which the user should be cognizant.
ncks: ERROR nco_rgr_map() reports datafile does not contain unstructured dimension name that matches a usual suspect (ncol, lndgrid, nCells, nEdges, sounding_id). HINT: Provide horizontal dimension name with "-r col_nm=foo"
ncremap: ERROR Failed to regrid. cmd_rgr[0] failed. Debug this:
ncks -t 2 -O --no_tmp_fl --gaa remap_script=ncremap --gaa remap_hostname=anvil.gsfc.nasa.gov --gaa remap_version="4.6.0" --hdr_pad=1000 --rgr lat_nm_out=lat --rgr lon_nm_out=lon --map=/tmp/ncremap_tmp_map_esmf_bilinear.nc.pid1934 /home/mathomp4/ncremap/Cube-to-Latlon/moist_c48.nc4 moist_c48_to_144.test.nc4
The error seems to point to the fact that my dst_fl isn't in the combined lat-lon format but in lat,lon format, right? Should I be using two SCRIP files in this case?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The error is because your input data file does not have a column (i.e., unstructured) dimension name that NCO recognizes. It defaults to ncol but looks for others, and tries to follow CF hints too. The message tells you how to provide it. If you think this message is incorrect, then please provide the SCRIP file and the datafile...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think I'm out of my depth. I'm going to add three links of where to find these data files. My guess is I do not have enough information on the files or enough files period. Or that our files aren't quite in the right style to use your tools.
But, if we can figure out how to do this, ncremap will be a powerful tool in our arsenal (especially as we move to more native cubed-sphere output...which you can't really view!).
PE48x288-CF.nc4 appears to be a SCRIP grid file for a cubed sphere. The other two files are "normal" 2D latxlon data files. If you had a cubed sphere data file, ncremap could project it onto either of the grids in your latxlon files. Right now, ncremap can put the latxlon files onto the cubed sphere grid. It can also re-project either of the latxlon files onto the grid used by the other latxlon file. It can also create mapfiles between any two of the three grids to accelerate subsequent remappings...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hmm. Well, the c48 file, if you look at it, is not really a latlon file in the real sense. As you can see it is lat=1,2,3,4,5... and lon=1,2,3,4,5... The data within is lat-lon in that sense, but it's really more of an i,j. I'm not sure how any utility could map its data onto the world because it doesn't have any handle into the world's lat-lon layout. We have an internal GrADS utility that works on CS files only because we know exactly how we layout our grid (point 1,1 is here, point 30,20 is here, etc.)
Now, I can ask our gurus to edit our model such that instead of lat=1,2,3,4... we actually put out lats and lons that correspond. Would ncremap like that a lot better? The lats and lons would look odd in that they wouldn't be "continuous" like a usual lat-lon file.
Sorry for the confusion. When it comes to output like this, we usually just regrid our output to latlon internally. But there is a push here to output directly on the cube so the data is "native" and then the only regridding would be done when post-run if needed by external utilities. As we always build ESMF and NCO, I thought ncremap could be ideal! (We don't build Tempest yet, so if that would be better, I could do that.)
Last edit: Matthew Thompson 2016-05-31
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Oh. OK. Yes, put the lat/lon coordinates and boundaries/vertices in the data files and/or a SCRIP gridfile (neither is trivial for a cubed-sphere grid, unfortunately). Then you can remap them.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Well, I do know that is a correct SCRIP file for our grid, I just don't know if it's enough data that ncremap needs to handle the transform. My hope is that if we provide enough information for ncremap, it can do it. If that means we need to generate SCRIPs, weights, etc., we'll give it a shot.
Given the files I sent you, is putting actual lats and lons the first step you'd recommend?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes. And you need to understand that the SCRIP file you sent defines a rank 1 unstructured grid whereas your datafiles are stored in rank 2 rectangular arrays, i.e., that particular SCRIP file appears to describe the native model 1d cube sphere grid not the 2d storage grid. Read the SCRIP docs for more info. But if you put the gridcell vertices and centers in the data files, then ncremap will know what to do...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Charlie, I'm over here with Matt and am the person that has dome the most with ESMF regridding here and jumping into this discussion. If I follow this here is what I have gotten from the duscussion thus far. Sorry for the long post but this is rather complex ...
1.) If we had cube-sphere data that was the in a 1-D unstructred layout identical to our scrip files describing our cube-sphere then NCO could in princple do this remapping.
What would the name of the 1-D dimension in the data file need to be for NCO to understand it?
2.) Adding the grid cells and vertices to our 2-D cubed-sphere output should make it possbile to regrid.
It is straightforward to add the cell center coordinates. I'm wondering what the format would need to be for the cell vertices, perhaps you have some insights based on your experience. The problem is the way we store our cube-sphere data. If we have a cN cube, we have 6 faces, each NxN cells. Now the way we put this in the file has no real relation to the grid topology. Basically we have two dimensions, the first is N, the second is 6N. So assuming you read this into a global fortran array the first (N,1:N) elements are face 1, elements (N,N+1:2N) are face 2, elements (N,2N+1:3N) are face 3, essentially we are stacking the 6 faces on top of each other.
It gets worse still, the points in row (N:N) which are face 1 are not connected to row (N:N+1) face 2 so the memory layout has no relationship to the tolopology, just a convention. The point of of all this is that there is no way to describe the corners with another 2D set of data say (N+1,6N+1). We could add another variable that was dimension (6,6N,4) and for every grid cell specify the 4 corners akin to the scrip file. Still no tolopology information but the script file doesn't have this either.
Assuming we can do this what kind of metadata, variable names is NCO looking for to figure out what variables are the cell centers and vertices?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The most straightforward choice is 'ncol' (='number of columns'). Any choice can be made to work with appropriate command line switches to NCO.
This makes my head want to explode. I suggest you pursue option 1. Option 2 can perhaps be made to work, though not exactly as you describe. SCRIP is fundamentally flawed in that it cannot hold connectivity information. But if you record every vertice of every gridcell, and treat it like a curvilinear grid, it might work.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It makes my head want to explode too. I was able to regrid our cube-sphere to Lat-Lon with ncremap after converting our wacky 2D storage to 1D and it worked great. One question, I got a log of extra "stuff" in the output file. Anyway flag to pass to not get all this in the output file?
doubleslat(slat);slat:long_name="latitude for staggered FV grid";slat:units="degrees_north";doubleslon(slon);slon:long_name="longitude for staggered FV grid";slon:units="degrees_east";doublelat_bnds(lat,nbnd);lat_bnds:long_name="gridcell latitude interfaces";doublelon_bnds(lon,nbnd);lon_bnds:long_name="gridcell longitude interfaces";doublegw(lat);gw:long_name="latitude quadrature weights (normalized to sum to 2.0 on global grids)";doublearea(lat,lon);area:long_name="solid angle subtended by gridcell";area:standard_name="cell_area";area:units="steradian";area:cell_methods="lat, lon: sum";
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Glad it works for you. slat/slon were added to make CESM/AMWG analysis easier. i can see why you wouldn't want them. gw has an unfortunate name but is useful for everyone in post-processing. lat_bnds/lon_bnds/area make the output (much) more CF-compliant. I'm reluctant to allow easy elimination of the last three. i will put adding a switch to eliminate slat/slon on the TODO list. for now, to eliminate what you don't want use ncks -x -v slat,slon,... in.nc out.nc
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Another question, if you do not specify the source grid script file with the -s option the documentation says it tries to infer the grid from the input file. Does this understand the ugrid convention that I see here in the ESMF documentation if I were to describe our cube with the format and embed the data in the file?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
No, ncremap cannot infer a grid from a ugrid specification. That is a long-term goal, beyond the next year. basically ncremap infers well when it sees a data file with simple rectangular geometries and/or when the coordinates provide their interface edges (lat_bnds, lon_bnds).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sorry for the generic Subject, but I'm not sure what I want to do...if I can do it. Namely, I decided to fiddle around with ncremap because, well, it's new! So, I took a lat-lon file of mine along with a grid SCRIP file (of a cubed-sphere grid) and passed it through ncremap...and it worked. Well, it didn't crash. As I have very little clue what I'm doing, I took this as a good first step. Now about the only way I have to explore cubed-sphere files is a GrADS script that can sort of display variables not on a lat-lon grid through magic (to me). But, it needs files where the variables are like:
But, what I got out had variables like:
My question is, is there an NCO way to "deconvolve" the combined lat-lon dimension into separate lat and lon?
You successfully mapped the lat-long grid file to the cubed-sphere. Now remap the original (or the cubed sphere data) to any other grid already embedded in a data file (not a SCRIP grid file) dst.nc with
ncremap -i in.nc -d dst.nc -o out.ncNotice how you did not need a grid/SCRIP file for the remapping? That is ncremap's signature feature.
cz
Hmm. Okay. That did seem to do something, but at first glance, I'm thinking it did something weird. Faces of the cube seem rotated.
Maybe I should ask the reverse question. I can always view a lat-lon file. I really want to use ncremap so I can convert a native cubed-sphere output to lat-lon so it can be opened in GrADS/panoply.
Let's say I wanted to go from a cubed-sphere file (which I have a SCRIP file, PE48x288-CF.nc4), into a lat-lon grid. How would I do that? I tried:
The error seems to point to the fact that my dst_fl isn't in the combined lat-lon format but in lat,lon format, right? Should I be using two SCRIP files in this case?
The error is because your input data file does not have a column (i.e., unstructured) dimension name that NCO recognizes. It defaults to ncol but looks for others, and tries to follow CF hints too. The message tells you how to provide it. If you think this message is incorrect, then please provide the SCRIP file and the datafile...
Charlie,
I think I'm out of my depth. I'm going to add three links of where to find these data files. My guess is I do not have enough information on the files or enough files period. Or that our files aren't quite in the right style to use your tools.
But, if we can figure out how to do this, ncremap will be a powerful tool in our arsenal (especially as we move to more native cubed-sphere output...which you can't really view!).
The links are:
https://www.dropbox.com/s/j1bh3zvuktbtkys/PE48x288-CF.nc4?dl=0
https://www.dropbox.com/s/tjdr7eew6olfx27/moist_c48.nc4?dl=0
https://www.dropbox.com/s/j5095k87a4ihkpy/moist_144.nc4?dl=0
(The two moist files are rather large and binary, so I figured it was better than attaching them here.)
Thanks,
Matt
PE48x288-CF.nc4 appears to be a SCRIP grid file for a cubed sphere. The other two files are "normal" 2D latxlon data files. If you had a cubed sphere data file, ncremap could project it onto either of the grids in your latxlon files. Right now, ncremap can put the latxlon files onto the cubed sphere grid. It can also re-project either of the latxlon files onto the grid used by the other latxlon file. It can also create mapfiles between any two of the three grids to accelerate subsequent remappings...
Hmm. Well, the c48 file, if you look at it, is not really a latlon file in the real sense. As you can see it is lat=1,2,3,4,5... and lon=1,2,3,4,5... The data within is lat-lon in that sense, but it's really more of an i,j. I'm not sure how any utility could map its data onto the world because it doesn't have any handle into the world's lat-lon layout. We have an internal GrADS utility that works on CS files only because we know exactly how we layout our grid (point 1,1 is here, point 30,20 is here, etc.)
Now, I can ask our gurus to edit our model such that instead of lat=1,2,3,4... we actually put out lats and lons that correspond. Would ncremap like that a lot better? The lats and lons would look odd in that they wouldn't be "continuous" like a usual lat-lon file.
Sorry for the confusion. When it comes to output like this, we usually just regrid our output to latlon internally. But there is a push here to output directly on the cube so the data is "native" and then the only regridding would be done when post-run if needed by external utilities. As we always build ESMF and NCO, I thought ncremap could be ideal! (We don't build Tempest yet, so if that would be better, I could do that.)
Last edit: Matthew Thompson 2016-05-31
Oh. OK. Yes, put the lat/lon coordinates and boundaries/vertices in the data files and/or a SCRIP gridfile (neither is trivial for a cubed-sphere grid, unfortunately). Then you can remap them.
Well, I do know that is a correct SCRIP file for our grid, I just don't know if it's enough data that ncremap needs to handle the transform. My hope is that if we provide enough information for ncremap, it can do it. If that means we need to generate SCRIPs, weights, etc., we'll give it a shot.
Given the files I sent you, is putting actual lats and lons the first step you'd recommend?
Yes. And you need to understand that the SCRIP file you sent defines a rank 1 unstructured grid whereas your datafiles are stored in rank 2 rectangular arrays, i.e., that particular SCRIP file appears to describe the native model 1d cube sphere grid not the 2d storage grid. Read the SCRIP docs for more info. But if you put the gridcell vertices and centers in the data files, then ncremap will know what to do...
Hi Charlie, I'm over here with Matt and am the person that has dome the most with ESMF regridding here and jumping into this discussion. If I follow this here is what I have gotten from the duscussion thus far. Sorry for the long post but this is rather complex ...
1.) If we had cube-sphere data that was the in a 1-D unstructred layout identical to our scrip files describing our cube-sphere then NCO could in princple do this remapping.
What would the name of the 1-D dimension in the data file need to be for NCO to understand it?
2.) Adding the grid cells and vertices to our 2-D cubed-sphere output should make it possbile to regrid.
It is straightforward to add the cell center coordinates. I'm wondering what the format would need to be for the cell vertices, perhaps you have some insights based on your experience. The problem is the way we store our cube-sphere data. If we have a cN cube, we have 6 faces, each NxN cells. Now the way we put this in the file has no real relation to the grid topology. Basically we have two dimensions, the first is N, the second is 6N. So assuming you read this into a global fortran array the first (N,1:N) elements are face 1, elements (N,N+1:2N) are face 2, elements (N,2N+1:3N) are face 3, essentially we are stacking the 6 faces on top of each other.
It gets worse still, the points in row (N:N) which are face 1 are not connected to row (N:N+1) face 2 so the memory layout has no relationship to the tolopology, just a convention. The point of of all this is that there is no way to describe the corners with another 2D set of data say (N+1,6N+1). We could add another variable that was dimension (6,6N,4) and for every grid cell specify the 4 corners akin to the scrip file. Still no tolopology information but the script file doesn't have this either.
Assuming we can do this what kind of metadata, variable names is NCO looking for to figure out what variables are the cell centers and vertices?
It makes my head want to explode too. I was able to regrid our cube-sphere to Lat-Lon with ncremap after converting our wacky 2D storage to 1D and it worked great. One question, I got a log of extra "stuff" in the output file. Anyway flag to pass to not get all this in the output file?
Glad it works for you. slat/slon were added to make CESM/AMWG analysis easier. i can see why you wouldn't want them. gw has an unfortunate name but is useful for everyone in post-processing. lat_bnds/lon_bnds/area make the output (much) more CF-compliant. I'm reluctant to allow easy elimination of the last three. i will put adding a switch to eliminate slat/slon on the TODO list. for now, to eliminate what you don't want use
ncks -x -v slat,slon,... in.nc out.ncAnother question, if you do not specify the source grid script file with the -s option the documentation says it tries to infer the grid from the input file. Does this understand the ugrid convention that I see here in the ESMF documentation if I were to describe our cube with the format and embed the data in the file?
No, ncremap cannot infer a grid from a ugrid specification. That is a long-term goal, beyond the next year. basically ncremap infers well when it sees a data file with simple rectangular geometries and/or when the coordinates provide their interface edges (lat_bnds, lon_bnds).