Menu
▾
▴
Re: [Geotools-devel] [netcdf-java] runtime aggregation
|
From: Niels C. <ni...@sc...> - 2016-09-23 14:23:44
|
HI Daniele, Thank you for your quick response. Two remarks though - I do seem to get from your explanation that it can also be decreasing rather than increasing, which means it should at least be possible to work the other way around, which was the case here. There are only two files, the latest in time is first. So at least the maximum/minimum of the two extremes should be picked. - I understand this is a convention for variables in general. But does this however count also for the order in which aggregated files are listed in an aggregation with an already pre-existing dimension (which is the case here)? Perhaps the issue is somewhere else in the code, and the values should be sorted after reading the list of files. Kind Regards Niels On 23-09-16 15:50, Daniele Romagnoli wrote: > Hi Niels, > > > On Fri, Sep 23, 2016 at 2:40 PM, Niels Charlier <ni...@sc... > <mailto:ni...@sc...>> wrote: > > Simone, Daniele, > > Do you have an opinion on point (3) of this email? > > It seems that one of you have written the code that expects a > chronological order of listed aggregation files. > > The logic is in > https://github.com/geotools/geotools/blob/master/modules/plugin/coverage-multidim/netcdf/src/main/java/org/geotools/imageio/netcdf/cv/TimeCoordinateVariable.java > <https://github.com/geotools/geotools/blob/master/modules/plugin/coverage-multidim/netcdf/src/main/java/org/geotools/imageio/netcdf/cv/TimeCoordinateVariable.java> > The getMinimum() and getMaximum() methods. > > The problem could easily be resolved by searching for the minimum > and maximum rather than expecting a certain order. > I can't find anything in the specs that says they should be in > order, so would you agree if I changed it to that regards? > > > I think that we developed this based on a best practice for the > NetCDF, assuming that coordinate axis/coordinate variables are monotonic. > https://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.html > > Quoting the CoordinateSystems section: > > /A ***coordinate variable*** is a one-dimensional variable with the > same name as a dimension, which names the coordinate values of the > dimension. It must not have any missing data (for example, no > |_FillValue| or |missing_value| attributes) *and must be strictly > monotonic (values increasing or decreasing).* > / > / > / > However I'm not against that improvement provided that the time to > index mapping still work and the search isn't too much expensive > (maybe we should cache the min and the max?). > > Cheers, > Daniele > > > Niels > > > On 08-09-16 14:51, Niels Charlier wrote: >> Ben, Sean, >> >> I am also including the geotools list in the discussion, because >> it regards both libraries. >> >> I got the aggregation file to work. Apart from some obvious bug >> fixes and additions to support the "runtime" dimension in >> geotools, I had to make a few changes to the xml file. >> What is still unclear to me is whether these changes >> - are actually required to work (i.e. the aggregation file is >> faulty). in that case we need to work on better error reporting >> in geotools (possibly also in netcdf-java) >> - should not actually be required, in which case we need to fix >> these as bugs in either geotools or netcdf-java >> >> I have included the .ncml file in the attachment to demonstrate. >> There are three such changes I made: >> >> 1. the Conventions global attribute,as discussed before. >> >> 2. removal of the "time" variable in the aggregation (see >> commented line in attached .ncml file) >> >> This issue is related to the previous one, it is again about >> /makeCoordinateSystemsImplicit/ versus >> /makeCoordinateSystemsMaximal/. >> >> * If the "time" variable is included as aggregation variable, >> it also gets the "runtime" dimension added to its dimensions. >> * As a consequence, netcdf-java does not recognise it as an AXIS. >> * As a consequence, /makeCoordinateSystemsImplicit /fails to >> include it in the CRS's for the actual variables (water_u, >> salinity, etc...). >> * As a consequence, /makeCoordinateSystemsMaximal /creates a >> CRS for these variables. However, it puts the dimensions in the >> order of the /dimension variable/ list >> (rather than the variable's own dimensions), which is >> completely arbitrary as far as I can see. >> * "runtime" ends up as the last axis instead of the first. >> This is inconsistent with the order of dimensions in the >> variable. Geotools fails on this inconsistency. >> >> I am not sure whether the fault here lies with the .ncml >> file, geotools or netcdf-java. >> >> >> 3. changing the order of the list of files in the inner >> aggregation tags. >> <netcdf location="hycom_glb_regp01_old_t003.nc <http://hycom_glb_regp01_old_t003.nc>"/> <netcdf location="hycom_glb_regp01_old_t000.nc <http://hycom_glb_regp01_old_t000.nc>"/> >> instead of <netcdf location="hycom_glb_regp01_old_t000.nc <http://hycom_glb_regp01_old_t000.nc>"/> <netcdf location="hycom_glb_regp01_old_t003.nc <http://hycom_glb_regp01_old_t003.nc>"/> >> Otherwise the one with the earliest time coordinates is last >> in the list instead of first... and geotools fails on this. Is >> this normal? This issue is definitely unrelated to >> netcdf-java, but I am not sure whether this is a bug in geotools >> or in the .ncml file. Regards Niels On 07-09-16 13:23, Niels >> Charlier wrote: >>> Ben, Sean, Changing the Convention has resolved this issue. >>> Thanks both for your tips. I am now dealing with other issues in >>> the same file... will keep you updated. Kind Regards Niels On >>> 02-09-16 03:00, Ben Caradoc-Davies wrote: >>>> Niels, one thing you could try is to change the Conventions >>>> global attribute of these files to "CF-1.6" to see if this >>>> changes the way they are handled (if these files are close >>>> enough to CF). Sean, do you expect aggregations to behave the >>>> same as single files with the same Conventions? It looks like >>>> Niels has identified a difference. Kind regards, Ben. On >>>> 30/08/16 03:29, Sean Arms wrote: >>>>> Greetings Niels, The class DefaultConvention is netCDF-Java's >>>>> way of trying to handle cases in which a convention is not >>>>> specified. It tries to be pretty forgiving in terms of files >>>>> where people have tried to be compliant with some convention, >>>>> but didn't set a convention attribute, or knew enough about >>>>> some conventions in general to sort of get things in the ball >>>>> park...for example, setting positive=up on a vertical >>>>> coordinate, but not getting the rest of the stuff correct (not >>>>> true for your case, but our code tries to handle the more >>>>> common case we come across). If a file is actually following >>>>> CF, netCDF-Java will use the class CF1Convention. Currently we >>>>> handle (or try to) around 30 specific conventions (not sure >>>>> all are actually documented conventions), but when all else >>>>> fails, we go to the default convention. NcML aggregations in >>>>> general can get pretty gnarly, and are very hard to get >>>>> correct. Have you looked into using the Forecast Model Run >>>>> Collection (FMRC) featureCollection element to serve these >>>>> data? >>>>> https://www.unidata.ucar.edu/software/thredds/current/tds/tutorial/FmrcFeatureCollectionsTutorial.html >>>>> <https://www.unidata.ucar.edu/software/thredds/current/tds/tutorial/FmrcFeatureCollectionsTutorial.html> >>>>> Cheers, Sean On Mon, Aug 29, 2016 at 6:11 AM, Niels Charlier >>>>> <ni...@sc...> <mailto:ni...@sc...> wrote: >>>>>> Hello, I am currently working on the netcdf extensions of >>>>>> geotools/geoserver, although I should say I am fairly new at >>>>>> netcdf, so I am still learning. I am currently trying to get >>>>>> some particular datasets to work with geoserver, and I think >>>>>> I might have hit against some limitations, perhaps even a >>>>>> bug, in netcdf-java, so I wanted to ask you for advice. We >>>>>> have a .ncml have that aggregate datasets based on 'runtime'. >>>>>> This is an explanation from the client: *"'runtime' is not a >>>>>> first class coordinate in the data produced by our production >>>>>> groups, so the initial approach to solve this using unidata >>>>>> mechanisms is to use NCML to aggregate * *two datasets with >>>>>> different runtimes together using a 'joinNew' aggregation, >>>>>> which adds a new coordinate axis 'runtime' to the dataset * >>>>>> *(this takes place in several of the .ncml files included in >>>>>> the test data). The runtime axis would then be exposed as a >>>>>> custom dimension in geoserver so that it can be used for >>>>>> subsets. This means that a reader would have to be prepared >>>>>> to handle dimension values that are possibly two >>>>>> dimensional(a set of values for each available 'runtime'), >>>>>> which I don't think the current reader can handle? "* My >>>>>> first question is what it would entail for building support >>>>>> for this in netcdf-java? In the meantime I have been >>>>>> extensively debugging what happens when I try to load the >>>>>> aggregation file, and I can give some detailed information >>>>>> about what goes wrong in the code. I think I might have found >>>>>> something else that is going wrong. * as part of the >>>>>> aggregation, the runtime dimension is being added to every >>>>>> other variable of each of the .nc files. * During >>>>>> *CoordSysBuilder.buildCoordinateSystems* some things happen >>>>>> as a consequence of that that is different from when I read >>>>>> the .nc files individually: -> In >>>>>> *makeCoordinateSystemsImplicit*, as a consequence of all >>>>>> variables having an additional dimension, the CRS with >>>>>> lacking runtime dimension is not considered sufficient and is >>>>>> not being set. -> However, in >>>>>> *makeCoordinateSystemsMaximal* the CRS's are being set, >>>>>> except for one variable called "surf_el", because it is being >>>>>> considered an AXIS (see line 873, call to* !vp.isData()*). * >>>>>> During index building (*NetCDFImageReader.init -> initIndex >>>>>> -> getCoordinateSystem*), a runtime exception is thrown >>>>>> because surf_el doesn't have a CRS. The thing is, surf_el is >>>>>> not an axis. The reason it is considered an axis is because >>>>>> it has a "positive" attribute. This happens in >>>>>> *DefaultConvention.findCoordinateAxes -> getAxisType -> >>>>>> getAxisTypeCoards* (see line 297, *positive != null*). To >>>>>> quote my colleague Ben who I asked advice about this: >>>>>> *"surf_el is a time-varying data variable and not any kind >>>>>> of coordinate. Surface elevation looks to me like an output >>>>>> of the HYCOM ocean model **<https://hycom.org/> >>>>>> <https://hycom.org/> <https://hycom.org/> >>>>>> <https://hycom.org/>**. Any logic that says that a variable >>>>>> is a coordinate if it has attribute "positive"="up" is >>>>>> broken. There is this statement in the CF conventions: * *"A >>>>>> vertical coordinate will be identifiable by: * * units of >>>>>> pressure; or * * the presence of the positive attribute >>>>>> with a value of up or down (case insensitive)." * >>>>>> *http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html >>>>>> <http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html> >>>>>> <http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html> >>>>>> <http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html>* >>>>>> *There are such things as time-varying multidimensional >>>>>> coordinate variables (think of the lat and lon grids for a >>>>>> satellite swathe), but this is not one of these. * *I think >>>>>> that, to be a coordinate variable, a variable must be >>>>>> mentioned in the "coordinates" attribute of another variable >>>>>> in the same file, or have a name that matches its dimension >>>>>> (i.e. a basic single-dimensional coordinate variable). " * So >>>>>> have I indeed identified a bug here? please advice. Thank you >>>>>> for your help. Kind Regards, Niels >>>>>> _______________________________________________ NOTE: All >>>>>> exchanges posted to Unidata maintained email lists are >>>>>> recorded in the Unidata inquiry tracking system and made >>>>>> publicly available through the web. Users who post to any of >>>>>> the lists we maintain are reminded to remove any personal >>>>>> information that they do not want to be made public. >>>>>> netcdf-java mailing list net...@un... >>>>>> <mailto:net...@un...> For list information or >>>>>> to unsubscribe, visit: >>>>>> http://www.unidata.ucar.edu/mailing_lists/ >>>>>> <http://www.unidata.ucar.edu/mailing_lists/> >>>>> _______________________________________________ NOTE: All >>>>> exchanges posted to Unidata maintained email lists are >>>>> recorded in the Unidata inquiry tracking system and made >>>>> publicly available through the web. Users who post to any of >>>>> the lists we maintain are reminded to remove any personal >>>>> information that they do not want to be made public. >>>>> netcdf-java mailing list net...@un... >>>>> <mailto:net...@un...> For list information or >>>>> to unsubscribe, visit: >>>>> http://www.unidata.ucar.edu/mailing_lists/ >>>>> <http://www.unidata.ucar.edu/mailing_lists/> >>> _______________________________________________ NOTE: All >>> exchanges posted to Unidata maintained email lists are recorded >>> in the Unidata inquiry tracking system and made publicly >>> available through the web. Users who post to any of the lists >>> we maintain are reminded to remove any personal information that >>> they do not want to be made public. netcdf-java mailing list >>> net...@un... >>> <mailto:net...@un...> For list information or to >>> unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/ >>> <http://www.unidata.ucar.edu/mailing_lists/> >> >> ------------------------------------------------------------------------------ >> >> _______________________________________________ >> GeoTools-Devel mailing list >> Geo...@li... >> <mailto:Geo...@li...> >> https://lists.sourceforge.net/lists/listinfo/geotools-devel >> <https://lists.sourceforge.net/lists/listinfo/geotools-devel> > > -- > == > GeoServer Professional Services from the experts! Visit > http://goo.gl/it488V for more information. == Ing. Daniele Romagnoli > Senior Software Engineer GeoSolutions S.A.S. Via di Montramito 3/A > 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 > 1660272 http://www.geo-solutions.it http://twitter.com/geosolutions_it > ------------------------------------------------------- > > *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003* > > Le informazioni contenute in questo messaggio di posta elettronica e/o > nel/i file/s allegato/i sono da considerarsi strettamente riservate. > Il loro utilizzo è consentito esclusivamente al destinatario del > messaggio, per le finalità indicate nel messaggio stesso. Qualora > riceviate questo messaggio senza esserne il destinatario, Vi preghiamo > cortesemente di darcene notizia via e-mail e di procedere alla > distruzione del messaggio stesso, cancellandolo dal Vostro sistema. > Conservare il messaggio stesso, divulgarlo anche in parte, > distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità > diverse, costituisce comportamento contrario ai principi dettati dal > D.Lgs. 196/2003. > > The information in this message and/or attachments, is intended solely > for the attention and use of the named addressee(s) and may be > confidential or proprietary in nature or covered by the provisions of > privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New > Data Protection Code).Any use not in accord with its purpose, any > disclosure, reproduction, copying, distribution, or either > dissemination, either whole or partial, is strictly forbidden except > previous formal approval of the named addressee(s). If you are not the > intended recipient, please contact immediately the sender by > telephone, fax or e-mail and delete the information in this message > that has been received in error. The sender does not give any warranty > or accept liability as the content, accuracy or completeness of sent > messages and accepts no responsibility for changes made after they > were sent or for other risks which arise as a result of e-mail > transmission, viruses, etc. > |