#5 Populating models to grids

[Allan Diegan]

I have GFE correctly installed and I do believe I am downloading model data, but how do I display the data into GFE? The options I do have 'populate' -> 'copy all grids from' are "ICS, LAPS, MSAS, Offical, RTMA, Restore, and ISC V". No GFS, or NAM options...

Thanks!

[Michael Foland]

Hey I got this fixed for you. It took some time once the script was running. It fetches the latest model data :) Would be nice to set this to a cron somehow for when the new models do come out.. and have more data like ECMWF and the HRRR

[Mike Romberg]

Good. I'm glad that is working. The 'gfei -m ingest' thing was a sorta quick experiment I did to replace the portion of the AWIPS ingest subsystem used by the gfe. The idea is to integrate it into the ifpServer at some point. I may rewrite it first whenever I find time. So, for now you need to run it manually.

It does seem to stay up for days/weeks at a time on my system. But it also downloads boatloads of data. I don't want to integrate it into the server without having some kind of user interface and have it off by default.

[Michael Foland]

Could there be an option for a cron job to fetch at every model run?

[Mike Romberg]

The gfei -m ifp.ingest script is intended to be run as it's own stand alone cron. Just start it and let it run forever (or until something in the grib decoder blows up which will happen after some days). This script is configured with the NAM and GFS run times. It checks hourly for new data at the NOMADS website and pulls new data as it appears.

I just run it in a terminal window when I'm interested in an up to date set of model data.   At some point I may have time to re-write this ingest code.   The current version was an experiment in python threading and multi-cpu support.   It did not turn out good enough for integration into the ifpServer.   Whenever I re-write it, I'll put it into ifpServer and no cron (or terminal window) will be needed.

[Michael Foland]

When loading the gfei script, it goes to dgrib and sometimes it will show up in GFE, and has pulled up data into the grids in the past. Now running the script, whether it be the NAM or the GFS... it will not pull into the GFE grids. Not sure what's going on here.

[Michael Foland]

Update on the gfei.. Getting some errors concerning NetCDF

close: /scratch/data/cdf/gfs/230/20180428_0000
Traceback (most recent call last):
File "libs/pylib/tasks/threadpool.py", line 21, in run
File "libs/pylib/tasks/threadpool.py", line 15, in runTask
File "libs/pylib/tasks/pipeline.py", line 135, in processStage
File "libs/pylib/ifp/ingest/nomads.py", line 283, in put
File "libs/pylib/ifp/ingest/nomads.py", line 312, in storeTopo
File "netCDF4/_netCDF4.pyx", line 2293, in netCDF4._netCDF4.Dataset.createVariable (netCDF4/_netCDF4.c:18399)
File "netCDF4/_netCDF4.pyx", line 3366, in netCDF4._netCDF4.Variable.init (netCDF4/_netCDF4.c:32393)
File "netCDF4/_netCDF4.pyx", line 3535, in netCDF4._netCDF4.Variable.ncattrs (netCDF4/_netCDF4.c:35458)
File "netCDF4/_netCDF4.pyx", line 1087, in netCDF4._netCDF4._get_att_names (netCDF4/_netCDF4.c:4184)
AttributeError: NetCDF: Not a valid ID
NoneType: None
open: /scratch/data/cdf/gfs/230/20180428_0000

Hope we aren't bothering you.. Just wanting to help you out on making the project work :)

[Mike Romberg]

When ifp.ingest runs it does not make any attempt to determine what is the latest model found on nomads right now. Instead it looks at the current time. Then it calculates what is the closest model run for the time right now. It then tries to find any files for the calculated model run and start downloading processing them. So, it may or may not immediatly start downloading files. It just depends on the current time and where we are in the model run cycle. It is not perfect.

Again, the idea of ifp.ingest is to just fire it up and leave it running for a long time (hours, days, weeks).   Also, it is pretty experimental.

  The above stack trace indicates that you have a corrupt netcdf file.  Just remove everything in /scratch/data/cdf and start the script again. 'rm -rf /scratch/data/cdf'

    The files in /scratch/data/cdf can be corrupted if the ifp.ingest script gets shutdown in some way while it is processing data.   So, it is not a good idea to stop it in the middle of grid decoding.   Sometimes it may stop itself as the ncep grid decoder can crash internally.  But this does not happen all that often.

      Also, thank you for reporting these issues.   It would be nice to get more of this working again after all the surgery I did to the code to bring it up to date with modern versions of the languages.   I don't have time to test everything and frankly am not familiar with using all of the features.  So, it helps to have bug reports from someone who is.

[Allan Diegan]

Please let me know if you find additional ways to ingest model data, as this could be a huge development! Further, I notice that after running the script, the GFS shows but the grids do not match what the GFS shows online for that day. For instance today (April 29), the GFS model viewed in GFE shows rain for Kentucky/Tennessee, but when you view the GFS model online for the same day, it shows no rain which is right because we are under high pressure.

The NAM from 12Z shows that it is in, but only loads one hour and it has been running since 5 am this morning. It does seem like the files get corrupted a lot, as I noticed if I delete the files and restart the script, the models show up in GFE.

Were you ever successfully able to populate the entire NAM 0-84 hours out into GFE?

Thanks again for the amazing work you have put into this, as I can see this being extremely popular once everything gets worked out!

[Michael Foland]

The models aren't populating all the way, and we are not getting the full model suite. The GFS is also pulling in things wrong (see attatched photo), and would appear to possibly be corrupted.

[Mike Romberg]

I would love to blow away the whole ingest system and create something much more straight forward and easy to deal with. It probably would not even be too hard to do. The major obstacle is finding model data that is:

• Available online and at no charge
• Well documented as to what it is.
• In a reasonably sane data format (grib barely counts)

So far, the best I've stumbled across is the NCEP stuff on NOMADS. But even it is less than ideal as it is a MAJOR headache to figure out what cryptic file names are part of each model run and what cryptic variable names mean. The documentation is sparse and incomplete.

But I'm just a hobby programmer and not a weather professional.  If anyone else knows a sorce of model data that seems like it might fall into the above three categories please let me know.   It is high on my list to find a better source of data.

  The whole ingest of model data into the gfe has always been a two stage affair.  First the model data is converted from raw grib format into netcdf files that were used by AWIPS I (and the gfe).   But the models never did (and still don't as far as I know) produce usefule fields such as wx, maxt, mint etc.  They are also not at the resolution used by the gfe.   So, a set of algorithms called ifpInit is run to create things like wx.   In the case of the global GFS I just patched the old gfs75 ifpInit code into the global gfs.  It does not surprise me at all that the wx grids are not kosher as many fields were flat out missing.   The GFS that is there now is just to have something there.  It probably is not right and may need adjustments.

  As far as ifp.Ingest goes, it does run for weeks at a time on my home system.   I mostly just run it with Nam12.   It ingests fully 72 hours or so for each run (which is all I think it has now days). The GFS goes further but has several disadvantages.  It is the global GFS.   The grids are large and low resolution in both space and vertical dimensions.  So, using the GFS is mostly interesting for non US locations.   The other models I can't either find on NOMADS or seem to be ensembles (which the gfe can't deal with).

    You should not shutdown the ifp.ingest script in the middle of processing.   It is safe to do so when it enters a "quite" state.  This is when it is printing "check for..." messages every hour.   But if it is displaying "decoding grib" stuff and gets shut down then there will be a mess left in /scratch/data/cdf.   If this does happen, it is best to delete the whole directory before restarting the script.   This is because there are more files in there than you see at first.   There are temporary files and database files with .name type names that also need to get cleaned out.   So, 'rm -rf /scratch/data/cdf/nam' is better than 'rm -f /scratch/data/cdf/nam/*'.   The later leaves behind a bunch of (possibly corrupted) databases.

[Allan Diegan]

Luckily most models are free of charge with exceptions being the EURO and maybe one or two others.

The only places I know of at the moment are the following:

http://nomads.ncep.noaa.gov:9090/dods/
This locations does have better explanations for what variable names are what, diminsions, etc...

and

https://catalog.data.gov/dataset/global-forecast-system-gfs-0-5-deg

The hard part is going to be finding the format that is easy to work in. I believe Mike Foland and I can help find other locations. I believe that assuming we can succesfully ingest the model data and degrib it then we can technically view any model parameter even things such as sea level pressure, and all sorts of other important meteorological parameters. If you ever need information on the meteorological side of things I would be glad to help out! I would personally be happy with the NAM for the short term...

[Michael Foland]

The free models out there are the HRRR, RAP, NAM, and GFS.

[Michael Foland]

While the ifp.ingest is a nice idea... the script wants to pull in the models and dgribs them... and continues to check... when it completes it shows up in GFE, HOWEVER it does not show in the grids. Is this a GFE issue? I did try the rm -rf to the cd directory as well.

[Mike Romberg]

I think there may possibly be an ncgen related issue going on with your build. See the other ticket related to this.

[Mike Romberg]

duplicate of still open ticket