#12 GFEI Errors / Importing NAM or GFS into GFE

[Michael Foland]

Tried the gfei script today and got the following:

[gfe@localhost ~]$ sh /home/gfe/gfeInstall/bin/gfei -m ifp.ingest --models mam
Check: http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/ nam 20180509 12z
download: nam.t12z.awphys16.tm00.grib2
download: nam.t12z.awphys30.tm00.grib2
degrib: nam.t12z.awphys30.tm00.grib2
download: nam.t12z.awip1235.tm00.grib2
update: /scratch/data/cdf/nam/218/template
open: /scratch/data/cdf/nam/218/20180509_1200
close: /scratch/data/cdf/nam/218/20180509_1200
Traceback (most recent call last):
File "libs/pylib/tasks/threadpool.py", line 21, in run
File "libs/pylib/tasks/threadpool.py", line 15, in runTask
File "libs/pylib/tasks/pipeline.py", line 135, in processStage
File "libs/pylib/ifp/ingest/nomads.py", line 304, in put
File "netCDF4/_netCDF4.pyx", line 4191, in netCDF4._netCDF4.Variable.setitem (netCDF4/_netCDF4.c:46816)
File "netCDF4/_netCDF4.pyx", line 3502, in netCDF4._netCDF4.Variable.shape.get (netCDF4/_netCDF4.c:34880)
File "netCDF4/_netCDF4.pyx", line 3447, in netCDF4._netCDF4.Variable._getdims (netCDF4/_netCDF4.c:34052)
RuntimeError: NetCDF: Not a valid ID
NoneType: None
open: /scratch/data/cdf/nam/218/20180509_1200
download: nam.t12z.awphys15.tm00.grib2
download: nam.t12z.awphys78.tm00.grib2
degrib: nam.t12z.awphys16.tm00.grib2
degrib: nam.t12z.awip1235.tm00.grib2
degrib: nam.t12z.awphys78.tm00.grib2
download: nam.t12z.awip1269.tm00.grib2
download: nam.t12z.awip1272.tm00.grib2
download: nam.t12z.awip1236.tm00.grib2
degrib: nam.t12z.awip1269.tm00.grib2
degrib: nam.t12z.awphys15.tm00.grib2
degrib: nam.t12z.awip1236.tm00.grib2
download: nam.t12z.awip1205.tm00.grib2
degrib: nam.t12z.awip1272.tm00.grib2
download: nam.t12z.awip1266.tm00.grib2
degrib: nam.t12z.awip1205.tm00.grib2
download: nam.t12z.awip1231.tm00.grib2
degrib: nam.t12z.awip1266.tm00.grib2
download: nam.t12z.awip1200.tm00.grib2
degrib: nam.t12z.awip1200.tm00.grib2
download: nam.t12z.awip1221.tm00.grib2
degrib: nam.t12z.awip1221.tm00.grib2
download: nam.t12z.awphys04.tm00.grib2
degrib: nam.t12z.awphys04.tm00.grib2
download: nam.t12z.awip1263.tm00.grib2
download: nam.t12z.awphys10.tm00.grib2
download: nam.t12z.awip1204.tm00.grib2
degrib: nam.t12z.awphys10.tm00.grib2
degrib: nam.t12z.awip1231.tm00.grib2
degrib: nam.t12z.awip1263.tm00.grib2
download: nam.t12z.awphys20.tm00.grib2
download: nam.t12z.awphys75.tm00.grib2
degrib: nam.t12z.awip1204.tm00.grib2
degrib: nam.t12z.awphys20.tm00.grib2
download: nam.t12z.awip1248.tm00.grib2
download: nam.t12z.awphys08.tm00.grib2
degrib: nam.t12z.awphys75.tm00.grib2
download: nam.t12z.awphys03.tm00.grib2
degrib: nam.t12z.awip1248.tm00.grib2
degrib: nam.t12z.awphys03.tm00.grib2
download: nam.t12z.awip1239.tm00.grib2
download: nam.t12z.awip1208.tm00.grib2
download: nam.t12z.awphys33.tm00.grib2
download: nam.t12z.awphys09.tm00.grib2
degrib: nam.t12z.awip1239.tm00.grib2
degrib: nam.t12z.awip1208.tm00.grib2
degrib: nam.t12z.awphys08.tm00.grib2
degrib: nam.t12z.awphys33.tm00.grib2
download: nam.t12z.awphys05.tm00.grib2
download: nam.t12z.awip1284.tm00.grib2
download: nam.t12z.awip1214.tm00.grib2
download: nam.t12z.awphys45.tm00.grib2
download: nam.t12z.awip1212.tm00.grib2
download: nam.t12z.awphys57.tm00.grib2
degrib: nam.t12z.awip1284.tm00.grib2
degrib: nam.t12z.awphys05.tm00.grib2
download: nam.t12z.awphys11.tm00.grib2
degrib: nam.t12z.awip1212.tm00.grib2
download: nam.t12z.awip1234.tm00.grib2
degrib: nam.t12z.awphys11.tm00.grib2
degrib: nam.t12z.awip1214.tm00.grib2
degrib: nam.t12z.awphys09.tm00.grib2
degrib: nam.t12z.awphys45.tm00.grib2
download: nam.t12z.awip1220.tm00.grib2
degrib: nam.t12z.awip1234.tm00.grib2
degrib: nam.t12z.awphys57.tm00.grib2
download: nam.t12z.awip1227.tm00.grib2
degrib: nam.t12z.awip1220.tm00.grib2
download: nam.t12z.awphys18.tm00.grib2
degrib: nam.t12z.awip1227.tm00.grib2
download: nam.t12z.awphys66.tm00.grib2
degrib: nam.t12z.awphys66.tm00.grib2
download: nam.t12z.awip1202.tm00.grib2
degrib: nam.t12z.awip1202.tm00.grib2
download: nam.t12z.awip1275.tm00.grib2
degrib: nam.t12z.awip1275.tm00.grib2
download: nam.t12z.awip1213.tm00.grib2
degrib: nam.t12z.awip1213.tm00.grib2
download: nam.t12z.awip1254.tm00.grib2
degrib: nam.t12z.awphys18.tm00.grib2
download: nam.t12z.awip1224.tm00.grib2
download: nam.t12z.awip1260.tm00.grib2
degrib: nam.t12z.awip1224.tm00.grib2
download: nam.t12z.awip1218.tm00.grib2
degrib: nam.t12z.awip1260.tm00.grib2
download: nam.t12z.awphys13.tm00.grib2
degrib: nam.t12z.awip1218.tm00.grib2
download: nam.t12z.awphys31.tm00.grib2
degrib: nam.t12z.awip1254.tm00.grib2
download: nam.t12z.awphys19.tm00.grib2
degrib: nam.t12z.awphys13.tm00.grib2
degrib: nam.t12z.awphys19.tm00.grib2
download: nam.t12z.awphys69.tm00.grib2
download: nam.t12z.awip1210.tm00.grib2
download: nam.t12z.awip1203.tm00.grib2
download: nam.t12z.awip1219.tm00.grib2
degrib: nam.t12z.awphys69.tm00.grib2
download: nam.t12z.awphys54.tm00.grib2
degrib: nam.t12z.awip1210.tm00.grib2
degrib: nam.t12z.awphys31.tm00.grib2
degrib: nam.t12z.awip1219.tm00.grib2
degrib: nam.t12z.awip1203.tm00.grib2
download: nam.t12z.awip1217.tm00.grib2
degrib: nam.t12z.awphys54.tm00.grib2
download: nam.t12z.awphys51.tm00.grib2
degrib: nam.t12z.awip1217.tm00.grib2
download: nam.t12z.awphys00.tm00.grib2
degrib: nam.t12z.awphys00.tm00.grib2
download: nam.t12z.awphys21.tm00.grib2
degrib: nam.t12z.awphys51.tm00.grib2
download: nam.t12z.awphys72.tm00.grib2
download: nam.t12z.awphys17.tm00.grib2
degrib: nam.t12z.awphys21.tm00.grib2
download: nam.t12z.awphys48.tm00.grib2
download: nam.t12z.awip1281.tm00.grib2
degrib: nam.t12z.awphys72.tm00.grib2
degrib: nam.t12z.awphys17.tm00.grib2
degrib: nam.t12z.awip1281.tm00.grib2
download: nam.t12z.awip1222.tm00.grib2
degrib: nam.t12z.awip1222.tm00.grib2
download: nam.t12z.awphys60.tm00.grib2
degrib: nam.t12z.awphys48.tm00.grib2
download: nam.t12z.awip1209.tm00.grib2
download: nam.t12z.awip1251.tm00.grib2
download: nam.t12z.awip1230.tm00.grib2
degrib: nam.t12z.awip1251.tm00.grib2
degrib: nam.t12z.awip1209.tm00.grib2
degrib: nam.t12z.awip1230.tm00.grib2
download: nam.t12z.awip1215.tm00.grib2
degrib: nam.t12z.awip1215.tm00.grib2
download: nam.t12z.awip1232.tm00.grib2
degrib: nam.t12z.awip1232.tm00.grib2
download: nam.t12z.awphys25.tm00.grib2
degrib: nam.t12z.awphys60.tm00.grib2
download: nam.t12z.awphys26.tm00.grib2
download: nam.t12z.awphys07.tm00.grib2
degrib: nam.t12z.awphys26.tm00.grib2
degrib: nam.t12z.awphys25.tm00.grib2
download: nam.t12z.awip1206.tm00.grib2
download: nam.t12z.awip1216.tm00.grib2
download: nam.t12z.awphys29.tm00.grib2
degrib: nam.t12z.awphys07.tm00.grib2
degrib: nam.t12z.awip1206.tm00.grib2
download: nam.t12z.awphys34.tm00.grib2
degrib: nam.t12z.awip1216.tm00.grib2
degrib: nam.t12z.awphys29.tm00.grib2
download: nam.t12z.awip1228.tm00.grib2
degrib: nam.t12z.awphys34.tm00.grib2
download: nam.t12z.awphys81.tm00.grib2
degrib: nam.t12z.awip1228.tm00.grib2
download: nam.t12z.awip1278.tm00.grib2
degrib: nam.t12z.awip1278.tm00.grib2
download: nam.t12z.awphys27.tm00.grib2
degrib: nam.t12z.awphys27.tm00.grib2
download: nam.t12z.awphys01.tm00.grib2
download: nam.t12z.awphys84.tm00.grib2
degrib: nam.t12z.awphys81.tm00.grib2
degrib: nam.t12z.awphys01.tm00.grib2
download: nam.t12z.awphys32.tm00.grib2
degrib: nam.t12z.awphys84.tm00.grib2
download: nam.t12z.awip1245.tm00.grib2
degrib: nam.t12z.awip1245.tm00.grib2
download: nam.t12z.awphys12.tm00.grib2
download: nam.t12z.awip1211.tm00.grib2
degrib: nam.t12z.awphys32.tm00.grib2
degrib: nam.t12z.awip1211.tm00.grib2
download: nam.t12z.awphys06.tm00.grib2
degrib: nam.t12z.awphys12.tm00.grib2
download: nam.t12z.awphys36.tm00.grib2
download: nam.t12z.awip1207.tm00.grib2
degrib: nam.t12z.awphys36.tm00.grib2
download: nam.t12z.awip1225.tm00.grib2
download: nam.t12z.awphys42.tm00.grib2
degrib: nam.t12z.awip1207.tm00.grib2
degrib: nam.t12z.awip1225.tm00.grib2
download: nam.t12z.awphys63.tm00.grib2
degrib: nam.t12z.awphys06.tm00.grib2
degrib: nam.t12z.awphys42.tm00.grib2
download: nam.t12z.awip1233.tm00.grib2
degrib: nam.t12z.awip1233.tm00.grib2
download: nam.t12z.awphys02.tm00.grib2
degrib: nam.t12z.awphys63.tm00.grib2
download: nam.t12z.awip1242.tm00.grib2
degrib: nam.t12z.awphys02.tm00.grib2
download: nam.t12z.awphys39.tm00.grib2
degrib: nam.t12z.awip1242.tm00.grib2
download: nam.t12z.awphys24.tm00.grib2
download: nam.t12z.awip1229.tm00.grib2
degrib: nam.t12z.awphys39.tm00.grib2
download: nam.t12z.awip1201.tm00.grib2
degrib: nam.t12z.awphys24.tm00.grib2
degrib: nam.t12z.awip1201.tm00.grib2
download: nam.t12z.awip1257.tm00.grib2
degrib: nam.t12z.awip1257.tm00.grib2
download: nam.t12z.awphys28.tm00.grib2
degrib: nam.t12z.awip1229.tm00.grib2
download: nam.t12z.awip1223.tm00.grib2
degrib: nam.t12z.awip1223.tm00.grib2
download: nam.t12z.awphys14.tm00.grib2
degrib: nam.t12z.awphys28.tm00.grib2
download: nam.t12z.awphys23.tm00.grib2
degrib: nam.t12z.awphys14.tm00.grib2
download: nam.t12z.awphys22.tm00.grib2
degrib: nam.t12z.awphys23.tm00.grib2
download: nam.t12z.awphys35.tm00.grib2
degrib: nam.t12z.awphys22.tm00.grib2
download: nam.t12z.awip1226.tm00.grib2
degrib: nam.t12z.awphys35.tm00.grib2
Check: http://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/ nam 20180509 12z
degrib: nam.t12z.awip1226.tm00.grib2
close: /scratch/data/cdf/nam/218/20180509_1200

When we are trying to bring in data, it does not want to load into the GFE. See the error at the beginning of the script. Not sure what necessarily is going on here, and when model data does ever try to come in, it shows as an option in GFE, but does not show on the grids.

[Michael Foland]

It would appear that gfei script is not telling the IFPServer that the data is in. When I restart IFP, it shows just local data... and grids manually created, and does not bring in the models.

[Mike Romberg]

It appears that the netcdf4 library is having an issue with the template file. The gfe source has several .cdl files which are a text files describing an exmpty netcdf file. Each cdl is "compiled" into a netcdf binary file durring the build by the /usr/bin/ncgen program. The empyt netcdf file is then used as a starting point for each new model run. It is the file called template mentioned in the above output.

I wonder if your /usr/bin/ncgen is the right version? What does this command report on your system?

/usr/bin/ncgen --verion

[Michael Foland]

The version says:
netcdf library version 4.4.1.1 of Aug 4 2017 03:49:07

Also I found where you ingest the models, and I was looking at them to see if I could specify and add the RAP, RTMA, etc :) I think you should make a gui or something for different options :)

[Mike Romberg]

OK. Your netcdf seems to be the correct version. So, that is ruled out as being the problem. My guess is that something got corrupted under /scratch/data/cdf. There is a file in there called template that is built from the .cdl in the tree at compile time. It is used as the empty starting point for each new model run. Either it is corrupt or the named cdf file in the above traceback became corrupt durring ingest.

A really, really good way for the netcdf files to become corrupt is to shutdown the ingest script while it is downloading and degribbing.   This is kinda like shutting down your computer in the middle of a software update.  It is safe to shutdown the ingest script when it is "quiet" and just printing "checking..." every hour.

So, start the ingest script and just leave it running.   It is meant to be running all the time.   It is not meant as a quick grab data off nomads right now sorta thing.

   Since the ingest script does seem to run fine for days and weeks at a time for me, lets try resetting your configuration and try again.  To reset:

1. Remove everything under /scratch/data/cdf (rm -rf /scratch/data/cdf)
2. Do, a clean build of the gfe (make clean; make links; make; make install)
3. start the ingest script and keep it running at least until it has downloaded one model run and starts to print "checking..." every hour.

[Allan Diegan]

Decided to post this error on this ticket as it more appropriate for this ticket. This came up last night and again today.

[Mike Romberg]

The second error (you just posted) is a clear sign that a pickle database is corrupt. If you go to the /scratch/data/cdf/name/218 directory and do 'ls -a' you will see that there are more files in there than just the ones you see with a vanilla 'ls' command.

In particular, there is a '.grib' directory which is used to hold the grib files while they are being downloaded and being decoded.   After they are decoded they *should* be deleted.   The second kind of file is of the form .XXXXXXXX_XXX.pkl which match the file name of the netcdf database.  What these files hold are a record of which grib files from nomads have already been downloaded (so there is some way to tell which ones have been done and which ones may be new).

  The second error is due to a .pkl file being corrupt.   So, removing all of these .pkl files should get things back up and running.   Or simply do a 'rm -rf /scratch/data/cdf/nam' and start with a clean slate.  Which is probably even better.

    The first error I think may be caused by either something funky going on with your system clock (large time step) or unusually slow processing of the grib file.   The netcdf file seems to be shutdown too soon.  It is suppose to be closed after five minutes of inactivity.  But from the looks of things, it is being shut down right after it is created and opened.  I'm looking into that.   The part that is breaking is updating the netcdf files inventory.   Which would explain why the ifpServer thinks there are not many (or any) grids.

[Allan Diegan]

With the fresh install, I now have more grids loading then I ever did before. However, its still hit or miss which hours load. I continue to let it run, although correct me if I'm wrong, but the script does not care if its loading the hours in order right? For example it seems that it grabs the hours out of order and not, 01, 02, 03,...24, 25...etc. I'm curious to see if every hour of the NAM will actually load before the script starts looking for the 18Z run.

Where does the degrib process take place in the code? Also, I hope you have had some luck with the stuff I sent you. If you have any additional questions about that please let me know and I will help you or point you in the right direction.

[Mike Romberg]

The nam is hourly for the first 37 grids. I guess this is the initialization and then 36 forecast grids. Then it is every three hours to the end.

You are correct in that the part of the ingest script that downloads files does not care at all about forecast times.   It uses a simple process of finding a list of all files in the remote directory.   It then filters those named files down to a list that apply to the current expected model.

  This sublist of files may be complete or incomplete as grib files may still be in the process of being written tehre.   So, the ingest script starts to download them in no particular order and feed them into a decoder.   Since none of the grib files by themselves are complete enough to generate any gfe surface grids anyway, the order does not really matter.

    After a grib file has been downloaded and processed, a record that it has been done is saved into the pickle (.pkl) file mentioned earlier.   This database is used a bit later.   After the while list of files that were found is downloaded and sent to be decoded, the downloder script rests for five minutes or so.

      When it wakes from a rest it checks the whole huge list again and creates another list of files for this model run.   It also uses the list of files that have already been processed to locate any new files that still need to be downloaded and processed.   If it wakes up for something like 15 minutes without finding any new files for this model run it assumes that is all there is and then sleeps for an hour.

        All this code is in the libs/pylib/ifp/ingest directory.  The decoding stuff is in nomads.py.

          If you are not getting full nam runs ingested perhaps you network connection does not have enough bandwidth to pull all the grib files.   It is alot of data.  How long does it take your browser to download a single file?