expand FL_LST_IN_MAX_LNG

Developers
2012-11-27
2013-10-17
  • Seth McGinnis
    Seth McGinnis
    2012-11-27

    I got the following error using ncecat:

    ncecat: ERROR Total length of fl_lst_in from stdin exceeds 1000000 characters. 
    Possible misuse of feature. If your input file list is really this long, send 
    request to help@nco.sf.net to expand FL_LST_IN_MAX_LNG
    

    The use case here is that I'm concatenating observations from individual stations into a single file for a region that covers most of North America.  I've got nearly 40,000 stations in the region.  The filenames look like this:

    var.ID.start-end.nc
    

    ,  where var, start and end are 4 characters each, and ID is an 11-character string that's basically random.

    The length limit is not a show-stopper for me - I can easily use the "technique of last resort" suggested in the NCO User's Guide - but I thought I'd mention this use case to you just to note it as one where it's possible to exceed the length limit without doing something completely unreasonable.  The file that results is still a manageable size, and I if I were working on a larger domain, like the whole globe, I think that even if I were to trim the filenames down to just "stationID.nc", I'd still exceed the limit and be forced to use the symlink approach.

    Also note that I tried to email this message to help@nco.sf.net as directed by the error message, but it bounced.

     
  • Charlie Zender
    Charlie Zender
    2012-11-27

    Thanks for following the instructions. No one ever does that. Sorry the list doesn't exist. Future NCO versions will point to a list that does exist.

    My list implementation needs a maximum length to prevent buffer overflow attacks. But it can exceed 1MB if you need it to.

    Bottom line: Give me a list length that would fit your anticipated needs and I'll up the max length to fit it.
    cz

     
  • Seth McGinnis
    Seth McGinnis
    2012-11-30

    If you increase the list length to 10M characters, you'll have this use case covered comfortably but not excessively.  Thanks!

     
  • Charlie Zender
    Charlie Zender
    2012-12-07

    Done. In daily snapshot. Will be in next release.
    c