Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

FL_LST_IN_MAX_LNG too small?

Developers
2014-07-11
2014-07-25
  • Christine Smit
    Christine Smit
    2014-07-11

    Hi! I was trying to use ncra on 10 years worth of hourly data files, which is 876972 files. Each file's full path is 192 characters long. I pipe the input file list to stdin, which means I'm sending 17 million characters in. I'm getting an error from NCO that says:

    ncra: ERROR Total length of fl_lst_in from stdin exceeds 10000000 characters. Possible misuse of feature. If your input file list is really this long, post request to developer's forum (http://sf.net/projects/nco/forums/forum/9831) to expand FL_LST_IN_MAX_LNG

    We actually have 30 years of data for this product, which means we could attempt to send about 50 million characters in to stdin. When you say "misuse of feature," what do you suggest that we do instead? Break up the average into chunks and then average the chunks?

     
  • Charlie Zender
    Charlie Zender
    2014-07-11

    Your usage is not a mis-use. The warning is in case users accidentally pipe garbage to stdin. I will expand the size of allowed filename lists, or figure out a way to make it completely dynamic, and post again when that is ready to test. We are happy to support large workflows, we just want to keep the memory profile as small as possible for other users too. No need to change your methodology if you can wait a few days.

     
  • Charlie Zender
    Charlie Zender
    2014-07-12

    Never mind. # hours = 87600 not 876972. factor of 10, meaning ~50 MB max, yes?

     
  • Charlie Zender
    Charlie Zender
    2014-07-12

    OK, the new limit will suffice for 30 yrs of hourly datasets each 192 bytes long. Will be in NCO 4.4.5 or compile daily snapshot yourself now. Please let me know if problems continue...

     
  • Christine Smit
    Christine Smit
    2014-07-14

    Whoops, you are right! I meant ~500 million characters, not 50 million for 30 years. Thanks!

     
  • Christine Smit
    Christine Smit
    2014-07-14

    Wait a minute. I think I must have mis-copied the number of files. There should be about 365x24x10 files in 10 years, which is about 87,600 files. At 192 characters a file, that's about 17 million characters. So 30 years is about 50 million characters, not 500 million. Sorry. Duh.

    We'll give the update a try and see what happens. I'm not sure how well our system will be able to handle a process using that many files.

    Thanks!

     
    Last edit: Christine Smit 2014-07-14
  • Christine Smit
    Christine Smit
    2014-07-17

    We are going to test out your daily build on our development box, but we'll obviously have to go with a release for our operations boxes. Do you have any idea when 4.5.5 will be released? The main site says June, but I'm guessing that you aren't really going to wait until next year, give the frequency of releases.

     
  • Charlie Zender
    Charlie Zender
    2014-07-25

    Hi Christine,
    Sorry for the delay. Have been traveling. Expect to release 4.4.5 in first half of August. Updated homepage with some features (not many new) that will be in it.
    c