#219 count(*) observations for better progress

Enhancement
open
David Benn
5
2013-03-16
2010-10-28
David Benn
No

The number of observations in the requested date range could be queried before the observations are read to:

1. Provide immediate information about HM obs are being read.
2. Better progress bar update via chunking of obs reads.
3. If too many obs, giving a warning or cutting off at some maximum (like Zapper does).

Discussion

  • Sara Beck
    Sara Beck
    2011-01-07

    This one summarizes what both Tracker #2866594 and #3001661 were about so I closed those.

     
  • Sara Beck
    Sara Beck
    2011-01-07

    I also merged Tracker #3080054 which said:

    Before reading an AID dataset, we could get a count of data items first to:

    1. Properly update the progress bar as we do for file reads by knowing how many lines there are.
    2. Ask the user whether to load all N observations, especially for "load-all-data" mode. If it is unlikely that we will be able to load all data items, we should warn the user and possibly limit the number of obs as Zapper does (to ~20,000).

    Before doing so, we should time how long this operation takes on large datasets with no SQL 'where' clause.