#219 count(*) observations for better progress

Enhancement
open
5
2013-03-16
2010-10-28
David Benn
No

The number of observations in the requested date range could be queried before the observations are read to:

1. Provide immediate information about HM obs are being read.
2. Better progress bar update via chunking of obs reads.
3. If too many obs, giving a warning or cutting off at some maximum (like Zapper does).

Discussion

  • Sara Beck

    Sara Beck - 2011-01-07

    This one summarizes what both Tracker #2866594 and #3001661 were about so I closed those.

     
  • Sara Beck

    Sara Beck - 2011-01-07

    I also merged Tracker #3080054 which said:

    Before reading an AID dataset, we could get a count of data items first to:

    1. Properly update the progress bar as we do for file reads by knowing how many lines there are.
    2. Ask the user whether to load all N observations, especially for "load-all-data" mode. If it is unlikely that we will be able to load all data items, we should warn the user and possibly limit the number of obs as Zapper does (to ~20,000).

    Before doing so, we should time how long this operation takes on large datasets with no SQL 'where' clause.

     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks