cclib-devel

[cclib-devel] progress support

[Adam T. <a-t...@st...>]

So I added an example progress class called TextProgress which basically 
prints out bars saying how much progress has been made, and made necessary 
changes to G03 parser to allow use of this class.

If you want to use it, first create an instance, and pass that instance to the 
G03 class along with the filename:

progress=TextProgress()
p=G03(filename,progress)
p.parse()

I changed the __init__ function of G03 to accept any amount of arguments, and 
pass that onto the baseclass. See the SVN diffs for more details. This allows 
the user of cclib to not specify a parser and continue as before.

Unfortunately, not all of this message is good news. Calling G03 with a 
TextParser adds a 20-25% overhead: 4 seconds becomes 5ish, 21 seconds becomes 
more like 26, etc. I think the problem is that it calls the 
progress.update(step) far too frequently. It may make more sense to identify 
the slower parts of the parsing class, and only add progress.update there, 
but I don't know for sure.

Any ideas on how to improve this?

Adam



-- 
Want a web browser that is fully customizable? 

Go to http://www.mozilla.org/products/firefox/

to download the newest version of Firefox and 
its available extentions.

Re: [cclib-devel] progress support

[Noel O'B. <no...@ca...>]

On Thu, 2006-03-09 at 12:03 -0800, Adam Tenderholt wrote:
> So I added an example progress class called TextProgress which basically 
> prints out bars saying how much progress has been made, and made necessary 
> changes to G03 parser to allow use of this class.
> 
> If you want to use it, first create an instance, and pass that instance to the 
> G03 class along with the filename:
> 
> progress=TextProgress()
> p=G03(filename,progress)
> p.parse()

Got it. Works fine, although doesn't play well with the logger, but you
can turn the logger off or get it to write to a file.

> I changed the __init__ function of G03 to accept any amount of arguments, and 
> pass that onto the baseclass. See the SVN diffs for more details. This allows 
> the user of cclib to not specify a parser and continue as before.
Yes, and seems to be just as fast if you don't specify a progress.

> Unfortunately, not all of this message is good news. Calling G03 with a 
> TextParser adds a 20-25% overhead: 4 seconds becomes 5ish, 21 seconds becomes 
> more like 26, etc. I think the problem is that it calls the 
> progress.update(step) far too frequently.
I don't know where the slow step is. I've added a call to random so that
it only calls it every so many steps, but there isn't a massive
improvement.

>  It may make more sense to identify 
> the slower parts of the parsing class, and only add progress.update there, 
> but I don't know for sure.
> 
> Any ideas on how to improve this?
Not really. I think this is the price you pay. A bit like Heisenberg's
uncertainty principle - by observing the progress you change it. :-) If
we could identify more clearly the slow step - for example, is it the
call to tell()? - then we could think about how to improve it. I tried
various things to speed up the textprogress, but they only made a minor
difference.

BTW, you may be interested in:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/299207
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/168639

Regards,
 Noel

Re: [cclib-devel] progress support

[Adam T. <a-t...@st...>]

> Got it. Works fine, although doesn't play well with the logger, but you
> can turn the logger off or get it to write to a file.

Yeah. This is done with calling G03 with a third argument of logging.ERROR or 
such.

> I don't know where the slow step is. I've added a call to random so that
> it only calls it every so many steps, but there isn't a massive
> improvement.

I also added optional arguments to parse so that you can specify how "fine" or 
"course" you want the update to be. The first option is the fine argument, 
which is used for checks against random() inside "slow" parts like parsing 
overlap and coefficients. The second option is the course argument, which is 
used for checks that aren't "slow".

> Not really. I think this is the price you pay. A bit like Heisenberg's
> uncertainty principle - by observing the progress you change it. :-) If
> we could identify more clearly the slow step - for example, is it the
> call to tell()? - then we could think about how to improve it. I tried
> various things to speed up the textprogress, but they only made a minor
> difference.

I agree that it is probably the price we pay. I've tried numerous things to 
speed it up, but nothing seems to be too significant. I think I'd rather know 
the progress and add on (at most) a few seconds, and of course, it's 
ultimately up to the developer using cclib.

> BTW, you may be interested in:
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/299207
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/168639

I didn't exactly take their work, but I did figure out how to make the text 
progress work like is expected. I haven't finished every part of it, but it 
should still be useful. Of course, the text progress is more an example for 
people to follow. I just wanted to establish the API used for any progress 
classes. Basically two functions need to be implemented:

1) initialize, which expects nsteps and text
2) update, which expects step and text

I've also added a few test files for ADF, although I haven't started working 
on the parser. Such a daunting task!

Adam