Re: [Lib2geom-devel] performance tests

[mathog <ma...@ca...>]

On 07-Nov-2013 14:09, Johan Engelen wrote:

> I highly recommend using Ninja for building instead of Make.

Not broken, don't fix.

> I'd like some input in how to improve the performance test framework. I
> believe we want separate .cpp files for each performance test. Each
> performance test should take a couple of 100 ms. And then each test
> should be done a couple of times (three times currently) to get some
> averaging going.

Very hard to generalize the best way to do performance testing on a 
library.

Assume we are talking about purely algorithmic testing, so that 
interference from screen synchronization events is eliminated.

The biggest issue you are likely to run into is that small very fast 
routines are extremely difficult to measure accurately.  As a rule of 
thumb, if the execution time within a routine is of the same order of 
magnitude as a typical function call, then the measurement will be poor.

The other issue you will hit is that the order of function tests can 
dramatically change the measurement.  This happens because depending on 
how the program executes data and code is progressively forced out of 
the fastest memory into slower and slower memory, in a really bad case, 
all the way into virtual memory swapping to disk.  These memory effects 
cover orders of magnitude in speed, which can totally swamp any intended 
measurements of algorithm speed.

All else being equal then, you want the test code to beat on a single 
routine very many times (at least thousands for fast routines, fewer for 
slow routines), before moving on to test the next routine.  Within one 
routine test loop it is OK to vary the parameters, so long as the method 
for doing so is itself very fast (relative to the execution speed within 
the routine).

3 is too small a number for any test.  Consider what happens when you 
run
an arbitrary fast test program 3 times right after the system boots up.  
The first run time will consist almost entirely of the OS pulling in 
libraries and other codes before it even starts to run the program.  The 
2nd and 3rd runs, assuming everything stays in memory, will not have 
this delay.  Wait a couple of hours and run it again.  Is it still in 
memory?  This is crude, but one way around that issue is to run the test 
program once without timing it, then start the timing loop.

Regards,

David Mathog
ma...@ca...
Manager, Sequence Analysis Facility, Biology Division, Caltech