Re: Test Harness Status

[Brian G. <bri...@da...>]

Jim Kring wrote:

> One of the main issues at the moment is that when I run the testers I 
> get slightly different results (primarily for the top entries).  I 
> have been running the tester on LV 7.0 and Windows XP.  I have reduced 
> the problems by disabling my network adapter and shutting down 
> anti-virus and other services.  However, there are still some problems.

Unfortunately I believe that you have found the limitation of the 
measurement system and perhaps the top entries are equal in 
performance?  Since we are using the built in MS timer we must run the 
code N times and average to get the actual time per iteration.  N must 
be chosen so that it is large enough such that the 1 msec resolution 
(and overhead of reading the MS) timer becomes insignificant as compared 
to the actual time per iteration.  But as N gets larger, the total time 
increases which gives a greater chance that the CPU may get hijacked for 
another Windows task.  You have minimized this by shutting down services 
but there will always be something.  (Don't move the mouse during testing)

If you have a DAQ card available, you could use one of the 
counter/timers to get a 20MHz timestamp which solves the resolution 
issue, but there is still the issue of  not knowing how much of the time 
for one iteration is due to reading the timer.  The attached VI shows 
the variability.

Christian introduced the concept of picking the fastest run and not 
averaging which I can agree with as long as memory leaks and other 
issues that could cause worsening performance over time are caught by 
another test.  If so, you could use the test harness as it is to get the 
VI's that are performing equally and then we could pick the one test 
that was most representative of the task and run it many times with the 
hardware timer.  The winner would be the one with the single fastest time?

Or since the top performers may be equal in performance, you could use 
one of the other criteria as the tie break

> Would it help if I made copies of all the entries available to those 
> of you who want to work on the test harness?  I could password protect 
> them all, first.

I've been using different versions of my own VI's so I don't need copies 
for development purposes.