[WWdevel] New diagnostic capabilities in function answer checker

[Davide P. C. <dp...@un...>]

Folks:

A few weeks ago, I mentioned an idea for helping problem writers  
analyze whether their functions were using good input ranges and  
tolerance values.  I finally had the chance to put something together  
for that, and have updated the Parser package to allow you to get  
diagnostic information for the Formula answer checker.  Since this is  
used by the Parser-based version of fun_cmp(), it is also available  
to it.

The diagnostics include a graph of the student answer and the correct  
answer (one one set of axes), plus graphs of the absolute and  
relative errors between the two.  It also shows the points actually  
used for the function comparison and the numeric values of the  
functions and errors.  Any of these items can be enabled/disabled  
separately.

Here is the message from the CVS log that says how to use it:

______________________________________________________________

Added a new experimental diagonstic function for the function answer
checker.  When enabled, it will produce graphs of the correct answer,
the student answer, and the absolute and relative errors, and will
list the data points used in the comparison, plus the numerical values
of the results and errors.

To enable the diagnostic, use

    ANS(fun_cmp($f,diagnostics=>1));

Note that only single-variable functions can be graphed at the moment,
so if you are using a multi-variable check, you need to disable the
graphing.  To do this use

    ANS(fun_cmp($f,vars=>['x','y'],diagnostics=>[showGraphs=>0]));

The diagnostic mode is only available for the Parser-based versions of
the function checker, and (of course) with the native Parser objects
as well:

    ANS(Formula($f)->cmp(diagnostics=>1));

There are now Context settings to control the diagnostics, which can
be set through Context()->diagnostics->set().  For example

    Context()->diagnostics->set(formulas=>{showGraphs=>0});

would turn off graphs for all functions comparisons.  Some of the
other values you can set are:

    formulas => {
      showTestPoints => 1,        # show the test points and function  
values
      showRelativeErrors => 1,    # show the relative errors for the  
student answer
      showAbsoluteErrors => 1,    # show the absolute errors for the  
student answer
      showGraphs => 1,            # show the various graphs
      graphRelativeErrors => 1,   #  show the relative error graph
      graphAbsoluteErrors => 1,   #  show the absolute error graph
      clipRelativeError => 5,     #  don't show relative errors above 5
      clipAbsoluteError => 5,     #  don't show absolute errors above 5
      plotTestPoints => 1,        #  include dots at the test points
      combineGraphs => 1,         #  show correct and student graphs  
in one image
    },
    graphs => {
      divisions => 75,       # the number of data points to plot
      limits => [-2,2],      # the lower and upper limit of the plot
                             #  (taken from the function limits if  
not provided)
      size => 250,           # pixel size of the image (could be  
[width,height])
      grid => [10,10],       # number of grid lines in each direction
      axes => [0,0],         # where to put axes relative to origin
    }

Any of these can be set in the Context(), or in the answer checker
itself.  If you set diagnostics to an array reference, the entries in
the array refer to element of the formulas hash.  If you set
diagonstics to a hash reference, then you can set values in either the
formulas or graphs hashes, as in:

    ANS(Formula($f)->cmp(diagnostics=>{
      formulas => {showAbsoluteErrors=>0},
      graphs => {size=>300, divisions=>100},
    }));

If you want all function checkers to show diagnostics, use

    Context()->diagonstics->set(formulas=>{show=>1});

The image file names are modified to include the current time so that
the names will be unique.  This avoids problems with the browser cache
showing a old image when a new one has been generated.  But this also
means that the temporary image directory will fill up fast, so you may
need to empty it if you use the diagnostic images frequently.

This is just a first attempt at a diagnostic feature.  I think it will
help when you are not sure if the tolerances are set properly, or if
you think a student answer should be markes correct but isn't, as it
will point out which point(s) are not being accepted.
______________________________________________________________

Note that the diagnostics are added to the page via the "warn"  
command, so the screen will show pink when they are enabled.  As a  
side effect, the diagnostics will go into the httpd error log as  
well, and since they are pretty long, this can increase the size of  
your log file if you use it a lot.  Is there a better way to add this  
kind of data to the output?  I can't just use TEXT, because it is  
generated when the answer checker runs (not when the page is set up),  
and I think that it is too late for that (though I didn't actually  
try it).

Anyway, give it a try and see what you think.  It may help diagnose  
those times when it is not clear why a student's answer is (or is  
not) being accepted when it shouldn't be.

Davide