[WWdevel] Re: extraAnswerEvaluators.pl

[Davide P.C. <dp...@un...>]

John and Gavin:

I'm rerouting this to the developer mailing list, since it now relates 
to feature decisions.

On Aug 24, 2005, at 12:19 AM, John Jones wrote:

> Davide P. Cervone wrote:
>
>>> For the num_cmp() evaluator, setting 'tol'=>0.5 gives the result
>>>   Entered    Answer Preview    Correct        Result
>>>   0            0        1/sqrt(7)    correct
>>> and taking out the tol gives
>>>   Entered    Answer Preview    Correct        Result
>>>   0.25        0.25        1/sqrt(7)    incorrect
>>>
>>> For the number_list_cmp evaluator, with the 'tol' option I get
>>>   Entered    Answer Preview    Correct    Result
>>>     -0, 0       -0,0    0, 0    correct
>>> and without,
>>>   Entered    Answer Preview    Correct            Result
>>>   -0.25, 0.25   -0.25,0.25    -0.37796.., 0.37796..    incorrect
>>
>>
>> The reason that the parser shows the student's answer as 0 is that 
>> when it stringifies a real number, it shows it as zero if it is 
>> "equal" to zero according to the current equality conditions.  Since 
>> your answers actually ARE equal to zero (within an absolute tolerance 
>> of 0.5), they get printed as zeros.  When a RELATIVE tolerance is 
>> used, there is a special tolerance for zero that gets used for 
>> checking against zero.  This is not the case for absolute tolerance, 
>> because you are specifying exactly how close you want the answer to 
>> be.
>
> I am not so sure it is good to simplify a student's answer to 0 for a 
> few reasons.  It exposes some of webwork's internal tricks to the 
> students.  In general, other answers are not simplified in this way.  
> For example, 1.0001 is not simplified to 1.  And, it would be 
> confusing to a student to see their answer not register.  I tried 
> setting tolType=>'strict' in num_cmp and taking the default setting.  
> Then, I submitted 0.09, webwork told me I submitted 0.
>

Well, first of all, tolType should be one of 'relative' or 'absolute' 
from what I can see.  (I am unable to find a reference to tolType being 
'strict' anywhere.)  num_cmp doesn't check the value of tolType to make 
sure it is legal, and perhaps it should.  What happens when you tolType 
to something other than 'relative' or 'absolute' is somewhat random, as 
some tests are for tolType equal to 'relative' and others are for 
'absolute' and both assume if it is not the one checked, then it is the 
other.  So when set to 'strict' you will sometimes get the absolute 
branch of the test and other times the relative branch.  For the 
example above, you end up getting an absolute test with the tolerance 
set to 1, so the answer 0.09 is properly considered to be 0.

The intent of changing the student answer to zero was to change answers 
of the form 1E-13 (which are confusing to some students) to 0.  In the 
case were RELATIVE tolerances are used, this is in fact the effect, 
since the relative checks have a special and much more restricted test 
for zero than absolute checks do (this is when the zeroLevel and 
zeroLevelTol values come into play), so only values less than 1E-12 get 
affected.  But when people are setting absolute tolerances on the order 
of 1 or .5, the behavior is a bit less satisfying.  I think the real 
problem with these examples is that the absolute tolerance is 
artificially large.

On the other hand, since it seems that it is easy to get absolute 
tolerances accidentally, I think it might be a good idea if the problem 
author were made aware of this by the fact that small answers are being 
treated as zero, otherwise he or she will be surprised when a student's 
answer of 0 is counted as correct for an answer of .09 (as happened in 
Gavin's problem).  The fact that the Parser shows you what is happening 
could be considered a GOOD thing in terms of problem authoring.  
Certainly it alerted Gavin to the error in his problem.

Perhaps changing to zero should be suppressed when absolute checks are 
being made, but I still think it works for relative checks.  I had 
considered using the tolerance when stringifying the number so that it 
would appear with the number of digits that where appropriate for the 
tolerance, so that an absolute tolerance of .01 would print 1.0001 as 
1.00 and relative tolerance of 1 percent would print 10001 as 10000, 
but couldn't decide what a tolerance of .05 meant in this setting, and 
thought that perhaps it was too slick.  But it has always bothered me 
that the answers showed large numbers of digits as though they were 
significant, when they aren't.  I'd be interested in hearing what Mike, 
Arnie and Sam have to say about these issues.

Davide