Re: [WWdevel] Subtractive cancellation

[Davide P.C. <dp...@un...>]

> I would vote for keeping the overhead low and the processing as simple 
> and straight forward as possible.  Since perl uses double precision, 
> errors are quite rare but certainly not so rare that they don't show 
> up.

Double precision, while it sometimes helps, is not enough to prevent 
it.  (My examples of 16 digit number are ALREADY double-precision 
numbers, that's 16 DECIMAL digits not binary digits.)  In the cases 
where subtractive cancellation are going to occur, more digits often 
don't help, since you just get more digits that cancel and are still 
left with just a couple of the least-significant digits.  After all, if 
the answer is really SUPPOSED to be zero, and it's ending up as 3.5E-14 
because of round-off errors due to holding only a fixed number of 
digits (and not because that's what the answer REALLY is supposed to 
be), then if you had twice as many digits, your round-off errors would 
be occurring twice as far from the decimal place, and you'd still end 
up with a result with the same precision in the end.

I've seen it happen probably 6 times this year.  Certainly not a huge 
amount, but it's something that could be detected and avoided 
automatically.

> At least they are reasonably understandable to someone who knows a 
> little numerical analysis.

My experience is that it's not something most people are going to think 
of.

> It's possible a more complicated system, in addition to possibly 
> slowing things down, might have unanticipated actions which will be 
> harder for people to decipher.

I agree with that in terms of the more aggressive "convert  to zero" 
idea, but in terms of discarding test points based on this, I'm not 
sure how that would happen.  (Unless EVERY test point had subtractive 
cancellation, and so not enough points could be found, but the problem 
developer might want to know that the function involved is numerically 
unstable, and I have ideas in mind for that anyway.)

Once the checker comes up with the test points, the rest of the test is 
the same as usual.  It's only in choosing the points in the first place 
that the cancellation is taken in to account (in my original 
suggestion).

> I very much like your idea of providing a numeric stability test.  I 
> think the place to put that test would be on the "Edit" problem page 
> (maybe renamed to "Edit and Test").

One thing to keep in mind is that the problem can include many answer 
checkers, and you may only want to be looking at one at a time (the 
current diagnostics include graphs of the functions, and other things 
that take up considerable space, so you might not want to see that for 
every function if you are trying to analyze only one answer blank.  So 
this may need to be tied to the answers more tightly, and so might need 
information that is not available to the editor (which is why I 
suggested the problem page).

We may also want to include a way of selecting what kinds of tests to 
perform (I have several in mind) and whether to include the graphs, and 
so on.  The interface to this needs to be thought out in more detail.

> Since I assume to use your tests, the problem has to load the parser, 
> you could have a bold message that says "Testing methods only 
> available to problems using the parser" if used on an old problem.  
> That way more people might be made aware of the availability and use 
> of the parser.

Since the function checker now calls the Parser behind the scenes 
unless you request the old answer macros (I don't remember what the 
default setting is for that), the diagnostics are available for all the 
old problems as well as new explicitly parser-based ones.  You don't 
need to load Parser.pl explicitly.

Just add diagnostics=>1 to a fun_cmp() call and submit an answer to 
that answer blank and you'll see the results.

Davide