Re: [Plplot-devel] Re: Embedded single precision in the matrix library

[Maurice L. <mj...@ga...>]

Alan W. Irwin writes:
 > Point well taken, but I have investigated further, and there is more to this
 > story that I didn't understand before.  From the above example $ y -
 > 3.123456 has access to the full ~17 digit precision of y before subtracting the
 > constant from it so there are actually still 10 significant digits left in
 > the result (note the last two "81" digits are incorrect because the internal
 > precision of y is roughly 17 digits.)
 > 
 > But look how the corresponding matrix result does not act this way.
 > 
 > pltcl> set tcl_precision 12
 > 12
 > pltcl> matrix a f 1 = {3.1234567890123456789}
 > a
 > pltcl> puts [expr [a 0] - 3.123456]
 > 7.89010000002e-07
 > 
 > There are 5 fewer significant digits in this result than for the y - const
 > result.
 > 
 > My mental model of what is going on here is as follows: unlike $y - const,
 > [a 0] - const only has access to tcl_precision digits of precision for
 > matrix values. In other words set tcl_precision effectively sets the
 > internal precision of matrix calculations rather than simply limiting the
 > number of digits emitted for results.

The story becomes more interesting.  I believe the two representations
(variable or tclMatrix) would've given the same result in pre-Tcl8.0.  The
reason being that in the old Tcl, "everything's a string".  But no longer --
the variable that you create with "set" is "dual-ported", meaning it has both
a binary and string representation.  The binary one is used in expression
evaluations and so is more accurate.

I've been poring over the documentation, and I think it's feasible to change
tclMatrix to emit objects instead of strings, which seems to be the right way
to really fix this.  Also will help a lot with performance.  The object is
equipped with procedure calls to convert it to strings when necessary.
Unfortunately this is a pretty major undertaking, and not one I care to get
into right now.

So as for a stopgap measure, I'll change it to carry 12 digits of precision,
both internally and externally.  I tried 17 but didn't like the result.  E.g.

pltcl> matrix c f 3 = {3.2 4.5 5.7}
c
pltcl> puts [c *]
3.2000000000000002 4.5 5.7000000000000002

which the tclvars man page warns about.

-- 
Maurice LeBrun    mj...@ga...