Re: [Plplot-devel] Re: Embedded single precision in the matrix library

[Maurice L. <mj...@ga...>]

Alan W. Irwin writes:
 > Point well taken, but I have investigated further, and there is more to this
 > story that I didn't understand before.  From the above example $ y -
 > 3.123456 has access to the full ~17 digit precision of y before subtracting the
 > constant from it so there are actually still 10 significant digits left in
 > the result (note the last two "81" digits are incorrect because the internal
 > precision of y is roughly 17 digits.)
 > 
 > But look how the corresponding matrix result does not act this way.
 > 
 > pltcl> set tcl_precision 12
 > 12
 > pltcl> matrix a f 1 = {3.1234567890123456789}
 > a
 > pltcl> puts [expr [a 0] - 3.123456]
 > 7.89010000002e-07
 > 
 > There are 5 fewer significant digits in this result than for the y - const
 > result.
 > 
 > My mental model of what is going on here is as follows: unlike $y - const,
 > [a 0] - const only has access to tcl_precision digits of precision for
 > matrix values. In other words set tcl_precision effectively sets the
 > internal precision of matrix calculations rather than simply limiting the
 > number of digits emitted for results.

You're right, it's still not really right.  Will look into it some more.

-- 
Maurice LeBrun    mj...@ga...