Re: [Plplot-devel] Re: Embedded single precision in the matrix library

[Alan W. I. <ir...@be...>]

On Tue, 15 Jan 2002, Maurice LeBrun wrote:

> Alan W. Irwin writes:
>  > Thanks for your fix.  At first I thought there was a 12-digit limit to the
>  > matrix precision, but that is just the default value of tcl_precision.  If I
>  > set it to 17 I get full double-precision results.
>  >
>  > BTW, tcl_precision doesn't seem to have universal application.  My tests
>  > indicated the internal precision of ordinary variables in pltcl was 17
>  > digits regardless of what tcl_precision value was set.  For example with
>  > the default (12) value of tcl_precision I get the following results:
>  >
>  > pltcl> set y 3.12345678901234567890
>  > 3.12345678901234567890
>  > pltcl> puts [expr $y - 3.123456]
>  > 7.89012345681e-07
>    1 234567890ab
>
> ..i.e. 12 digits of precision.

Point well taken, but I have investigated further, and there is more to this
story that I didn't understand before.  From the above example $ y -
3.123456 has access to the full ~17 digit precision of y before subtracting the
constant from it so there are actually still 10 significant digits left in
the result (note the last two "81" digits are incorrect because the internal
precision of y is roughly 17 digits.)

But look how the corresponding matrix result does not act this way.

pltcl> set tcl_precision 12
12
pltcl> matrix a f 1 = {3.1234567890123456789}
a
pltcl> puts [expr [a 0] - 3.123456]
7.89010000002e-07

There are 5 fewer significant digits in this result than for the y - const
result.

My mental model of what is going on here is as follows: unlike $y - const,
[a 0] - const only has access to tcl_precision digits of precision for
matrix values. In other words set tcl_precision effectively sets the
internal precision of matrix calculations rather than simply limiting the
number of digits emitted for results.

>  > I would prefer matrix to act the same way regardless ot tcl_precision.
>  > Therefore, could you just set the matrix precision to 17 digits when PLFLT
>  > is double and 7 digits otherwise?  The other alternative I am considering is
>  > to set tcl_precision to 17 in each of the examples, but I prefer not to do
>  > that if there is an automatic PLFLT-style solution you can use for
>  > libmatrix.
>
> I really think tying it to tcl_precision is the right thing to do.
> It doesn't have any effect on internal representation, just the number
> of digits emitted when converting the PLFLT to a string.

That would be okay with me, but as you can see from the above example it
is affecting the (effective) internal precision, and not just the digits
being emitted.

Hope you can get this matrix problem sorted out.

Alan