Re: [Plplot-devel] Re: Embedded single precision in the matrix library

[Maurice L. <mj...@ga...>]

Alan W. Irwin writes:
 > Thanks for your fix.  At first I thought there was a 12-digit limit to the
 > matrix precision, but that is just the default value of tcl_precision.  If I
 > set it to 17 I get full double-precision results.
 > 
 > BTW, tcl_precision doesn't seem to have universal application.  My tests
 > indicated the internal precision of ordinary variables in pltcl was 17
 > digits regardless of what tcl_precision value was set.  For example with
 > the default (12) value of tcl_precision I get the following results:
 > 
 > pltcl> set y 3.12345678901234567890
 > 3.12345678901234567890
 > pltcl> puts [expr $y - 3.123456]
 > 7.89012345681e-07
   1 234567890ab

..i.e. 12 digits of precision.  Further examples:

pltcl> set tcl_precision 6
6
pltcl> puts [expr $y - 3.123456] 
7.89012e-07
pltcl> set tcl_precision 4
4
pltcl> puts [expr $y - 3.123456] 
7.89e-07

 > I would prefer matrix to act the same way regardless ot tcl_precision.
 > Therefore, could you just set the matrix precision to 17 digits when PLFLT
 > is double and 7 digits otherwise?  The other alternative I am considering is
 > to set tcl_precision to 17 in each of the examples, but I prefer not to do
 > that if there is an automatic PLFLT-style solution you can use for
 > libmatrix.

I really think tying it to tcl_precision is the right thing to do.
It doesn't have any effect on internal representation, just the number
of digits emitted when converting the PLFLT to a string.

-- 
Maurice LeBrun    mj...@ga...