From: Michael Kay <mike@sa...>  20070318 19:10:23

If you don't want rounding errors for numbers with more than 16 digits, then don't use xs:double arithmetic, which is defined to work with a given precision (Saxon doesn't even have the option of using higher precision). Work instead with xs:integer or xs:decimal. The EXSLT math library is defined to work on xs:double as that's the only thing that was available in XSLT 1.0. One could define a similar library for xs:decimal or xs:integer, though I'm not sure which of the functions would be needed  I can't see anyone except numerical analysts wanting trigonometry functions in xs:decimal. Michael Kay http://www.saxonica.com/ > Original Message > From: saxonhelpbounces@... > [mailto:saxonhelpbounces@...] On Behalf > Of Abel Braaksma > Sent: 18 March 2007 17:56 > To: Mailing list for SAXON XSLT queries > Subject: [saxon] Unexpected rounding problems with > math:power(x, 0) and idiv with idivoperand gt 2^53 > > Hi List, > > I struggled with some inexplainable errors when calculating > from decimal to hexadecimal in my UUID post on the xsllist: > http://www.biglist.com/lists/xsllist/archives/200703/msg00434 > .html, for which I used the math:power EXSLT function of > Saxon. In the post, I resolved the issue explained below with > explicit casting of the math:power result to an xs:integer. > > I used the following functions to test erroneous behavior > when I suspected that math:power(xx, 0) crapped my results: > > <! using power fu that returns 1.0 > <xsl:function > name="f:pwr" as="xs:integer"> > <xsl:param name="nr" /> > <xsl:sequence select="$nr idiv math:power(16, 0)" /> > </xsl:function> > > <! using idiv without power fu > > <xsl:function name="f:dbl" as="xs:integer"> > <xsl:param name="nr" /> > <xsl:sequence select="$nr idiv 1.0" /> </xsl:function> > > The result of these are equal for all numbers below or equal to > 9007199254740994 (which happens to be 2^53, which happens to > be the highest integer representable in a IEEE754 floating > point double without rounding problems. > > When input is: 9007199254740995 then: > f:pwr >> 9007199254740996 > f:dbl >> 9007199254740995 > > When input is: 13393530584140003 > f:pwr >> 13393530584140004 > f:dbl >> 13393530584140003 > > When input is: 133935305841400005 > f:pwr >> 133935305841400000 > f:dbl >> 133935305841400005 > > > I know that the XPath standard does not require any processor > to work more correct than possible with IEEE 754 arithmetic. > I also know that the underlying language is Java and the JVM > will likely have its constraints. However, I was suprised > that the result was influenced by the math:power function, > and that the results are different when using floating points > in another way. I reckoned that math:power(xx, 0) would > return 1.0 (representable as a precise number with floating point > representation) and that any further calculation would equal > the calculations done with the number '1.0' itself. > > I did see some other differences with other values in > math:power, but I did not investigate it too much further. > When surrounding the > math:power() with a cast to xs:integer, all problems vanish. > I think I expected the result of math:power(int, int) to be > optimized to integer arithmetic. Not sure whether that is > feasible, though. > > Well, Michael, this time I don't have an obscure bug, I just > have an observation that struck me as odd, which isn't wrong > or a bug per se, but may raise questions when anybody > encounters it, which is why I felt to file it to the archives ;) > > Cheers, >  Abel Braaksma > http://www.nuntia.nl > >  >  > Take Surveys. Earn Cash. Influence the Future of IT Join > SourceForge.net's Techsay panel and you'll get the chance to > share your opinions on IT & business topics through brief > surveysand earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge > &CID=DEVDEV > _______________________________________________ > saxonhelp mailing list > saxonhelp@... > https://lists.sourceforge.net/lists/listinfo/saxonhelp 