Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project!

## [saxon] Unexpected rounding problems with math:power(x, 0) and idiv with idiv-operand gt 2^53

 [saxon] Unexpected rounding problems with math:power(x, 0) and idiv with idiv-operand gt 2^53 From: Abel Braaksma - 2007-03-18 17:57:34 ```Hi List, I struggled with some inexplainable errors when calculating from decimal to hexadecimal in my UUID post on the xsl-list: http://www.biglist.com/lists/xsl-list/archives/200703/msg00434.html, for which I used the math:power EXSLT function of Saxon. In the post, I resolved the issue explained below with explicit casting of the math:power result to an xs:integer. I used the following functions to test erroneous behavior when I suspected that math:power(xx, 0) crapped my results: The result of these are equal for all numbers below or equal to 9007199254740994 (which happens to be 2^53, which happens to be the highest integer representable in a IEEE-754 floating point double without rounding problems. When input is: 9007199254740995 then: f:pwr >> 9007199254740996 f:dbl >> 9007199254740995 When input is: 13393530584140003 f:pwr >> 13393530584140004 f:dbl >> 13393530584140003 When input is: 133935305841400005 f:pwr >> 133935305841400000 f:dbl >> 133935305841400005 I know that the XPath standard does not require any processor to work more correct than possible with IEEE 754 arithmetic. I also know that the underlying language is Java and the JVM will likely have its constraints. However, I was suprised that the result was influenced by the math:power function, and that the results are different when using floating points in another way. I reckoned that math:power(xx, 0) would return 1.0 (representable as a precise number with floating point representation) and that any further calculation would equal the calculations done with the number '1.0' itself. I did see some other differences with other values in math:power, but I did not investigate it too much further. When surrounding the math:power() with a cast to xs:integer, all problems vanish. I think I expected the result of math:power(int, int) to be optimized to integer arithmetic. Not sure whether that is feasible, though. Well, Michael, this time I don't have an obscure bug, I just have an observation that struck me as odd, which isn't wrong or a bug per se, but may raise questions when anybody encounters it, which is why I felt to file it to the archives ;) Cheers, -- Abel Braaksma http://www.nuntia.nl ```

 [saxon] Unexpected rounding problems with math:power(x, 0) and idiv with idiv-operand gt 2^53 From: Abel Braaksma - 2007-03-18 17:57:34 ```Hi List, I struggled with some inexplainable errors when calculating from decimal to hexadecimal in my UUID post on the xsl-list: http://www.biglist.com/lists/xsl-list/archives/200703/msg00434.html, for which I used the math:power EXSLT function of Saxon. In the post, I resolved the issue explained below with explicit casting of the math:power result to an xs:integer. I used the following functions to test erroneous behavior when I suspected that math:power(xx, 0) crapped my results: The result of these are equal for all numbers below or equal to 9007199254740994 (which happens to be 2^53, which happens to be the highest integer representable in a IEEE-754 floating point double without rounding problems. When input is: 9007199254740995 then: f:pwr >> 9007199254740996 f:dbl >> 9007199254740995 When input is: 13393530584140003 f:pwr >> 13393530584140004 f:dbl >> 13393530584140003 When input is: 133935305841400005 f:pwr >> 133935305841400000 f:dbl >> 133935305841400005 I know that the XPath standard does not require any processor to work more correct than possible with IEEE 754 arithmetic. I also know that the underlying language is Java and the JVM will likely have its constraints. However, I was suprised that the result was influenced by the math:power function, and that the results are different when using floating points in another way. I reckoned that math:power(xx, 0) would return 1.0 (representable as a precise number with floating point representation) and that any further calculation would equal the calculations done with the number '1.0' itself. I did see some other differences with other values in math:power, but I did not investigate it too much further. When surrounding the math:power() with a cast to xs:integer, all problems vanish. I think I expected the result of math:power(int, int) to be optimized to integer arithmetic. Not sure whether that is feasible, though. Well, Michael, this time I don't have an obscure bug, I just have an observation that struck me as odd, which isn't wrong or a bug per se, but may raise questions when anybody encounters it, which is why I felt to file it to the archives ;) Cheers, -- Abel Braaksma http://www.nuntia.nl ```
 Re: [saxon] Unexpected rounding problems with math:power(x, 0) and idiv with idiv-operand gt 2^53 From: Dimitre Novatchev - 2007-03-18 18:44:27 ```> I know that the XPath standard does not require any processor to work > more correct than possible with IEEE 754 arithmetic. I also know that > the underlying language is Java and the JVM will likely have its > constraints. However, I was suprised that the result was influenced by > the math:power function This has nothing to do with math:power. Take for example: 9007199254740995 idiv xs:double(1) the result is: 9007199254740996 -- Cheers, Dimitre Novatchev --------------------------------------- Truly great madness cannot be achieved without significant intelligence. --------------------------------------- To invent, you need a good imagination and a pile of junk ------------------------------------- You've achieved success in your field when you don't know whether what you're doing is work or play On 3/18/07, Abel Braaksma wrote: > Hi List, > > I struggled with some inexplainable errors when calculating from decimal > to hexadecimal in my UUID post on the xsl-list: > http://www.biglist.com/lists/xsl-list/archives/200703/msg00434.html, for > which I used the math:power EXSLT function of Saxon. In the post, I > resolved the issue explained below with explicit casting of the > math:power result to an xs:integer. > > I used the following functions to test erroneous behavior when I > suspected that math:power(xx, 0) crapped my results: > > > > > > > > > > > > > > The result of these are equal for all numbers below or equal to > 9007199254740994 (which happens to be 2^53, which happens to be the > highest integer representable in a IEEE-754 floating point double > without rounding problems. > > When input is: 9007199254740995 then: > f:pwr >> 9007199254740996 > f:dbl >> 9007199254740995 > > When input is: 13393530584140003 > f:pwr >> 13393530584140004 > f:dbl >> 13393530584140003 > > When input is: 133935305841400005 > f:pwr >> 133935305841400000 > f:dbl >> 133935305841400005 > > > I know that the XPath standard does not require any processor to work > more correct than possible with IEEE 754 arithmetic. I also know that > the underlying language is Java and the JVM will likely have its > constraints. However, I was suprised that the result was influenced by > the math:power function, and that the results are different when using > floating points in another way. I reckoned that math:power(xx, 0) would > return 1.0 (representable as a precise number with floating point > representation) and that any further calculation would equal the > calculations done with the number '1.0' itself. > > I did see some other differences with other values in math:power, but I > did not investigate it too much further. When surrounding the > math:power() with a cast to xs:integer, all problems vanish. I think I > expected the result of math:power(int, int) to be optimized to integer > arithmetic. Not sure whether that is feasible, though. > > Well, Michael, this time I don't have an obscure bug, I just have an > observation that struck me as odd, which isn't wrong or a bug per se, > but may raise questions when anybody encounters it, which is why I felt > to file it to the archives ;) > > Cheers, > -- Abel Braaksma > http://www.nuntia.nl > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > saxon-help mailing list > saxon-help@... > https://lists.sourceforge.net/lists/listinfo/saxon-help > ```
 Re: [saxon] Unexpected rounding problems with math:power(x, 0) and idiv with idiv-operand gt 2^53 From: Abel Braaksma - 2007-03-18 23:28:38 ```Dimitre Novatchev wrote: > This has nothing to do with math:power. > > Take for example: > > 9007199254740995 idiv xs:double(1) > > the result is: > > 9007199254740996 I see now, thanks. I used 1.0 as comparison, which is an xs:decimal (though I thought it was an xs:double, I need to reread my textbook, I think). -- Abel ```
 Re: [saxon] Unexpected rounding problems with math:power(x, 0) and idiv with idiv-operand gt 2^53 From: Michael Kay - 2007-03-18 19:10:23 ```If you don't want rounding errors for numbers with more than 16 digits, then don't use xs:double arithmetic, which is defined to work with a given precision (Saxon doesn't even have the option of using higher precision). Work instead with xs:integer or xs:decimal. The EXSLT math library is defined to work on xs:double as that's the only thing that was available in XSLT 1.0. One could define a similar library for xs:decimal or xs:integer, though I'm not sure which of the functions would be needed - I can't see anyone except numerical analysts wanting trigonometry functions in xs:decimal. Michael Kay http://www.saxonica.com/ > -----Original Message----- > From: saxon-help-bounces@... > [mailto:saxon-help-bounces@...] On Behalf > Of Abel Braaksma > Sent: 18 March 2007 17:56 > To: Mailing list for SAXON XSLT queries > Subject: [saxon] Unexpected rounding problems with > math:power(x, 0) and idiv with idiv-operand gt 2^53 > > Hi List, > > I struggled with some inexplainable errors when calculating > from decimal to hexadecimal in my UUID post on the xsl-list: > http://www.biglist.com/lists/xsl-list/archives/200703/msg00434 > .html, for which I used the math:power EXSLT function of > Saxon. In the post, I resolved the issue explained below with > explicit casting of the math:power result to an xs:integer. > > I used the following functions to test erroneous behavior > when I suspected that math:power(xx, 0) crapped my results: > > name="f:pwr" as="xs:integer"> > > > > > > > > > > The result of these are equal for all numbers below or equal to > 9007199254740994 (which happens to be 2^53, which happens to > be the highest integer representable in a IEEE-754 floating > point double without rounding problems. > > When input is: 9007199254740995 then: > f:pwr >> 9007199254740996 > f:dbl >> 9007199254740995 > > When input is: 13393530584140003 > f:pwr >> 13393530584140004 > f:dbl >> 13393530584140003 > > When input is: 133935305841400005 > f:pwr >> 133935305841400000 > f:dbl >> 133935305841400005 > > > I know that the XPath standard does not require any processor > to work more correct than possible with IEEE 754 arithmetic. > I also know that the underlying language is Java and the JVM > will likely have its constraints. However, I was suprised > that the result was influenced by the math:power function, > and that the results are different when using floating points > in another way. I reckoned that math:power(xx, 0) would > return 1.0 (representable as a precise number with floating point > representation) and that any further calculation would equal > the calculations done with the number '1.0' itself. > > I did see some other differences with other values in > math:power, but I did not investigate it too much further. > When surrounding the > math:power() with a cast to xs:integer, all problems vanish. > I think I expected the result of math:power(int, int) to be > optimized to integer arithmetic. Not sure whether that is > feasible, though. > > Well, Michael, this time I don't have an obscure bug, I just > have an observation that struck me as odd, which isn't wrong > or a bug per se, but may raise questions when anybody > encounters it, which is why I felt to file it to the archives ;) > > Cheers, > -- Abel Braaksma > http://www.nuntia.nl > > -------------------------------------------------------------- > ----------- > Take Surveys. Earn Cash. Influence the Future of IT Join > SourceForge.net's Techsay panel and you'll get the chance to > share your opinions on IT & business topics through brief > surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge > &CID=DEVDEV > _______________________________________________ > saxon-help mailing list > saxon-help@... > https://lists.sourceforge.net/lists/listinfo/saxon-help ```
 Re: [saxon] Unexpected rounding problems with math:power(x, 0) and idiv with idiv-operand gt 2^53 From: Abel Braaksma - 2007-03-18 23:25:25 ```Michael Kay wrote: > If you don't want rounding errors for numbers with more than 16 digits, then > don't use xs:double arithmetic, which is defined to work with a given > precision (Saxon doesn't even have the option of using higher precision). > Work instead with xs:integer or xs:decimal. > > The EXSLT math library is defined to work on xs:double as that's the only > thing that was available in XSLT 1.0. One could define a similar library for > xs:decimal or xs:integer, though I'm not sure which of the functions would > be needed - I can't see anyone except numerical analysts wanting > trigonometry functions in xs:decimal. Thanks for the clarification, I'll consider xs:decimal more often. I see now that I was focused on xs:double / xs:float as being the counterparts of their IEEE 754 equivalents, and that, when using just '1.0' instead, I unconsciously used an xs:decimal (where I thought that the correct outcome was due to optimization to xs:integer or something). About trigonometry, you are probably right, except that making charts in SVG might be a common task for XSLT, though I guess that float/double will give enough precision. However, with math:power as an aid for calculating hex/dec/bin/oct conversions of any kind, 16-digit dec numbers will suffer from this rounding error. But of course, there are easy and plenty workarounds. -- Abel ```