Rounding bug in format-number() ?

Help
JMescheder
2010-09-01
2012-10-08
  • JMescheder

    JMescheder - 2010-09-01

    Hello,

    I have got problems with rounding errors in format-number(). This seems to
    affect all versions > 6.5.5

    Example:

    <xsl:value-of select="format-number($testval,'#.##0,')"/>
    1.234.567.890
    <xsl:value-of select="format-number($testval,'#.##0,#')"/>
    1.234.567.890,1
    <xsl:value-of select="format-number($testval,'#.##0,0')"/>
    1.234.567.890,1
    <xsl:value-of select="format-number($testval,'#.##0,##')"/>
    1.234.567.890,12
    <xsl:value-of select="format-number($testval,'#.##0,0#')"/>
    1.234.567.890,12
    <xsl:value-of select="format-number($testval,'#.##0,###')"/>
    1.234.567.890,12
    <xsl:value-of select="format-number($testval,'#.##0,00#')"/>
    1.234.567.890,12
    <xsl:value-of select="format-number($testval,'#.##0,####')"/>
    1.234.567.890,12
    <xsl:value-of select="format-number($testval,'#.##0,000#')"/>
    1.234.567.890,120
    <xsl:value-of select="format-number($testval,'#.##0,#####')"/>
    1.234.567.890,12
    <xsl:value-of select="format-number($testval,'#.##0,0000#')"/>
    1.234.567.890,1200
    <xsl:value-of select="format-number($testval,'#.##0,######')"/>
    1.234.567.890,12
    <xsl:value-of select="format-number($testval,'#.##0,00000#')"/>
    1.234.567.890,12000
    <xsl:value-of select="format-number($testval,'#.##0,#######')"/>
    1.234.567.890,1199999
    <xsl:value-of select="format-number($testval,'#.##0,000000#')"/>
    1.234.567.890,1199999
    <xsl:value-of select="format-number($testval,'#.##0,########')"/>
    1.234.567.890,11999989
    <xsl:value-of select="format-number($testval,'#.##0,0000000#')"/>
    1.234.567.890,11999989
    <xsl:value-of select="format-number($testval,'#.##0,#########')"/>
    1.234.567.890,119999886
    <xsl:value-of select="format-number($testval,'#.##0,00000000#')"/>
    1.234.567.890,119999886
    <xsl:value-of select="format-number($testval,'#.##0,##########')"/>
    1.234.567.890,1199998856
    <xsl:value-of select="format-number($testval,'#.##0,00########')"/>
    1.234.567.890,1199998856
    <xsl:value-of select="format-number($testval,'#.##0,000000000#')"/>
    1.234.567.890,1199998856

    Regards, J.Mescheder

     
  • Michael Kay

    Michael Kay - 2010-09-01

    You haven't said how $testval was initialised, but I imagine you did something
    like this:

    <xsl:variable name="testval" select="1234567890.12e0"/>

    in which case $testval will have a value that is the nearest xs:double value
    to 1234567890.12. Because no xs:double is exactly equal to this number, the
    nearest xs:double is probably something like 123456890.1199998856. The format-
    number() function will then format this value to as many decimal places as you
    request. So the rounding error is not coming from format-number(), it is
    coming from the string-to-double conversion that happened when the variable
    was initialized. If you need to avoid these problems, use xs:decimal rather
    than xs:double.

     
  • JMescheder

    JMescheder - 2010-09-01

    Thanks for the quick reply!

    Sorry, forgot to post the initialisation line:

    <xsl:variable name="testval">1234567890.12</xsl:variable>

    How should I alter the initialisation to avoid the problem?

     
  • JMescheder

    JMescheder - 2010-09-01

    Got it.

    <xsl:variable name="testval" as="xs:decimal">1234567890.12</xsl:variable>

    works as expected.

    Thanks for the hint!

     
  • Michael Kay

    Michael Kay - 2010-09-01

    I would write it as

    <xsl:variable name="testval" as="xs:decimal" select="1234567890.12"/>

    It's generally better to use the select attribute on xsl:variable rather than
    using child nodes, unless you actually want the value to be a tree structure.
    The as="xs:decimal" ensures that the value is a decimal rather than a double.
    In your code the value is actually a document node containing a text node;
    when you supply that as input to format-number it is atomized, giving an
    untypedAtomic value, and this is then converted to an xs:double.

     
  • JMescheder

    JMescheder - 2010-09-01

    Many thanks for your advice and explaination!

     

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks