## RE: [saxon] Rounding problems with decimal format

 RE: [saxon] Rounding problems with decimal format From: Michael Kay - 2003-01-08 15:11:03 ```XSLT defines format-number() in terms of the JDK DecimalFormat specification, and this is the way the JDK works, though it isn't well documented. The algorithm is called "half-even rounding", the idea is that if you round .5 to the nearest even number then you will round up 50% of the time and down the other 50%. This means that the statistical mean of a set of numbers remains unchanged, and means in banking applications that there's no bias to one party in the transaction. So there are good reasons for it, even though it's not always the rounding algorithm that you want. Michael Kay Software AG home: Michael.H.Kay@... work: Michael.Kay@...=20 > -----Original Message----- > From: saxon-help-admin@...=20 > [mailto:saxon-help-admin@...] On Behalf Of=20 > Haaland =D8istein > Sent: 07 January 2003 11:14 > To: 'saxon-help@...' > Subject: [saxon] Rounding problems with decimal format >=20 >=20 > I have some problems with decimal-format which rounds in a=20 > rather inconsistent way, a least to me. >=20 > xslt: >=20 > xmlns:xsl=3D"http://www.w3.org/1999/XSL/Transform"=20 > extension-element-prefixes=3D"saxon" > xmlns:saxon=3D"http://icl.com/saxon">; >=20 > group-separator=3D"" NaN=3D"" />=09 >=20 > =09 >=20 > | > select=3D"format-number('1.00015', '#0.0###', 'ecdefault')"/>=09 > | > select=3D"format-number('1.00025', '#0.0###', 'ecdefault')"/> =09 > | =09 > select=3D"format-number('1.00035', '#0.0###', 'ecdefault')"/>=09 > | > select=3D"format-number('1.00045', '#0.0###', 'ecdefault')"/>=09 > | > select=3D"format-number('1.00055', '#0.0###', 'ecdefault')"/>=09 > | > select=3D"format-number('1.00065', '#0.0###', 'ecdefault')"/>=09 > | > > =09 >=20 > Result: >=20 > >=20 > | > 1.0002 > | > 1.0002 > | > 1.0004 > | > 1.0004 > | > 1.0006 > | > 1.0006 > | >=20 > I would have expected it to be, or aleast wanted it to be: >=20 > | > 1.0002 > | > 1.0003 > | > 1.0004 > | > 1.0005 > | > 1.0006 > | > 1.0007 > | =20 >=20 > I'm using the saxon version 6.5.2 and jdk 1.3.1. >=20 > Oistein >=20 >=20 >=20 >=20 > ------------------------------------------------------- > This SF.NET email is sponsored by: > SourceForge Enterprise Edition + IBM + LinuxWorld =3D Something=20 > 2 See! http://www.vasoftware.com=20 > _______________________________________________ > saxon-help mailing list > saxon-help@...=20 > https://lists.sourceforge.net/lists/listinfo/s>; axon-help >=20 ```

 [saxon] Rounding problems with decimal format From: - 2003-01-07 11:19:35 ```I have some problems with decimal-format which rounds in a rather inconsistent way, a least to me. xslt: ; | | | | | | | Result: | 1.0002 | 1.0002 | 1.0004 | 1.0004 | 1.0006 | 1.0006 | I would have expected it to be, or aleast wanted it to be: | 1.0002 | 1.0003 | 1.0004 | 1.0005 | 1.0006 | 1.0007 | I'm using the saxon version 6.5.2 and jdk 1.3.1. Oistein ```
 Re: [saxon] Rounding problems with decimal format From: Trevor Nash - 2003-01-08 12:56:56 ```On Tue, 7 Jan 2003 12:14:18 +0100, you wrote: >I have some problems with decimal-format which rounds in a rather >inconsistent way, a least to me. We've had this one before, see 17/10/02. To quote Mike: >It's not at all unreasonable that numbers ending in .5 should sometimes >round up and sometimes down. Firstly, the actual value is in binary and >is likely to be slightly more than or less than the decimal value as >written. Secondly, rounding algorithms often deliberately have rules >giving .5 an equal probability of rounding up or down, to eliminate >statistical bias when averaging a large set of numbers. This isn't peculiar to XSLT: in general when you are using fractional values you have to choose your algorithms carefully. Trevor Nash Melvaig Software Engineering Limited voice: +44 (0) 1445 771 271=20 email: tcn@... web: http://www.melvaig.co.uk ```
 RE: [saxon] Rounding problems with decimal format From: Michael Kay - 2003-01-08 15:11:03 ```XSLT defines format-number() in terms of the JDK DecimalFormat specification, and this is the way the JDK works, though it isn't well documented. The algorithm is called "half-even rounding", the idea is that if you round .5 to the nearest even number then you will round up 50% of the time and down the other 50%. This means that the statistical mean of a set of numbers remains unchanged, and means in banking applications that there's no bias to one party in the transaction. So there are good reasons for it, even though it's not always the rounding algorithm that you want. Michael Kay Software AG home: Michael.H.Kay@... work: Michael.Kay@...=20 > -----Original Message----- > From: saxon-help-admin@...=20 > [mailto:saxon-help-admin@...] On Behalf Of=20 > Haaland =D8istein > Sent: 07 January 2003 11:14 > To: 'saxon-help@...' > Subject: [saxon] Rounding problems with decimal format >=20 >=20 > I have some problems with decimal-format which rounds in a=20 > rather inconsistent way, a least to me. >=20 > xslt: >=20 > xmlns:xsl=3D"http://www.w3.org/1999/XSL/Transform"=20 > extension-element-prefixes=3D"saxon" > xmlns:saxon=3D"http://icl.com/saxon">; >=20 > group-separator=3D"" NaN=3D"" />=09 >=20 > =09 >=20 > | > select=3D"format-number('1.00015', '#0.0###', 'ecdefault')"/>=09 > | > select=3D"format-number('1.00025', '#0.0###', 'ecdefault')"/> =09 > | =09 > select=3D"format-number('1.00035', '#0.0###', 'ecdefault')"/>=09 > | > select=3D"format-number('1.00045', '#0.0###', 'ecdefault')"/>=09 > | > select=3D"format-number('1.00055', '#0.0###', 'ecdefault')"/>=09 > | > select=3D"format-number('1.00065', '#0.0###', 'ecdefault')"/>=09 > | > > =09 >=20 > Result: >=20 > >=20 > | > 1.0002 > | > 1.0002 > | > 1.0004 > | > 1.0004 > | > 1.0006 > | > 1.0006 > | >=20 > I would have expected it to be, or aleast wanted it to be: >=20 > | > 1.0002 > | > 1.0003 > | > 1.0004 > | > 1.0005 > | > 1.0006 > | > 1.0007 > | =20 >=20 > I'm using the saxon version 6.5.2 and jdk 1.3.1. >=20 > Oistein >=20 >=20 >=20 >=20 > ------------------------------------------------------- > This SF.NET email is sponsored by: > SourceForge Enterprise Edition + IBM + LinuxWorld =3D Something=20 > 2 See! http://www.vasoftware.com=20 > _______________________________________________ > saxon-help mailing list > saxon-help@...=20 > https://lists.sourceforge.net/lists/listinfo/s>; axon-help >=20 ```