Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

#373 Format bif does not round some small numbers

open
Mark Hessling
None
5
2012-08-27
2012-07-10
Felix Hofmann
No

Running under xp home sp3 and v3.5 & 3.6 of Regina
the format bif converts some small numbers to exponential
notation rather than rounding them
the follow snippet demonstrates the problem

say format(2.499999999,1,0); say
say format(2.500000000,1,0); say
say format(0.000000001,1,0) ' error'; say
say format(0.00000001,1,0) ' error'; say
say format(0.0000001,1,0) ' error'; say
say format(0.0000009,1,0) ' error'; say
say format(0.000001,1,0); say

Discussion

  • Mark Hessling
    Mark Hessling
    2012-08-27

    I've run each of the examples through the ANSI BIF tester: http://www.rexxla.org/cgi/ANSI/ansibifs.rex and Regina matches the results except for the first one, where Regina returns a value of 2 whereas ANSI returns 3. All tests done with the default NUMERIC DIGITS of 9.
    Now the first result is interesting; I believe Regina's answer of 2 to be correct. I will have to check the ANSI standard and the BIF tester code.

     
  • Mark Hessling
    Mark Hessling
    2012-08-27

    • assigned_to: nobody --> rexx
     
  • Mark Hessling
    Mark Hessling
    2012-08-28

    The ANSI BIF tester is correct and the answer to the first example is indeed 3. Regina does produce an answer of 3 if the OPTION; STRICT_ANSI is set.