#957 format(1.2345678E-12,,2) = 0.00000000000123

None
open
nobody
none
5
2013-11-22
2010-11-09
No

The format function returns the wrong result for very small numbers with enough digits. E.g.,

format( 1.234567E-11 ,,2) = 0.00 -- ok
format( 1.2345678E-11 ,,2) = 0.00 -- ok
format( 1.234567E-12 ,,2) = 0.00 -- ok
format( 1.2345678E-12 ,,2) = 0.00000000000123 -- wrong
format( 1.234567E-13 ,,2) = 0.000000000000123 -- wrong
format( 1.2345678E-13 ,,2) = 0.000000000000123 -- wrong

Discussion

  • DangerDan

    DangerDan - 2010-11-22

    Depends on value of Numeric Digits {digits()}
    Numeric Digits 12 --all of these formats are correct =0.00
    Numeric Digits 4 --all of these formats are wrong =1.24E-??, where ??=11 to 13
    Numeric Digits 9 --(the default), behavior as initially described

     
  • Rick McGuire

    Rick McGuire - 2013-11-22
    • Pending work items: --> none
    • Group: v4.0 --> None
     


Anonymous

Cancel  Add attachments





Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks