SourceForge has been redesigned. Learn more.

#67 Strange results in simple operations with floating point


When doing simple math operations with numbers in floating point, galculator output strange results very near to the exact result, but still not the correct result.

1000,11 - 1000,10 = 0,00999999999999

This bug was reported on Debian -


  • Simon

    Simon - 2009-01-11

    This is an numerical issue. The result 0.01 can not be represented in an exact way by base 2 floating point numbers (see for example\). Latest 1.3.3 release contains a work around for this problem. Thanks for reporting.

  • Simon

    Simon - 2009-01-11
    • assigned_to: nobody --> chimaira
    • status: open --> closed
  • Nobody/Anonymous

    The error persists on 1.3.4: 1*(0,5-0,4-0,1) → -2,77555756156e-17

  • Simon

    Simon - 2009-03-04

    Thank you for your comment. The 1000.11-1000.10 problem was a numerical issue raised by a conversion specific to galculator to fit display requirements. That one could be resolved by tuning the corresponding conversion code. However, the 1*(0,5-0,4-0,1) inaccuracy is a problem inherent to IEEE floating point numbers. octave, matlab all give you the same result.

    It would be possible to get the correct result by supporting galculator's arithmetic core with a multiple precision library such as gmp. however, there are no plans towards such a big change at the moment; galculator is (only) a double precision.calculator.

    thanks, simon


Log in to post a comment.