SourceForge has been redesigned. Learn more.
Close

#67 Strange results in simple operations with floating point

closed
Simon
None
5
2009-01-11
2009-01-09
No

When doing simple math operations with numbers in floating point, galculator output strange results very near to the exact result, but still not the correct result.

Example:
1000,11 - 1000,10 = 0,00999999999999

This bug was reported on Debian - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=459456

Discussion

  • Simon

    Simon - 2009-01-11

    This is an numerical issue. The result 0.01 can not be represented in an exact way by base 2 floating point numbers (see for example http://docs.python.org/tutorial/floatingpoint.html\). Latest 1.3.3 release contains a work around for this problem. Thanks for reporting.

     
  • Simon

    Simon - 2009-01-11
    • assigned_to: nobody --> chimaira
    • status: open --> closed
     
  • Nobody/Anonymous

    The error persists on 1.3.4: 1*(0,5-0,4-0,1) → -2,77555756156e-17

     
  • Simon

    Simon - 2009-03-04

    Thank you for your comment. The 1000.11-1000.10 problem was a numerical issue raised by a conversion specific to galculator to fit display requirements. That one could be resolved by tuning the corresponding conversion code. However, the 1*(0,5-0,4-0,1) inaccuracy is a problem inherent to IEEE floating point numbers. octave, matlab all give you the same result.

    It would be possible to get the correct result by supporting galculator's arithmetic core with a multiple precision library such as gmp. however, there are no plans towards such a big change at the moment; galculator is (only) a double precision.calculator.

    thanks, simon

     

Log in to post a comment.