Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

#14 Calculation error

Win32 v4.0
open
Mike Arrison
9
2004-09-29
2004-09-29
Anonymous
No

Hi,

I run the GraphCalc 4.0.1 on Win2000 SP4 and have the follwing problem:
If I do the simple substraction like
123456.5 - 123456.1
I would expect 0.4 as a result but I get
0.39999999999418

OK it is very close but that is not good enough.

Any clue where could that come from?

Regards.

Romain Cazalis
r.cazalis@gestech.com.au

Discussion

  • Mike Arrison
    Mike Arrison
    2004-09-29

    Logged In: YES
    user_id=668541

    More examples
    --------------
    123456.5 - 123456.1=0.39999999999418
    1000000.1-1000000=0.09999999997671
    1000000000-1000000000=0
    1000000000.1-1000000000=0.10000002384186
    1000000000000.1-1000000000000=0.0999755859375

     
  • Mike Arrison
    Mike Arrison
    2004-09-29

    • priority: 5 --> 9
     
  • Jeremy
    Jeremy
    2005-01-10

    Logged In: YES
    user_id=594550

    Computers cannot store 0.1 exactly in binary. 0.4 is
    probably the same. This is no excuse though, this problem
    should be fixed.

     
  • Logged In: NO

    I have similar problem,

    try do a (5.5-5.4)*1000 = 99.9999999999997

    not nice...

     
  • Logged In: NO

    This type of problem is inherent in the standard floating-point data type. (jsm's explanation is a little too simple, but I'm not sure I can explain it perfectly either) Without creating a string-based numerical data type, only occasional tricks can be used to solve this, like doing the integer and fraction parts as separate calculations, or for trig functions, finding the remainder when you divide by 2*pi. More complicated tricks could bemultiple levels of calculation within expressions to reduce error, or somehow splitting up even the fractional part.

    My example that involves showing how the earlier tricks aren't perfect: ((((100*pi)+.1)/pi)-100)*pi. Doing that in multiple steps involving the use of the ans variable produced a much worse number.

    Try typing in just: 0.10000000000008.

    But these tricks, even though they reduce predictability of accuracy, are probably worth whatever effort it would take to add the code. I'm not sure that they should be up to string-based floats, though.

    But they could at least make sure that the ans variable was string based. That'd be easy.