#14 Calculation error

[Anonymous]

Hi,

I run the GraphCalc 4.0.1 on Win2000 SP4 and have the follwing problem:
If I do the simple substraction like
123456.5 - 123456.1
I would expect 0.4 as a result but I get
0.39999999999418

OK it is very close but that is not good enough.

Any clue where could that come from?

Regards.

Romain Cazalis
r.cazalis@gestech.com.au

[Mike Arrison]

Logged In: YES
user_id=668541

More examples
--------------
123456.5 - 123456.1=0.39999999999418
1000000.1-1000000=0.09999999997671
1000000000-1000000000=0
1000000000.1-1000000000=0.10000002384186
1000000000000.1-1000000000000=0.0999755859375

[Jeremy]

Logged In: YES
user_id=594550

Computers cannot store 0.1 exactly in binary. 0.4 is
probably the same. This is no excuse though, this problem
should be fixed.

[Nobody/Anonymous]

Logged In: NO

I have similar problem,

try do a (5.5-5.4)*1000 = 99.9999999999997

not nice...

[Nobody/Anonymous]

Logged In: NO

This type of problem is inherent in the standard floating-point data type. (jsm's explanation is a little too simple, but I'm not sure I can explain it perfectly either) Without creating a string-based numerical data type, only occasional tricks can be used to solve this, like doing the integer and fraction parts as separate calculations, or for trig functions, finding the remainder when you divide by 2*pi. More complicated tricks could bemultiple levels of calculation within expressions to reduce error, or somehow splitting up even the fractional part.

My example that involves showing how the earlier tricks aren't perfect: ((((100*pi)+.1)/pi)-100)*pi. Doing that in multiple steps involving the use of the ans variable produced a much worse number.

Try typing in just: 0.10000000000008.

But these tricks, even though they reduce predictability of accuracy, are probably worth whatever effort it would take to add the code. I'm not sure that they should be up to string-based floats, though.

But they could at least make sure that the ans variable was string based. That'd be easy.