#13 Calculation error with fractions and floor

[Grósz Dániel]

I don't know whether you follow reports a http://www.getjar.com/products/6371/Calc . I can reproduce the following bug reported there on my Motorola V3xx:

Try the following:
2.6 ENTER 9 * 3.4 -
this gives exactly 20. Now
mathmiscintfloor
and the result is 19.
Also if you type frac it gives 1 !!!!!
Why?????

[Roar Lauritzsen]

Logged In: YES
user_id=1189103
Originator: NO

Decimal fractions such as 2.6 and 3.4 cannot be represented exactly in binary (they are 2.9999999(hex) and 3.6666666(hex) respectively). After the calculation the result you are viewing is probably 19.9999999, but it is rounded to 20 in the display, since you can only view 16 digits.

[You may further argue that if you subtract 20 from the seen result you get exactly 0, and not something like 1.64e-19 as should perhaps have been the case. That is because some "magic rounding" has been added to the subtract operation for this specific and very common case (i.e. "if a-b < a*1e.18 then result=0"). The same kind of magic rounding is not applied to "floor".]

[Grósz Dániel]

Logged In: YES
user_id=1192308
Originator: YES

OK, I understand. In effect, especially if we know this behavior, it cannot cause problems. On the other hand, couldn't be that rounding algorithm applied to cases like this?
(Btw I like very much your program. It's very comfortable and powerful.)