#4968 Comparing a float or bigfloat against an integer converts the integer to a float, so equal is wrong and max/min are swapped

[David Scherfgen]

equal(1.0e16, 10000000000000001) returns true, max returns the smaller of the two, and equality is not transitive

Maxima 5.49post, SBCL 2.6.7.

The double 1.0e16 has the exact value 10000000000000000, so the two numbers below are distinct integers one apart. Comparison appears to convert the integer to a double, after which they collide.

(%i1) display2d:false$
(%i2) rationalize(1.0e16);
(%o2) 10000000000000000
(%i3) is(1.0e16 < 10000000000000001);
(%o3) false
(%i4) is(10000000000000001 > 1.0e16);
(%o4) false
(%i5) is(equal(1.0e16, 10000000000000001));
(%o5) true
(%i6) is(equal(1.0e16, 9999999999999999));
(%o6) true
(%i7) is(equal(10000000000000001, 9999999999999999));
(%o7) false
(%i8) max(1.0e16, 10000000000000001);
(%o8) 1.0e16
(%i9) min(1.0e16, 10000000000000001);
(%o9) 10000000000000001

Expected: (%o3) and (%o4) should be true; (%o5) and (%o6) should be false; (%o8) should be 10000000000000001 and (%o9) should be 1.0e16.

(%o5), (%o6) and (%o7) together break transitivity of equal: one float is reported equal to two integers that are not equal to each other. (%o8) and (%o9) are exchanged — max returns the smaller value and min the larger.

Bigfloats are affected the same way, and the outcome depends on fpprec:

(%i10) block([fpprec:16], bfloat(10^20)-10^20);
(%o10) 0.0b0
(%i11) block([fpprec:16], is(bfloat(10^20) < 10^20+1));
(%o11) false
(%i12) block([fpprec:16], is(equal(bfloat(10^20), 10^20+1)));
(%o12) true
(%i13) block([fpprec:40], is(bfloat(10^20) < 10^20+1));
(%o13) true

(%o10) shows the bigfloat is exactly 10^20 at that precision, so (%o11) and (%o12) are wrong; (%o13) shows the same comparison is correct once fpprec is raised.