#4854 fpintpart's precision check is dead code - floor/ceiling of large bfloats silently wrong, heap exhaustion for huge exponents

[David Scherfgen]

By Claude:

The exponent check in fpintpart (src/float.lisp) can never fire, so floor, ceiling, entier, fix and round of a bigfloat whose exponent exceeds fpprec (in bits) silently fabricate digits, and fail with raw Lisp errors or heap exhaustion for very large exponents. This is a regression of bug #2784.

With default fpprec: 16:

(%i1) floor(1.0b30);
(%o1) 1000000000000000002292438794240
(%i2) floor(1.0b30) - 10^30;
(%o2) 2292438794240

Only the leading ~17 digits are meaningful; the tail is the 2^44 quantization residue of the 56-bit mantissa, presented as exact integer digits.

For huge exponents, the internal (expt 2 (- m)) blows up. On SBCL, exponents beyond the maximum bignum length give an opaque caught Lisp error instead of the intended Maxima error:

(%i3) floor(1.0b100000000000);

Maxima encountered a Lisp error:

 can't represent result of left shift

For exponents whose bignum is representable but larger than the heap (roughly 1.0b2000000000 up to 1.0b60000000000 depending on dynamic-space size; e.g. floor(1.0b30000000000) needs a ~12 GiB integer), SBCL attempts the allocation and dies with heap exhaustion — the original #2784 failure mode. Other Lisps will behave differently but have no protection either.

Root cause: commit 576c7508 made fpintpart signal an error when exponent >= fpprec. Commit 41cc6dc5f (2016, for bug #3056) added skip-exponent-check-p so that internal callers (fpexp, fpsin) can bypass the check while external callers keep it. But the rewrite placed the guard in the branch where its condition is always false:

(let ((m (- fpprec exponent)))
  (if (plusp m)
      (quotient mantissa (expt 2 (- fpprec exponent)))
      ;; here exponent >= fpprec always holds, so the test below is never true:
      (if (and (not skip-exponent-check-p) (< exponent fpprec))
          (merror "~M doesn't have enough precision ...")
          (* mantissa (expt 2 (- m))))))

(< exponent fpprec) contradicts the branch it lives in, so the merror is unreachable and every caller behaves as if it passed :skip-exponent-check-p t. The regression test added in 41cc6dc5f (tests/rtest16.mac, the ceiling of the %e^-563501581931 expression) still passes, but only because pretty-good-floor-or-ceiling in nummod.lisp bails out on that input before any bigfloat reaches fpentier; a bare bigfloat argument goes straight through simp-floor's $bfloatp branch with no guard.

Suggested fix, restoring the semantics both commits agreed on (error iff exponent >= fpprec and the caller did not opt out):

(let ((m (- fpprec exponent)))
  (if (plusp m)
      (quotient mantissa (expt 2 m))
      (if skip-exponent-check-p
          (* mantissa (expt 2 (- m)))
          (merror "~M doesn't have enough precision to compute its integer part"
                  `((bigfloat ,fpprec) ,mantissa ,exponent)))))

Notes on fallout: with the check restored, floor(1.0b30) signals an error again, as in 5.35–5.37 (intended per 576c7508, but user-visible). fpration1 and exptbigfloat also call fpentier without the keyword, yet deliberately treat the bigfloat as an exact rational — fpentier should probably grow an optional skip flag passed as true from those two sites so that e.g. rat(1.0b30) keeps working, while the nummod.lisp callers stay strict. The fpentier calls in cpoly.lisp have tiny exponents and are unaffected. It would also be worth adding a bare floor/ceiling-of-bigfloat case to rtest16.mac, since the existing test no longer exercises this path.