in src/assembly/x86[-64]/arith.lisp in
(define-generic-arith-routine (- 10) ...)
one finds the following comment:
;; FIXME: This is screwed up.
;;; I can't figure out the flags on subtract. Overflow never gets
;;; set and carry always does. (- 0 most-negative-fixnum) can't be
;;; easily detected so just let the upper level stuff do it.
To unupscrew this, a "complement carry" is all that is needed:
(inst sub res y)
(inst jmp :no OKAY)
(inst cmc) ; carry has correct sign now
(inst rcr res 1)
(inst sar res 2) ; remove type bits
Please find attached a patch that implements this for both x86-64
To the one who wrote "I can't figure out the flags on subtract" and
for other curious party here is an explanation of why this works:
A good description of the behaviour of the flags can be found in the
manual "AMD64 Architecture Programmer's Manual Volume 1: Application
Programming" in chapter 3.1.4 "Flags Register". In essence, it says:
The overflow flag (OF) is set if the result of the subtraction doesn't
fit in a signed word, i.e., if it is larger than 2^63-1 or smaller than
-2^63. The carry flag (CF) is set if the subtraction results in a
borrow when the arguments are interpreted as unsigned words.
Let x and y be the arguments of the subtraction. Of the two cases where
OF = 1 let's first look at the one where x - y is too large to fit in a
signed word. Since both arguments are signed words this can only happen
if x >= 0 and y < 0, thus the sign bit of x is 0 and the sign bit of y
is 1. Thus, compared as unsigned numbers, x < y holds, the unsigned
subtraction results in a borrow and thus CF = 1.
A similar argument shows that if x - y is too small to fit in a signed
word, then CF = 0.
Thus: If OF = 1 then the sign of the result is the complement of CF.