From: <lut...@fr...> - 2005-02-13 16:20:16
|
Dear SBCL developers, under the subject "x86-64 (aka AMD64, EMT-64) support" Christophe Rhodes wrote on 2005-01-06: > There is still plenty of stuff which could do with work on the > x86-64 platform [...] the disassembler is somewhat selective in > its understanding of the instruction set [...] I would like to help improving the disassembler. I have investigated its current state and done some experiments, and would like to propose my ideas. As these would further cement a hack in the disassembler and I don't know whether someone has conflicting plans for its development, I would like to get some feedback before putting more work into this. The deficiency I would like to address is the incomplete treatment of the REX prefix, namely the disassembler often outputs one of the first 8 register names when the instruction references one of the second 8 registers, e.g. for base and index registers in effective addresses or even for such simple cases as "XOR reg, reg" or "MOV reg, imm". E.g. in (disassemble 'string=): ; 6C5: 4D31F6 XOR R14, RSI ; Should be XOR R14, R14. Additionally, operand sizes sometimes leak from one instruction to the next. Instead of simply complicating the instruction format definitions and extending prefilter and printer functions I would like to suggest somewhat larger changes. Currently, the rex bits are taken into account by two prefilters, prefilter-word-reg and prefilter-reg/mem and by extending the instruction formats, e.g.: (sb!disassem:define-instruction-format (reg-no-width 8 :default-printer '(:name :tab reg)) ... (reg :field (byte 3 0) :type 'word-reg) ...) (sb!disassem:define-instruction-format (rex-reg-no-width 16 :default-printer '(:name :tab reg)) (rex :field (byte 4 4) :value #b0100) ... (reg :fields (list (byte 3 8) (byte 4 0)) :type 'word-reg) ...) (defun prefilter-word-reg (value dstate) (declare (type (or full-reg list) value)) (if (atom value) value (let ((reg (first value)) (rex.wrxb (second value))) ... ;; Code setting the REG-WIDTH property in DSTATE ...))) This, and prefilter-width, already (ab)uses the prefilters to communicate information from a prefilter of one instruction field to the prefilters/ printers of others. (This is the hack I mean.) They use the PROPERTIES slot of the DSTATE object for this. I believe that this solution -- to use side-effects of prefilters -- nevertheless is the best way to go, but IMHO it can be made more complete and somewhat more elegant as follows: As the PROPERTIES slot of the DSTATE is used only in the x86 and the x86-64 ports, throw it away. Instead put in a slot INST-PROPERTIES that is set to nil at the start of each iteration in the loop in map-segment-instructions. This makes it possible to have several prefilters set properties and let these properties accumulate for use by all following prefilters/printers of the same instruction, without the current contortions in prefilter-width to clean up (which, besides, doesn't work for instruction formats that don't have a width field). Also, I suggest to separate the fields that have prefilters setting properties from the ones using them where possible. E.g. change the instruction format definition shown in the example above to: (sb!disassem:define-instruction-format (rex-reg-no-width 16 :default-printer '(:name :tab reg)) (rex :field (byte 4 4) :value #b0100) (wrxb :field (byte 4 0) :type 'wrxb) (op :field (byte 5 11)) (reg :field (byte 3 8) :type 'word-reg) ...) This uses an argument-type "wrxb" that has a prefilter that stores the rex bits into the properties. In the end there should be prefilters for the rex prefix, the #x66 prefix and the width bit that set corresponding properties. The properties are then used implicitly by the prefilters and/or printers of all fields that need them, e.g. the "reg" field above, but also the sib byte and others. This makes the instruction format definitions and the prefilter functions simpler. (Consider extending the current solution -- using an atom versus a list to indicate optional arguments to the prefilter function -- to handle the two independently optional prefixes rex and #x66. I see no nice way to do that, given the restrictions on constructing the prefilter's argument list that define-instruction-format imposes. To create different argument types and prefilter functions for each possible prefix combination instead sounds like copy-paste programming, if not a combinatorial explosion. Also note that in many instructions the same prefix information (rex.w set?, #x66 present?) is needed by the prefilters/printers of several fields.) I have implemented these ideas to such a large part that I know they work. Shall I pursue this further? Yours Lutz Euler |
From: Cheuksan E. W. <wan...@gm...> - 2005-02-14 01:12:34
|
Sounds good to me, given the ugliness of the x86-64 instruction set. I won't be hacking the x86-64 disassembler for the next several days. I don't know about kmr, jsnell, or anyone else. Can you send the diff you have now to the list so we can take a closer look? Thanks Cheuksan Wang On Sun, 13 Feb 2005 17:19:23 +0100, Lutz Euler <lut...@fr...> wrote: > Dear SBCL developers, > > under the subject "x86-64 (aka AMD64, EMT-64) support" Christophe Rhodes > wrote on 2005-01-06: > > There is still plenty of stuff which could do with work on the > > x86-64 platform [...] the disassembler is somewhat selective in > > its understanding of the instruction set [...] > > I would like to help improving the disassembler. I have investigated > its current state and done some experiments, and would like to propose > my ideas. As these would further cement a hack in the disassembler and > I don't know whether someone has conflicting plans for its development, > I would like to get some feedback before putting more work into this. > > The deficiency I would like to address is the incomplete treatment of > the REX prefix, namely the disassembler often outputs one of the first > 8 register names when the instruction references one of the second 8 > registers, e.g. for base and index registers in effective addresses or > even for such simple cases as "XOR reg, reg" or "MOV reg, imm". E.g. in > (disassemble 'string=): > ; 6C5: 4D31F6 XOR R14, RSI ; Should be XOR R14, R14. > Additionally, operand sizes sometimes leak from one instruction to the next. > Instead of simply complicating the instruction format definitions and > extending prefilter and printer functions I would like to suggest somewhat > larger changes. > > Currently, the rex bits are taken into account by two prefilters, > prefilter-word-reg and prefilter-reg/mem and by extending the instruction > formats, e.g.: > > (sb!disassem:define-instruction-format (reg-no-width 8 > :default-printer '(:name :tab reg)) > ... > (reg :field (byte 3 0) :type 'word-reg) > ...) > > (sb!disassem:define-instruction-format (rex-reg-no-width 16 > :default-printer '(:name :tab reg)) > (rex :field (byte 4 4) :value #b0100) > ... > (reg :fields (list (byte 3 8) (byte 4 0)) :type 'word-reg) > ...) > > (defun prefilter-word-reg (value dstate) > (declare (type (or full-reg list) value)) > (if (atom value) > value > (let ((reg (first value)) > (rex.wrxb (second value))) > ... > ;; Code setting the REG-WIDTH property in DSTATE > ...))) > > This, and prefilter-width, already (ab)uses the prefilters to communicate > information from a prefilter of one instruction field to the prefilters/ > printers of others. (This is the hack I mean.) They use the PROPERTIES > slot of the DSTATE object for this. > > I believe that this solution -- to use side-effects of prefilters -- > nevertheless is the best way to go, but IMHO it can be made more complete > and somewhat more elegant as follows: > > As the PROPERTIES slot of the DSTATE is used only in the x86 and the x86-64 > ports, throw it away. Instead put in a slot INST-PROPERTIES that is set to > nil at the start of each iteration in the loop in map-segment-instructions. > This makes it possible to have several prefilters set properties and let > these properties accumulate for use by all following prefilters/printers > of the same instruction, without the current contortions in prefilter-width > to clean up (which, besides, doesn't work for instruction formats that don't > have a width field). > > Also, I suggest to separate the fields that have prefilters setting > properties from the ones using them where possible. E.g. change the > instruction format definition shown in the example above to: > > (sb!disassem:define-instruction-format (rex-reg-no-width 16 > :default-printer '(:name :tab reg)) > (rex :field (byte 4 4) :value #b0100) > (wrxb :field (byte 4 0) :type 'wrxb) > (op :field (byte 5 11)) > (reg :field (byte 3 8) :type 'word-reg) > ...) > > This uses an argument-type "wrxb" that has a prefilter that stores the rex > bits into the properties. In the end there should be prefilters for the rex > prefix, the #x66 prefix and the width bit that set corresponding properties. > The properties are then used implicitly by the prefilters and/or printers of > all fields that need them, e.g. the "reg" field above, but also the sib byte > and others. This makes the instruction format definitions and the prefilter > functions simpler. (Consider extending the current solution -- using an atom > versus a list to indicate optional arguments to the prefilter function -- > to handle the two independently optional prefixes rex and #x66. I see no > nice way to do that, given the restrictions on constructing the prefilter's > argument list that define-instruction-format imposes. To create different > argument types and prefilter functions for each possible prefix combination > instead sounds like copy-paste programming, if not a combinatorial explosion. > Also note that in many instructions the same prefix information (rex.w set?, > #x66 present?) is needed by the prefilters/printers of several fields.) > > I have implemented these ideas to such a large part that I know they work. > Shall I pursue this further? > > Yours > > Lutz Euler > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real users. > Discover which products truly live up to the hype. Start reading now. > http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click > _______________________________________________ > Sbcl-devel mailing list > Sbc...@li... > https://lists.sourceforge.net/lists/listinfo/sbcl-devel > |
From: <lut...@fr...> - 2005-02-16 19:53:05
Attachments:
patch1
|
Cheuksan Edward Wang wrote: > Can you send the diff you have now to the list so we can > take a closer look? Please find it attached. The changes to disassem.lisp and target-disassem.lisp are already complete (modulo dropping of dstate-properties). The changes to x86-64/insts.lisp are just the beginning. I stopped as soon as I got 4D31F6 XOR R14, R14 49BA1700004000000000 MOV R10, 1073741847 instead of 4D31F6 XOR R14, RSI 48BA1700004000000000 MOV RDX, 1073741847. Yours Lutz Euler |
From: Juho S. <js...@ik...> - 2005-02-17 23:30:26
|
On Wed, Feb 16, 2005 at 08:47:46PM +0100, Lutz Euler wrote: > Please find it attached. The changes to disassem.lisp and > target-disassem.lisp are already complete (modulo dropping of > dstate-properties). The changes to x86-64/insts.lisp are just > the beginning. Looks good, at least to my limited understanding of the disassembler :-) Would you prefer that this be commited to CVS immediately, or rather submit a bigger patch later? -- Juho Snellman |
From: <lut...@fr...> - 2005-02-18 22:13:38
|
Juho Snellman wrote: > Looks good, [...] Glad to read that! > Would you prefer that this be commited to CVS immediately, or rather > submit a bigger patch later? Neither of these ;-). I'd rather have the changes to insts.lisp not go in now and would like to polish up the other changes a bit before they get committed. I will start working on this now and send in a new patch. It would be great if you could commit that part then. Afterwards, working on insts.lisp will take me some time. If the changes I make there can be split up nicely I would prefer to send in the parts separately and get each one committed on its own. Regards Lutz Euler |
From: <lut...@fr...> - 2005-02-19 13:20:39
Attachments:
patch2
|
Hi, I wrote: > I [...] would like to polish up the other changes a bit before > they get committed. I will start working on this now and send in a > new patch. It would be great if you could commit that part then. The patch is attached. It contains only the infrastructure needed for the further changes, thus the disassembler's behaviour remains completely unchanged. Yours Lutz Euler |
From: <lut...@fr...> - 2005-03-06 14:55:26
Attachments:
patch
|
Hi, here is a big patch to the x86-64 disassembler that implements the changes I announced earlier. It supersedes the patch I sent on 2005-02-19. It provides and uses a framework to correctly use the prefixes and the width bit to determine the operand size and to correctly extend the register fields of the instructions. To test this I had to work through all printers, prefilters, arg-type and instruction-format definitions and rewrite some of them. In the course of this I took the liberty to fix several bugs that happened to cross my way. For details see below. I did one change that may be controversial: Because the processor sign-extends 32-bit immediate values to 64 bits when operating on qwords it was more regular to print most immediates as signed values. I will send a separate mail about this lest this mail gets too long. There are lots of things left to do. At least: Adding the REX prefix to more instructions, adding more support for the #x66 prefix and adding support for the SSE instructions. Nevertheless the patch is complete in itself and IMHO the disassembler is in a much better shape with it than without it. With kind regards, and see below for the details and the patch Lutz Euler The detailed changes (to insts.lisp, where not otherwise mentioned, and not repeating what I wrote on 2005-02-19) with examples of their effects: - General: * correct typos in the comments for deftype reg and deftype full-reg * correct one or two other typos in comments * replace c-ism by lisp-ism in comment for +default-operand-size+ * remove instruction-format reg-dir (unused) * add type declarations to function print-reg * extend *byte-reg-names*, *word-reg-names* and *dword-reg-names* to all 16 registers each (rationale: if the disassembler gets out of sync or is offered data instead of instruction bytes, don't signal an array indexing error so easily, instead decode even registers that the compiler/assembler does not use) - To correctly determine the address size: Currently: 408A41F1 MOV EAX, [ECX-15] With the patch: 408A41F1 MOV AL, [RCX-15] * replace *default-address-size* with +default-address-size+ and correct its value to :qword, correct comment * add functions inst-operand-size and inst-operand-size-default-qword * change print-addr-reg to always use +default-address-size+ * remove arg-type addr-reg (no longer needed) - Treatment of REX prefix and width bit to determine operand size and to extend register fields: Currently: 4D31F6 XOR R14, RSI 4F8B440101 MOV RAX, [RCX+RAX+1] 408A41F1 MOV EAX, [ECX-15] With the patch: 4D31F6 XOR R14, R14 4F8B440101 MOV R8, [R9+R8+1] 408A41F1 MOV AL, [RCX-15] * add prefilters prefilter-wrxb, prefilter-reg-r, prefilter-reg-b * new arg-type wrxb * change prefilter-reg/mem to use the new way to get at the REX bits. Make it use REX.B and REX.X to extend the base and index field at all. * same with prefilter-width, move it towards the beginning of the file to have the prefilters in the order their fields are in the instructions * adapt arg-type width to this, move it towards the front, too * change print-reg to use inst-operand-size * add arg-type reg-b (we need to distinguish two register arg-types depending on which REX bit extends them) * change all define-instruction-formats with rex to use the new wrxb field and arg-type * remove arg-type rex-reg/mem and sized-rex-reg/mem, use reg/mem and sized-reg/mem instead * remove function print-rex-reg/mem (no longer used) - Clean up word-... things: (word-reg and reg are treated no differently already without my patch; that they both exist seems to be a relict of an earlier incomplete attempt to correctly treat the width bit and the #x66 prefix, already in the x86 port. Same with word-accum.) * remove prefilter-word-reg * remove arg-type word-reg/mem and word-reg, use reg/mem and reg or reg-b instead * remove functions print-word-reg and print-word-reg/mem * remove arg-type word-accum, use accum instead - Treatment of instructions with a default operand size of 64 bits: Currently: 48FF5009 CALL BYTE PTR [RAX+9] 48FF75F0 PUSH BYTE PTR [RBP-16] 57 PUSH EDI With the patch: 48FF5009 CALL QWORD PTR [RAX+9] 48FF75F0 PUSH QWORD PTR [RBP-16] 57 PUSH RDI * new function print-reg/mem-with-width * new printer function print-sized-reg/mem-default-qword * new arg-type sized-reg/mem-default-qword * new instruction formats reg/mem-default-qword and rex-reg/mem-default-qword * use these in push, pop, call, jmp * add function print-reg-default-qword, arg-type reg-b-default-qword, instruction formats reg-no-width-default-qword and rex-reg-no-width-default-qword and use them in push and pop. (This corrects printing of the register size for the variants of these instructions without REX prefix. These are currently not used by the assembler but should be to reduce code size). - Sizes of and sign extension of immediate data: Currently: Wrong length of immediate, losing sync: 48B908000000 MOV RCX, 8 0000 ADD [EAX], EAX 0000 ADD [EAX], EAX Missing sign extension of 32-bit immediate to 64 bits: 48C745E8F8FFFFFF MOV QWORD PTR [RBP-24], 4294967288 With the patch: 48B90800000000000000 MOV RCX, 8 48C745E8F8FFFFFF MOV QWORD PTR [RBP-24], -8 * remove arg-type imm-data-upto-dword and imm-data * add arg-type signed-imm-data, signed-imm-data-upto-qword and signed-imm-data-default-qword, make them use inst-operand-size[-default-qword] * adapt all uses of these fields * remove arg-type signed-imm-dword * (in disassem.lisp) allow length 64 in read-signed-suffix * add arg-type imm-byte * use it in shift-inst-printer-list instead of signed-imm-byte (not that that makes a difference for the expected values of the immediate of 0 ... 63, but in case someone puts in some unreasonable value ...) - To be able to correctly print operand sizes in memory references: (for examples see above and movsx below) * (in target-insts.lisp) change print-mem-access: change argument order to honor conventions from insts.lisp, move determination of size out of this function and therefore rename the argument print-size-p to width, extend comment * adapt the caller in insts.lisp and make it use inst-operand-size or inst-operand-size-default-qword as appropriate - Clean up confusion about length of op fields with and without width bit: Currently: The shift-inst printer misdetects IMUL as RCL, losing sync: 4869D0 RCL RAX, CL 0001 ADD [ECX], EAX 0000 ADD [EAX], EAX With the patch: 4869D000010000 IMUL RDX, RAX, 256 * add width field to the instruction-format rex-reg, shrink op field to 4 bits. * same in rex-reg-reg/mem and rex-reg/mem (op is 7 bits instead of 8) * adapt usage in mov, lea, test and xchg instructions * correct rex-reg/mem-imm usage in the shift instructions and in mov and remove comment saying it doesn't work for 8-bit register yet (it does now) * same in arith-inst-printer-list (two places, #x80/81 and #x82/83) * remove #x82 encoding in arith-inst-printer-list (invalid in 64-bit mode) - Fix bit test instructions: (the immediate variant always uses an unsigned byte independent of the operand size) * use arg-type imm-byte instead of imm-data - Fix movsx[d] and movzx: (the size of the source (reg/mem) should be determined by the opcode, the size of the destination (reg) by the operand size) Currently: 48 BYTE #X48 0FB6C0 MOVZX EAX, EAX 48 BYTE #X48 63 BYTE #X63 C8488BC1 ENTER 35656, 193 With the patch: 480FB6C0 MOVZX RAX, AL 4863C8 MOVSXD RCX, EAX * rename print-byte-reg/mem to print-sized-byte-reg/mem, replace arg-type byte-reg/mem with sized-byte-reg/mem (was used only in SETcc, which can use the sized version, so we avoid adding another arg-type) * new functions print-sized-[d]word-reg/mem and arg-types sized-[d]word-reg/mem * add instruction-formats [rex-]ext-reg-reg/mem-no-width * remove (reg nil :type 'reg) in the define-instructions for movsx, movzx and movsxd (it is superfluous) * extend the printers in movsx and movzx to use the fixed-size arg-types * movsxd needs only the rex-variant because it should not be used without a REX prefix with the W bit set (it would do a movzxd otherwise, see the AMD docs) - Fix imul: Currently: Incorrectly identified as IMUL: 0FAE5DE8 IMUL EBX, [EBP-24] With the patch: Not decoded, should be LDMXCSR (printer not defined yet) 0F BYTE #X0F AE SCASB 5D POP RBP * formerly, 0F AE was incorrectly interpreted as imul (should be group 15 instead). Therefore change the corresponding imul printer by replacing arg-type ext-reg-reg/mem with ext-reg-reg/mem-no-width and adapting the op field. * replace the arg-type imm-word with signed-imm-data in the imul printer for opcode #x69 (to not resurrect bug 245a and more after the previous changes) * add rex-variants to the printers for the opcodes #x69 and #x6B * remove arg-type imm-word (no longer used) |
From: <lut...@fr...> - 2005-04-05 20:35:11
Attachments:
patch
|
Hi, I would like to offer another patch to make the x86-64 disassembler more complete. Please find it attached. What it contains: Fixed some glitches in my previous patch (from 2005-03-06): - deleted the rex and wrxb fields in the instruction format definition for rex-reg-reg/mem-dir; they are unchanged from the included definition and thus redundant. - in the arith-inst-printer-list added a missing (width 1) to rex-reg/mem-imm. - resurrected the printer for the movsxd variant without rex prefix (this variant is used to do zero extension dword -> qword and is generated by the movzxd pseudo instruction.) Added support for the variants of several instructions with a rex prefix: - neg, not, cmov (new instruction-format rex-cond-move) - bsf, bsr (here I additionally changed the instruction-format to ...-no-width, which seems clearer to me) - mul, imul, div, idiv (new instruction-format rex-accum-reg/mem) Added printers for some instructions: - cbw, cwd, cqo (new instruction-formats x66-byte and rex-byte) The following changes affect the assembler, too: - added the instruction cdqe (for completeness) - deleted the instructions pusha and popa (they are illegal in 64-bit mode) Yours, Lutz Euler |
From: Juho S. <js...@ik...> - 2005-04-06 01:55:27
|
On Tue, Apr 05, 2005 at 10:32:44PM +0200, Lutz Euler wrote: > Hi, > > I would like to offer another patch to make the x86-64 disassembler > more complete. Please find it attached. What it contains: Thank you very much. This has been committed in 0.8.21.19. -- Juho Snellman |
From: Juho S. <js...@ik...> - 2005-03-10 06:40:22
|
On Sun, Mar 06, 2005 at 03:49:54PM +0100, Lutz Euler wrote: > here is a big patch to the x86-64 disassembler that implements the > changes I announced earlier. It supersedes the patch I sent on > 2005-02-19. Thank you very much. This has been committed in 0.8.20.14. > I did one change that may be controversial: Because the processor > sign-extends 32-bit immediate values to 64 bits when operating on qwords > it was more regular to print most immediates as signed values. This seems appropriate for 32-bit immediates, but I find it a bit disorienting for bytes: ; 42E: 403CA2 CMP AL, -94 -- Juho Snellman |
From: <lut...@fr...> - 2005-03-12 17:27:21
|
Hi, thanks for applying the patch! I wrote: > I did one change that may be controversial: Because the processor > sign-extends 32-bit immediate values to 64 bits when operating on > qwords it was more regular to print most immediates as signed values. Juho Snellman answered: > This seems appropriate for 32-bit immediates, but I find it a bit > disorienting for bytes: > > ; 42E: 403CA2 CMP AL, -94 I agree with this feeling. Exactly this example led me to make the remark about controversiality. CMP on bytes is often used for tag tests and looks to me to be more difficult to read this way, since the programmer thinks of tags as unsigned values. So, it seems better to make all byte immediates unsigned. Then the question remains: Which sizes of immediates should be printed as signed numbers? All that are larger than bytes, i.e., 16, 32 and 64 bits, or only 32 and 64 bits, or only 64 bits, or none? (I understand the size of an immediate to be the operand size of the instruction, independent of the encoding of the immediate. E.g., ; 9A8: 4883C0FA ADD RAX, -6 is a 64-bit immediate, which could be printed signed, as here, or unsigned as ; 9A8: 4883C0FA ADD RAX, 18446744073709551610) If there is an agreement on what is wanted, I will happily work on changing the disassembler suitably. Yours Lutz Euler |
From: <lut...@fr...> - 2006-05-28 16:39:15
Attachments:
patch-x66prefix
|
Hi, in March 2005 I wrote about my changes to the x86-64 disassembler: > There are lots of things left to do. [...] adding more support for the > #x66 prefix [...] Here is a patch that adds support for the #x66 prefix (which overwrites the operand size to 16 bits) to the disassembler. An example for what this changes: (defun f (x y) (declare (optimize speed (debug 0) (safety 0)) (type (simple-array (unsigned-byte 16)) x y)) (setf (aref x 0) (aref y 0))) (disassemble 'f) Currently: ; 0268F5F7: 480FB74701 MOVZX RAX, WORD PTR [RDI+1] ; 5FC: 66 BYTE #X66 ; 5FD: 894201 MOV [RDX+1], EAX With the patch: ; 021DBCF7: 480FB74701 MOVZX RAX, WORD PTR [RDI+1] ; CFC: 66894201 MOV [RDX+1], AX The patch contains the general support for this prefix, that is, a prefilter function and an argument-type. Furthermore, as the only instruction I found being used with a 16-bit operand size is the "register to/from register/memory" form of the MOV instruction, I added only the instruction-formats and printer clauses needed for this. With kind regards Lutz Euler |
From: Juho S. <js...@ik...> - 2006-06-01 09:08:20
|
lut...@fr... (Lutz Euler) writes: > Here is a patch that adds support for the #x66 prefix (which overwrites > the operand size to 16 bits) to the disassembler. An example for what > this changes: Thanks, committed as 0.9.13.15. -- Juho Snellman |