[Valgrind-developers] A patch for Valgrind to support VMware backdoor (plain-text formatted)

[MoJiong Q. <qm...@ho...>]

Please forget previous html-formatted mail.
--------------------------------------------------

Hello guys, 
   
   I was playing Valgrind in VMware virtual machine, and found Valgrind doesn't work with VMware backdoor. (Please see below for a brief introduction of VMware backdoor.) Specifically, Valgrind reports two kinds of errors when running open-vm-tools and programs that use VMware backdoors: Segementation fault, due to in/out; And illegal instruction, due to ins/outs not handled in Valgrind.

   The attachment is a patch to fix the problem. I used Valgrind-3.1.1 (http://valgrind.org/downloads/valgrind-3.3.1.tar.bz2) as a code base. Roughly three changes in the patch:
   - Support ins/outs.  Added corresponding decoding stub in VEX/priv/guest-x86(amd64)/toIR.c, and dirtyhelpers in ghelper.c
   - Added a new function process_vmware_backdoor in ghelper.c. The dirtyhelpers for in, out, ins and outs call process_vmware_backdoor. 
   - Added code to process 8-bit memory chunk in memcheck/mc_translate.c/do_shadow_Dirty, because ins/outs produce 8-bit memory inputs.

   This patch doesn't change the original behavior when running on non-VMware systems. And it doesn't bring performance problem, since no normal application does port I/O. Considering VMware virtual machines are widely used, I think it is deservable for Valgrind to support VMware virtual machines.

   It would be great if you can take a look at the patch and finally merged it into Valgrind.


About VMware backdoor:
A user program running in a VMware virtual machine can communicate with VMM through so-called VMware backdoor. The commnucation is done by accessing a special I/O port and setting pre-defined value to EAX. VMware VMM redefines in/out/ins/outs meeting these two conditions to take arbitrary other registers as input and allow these four instructions to write to arbitrary other registers as output. 

The link (http://chitchat.at.infoseek.co.jp/vmware/backdoor.html) has some description. It talked about backdoor through in/out, but not ins/outs, which are actually also used by VMware as backdoor triggering instrucitons.


Thanks,
Mojiong

_________________________________________________________________
Explore the seven wonders of the world
http://search.msn.com/results.aspx?q=7+wonders+world&mkt=en-US&form=QBRE