Valgrind for Windows Wiki
Status: Pre-Alpha
Brought to you by:
cschwarz1
Menu
▾
▴
CallbacksFromSystemCalls
USER32 Callbacks
Disclaimer: All following information and changes were acquired and tested on x86/WOW64 Win7 system only. Various aspects might be different on different OS and architecture.
Overview
User callbacks start in user space at a ntdll!KiUserCallbackDispatcher. This function has - callback ID, callback code address, pointer to input data and size of input data.
The input data usually follows the arguments on the stack with just a little padding. The data seems to always start with an undocumented structure CAPTUREBUF with details about the call. The individual kinds of callbacks have their customized structures, all having CAPTUREBUF at their beginning.
The kernel stores everything on the user stack, so the CAPTUREBUF provides possibility for fixing any self-referential pointers in the data.
Valgrind support
In order to let e.g. Memcheck to see the passed data correctly initialized etc., Valgrind would need to copy the whole callback parameters block to e.g. the guest stack, fix any any self-referential pointers in the contents and execute the original callback.
This would be possible, because this is exactly what the system is created to do (CAPTUREBUF for "captured pointers"). Unfortunately, in the probably most widely used environment, WOW64, wow64user.dll does the conversion from kernel x64 structures to userland x86 ones and both the full input data size and the array of captured pointers is lost in the process, because the data size is not checked by anything down the road and any pointers can refer to the original values.
So at least for WOW64, every user callback needs to be analyzed and forwarded correctly to the original handler. But it has the advantage of knowing what's going in during the callback, which might prove useful for some new Valgrind tools.
Implementation details
In order to know what kind of marshalling to apply even though the IDs can change any time, one needs to know at least the name of the callback. This is done by tracing pointers in user32!_apfnDispatch[].
To leave the guest process more pristine, this symbol loading is again done in the loader, with the communication using shared memory and events. There is one command to get the address of _apfnDispatch, the second one resolves the function pointers located there. All done during global init.
A table of known user callbacks and their data contents has been introduced. It uses silly textual format to be able to define new structure layouts rapidly. However it still needs the manual analysis part, which sucks.
If the callbacks is known, it is forwarded properly to the guest stack, taking care of any pointers (mostly UNICODE_STRINGs). If the callback is not known, the original input data buffer is passed by pointer only, the memory is not marked as readable for V etc. This seems to work, but produces memcheck warnings, of course.
NtCallbackReturn() wrapper was modified to save the syscall parameters before popping the syscall stack, otherwise the return values from the callback would be invalid.
Hooks
This section describes how Valgrind for Windows intercepts KiUserCallbackDispatcher.
General:
- Valgrind modifies (overwrites a part of) the ntdll!KiUserCallbackDispatcher code to call an assembler stub
- the assembler stub executes the overwritten instructions and then calls a C function for further callback handling
- the C function then sets up the client state and transfers control to the virtual CPU
- the C function may decide to run the callback on the real CPU, by returning to the assembler stub (this is not used at the moment)
WIN32/WOW64:
- assembler stub: VGASM_(catch_user_callback_dispatcher) in coregrind/m_syswrap/syscall-x86-windows.S
- WIN32: C function: VG_(win_sc_win32_user_callback_dispatch) in coregrind/m_windows/win-syscall.c
- WOW64: C function: VG_(win_sc_wow64_user_callback_dispatch) in coregrind/m_windows/win-syscall.c
WIN64:
- assembler stub: VGASM_(catch_user_callback_dispatcher) in coregrind/m_syswrap/syscall-and64-windows.S
- C function: VG_(win_sc_win64_user_callback_dispatch) in coregrind/m_windows/win-syscall.c
WIN32
==== ============ =================== ==== ======== ======================== Original ntdll!KiUserCallbackDispatcher Hooked ntdll!KiUserCallbackDispatcher ==== ============ =================== ==== ======== ======================== 0000 83c404 add esp,4 0000 ff15 call \[user_callback_ 0003 5a pop edx xxxxxxxx dispatcher_ptr\] 0004 64a118000000 mov eax,fs:\[18h] 0006 90909090 nop\*4 000a 8b4030 mov eax,\[eax+30h] 000a 909090 nop\*3 000d 8b402c mov eax,\[eax+2Ch] 000d 909090 nop\*3 0010 ff1490 call \[eax+edx\*4] 0010 ff1490 call \[eax+edx\*4] 0013 33c9 xor ecx,ecx 0013 33c9 xor ecx,ecx 0015 33d2 xor edx,edx 0015 33d2 xor edx,edx 0017 cd2b int 2Bh 0017 cd2b int 2Bh
======== =========================== ======== =======================================
Stack layout at the entry time of Stack layout at the entry time of
KiUserCallbackDispatcher VG_(win_sc_win32_user_callback_dispatch)
======== =========================== ======== =======================================
ESP+0 return address (into
VG_(catch_user_callback_dispatcher))
+0x04 address of dispatcher table (->EAX)
ESP+0 bogus return address +0x08 KiUserCallbackDispatcher+0x0006 (->EIP)
+0x04 ID of the callback handler +0x0c ID of the callback handler (->EDX)
+0x08 ptr to CAPTUREBUF (ESP+0x10) +0x10 ptr to CAPTUREBUF (ESP+0x18)
+0x0c size of the CAPTUREBUF (S) +0x14 size of the CAPTUREBUF (S)
+0x10 start of CAPTUREBUF +0x18 start of CAPTUREBUF
... ...
+0x0c+S last word in CAPTUREBUF +0x14+S last word in CAPTUREBUF
+0x10+S caller of system call stub +0x18+S caller of system call stub
+0x14+S 1st syscall argument +0x1c+S 1st syscall argument
Notes:
- (S) denotes the size of the CAPTUREBUF in bytes
WOW64
==== ============== ===================== ==== ============== =====================
Original ntdll!KiUserCallbackDispatcher Hooked ntdll!KiUserCallbackDispatcher
==== ============== ===================== ==== ============== =====================
0000 648b0d00000000 mov ecx,fs:\[0h] 0000 648b0d00000000 mov ecx,fs:\[0h]
0007 ba8000e877 mov edx,offset 0007 ba8000e877 mov edx,offset
!KiUserApcDispatcher+0x48 !KiUserApcDispatcher+0x48
000c 8d442410 lea eax,\[esp+10h] 000c 8d442410 lea eax,\[esp+10h]
0010 894c2410 mov \[esp+10h],ecx 0010 894c2410 mov \[esp+10h],ecx
0014 89542414 mov \[esp+14h],edx 0014 89542414 mov \[esp+14h],edx
0018 64a300000000 mov fs:\[0h],eax 0018 64a300000000 mov fs:\[0h],eax
001e 83c404 add esp,4 001e ff15xxxxxxxx call \[user_callback_
0021 5a pop edx dispatcher_ptr\]
0022 64a130000000 mov eax,fs:\[30h] 0024 90909090 nop\*4
0028 8b402c mov eax,\[eax+2Ch] 0028 909090 nop\*3
002b ff1490 call \[eax+edx\*4] 002b ff1490 call \[eax+edx\*4]
002e 50 push eax 002e 50 push eax
002f 6a00 push 0 002f 6a00 push 0
0031 6a00 push 0 0031 6a00 push 0
0033 e8a4f70000 call ZwCallbackReturn 0033 e8a4f70000 call ZwCallbackReturn
======== =========================== ======== =======================================
Stack layout at the entry time of Stack layout at the entry time of
KiUserCallbackDispatcher VG_(win_sc_wow64_user_callback_dispatch)
======== =========================== ======== =======================================
ESP+0 return address (into
VG_(catch_user_callback_dispatcher))
+0x04 address of dispatcher table (->EAX)
ESP+0 0 (bogus return address) +0x08 KiUserCallbackDispatcher+0x0024 (->EIP)
+0x04 ID of the callback handler +0x0c ID of the callback handler (->EDX)
+0x08 ptr to CAPTUREBUF (ESP+0x18) +0x10 ptr to CAPTUREBUF (ESP+0x20)
+0x0c 0 (no size S) +0x14 0 (no size S)
+0x10 ?fs:\[0] +0x18 ?fs:\[0]
+0x14 ?KiUserApcDispatcher+0x48 +0x1c ?KiUserApcDispatcher+0x48
+0x18 start of CAPTUREBUF +0x20 start of CAPTUREBUF
... ...
+0x14+S last word in CAPTUREBUF +0x1c+S last word in CAPTUREBUF
Notes:
- (S) denotes the size of the CAPTUREBUF in bytes, but is always 0. The size of the CAPTUREBUF is unknown, therefore the caller of the system call can not be determined.
WIN64
==== ========== ===================== ==== ============== =====================
Original ntdll!KiUserCallbackDispatcher Hooked ntdll!KiUserCallbackDispatcher
==== ========== ===================== ==== ============== =====================
0000 488b4c2420 mov rcx,\[rsp+20h] 0000 90 nop
0005 8b542428 mov edx,\[rsp+28h] 0001 ff1425nnnnnnnn call \[user_callback_
dispatcher_ptr\]
0009 448b44242c mov r8d,\[rsp+2Ch] 0008 909090909090 nop*6
000e 65488b0425 mov rax,gs:\[60h] 000e 65488b0425 mov rax,gs:\[60h]
60000000 60000000
0017 4c8b4858 mov r9,\[rax+58h] 0017 4c8b4858 mov r9,\[rax+58h]
001b 43ff14c1 call \[r9+r8\*8] 001b 43ff14c1 call \[r9+r8\*8]
001f 33c9 xor ecx,ecx 001f 33c9 xor ecx,ecx
0021 33d2 xor edx,edx 0021 33d2 xor edx,edx
0023 448bc0 mov r8d,eax 0023 448bc0 mov r8d,eax
0026 e82f010000 call ZwCallbackReturn 0026 e82f010000 call ZwCallbackReturn
======== ============================== ======== =====================================
Stack layout at the entry time of Stack layout at the entry time of
KiUserCallbackDispatcher VG_(win_sc_win64_user_callback_dispatch)
======== ============================== ======== =====================================
RSP+0 return address (into
VG_(catch_user_callback_dispatcher))
+0x08 ? rcx=buffer
+0x10 ? rdx=length
+0x18 ? r8=callback ID
+0x20 ? r9=KiUserCallbackDispatcher+8 ->RIP
RSP+0 ? (1) +0x28 ? (1)
+0x08 ? +0x30 ?
+0x10 ? +0x38 ?
+0x18 ? +0x40 ?
+0x20 ptr to CAPTUREBUF (RSP+0x58) +0x48 ptr to CAPTUREBUF (RSP+0x80)
+0x28 (32bit) size of CAPTUREBUF (S) +0x50 (32bit) size of CAPTUREBUF (S)
+0x2c (32bit) ID of callback handler +0x54 (32bit) ID of callback handler
+0x30 caller of system call +0x58 caller of system call
+0x38 ? (2) +0x60 ? (2)
+0x40 ? +0x68 ?
+0x48 ? +0x70 ?
+0x50 ? +0x78 ?
+0x58 start of CAPTUREBUF +0x80 start of CAPTUREBUF
... ...
+0x50+S last word in CAPTUREBUF +0x78+S last word in CAPTUREBUF
Notes:
- (S) denotes the size of the CAPTUREBUF in bytes
- (1) this could be stack space reserved for calling the handler function (x64 calling convention: space for 4 register arguments must be reserved even if function takes less than 4 arguments)
- (2) this could be stack space reserved for calling the system call; kernel apparently extends this stack frame from the original SYSCALL to squeeze the CAPTUREBUF in
