#1460 Incorrect "Blocking by NMI" bit when handling NMI VM-exit

[Eric Li]

Bochs version: self-compiled, GitHub b46fc681d2f4624c3c7ed15c327cf7d21650e29b
Host Platform: Debian 12, kernel 6.1.0-12-amd64
Bochs build command:

./configure \
            --enable-all-optimizations \
            --enable-cpu-level=6 \
            --enable-x86-64 \
            --enable-vmx=2 \
            --enable-clgd54xx \
            --enable-busmouse \
            --enable-show-ips \
            --enable-smp \
            --disable-docbook \
            --with-x --with-x11 --with-term --with-sdl2
make

bochsrc

cpu: model=corei7_sandy_bridge_2600k, count=2, ips=50000000, reset_on_triple_fault=1, ignore_bad_msrs=1, msrs="msrs.def"
memory: guest=512, host=256
romimage: file=$BXSHARE/BIOS-bochs-latest, options=fastboot
vgaromimage: file=$BXSHARE/VGABIOS-lgpl-latest
ata0: enabled=1, ioaddr1=0x1f0, ioaddr2=0x3f0, irq=14
ata1: enabled=1, ioaddr1=0x170, ioaddr2=0x370, irq=15
ata0-master: type=cdrom, path="grub.iso", status=inserted
boot: cdrom, disk
log: log.txt
panic: action=ask
error: action=report
info: action=report
debug: action=ignore, pci=report # report BX_DEBUG from module 'pci'
debugger_log: -
com1: enabled=1, mode=file, dev=/dev/stdout

Attached with this bug report is grub.iso. It is a CD image that boots a micro-hypervisor I wrote (called SHV). The attachment is compressed with xz. SHV's source code can be found at https://github.com/lxylxy123456/shv/tree/fa0d6a59b281a8c91cb80d85aada22c1e122bb93
Bochs' serial output (printed to /dev/stdout):

...
g_nmi_opt: 0x105
g_nmi_exp: 0x81
exp_mask: 0x80
Experiment: 7
  Enter host, exp=7, state=0
    hlt_wait() begin, source =  EXIT_NMI_H   (5)
      Inject NMI
      Interrupt recorded:       EXIT_NMI_H   (5)
    hlt_wait() end
    hlt_wait() begin, source =  EXIT_TIMER_H (6)
      Inject interrupt
      Interrupt recorded:       EXIT_TIMER_H (6)
    hlt_wait() end
    hlt_wait() begin, source =  EXIT_TIMER_H (6)
      Inject NMI
      Inject interrupt
      Interrupt recorded:       EXIT_TIMER_H (6)
    hlt_wait() end
  Leave host
      Interrupt recorded:       EXIT_VMEXIT  (7)
  Enter host, exp=7, state=1

TEST_ASSERT '!get_blocking_by_nmi()' failed, line 713, file ../tools/../src/shv-nmi.c

Expected output (reproduced on KVM and real Intel hardware, with >= 2 CPUs):

g_nmi_opt: 0x105
g_nmi_exp: 0x81
exp_mask: 0x80
Experiment: 7
  Enter host, exp=7, state=0
    hlt_wait() begin, source =  EXIT_NMI_H   (5)
      Inject NMI
      Interrupt recorded:       EXIT_NMI_H   (5)
    hlt_wait() end
    hlt_wait() begin, source =  EXIT_TIMER_H (6)
      Inject interrupt
      Interrupt recorded:       EXIT_TIMER_H (6)
    hlt_wait() end
    hlt_wait() begin, source =  EXIT_TIMER_H (6)
      Inject NMI
      Strange wakeup from HLT
      Inject interrupt
      Interrupt recorded:       EXIT_TIMER_H (6)
    hlt_wait() end
  Leave host
      Interrupt recorded:       EXIT_VMEXIT  (7)
  Enter host, exp=7, state=1
    iret_wait() begin, source = EXIT_MEASURE (1)
    iret_wait() end
  Leave host
Sequential experiments pass
ALL TEST PASS

Bochs log file:

...
00274934631i[APIC1 ] Deliver NMI
00325000607i[APIC1 ] Deliver NMI
00375001417i[APIC1 ] Deliver NMI
00401085478i[CPU1  ] WARNING: HLT instruction with IF=0!

Explanation:
The code in LHV performs an experiment (called "Experiment 7" in serial output) on CPU 0 to test the behavior of NMI blocking. The experiment steps are:
1. Prepare state such that the CPU is currently in host mode (i.e. VMX root mode), and NMI is blocked
2. Modify VMCS to make sure that guest mode (i.e. VMX non-root mode) has NMI exiting = 1 and virutal NMIs = 0. The guest mode does not block NMI (Blocking by NMI = 0)
3. Wait until the CPU receives an NMI interrupt.
4. VM entry to guest mode. This immediately causes a VM exit to host mode since NMI is delivered.
5. VM entry to guest mode.
6. VM exit to host mode.
7. Check the "Blocking by NMI" bit of "Guest interruptibility state" in VMCS.

On real hardware, step 7 should give 0. However, on Bochs, step 7 gives 1.

The expected behavior is that after step 4, the "Blocking by NMI" bit of "Guest interruptibility state" in VMCS is set to 0. However, when running on Bochs, this bit is set to 1. This bit does not change in step 5 and 6, so Bochs fails the experiment at step 7.

FYI: another NMI and VMX related bug I reported is https://sourceforge.net/p/bochs/bugs/1456/.

[Stanislav Shwartsman]

Fix was pushed to github

Branch: refs/heads/master
Home: https://github.com/bochs-emu/Bochs
Commit: 6b48d6e33ff8d5a5f8b1986e01aa34c901faffe3
https://github.com/bochs-emu/Bochs/commit/6b48d6e33ff8d5a5f8b1986e01aa34c901faffe3

[Eric Li]

Thank you for fixing this bug! On my end I can confirm that this bug is fixed.