1.24-Update2 to 1.29 - VM firmware failure

[Bugsy]

Hi All,

After uprade from 1.24-Update2 to 1.29 (latest) the VM doesn't boot anymore past successful authorization (password accepted and start 0 ... message presented - such as volume is opened). It raises VMware level error

The firmware encountered an unexpected exception. The virtual machine cannot boot.

Exact logs from the VM machine log file below.

Before going into details - is it possible to downgrade without a harm back to 1.24-Update2? It's easy for me to replace UEFI level files -it is more at the Windows level which is big unknown if and how to do it?

Details of the problem are interesting and very weird - all are below.
In short - boot via VM fails, same OS booting of HW resources works fine. It could be something wrong with VMware, though with 1.24-Update2 it all worked fine.

Details:
Nothing else has been changed on the system (both host/vmware side, same as guest side).
The system setup is a little bit weird, as the VM uses physical disc for storage and this is very same partition as am booting normally without VM - all worked fine.
I can still boot the system directly (without VM, directly from UEFI => VeraCrypt password, boots Windows).

To make it even more weird, I do have two Windows deployments, which are booted depending on the password.
Both boot fine when booted on Physical system, not via VM.
Both fail when booting from VM level.
There's additional segregation at VM level, where there are two separate VM definitions, where each has access only to their own partitions allowing them to run in parallel via VM.
The only portion these two share is EFI partition, Microsoft Reserved and recovery partitions.
Windows system partitions are separate. EFI partitions are mapped for VMs to file. Though I've tried using real partitions and got same effect - compared content and there's not much visible change.

The only change was that upgrade of VeraCrypt. On the firs reboot, it failed with the message as stated.
Rebooting that instance of Windows directly, worked fine, though complained about different version being on boot loaded and on the system/app. App level was showing already 1.29.
Did reinstall VeraCrypt whilst Windows was booted on physical resources. To no effect. Only after forcing from Linux level, new binaries from 1.29 recovery disk, whilst keeping DcsProp file, it made it happy.
No complaints on Windows side after booting on physical resources - both Windows systems (separate partitions).

At the same time, I can't get around that "firmware" problem when booting from VM level.

Weird feeling is that it has nothing to do with UEFI level, as it loads fine, both VM and on physical side and only crashes after opening encrypted volume.

Without knowing inner works of VeraCrypt, could it be that something is loaded at the very beginning, VeraCrypt relates, driver?, which executes that "NVMe" reset function as reported by VMware and that is something new on 1.29 (or since 1.24-Update2)?
If it's ok to downgrade back to 1.24-Update2 - I'd be happy to give it a try, as I really need to be able to run these Windows as VM whilst keeping option to run them natively from HW when needed.

Thanks for going through this long read.

2022-10-13T17:08:36.206+02:00| vcpu-0| I005: Guest: Status upon boot failure: No Media
2022-10-13T17:08:36.237+02:00| vcpu-0| I005: Guest: About to do EFI boot: virtual VeraCrypt
2022-10-13T17:08:50.286+02:00| vcpu-0| I005: NVME-VMM: Controller level reset via CC.EN bit transition on nvme0
2022-10-13T17:08:50.286+02:00| vcpu-0| I005: NVME-CORE: Doing a partial reset of controller regs and queues.
2022-10-13T17:08:50.289+02:00| vcpu-0| I005: NVME-VMM: Controller level reset via CC.EN bit transition on nvme0
2022-10-13T17:08:50.289+02:00| vcpu-0| I005: NVME-CORE: Doing a partial reset of controller regs and queues.
2022-10-13T17:08:50.292+02:00| vcpu-0| I005: NVME-VMM: Controller level reset via CC.EN bit transition on nvme0
2022-10-13T17:08:50.292+02:00| vcpu-0| I005: NVME-CORE: Doing a partial reset of controller regs and queues.
2022-10-13T17:08:50.295+02:00| vcpu-0| I005: NVME-VMM: Controller level reset via CC.EN bit transition on nvme0
2022-10-13T17:08:50.295+02:00| vcpu-0| I005: NVME-CORE: Doing a partial reset of controller regs and queues.
2022-10-13T17:08:50.297+02:00| vcpu-0| I005: NVME-VMM: Controller level reset via CC.EN bit transition on nvme0
2022-10-13T17:08:50.297+02:00| vcpu-0| I005: NVME-CORE: Doing a partial reset of controller regs and queues.
2022-10-13T17:08:50.300+02:00| vcpu-0| I005: NVME-VMM: Controller level reset via CC.EN bit transition on nvme0
2022-10-13T17:08:50.300+02:00| vcpu-0| I005: NVME-CORE: Doing a partial reset of controller regs and queues.
2022-10-13T17:08:50.303+02:00| vcpu-0| I005: NVME-VMM: Controller level reset via CC.EN bit transition on nvme0
2022-10-13T17:08:50.303+02:00| vcpu-0| I005: NVME-CORE: Doing a partial reset of controller regs and queues.
2022-10-13T17:08:50.305+02:00| vcpu-0| I005: NVME-VMM: Controller level reset via CC.EN bit transition on nvme0
2022-10-13T17:08:50.306+02:00| vcpu-0| I005: NVME-CORE: Doing a partial reset of controller regs and queues.
2022-10-13T17:08:50.308+02:00| vcpu-0| I005: NVME-VMM: Controller level reset via CC.EN bit transition on nvme0
2022-10-13T17:08:50.309+02:00| vcpu-0| I005: NVME-CORE: Doing a partial reset of controller regs and queues.
2022-10-13T17:08:50.311+02:00| vcpu-0| I005: NVME-VMM: Controller level reset via CC.EN bit transition on nvme0
2022-10-13T17:08:50.311+02:00| vcpu-0| I005: NVME-CORE: Doing a partial reset of controller regs and queues.
2022-10-13T17:08:50.314+02:00| vcpu-0| I005: NVME-VMM: Controller level reset via CC.EN bit transition on nvme0
2022-10-13T17:08:50.314+02:00| vcpu-0| I005: NVME-CORE: Doing a partial reset of controller regs and queues.
2022-10-13T17:08:50.317+02:00| vcpu-0| I005: NVME-VMM: Controller level reset via CC.EN bit transition on nvme0
2022-10-13T17:08:50.317+02:00| vcpu-0| I005: NVME-CORE: Doing a partial reset of controller regs and queues.
2022-10-13T17:08:50.335+02:00| vcpu-0| I005: Msg_Post: Error
2022-10-13T17:08:50.335+02:00| vcpu-0| I005: [msg.efi.exception] The firmware encountered an unexpected exception. The virtual machine cannot boot.
2022-10-13T17:08:50.335+02:00| vcpu-0| I005: ----------------------------------------

DcsProps - nothing has been changed, just noticed new entry postexec and bootmgfw_ms.vc)

[cut standard]
<config key="ActionSuccess">postexec file(EFI\Microsoft\Boot\bootmgfw_ms.vc)</config>
[cut standard]

ls -l bootmgfw.efi bootmgfw_ms.vc bootmgr.efi 
-rwxr-xr-x 1 root root   24088 Mar  5  2020 bootmgfw.efi
-rwxr-xr-x 1 root root 1528144 Jul 22  2021 bootmgfw_ms.vc
-rwxr-xr-x 1 root root 1511760 Jul 22  2021 bootmgr.efi

[Bugsy]

Further investigation shows that it fails somewhere after (and I don't have enough experience/knowledge on VC to identify where it fails):
https://github.com/veracrypt/VeraCrypt-DCS/blob/master/DcsBoot/DcsBoot.c#L195

What is really weird is that it boots fine on HW and only fails when booted from VM level, what's worse if fails for booting both Windows.

@idrassi, should I downgrade back to 1.24-Update2? Downgrades are said to not be allowed, hence am troubled on next steps, at least some troubleshooting steps to better understand the issue.

[Alexandre Takacs]

Sorry not clear (to me :) from your message - what Hypervisor are you using ?
Do you boot the same disk directly from your PC or as a VM ?

[Bugsy]

Vmware Workstation is the hypervisor (earlier v12, v14, recently v16).
Yes, same physical disk is booted directly and via VM. It all worked fine until above described situation.