Re: [tboot-devel] [PATCH] x86: add a new SMP bring up way for tboot case

[Ren, Q. <qia...@in...>]

On 2013-05-14, H. Peter Anvin wrote:
> On 05/14/2013 02:21 PM, Qiaowei Ren wrote:
>> tboot provides a better AP wakeup mechanism based on cpu MWAIT
>> feature for OS/VMM. With this mechanism, system will boot faster and
>> will NOT require VT to be enabled. But it requires that OS/VMM must
>> have support it, otherwise system can never boot up.
>> 
>> Once this mechanism is enabled, tboot will put APs waiting in MWAIT
>> loops before launching kernel. kernel can check the new flag field
>> in
>> v6 tboot shared page for the hint. If the bit
>> TB_FLAG_AP_WAKE_SUPPORT in flag field is set, kernel BSP has to
>> write the monitored memory
>> (tboot->ap_wake_trigger) to bring APs out of MWAIT loops. The sipi
>> vector should be written in
>> tboot->ap_wake_addr before waking up APs.
>> 
> 
> This really needs a *detailed* specification about the state the CPU is parked in.
> Most BIOSes do in fact park the CPUs in an mwait loop, but we can't
> use it because the CPU state they are parked in is ill-defined.
> 
> This is a good idea, but please write (or point to) a spec about what
> the parked CPU state looks like and how the OS gets control.  From the
> *looks* of the code I assume it is entered in 16-bit real mode but
> then it is important to know what parts of the register state are well-defined.

The following is how to do mwait for tboot & kernel:

For bootstrap processor (BSP), "tboot TXT pre-launch" is executed after BIOS. In this stage, tboot will issue GETSEC[SENTER], which broadcasts messages to the chipset and other physical or logical processors in the platform. In response, other logical processors perform basic cleanup and other tasks, and then finally enter SENTER sleep state.

Next, for BSP, SINIT will run and then enter "tboot post-launch", which will start all sleeping APs. If tboot command line option " ap_wake_mwait=true" is set, APs will do some work and then enter mwait loop. Kernel will be launched in BSP by tboot post-launch, and bring APs out of mwait loop.

Tboot works in protected mode (but paging is disabled), and closes interrupt. For APs, MONITOR and MWAIT related code in tboot is as follows:
    while ( _tboot_shared.ap_wake_trigger != cpuid ) {
        cpu_monitor(&_tboot_shared.ap_wake_trigger, 0, 0);
        mb();
        if ( _tboot_shared.ap_wake_trigger == cpuid )
            break;
        cpu_mwait(0, 0);
    }
Their extension and hint are all 0. According Intel manual:
	Extension=0: Treat interrupts as break events even if masked (e.g., even if EFLAGS.IF=0).
	Hint=0: the preferred optimized state the processor should enter is C0.
So, when "tboot->ap_wake_trigger" is set by kernel, APs can exit from mwait loop.

Peter, I don't know whether I explain your problem. What do you think about it?

Thanks,
Qiaowei