Re: [tboot-devel] VT-d PMRs

[Jonathan M. M. <jon...@cm...>]

Hi Joe, list,

I started digging through this.

In the VT-d spec, the very beginning of chapter 7 says that each DMA
remapping hardware unit is placed at a system-specific 4K-aligned memory
mapped address, and that the BIOS will report where these are.

I realized I don't know what these addresses are, and Ch. 5 told me to
look at the DMAR table.

I wrote some code to decode the DMAR table, and the subsequent DRHD /
RMRR entries it contains, and the subsequent Device Scope Structures
that they contain.

When run on my system following execution of SENTER, the big long
printout at the end of this email ensues.  I made an educated guess that
the DRHD structure with flags = 1 (see Table 5-3 in the VT-d spec) might
be the DMA remapping hardware unit??? of choice.  I tried to print its
values at the relevant offsets for the PMRs (see Table 7-2), but I get
all 0s:

TBOOT: ACPI VT-d DRHD structure @ 7c6a3e78:
TBOOT:  Type: 0
TBOOT:  Length: 0x10 (16)
TBOOT:  Flags: 1
TBOOT:  Register Base Address: 0xfed93000
TBOOT:          DMAR DRHD Registers @ fed93000:
TBOOT:                  PMR Enable: 0x0
TBOOT:                  PMR Low Base: 0x0
TBOOT:                  PMR Low Limit: 0x0
TBOOT:                  PMR High Base: 0x0
TBOOT:                  PMR High Limit: 0x0


Am I on the right track here?  How do I know which of these "DMA
remapping hardware units" is the right one?

Thanks!
-Jon





TBOOT: acpi_dmar = 7c6a3df0
TBOOT: VT-d DMAR @ 7c6a3df0 (len 360 bytes):
44 4d 41 52 68 01 00 00 01 c7 31 30 31 36 30 37
4f 45 4d 44 4d 41 52 20 01 00 00 00 4d 53 46 54
97 00 00 00 23 00 00 00 00 00 00 00 00 00 00 00
00 00 18 00 00 00 00 00 00 00 d9 fe 00 00 00 00
01 08 00 00 00 00 1b 00 00 00 18 00 00 00 00 00
00 10 d9 fe 00 00 00 00 01 08 00 00 00 00 02 00
00 00 28 00 00 00 00 00 00 20 d9 fe 00 00 00 00
01 08 00 00 00 00 03 00 01 08 00 00 00 00 03 02
01 08 00 00 00 00 03 03 00 00 10 00 01 00 00 00
00 30 d9 fe 00 00 00 00 01 00 58 00 00 00 00 00
00 d0 0e 00 00 00 00 00 ff ff 0e 00 00 00 00 00
01 08 00 00 00 00 1d 00 01 08 00 00 00 00 1d 01
01 08 00 00 00 00 1d 02 01 08 00 00 00 00 1d 07
01 08 00 00 00 00 1a 00 01 08 00 00 00 00 1a 01
01 08 00 00 00 00 1a 02 01 08 00 00 00 00 1a 07
01 00 58 00 00 00 00 00 00 00 2f 7d 00 00 00 00
ff ff 2f 7d 00 00 00 00 01 08 00 00 00 00 1d 00
01 08 00 00 00 00 1d 01 01 08 00 00 00 00 1d 02
01 08 00 00 00 00 1d 07 01 08 00 00 00 00 1a 00
01 08 00 00 00 00 1a 01 01 08 00 00 00 00 1a 02
01 08 00 00 00 00 1a 07 01 00 20 00 00 00 00 00
00 00 60 7d 00 00 00 00 ff ff ff 7d 00 00 00 00
01 08 00 00 00 00 02 00
TBOOT: ACPI VT-d DMAR table @ 7c6a3df0 (len 48 bytes):
TBOOT:  Signature: DMAR
TBOOT:  Length: 0x168 (360)
TBOOT:  Revision: 1
TBOOT:  Checksum: c7
TBOOT:  OEMID: 101607
TBOOT:  OEM Table ID: OEMDMAR
TBOOT:  OEM Revision: 1
TBOOT:  Creator ID: MSFT
TBOOT:  Creator Revision: 0x97
TBOOT:  Host Address Width: 0x23 (35)
TBOOT: ACPI VT-d DRHD structure @ 7c6a3e20:
TBOOT:  Type: 0
TBOOT:  Length: 0x18 (24)
TBOOT:  Flags: 0
TBOOT:  Register Base Address: 0xfed90000
TBOOT:  Device Scope entry @ 7c6a3e30:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1b 00

TBOOT: ACPI VT-d DRHD structure @ 7c6a3e38:
TBOOT:  Type: 0
TBOOT:  Length: 0x18 (24)
TBOOT:  Flags: 0
TBOOT:  Register Base Address: 0xfed91000
TBOOT:  Device Scope entry @ 7c6a3e48:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 02 00

TBOOT: ACPI VT-d DRHD structure @ 7c6a3e50:
TBOOT:  Type: 0
TBOOT:  Length: 0x28 (40)
TBOOT:  Flags: 0
TBOOT:  Register Base Address: 0xfed92000
TBOOT:  Device Scope entry @ 7c6a3e60:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 03 00

TBOOT:  Device Scope entry @ 7c6a3e68:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 03 02

TBOOT:  Device Scope entry @ 7c6a3e70:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 03 03

TBOOT: ACPI VT-d DRHD structure @ 7c6a3e78:
TBOOT:  Type: 0
TBOOT:  Length: 0x10 (16)
TBOOT:  Flags: 1
TBOOT:  Register Base Address: 0xfed93000
TBOOT:          DMAR DRHD Registers @ fed93000:
TBOOT:                  PMR Enable: 0x0
TBOOT:                  PMR Low Base: 0x0
TBOOT:                  PMR Low Limit: 0x0
TBOOT:                  PMR High Base: 0x0
TBOOT:                  PMR High Limit: 0x0
TBOOT: ACPI VT-d RMRR structure @ 7c6a3e88:
TBOOT:  Type: 0x1 (1)
TBOOT:  Length: 0x58 (88)
TBOOT:  Flag: 0x0
TBOOT:  RMR Base Address:  ed000
TBOOT:  RMR Limit Address: effff
TBOOT:  Device Scope entry @ 7c6a3ea0:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 00

TBOOT:  Device Scope entry @ 7c6a3ea8:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 01

TBOOT:  Device Scope entry @ 7c6a3eb0:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 02

TBOOT:  Device Scope entry @ 7c6a3eb8:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 07

TBOOT:  Device Scope entry @ 7c6a3ec0:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 00

TBOOT:  Device Scope entry @ 7c6a3ec8:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 01

TBOOT:  Device Scope entry @ 7c6a3ed0:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 02

TBOOT:  Device Scope entry @ 7c6a3ed8:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 07

TBOOT: ACPI VT-d RMRR structure @ 7c6a3ee0:
TBOOT:  Type: 0x1 (1)
TBOOT:  Length: 0x58 (88)
TBOOT:  Flag: 0x0
TBOOT:  RMR Base Address:  7d2f0000
TBOOT:  RMR Limit Address: 7d2fffff
TBOOT:  Device Scope entry @ 7c6a3ef8:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 00

TBOOT:  Device Scope entry @ 7c6a3f00:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 01

TBOOT:  Device Scope entry @ 7c6a3f08:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 02

TBOOT:  Device Scope entry @ 7c6a3f10:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 07

TBOOT:  Device Scope entry @ 7c6a3f18:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 00

TBOOT:  Device Scope entry @ 7c6a3f20:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 01

TBOOT:  Device Scope entry @ 7c6a3f28:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 02

TBOOT:  Device Scope entry @ 7c6a3f30:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 07

TBOOT: ACPI VT-d RMRR structure @ 7c6a3f38:
TBOOT:  Type: 0x1 (1)
TBOOT:  Length: 0x20 (32)
TBOOT:  Flag: 0x0
TBOOT:  RMR Base Address:  7d600000
TBOOT:  RMR Limit Address: 7dffffff
TBOOT:  Device Scope entry @ 7c6a3f50:
TBOOT:          Type: 0x1
TBOOT:          Length: 0x8
TBOOT:          Segment Number: 0x0
TBOOT:          Starting Bus Number: 0x0
TBOOT:          Raw PCI Path (Length - 4): 00 00 02 00

TBOOT: VT-d DMAR table OK


Cihula, Joseph wrote:
>> From: Jonathan M. McCune [mailto:jon...@cm...]
>> Sent: Thursday, October 09, 2008 10:27 AM
>>
>> Hi Joe,
>>
>> Cihula, Joseph wrote:
>>> It is really:
>>> 	Current values of VT-d PMR registers do not match requested
>>> values in SinitMleData
>>>
>>> which means that some code has already programmed the PMRs but not
> to
>>> the same values that the MLE is requesting.  Because the PMRs cannot
>> be
>>> changed reliably once they are set/enabled, this is not an allowed
>>> condition.
>>
>> I am successfully invoking SENTER / SEXIT and then returning control
> to
>> the legacy OS once per boot cycle.  When I try to execute SENTER a
>> second time, the system reboots and LT.ERRORCODE is populated with the
>> above error.  However, I cannot figure out what is wrong with my PMR
>> values.
>>
>> Is there a way to read the current values, so that I can see how they
>> are set following the first SENTER? Looking through the MLE manual and
>> the Sw Dev Manual Vol 2b has left me without much insight.
> 
> VT-d registers are described in the VT-d spec at:
> http://download.intel.com/technology/computing/vptech/Intel(r)_VT_for_Di
> rect_IO.pdf
> 
>> My MLE resides at physical address 0x00c00000 and consumes less than
>> 0x200000 (2MB) bytes.
>>
>> TBOOT:   vtd_pmr_lo_base: 0xc00000
>> TBOOT:   vtd_pmr_lo_size: 0x200000
>> TBOOT:   vtd_pmr_hi_base: 0x0
>> TBOOT:   vtd_pmr_hi_size: 0x0
>>
>> I'm fairly confident that the os_sinit_data_t.vtd_pmr_* values are
>> being
>> set identically prior to both invocations of SENTER (when I print them
>> out, the above is what I see).
> 
> If you could use the VT-d spec to read and output the PMRs before
> launch, that should determine if they are somehow getting changed
> between invocations.
> 
>> Thanks,
>> -Jon
>