Re: [tboot-devel] VT-d PMRs

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

> From: Jonathan M. McCune [mailto:jon...@cm...]
> Sent: Friday, October 10, 2008 3:11 PM
> 
> Hi Joe, list,
> 
> I started digging through this.
> 
> In the VT-d spec, the very beginning of chapter 7 says that each DMA
> remapping hardware unit is placed at a system-specific 4K-aligned
> memory mapped address, and that the BIOS will report where these are.
> 
> I realized I don't know what these addresses are, and Ch. 5 told me to
> look at the DMAR table.
> 
> I wrote some code to decode the DMAR table, and the subsequent DRHD /
> RMRR entries it contains, and the subsequent Device Scope Structures
> that they contain.
> 
> When run on my system following execution of SENTER, the big long
> printout at the end of this email ensues.  I made an educated guess
> that the DRHD structure with flags = 1 (see Table 5-3 in the VT-d
spec)
> might be the DMA remapping hardware unit??? of choice.  I tried to
print its
> values at the relevant offsets for the PMRs (see Table 7-2), but I get
> all 0s:
> 
> TBOOT: ACPI VT-d DRHD structure @ 7c6a3e78:
> TBOOT:  Type: 0
> TBOOT:  Length: 0x10 (16)
> TBOOT:  Flags: 1
> TBOOT:  Register Base Address: 0xfed93000
> TBOOT:          DMAR DRHD Registers @ fed93000:
> TBOOT:                  PMR Enable: 0x0
> TBOOT:                  PMR Low Base: 0x0
> TBOOT:                  PMR Low Limit: 0x0
> TBOOT:                  PMR High Base: 0x0
> TBOOT:                  PMR High Limit: 0x0
> 
> 
> Am I on the right track here?  How do I know which of these "DMA
> remapping hardware units" is the right one?

These entries are not 4k aligned, so they aren't the right places.  The
easiest thing to do is to look at the Xen code that parses the tables
and reads the registers to see how to do it.

> 
> Thanks!
> -Jon
> 
> 
> 
> 
> 
> TBOOT: acpi_dmar = 7c6a3df0
> TBOOT: VT-d DMAR @ 7c6a3df0 (len 360 bytes):
> 44 4d 41 52 68 01 00 00 01 c7 31 30 31 36 30 37
> 4f 45 4d 44 4d 41 52 20 01 00 00 00 4d 53 46 54
> 97 00 00 00 23 00 00 00 00 00 00 00 00 00 00 00
> 00 00 18 00 00 00 00 00 00 00 d9 fe 00 00 00 00
> 01 08 00 00 00 00 1b 00 00 00 18 00 00 00 00 00
> 00 10 d9 fe 00 00 00 00 01 08 00 00 00 00 02 00
> 00 00 28 00 00 00 00 00 00 20 d9 fe 00 00 00 00
> 01 08 00 00 00 00 03 00 01 08 00 00 00 00 03 02
> 01 08 00 00 00 00 03 03 00 00 10 00 01 00 00 00
> 00 30 d9 fe 00 00 00 00 01 00 58 00 00 00 00 00
> 00 d0 0e 00 00 00 00 00 ff ff 0e 00 00 00 00 00
> 01 08 00 00 00 00 1d 00 01 08 00 00 00 00 1d 01
> 01 08 00 00 00 00 1d 02 01 08 00 00 00 00 1d 07
> 01 08 00 00 00 00 1a 00 01 08 00 00 00 00 1a 01
> 01 08 00 00 00 00 1a 02 01 08 00 00 00 00 1a 07
> 01 00 58 00 00 00 00 00 00 00 2f 7d 00 00 00 00
> ff ff 2f 7d 00 00 00 00 01 08 00 00 00 00 1d 00
> 01 08 00 00 00 00 1d 01 01 08 00 00 00 00 1d 02
> 01 08 00 00 00 00 1d 07 01 08 00 00 00 00 1a 00
> 01 08 00 00 00 00 1a 01 01 08 00 00 00 00 1a 02
> 01 08 00 00 00 00 1a 07 01 00 20 00 00 00 00 00
> 00 00 60 7d 00 00 00 00 ff ff ff 7d 00 00 00 00
> 01 08 00 00 00 00 02 00
> TBOOT: ACPI VT-d DMAR table @ 7c6a3df0 (len 48 bytes):
> TBOOT:  Signature: DMAR
> TBOOT:  Length: 0x168 (360)
> TBOOT:  Revision: 1
> TBOOT:  Checksum: c7
> TBOOT:  OEMID: 101607
> TBOOT:  OEM Table ID: OEMDMAR
> TBOOT:  OEM Revision: 1
> TBOOT:  Creator ID: MSFT
> TBOOT:  Creator Revision: 0x97
> TBOOT:  Host Address Width: 0x23 (35)
> TBOOT: ACPI VT-d DRHD structure @ 7c6a3e20:
> TBOOT:  Type: 0
> TBOOT:  Length: 0x18 (24)
> TBOOT:  Flags: 0
> TBOOT:  Register Base Address: 0xfed90000
> TBOOT:  Device Scope entry @ 7c6a3e30:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1b 00
> 
> TBOOT: ACPI VT-d DRHD structure @ 7c6a3e38:
> TBOOT:  Type: 0
> TBOOT:  Length: 0x18 (24)
> TBOOT:  Flags: 0
> TBOOT:  Register Base Address: 0xfed91000
> TBOOT:  Device Scope entry @ 7c6a3e48:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 02 00
> 
> TBOOT: ACPI VT-d DRHD structure @ 7c6a3e50:
> TBOOT:  Type: 0
> TBOOT:  Length: 0x28 (40)
> TBOOT:  Flags: 0
> TBOOT:  Register Base Address: 0xfed92000
> TBOOT:  Device Scope entry @ 7c6a3e60:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 03 00
> 
> TBOOT:  Device Scope entry @ 7c6a3e68:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 03 02
> 
> TBOOT:  Device Scope entry @ 7c6a3e70:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 03 03
> 
> TBOOT: ACPI VT-d DRHD structure @ 7c6a3e78:
> TBOOT:  Type: 0
> TBOOT:  Length: 0x10 (16)
> TBOOT:  Flags: 1
> TBOOT:  Register Base Address: 0xfed93000
> TBOOT:          DMAR DRHD Registers @ fed93000:
> TBOOT:                  PMR Enable: 0x0
> TBOOT:                  PMR Low Base: 0x0
> TBOOT:                  PMR Low Limit: 0x0
> TBOOT:                  PMR High Base: 0x0
> TBOOT:                  PMR High Limit: 0x0
> TBOOT: ACPI VT-d RMRR structure @ 7c6a3e88:
> TBOOT:  Type: 0x1 (1)
> TBOOT:  Length: 0x58 (88)
> TBOOT:  Flag: 0x0
> TBOOT:  RMR Base Address:  ed000
> TBOOT:  RMR Limit Address: effff
> TBOOT:  Device Scope entry @ 7c6a3ea0:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 00
> 
> TBOOT:  Device Scope entry @ 7c6a3ea8:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 01
> 
> TBOOT:  Device Scope entry @ 7c6a3eb0:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 02
> 
> TBOOT:  Device Scope entry @ 7c6a3eb8:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 07
> 
> TBOOT:  Device Scope entry @ 7c6a3ec0:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 00
> 
> TBOOT:  Device Scope entry @ 7c6a3ec8:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 01
> 
> TBOOT:  Device Scope entry @ 7c6a3ed0:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 02
> 
> TBOOT:  Device Scope entry @ 7c6a3ed8:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 07
> 
> TBOOT: ACPI VT-d RMRR structure @ 7c6a3ee0:
> TBOOT:  Type: 0x1 (1)
> TBOOT:  Length: 0x58 (88)
> TBOOT:  Flag: 0x0
> TBOOT:  RMR Base Address:  7d2f0000
> TBOOT:  RMR Limit Address: 7d2fffff
> TBOOT:  Device Scope entry @ 7c6a3ef8:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 00
> 
> TBOOT:  Device Scope entry @ 7c6a3f00:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 01
> 
> TBOOT:  Device Scope entry @ 7c6a3f08:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 02
> 
> TBOOT:  Device Scope entry @ 7c6a3f10:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1d 07
> 
> TBOOT:  Device Scope entry @ 7c6a3f18:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 00
> 
> TBOOT:  Device Scope entry @ 7c6a3f20:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 01
> 
> TBOOT:  Device Scope entry @ 7c6a3f28:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 02
> 
> TBOOT:  Device Scope entry @ 7c6a3f30:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 1a 07
> 
> TBOOT: ACPI VT-d RMRR structure @ 7c6a3f38:
> TBOOT:  Type: 0x1 (1)
> TBOOT:  Length: 0x20 (32)
> TBOOT:  Flag: 0x0
> TBOOT:  RMR Base Address:  7d600000
> TBOOT:  RMR Limit Address: 7dffffff
> TBOOT:  Device Scope entry @ 7c6a3f50:
> TBOOT:          Type: 0x1
> TBOOT:          Length: 0x8
> TBOOT:          Segment Number: 0x0
> TBOOT:          Starting Bus Number: 0x0
> TBOOT:          Raw PCI Path (Length - 4): 00 00 02 00
> 
> TBOOT: VT-d DMAR table OK
> 
> 
> Cihula, Joseph wrote:
> >> From: Jonathan M. McCune [mailto:jon...@cm...]
> >> Sent: Thursday, October 09, 2008 10:27 AM
> >>
> >> Hi Joe,
> >>
> >> Cihula, Joseph wrote:
> >>> It is really:
> >>> 	Current values of VT-d PMR registers do not match requested
> >>> values in SinitMleData
> >>>
> >>> which means that some code has already programmed the PMRs but not
> > to
> >>> the same values that the MLE is requesting.  Because the PMRs
> cannot
> >> be
> >>> changed reliably once they are set/enabled, this is not an allowed
> >>> condition.
> >>
> >> I am successfully invoking SENTER / SEXIT and then returning
control
> > to
> >> the legacy OS once per boot cycle.  When I try to execute SENTER a
> >> second time, the system reboots and LT.ERRORCODE is populated with
> the
> >> above error.  However, I cannot figure out what is wrong with my
PMR
> >> values.
> >>
> >> Is there a way to read the current values, so that I can see how
> they
> >> are set following the first SENTER? Looking through the MLE manual
> and
> >> the Sw Dev Manual Vol 2b has left me without much insight.
> >
> > VT-d registers are described in the VT-d spec at:
> >
>
http://download.intel.com/technology/computing/vptech/Intel(r)_VT_for_D
> i
> > rect_IO.pdf
> >
> >> My MLE resides at physical address 0x00c00000 and consumes less
than
> >> 0x200000 (2MB) bytes.
> >>
> >> TBOOT:   vtd_pmr_lo_base: 0xc00000
> >> TBOOT:   vtd_pmr_lo_size: 0x200000
> >> TBOOT:   vtd_pmr_hi_base: 0x0
> >> TBOOT:   vtd_pmr_hi_size: 0x0
> >>
> >> I'm fairly confident that the os_sinit_data_t.vtd_pmr_* values are
> >> being
> >> set identically prior to both invocations of SENTER (when I print
> them
> >> out, the above is what I see).
> >
> > If you could use the VT-d spec to read and output the PMRs before
> > launch, that should determine if they are somehow getting changed
> > between invocations.
> >
> >> Thanks,
> >> -Jon
> >