#494 Intel PCI NIC passthrough problem

[Anonymous]

Host:
- CPU Core2Duo E6300, Intel q45 chipset, VT-d enabled
- Ubuntu 9.10 x86_64, kernel 2.6.31.4 recompiled with CONFIG_DMAR=y and CONFIG_INTR_REMAP=y according to http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
- in-kernel kvm module
- qemu-kvm commit c04b2aebf50c7d8cba883b86d1b872ccfc8f2249
- intel_iommu=igfx_off
- Qemu command line: /usr/local/bin/qemu-system-x86_64 -m 512 -k en-us -drive if=ide,file=/dev/server2/ubuntu,boot=on -cdrom /home/install/Linux/i386/ubuntu-9.10-desktop-i386.iso -boot d -vga std -pcidevice host=01:00.0 -net none -vnc :15 -daemonize

Guest OSes:
- Ubuntu 9.10 live CD i386, Windows Server 2003 i386, Windows Server 2008 x86_64, MacOS X 10.5.x i386

The device on the host:
01:00.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
Subsystem: Intel Corporation Device 002e
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort-="">SERR- <PERR- INTx-
Latency: 32 (63750ns min), Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 31
Region 0: Memory at d0540000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at d0520000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at d000 [size=64]
Expansion ROM at bf000000 [disabled] [size=128K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [e4] PCI-X non-bridge device
Command: DPERE- ERO+ RBC=512 OST=1
Status: Dev=00:00.0 64bit- 133MHz- SCD- USC- DC=simple DMMRBC=2048 DMOST=1 DMCRS=16 RSCEM- 266MHz- 533MHz-
Capabilities: [f0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable+
Address: 00000000fee0300c Data: 41e9
Kernel driver in use: pci-stub
Kernel modules: e1000

Symptoms:
The device is found and initialized in all OS'es. The driver properly detects the link speed, and does detect when I plug the cable in or out. However, it does not send or receive any packets, all the counters are always at 0's though there must be more at least due to DHCP activity. dmesg does not show any errors neither on the host or on the guest.

ifconfig on the guest:
eth0 Link encap:Ethernet HWaddr 00:07:e9:0f:c8:10
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

lspci -vvv on the guest:
....
00:03.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 02)
Subsystem: Intel Corporation Device 002e
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort-="">SERR- <PERR- INTx-
Latency: 32 (63750ns min), Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 11
Region 0: Memory at f0000000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at f0020000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at c040 [size=64]
Expansion ROM at 20000000 [disabled] [size=128K]
Capabilities: [40] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable-
Address: 00000000 Data: 0000
Kernel driver in use: e1000
Kernel modules: e1000

cat /proc/interrupts on the guest:
CPU0
0: 89 IO-APIC-edge timer
1: 1088 IO-APIC-edge i8042
4: 2 IO-APIC-edge
6: 2 IO-APIC-edge floppy
7: 4 IO-APIC-edge parport0
8: 0 IO-APIC-edge rtc0
9: 0 IO-APIC-fasteoi acpi
11: 0 IO-APIC-fasteoi eth0
12: 1333 IO-APIC-edge i8042
14: 98 IO-APIC-edge ata_piix
15: 9874 IO-APIC-edge ata_piix
NMI: 0 Non-maskable interrupts
LOC: 59892 Local timer interrupts
SPU: 0 Spurious interrupts
CNT: 0 Performance counter interrupts
PND: 0 Performance pending work
RES: 0 Rescheduling interrupts
CAL: 0 Function call interrupts
TLB: 0 TLB shootdowns
TRM: 0 Thermal event interrupts
THR: 0 Threshold APIC interrupts
MCE: 0 Machine check exceptions
MCP: 7 Machine check polls
ERR: 0
MIS: 0

[Anonymous]

I should note that I also tried to pass through the built-in Intel Ethernet adapter of this board and got the same.

[Anonymous]

Note: with intel_iommu=on, same problem.

[Anonymous]

qemu-kvm-0.11 stable - same result.
Anybody managed to passthrough an Intel PCI/PCI-e NIC?

[Anonymous]

Update. Tested with three different Intel cards: PCI (e1000), PCI-e (e1000e) and the onboard one (e1000e). Actually, it turned out to be two different issues.

1) With intel_iommu=igfx_off, none of them are working. Did I misunderstood the purpose of igfx_off? I thought it only disables IOMMU for graphics devices. With intel_iommu=on, all of them work, but...

2) ...with the e1000 it does not work when the guest is Linux. The host prints messages telling about a DMAR fault accessing the PCIe-to-PCI bridge device. I assume it may be a bad behavior of the e1000 linux driver which tries to access the device parent bridge while it is not allowed to. Should I file a bug on the e1000 driver? This card works however with Windows and MacOS when intel_iommu=on.

[Anonymous]

Compiled latest kvm-git. Now I do not try igfx_off as it is not needed to run X anymore.

I've found out that #2 is actually a combination two different issues:

2a) Sometimes (randomly) the card just do not work with none of the OS's. Same symptoms as described - detects cable but no traffic. This correlates with DMAR failures accessing the PCI bridge. Most of the time the card works with Windows and MacOS X guests, and when it works, there are no DMAR messages.

2b) The card just never works with the Ubuntu guest. All counters are at 0's, interrupt counter in /proc/interrupts is at 0. In some runs, there were DMAR fault messages, at some there were none, so I now assume DMAR faults are not the reason of this issue.