#403 iproute2 vs. ixgbe SR-IOV on >=3.8 kernels

closed
nobody
None
in-kernel_driver
1
7 hours ago
2014-02-21
Joshua Dotson
No

I'm having an issue on late kernels (>=3.8) on both CentOS 6.5 and Ubuntu 12.04.4 64-bit. The issue is that SR-IOV VF(s) are not displayed below their PF(s) as is expected, when running an 'ip link show'. I did an A/B of kernel 2.6 vs. kernel 3.13 (via elrepo's kernel-ml and kmod-ixgbe packages). The VF(s) show properly using 'ip link show' on kernel 2.6 on CentOS 6.5.

Here is some output from a fully updated Ubuntu 12.04.4 LTS 64-bit box (all stock packages). Please help me figure this out. I'm building a 72-node cluster and I have exhausted many avenues of support (mailing lists, IRC, docs, videos, much Googling).

Thanks,
Joshua


@ Wright State University

root@tmp-n1:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:25:90:d8:fa:88 brd ff:ff:ff:ff:ff:ff
3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:25:90:d8:fa:89 brd ff:ff:ff:ff:ff:ff
root@tmp-n1:~# modprobe ixgbe max_vfs=2
root@tmp-n1:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:25:90:d8:fa:88 brd ff:ff:ff:ff:ff:ff
3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:25:90:d8:fa:89 brd ff:ff:ff:ff:ff:ff
11: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 00:25:90:cb:51:08 brd ff:ff:ff:ff:ff:ff
12: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether ba:38:ad:e8:03:70 brd ff:ff:ff:ff:ff:ff
13: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 7e:95:3f:2f:29:78 brd ff:ff:ff:ff:ff:ff
root@tmp-n1:~# lspci|grep 82599
04:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
04:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
04:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
root@tmp-n1:~# uname -a
Linux tmp-n1 3.8.0-35-generic #52~precise1-Ubuntu SMP Thu Jan 30 17:24:40 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
root@tmp-n1:~# ip -V
ip utility, iproute2-ss111117
root@tmp-n1:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.4 LTS
Release:    12.04
Codename:   precise
root@tmp-n1:~# apt-get upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@tmp-n1:~#

Here's the kind of thing I should see:

[root@core-n5 network-scripts]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:14:d1:da:08:59 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:25:90:aa:16:ec brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:25:90:aa:16:ed brd ff:ff:ff:ff:ff:ff
8: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
    link/ether 00:25:90:cb:50:e1 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:a1, vlan 30, tx rate 4000 (Mbps)
    vf 1 MAC 00:00:00:00:00:a2, vlan 40, tx rate 6000 (Mbps)
    vf 2 MAC 00:00:00:00:00:00
    vf 3 MAC 00:00:00:00:00:00
9: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
    link/ether 00:00:00:00:00:a1 brd ff:ff:ff:ff:ff:ff
10: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
    link/ether 00:00:00:00:00:a2 brd ff:ff:ff:ff:ff:ff
11: eth6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 36:05:20:0c:64:c4 brd ff:ff:ff:ff:ff:ff
12: eth7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether aa:7e:2d:0d:3a:56 brd ff:ff:ff:ff:ff:ff
[root@core-n5 network-scripts]#

Related

Bugs: #403

Discussion

  • Joshua Dotson
    Joshua Dotson
    2014-02-21

    • Description has changed:

    Diff:

    --- old
    +++ new
    @@ -1,4 +1,4 @@
    -I'm having an issue on late kernels (>=3.8) on both CentOS 6.5 _and_ Ubuntu 12.04.4 64-bit.  The issue is that SR-IOV VF(s) are not show below their PF(s) as is expected, when running an 'ip link show'.  I did an A/B of kernel 2.6 vs. kernel 3.13 (via elrepo's kernel-ml and kmod-ixgbe packages).  The VF(s) show properly using 'ip link show' on kernel 2.6 on CentOS 6.5.
    +I'm having an issue on late kernels (>=3.8) on both CentOS 6.5 _and_ Ubuntu 12.04.4 64-bit.  The issue is that SR-IOV VF(s) are not displayed below their PF(s) as is expected, when running an 'ip link show'.  I did an A/B of kernel 2.6 vs. kernel 3.13 (via elrepo's kernel-ml and kmod-ixgbe packages).  The VF(s) show properly using 'ip link show' on kernel 2.6 on CentOS 6.5.
    
     Here is some output from a fully updated Ubuntu 12.04.4 LTS 64-bit box (all stock packages).  Please help me figure this out.  I'm building a 72-node cluster and I have exhausted many avenues of support (mailing lists, IRC, docs, videos, much Googling).
    
     
  • Emil Tantilov
    Emil Tantilov
    2014-02-21

    After you updated the kernel did you also update iproute2? In general it is a good idea to keep both updated. If you can reproduce this with an updated iproute2 please post the iproute2 and kernel versions. I did see a similar issue on my setup and it was resolved after updating iproute2.

    I can see the VF information with tip of tree net-next (3.14 based) and:

    ip -V

    ip utility, iproute2-ss131122

     
    • Joshua Dotson
      Joshua Dotson
      2014-02-21

      Emil,

      Compiling iproute2 from latest git fixed this issue on Ubuntu 12.04.4
      LTS. I expect the same will be true for CentOS 6.5 + kernel-ml.

      Thanks very much.

      • Joshua

      On Fri, Feb 21, 2014 at 3:05 PM, Emil Tantilov emiltan@users.sf.net wrote:

      After you updated the kernel did you also update iproute2? In general it
      is a good idea to keep both updated. If you can reproduce this with an
      updated iproute2 please post the iproute2 and kernel versions. I did see a
      similar issue on my setup and it was resolved after updating iproute2.

      I can see the VF information with tip of tree net-next (3.14 based) and:
      ip -V

      ip utility, iproute2-ss131122

      Status: open
      Created: Fri Feb 21, 2014 07:44 PM UTC by Joshua Dotson
      Last Updated: Fri Feb 21, 2014 07:45 PM UTC
      Owner: nobody

      I'm having an issue on late kernels (>=3.8) on both CentOS 6.5 andUbuntu 12.04.4 64-bit. The issue is that SR-IOV VF(s) are not displayed
      below their PF(s) as is expected, when running an 'ip link show'. I did an
      A/B of kernel 2.6 vs. kernel 3.13 (via elrepo's kernel-ml and kmod-ixgbe
      packages). The VF(s) show properly using 'ip link show' on kernel 2.6 on
      CentOS 6.5.

      Here is some output from a fully updated Ubuntu 12.04.4 LTS 64-bit box
      (all stock packages). Please help me figure this out. I'm building a
      72-node cluster and I have exhausted many avenues of support (mailing
      lists, IRC, docs, videos, much Googling).

      Thanks,
      Joshua


      @ Wright State University

      root@tmp-n1:~# ip link show1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:002: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
      link/ether 00:25:90:d8:fa:88 brd ff:ff:ff:ff:ff:ff3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
      link/ether 00:25:90:d8:fa:89 brd ff:ff:ff:ff:ff:ffroot@tmp-n1:~# modprobe ixgbe max_vfs=2root@tmp-n1:~# ip link show1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:002: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
      link/ether 00:25:90:d8:fa:88 brd ff:ff:ff:ff:ff:ff3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
      link/ether 00:25:90:d8:fa:89 brd ff:ff:ff:ff:ff:ff11: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
      link/ether 00:25:90:cb:51:08 brd ff:ff:ff:ff:ff:ff12: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
      link/ether ba:38:ad:e8:03:70 brd ff:ff:ff:ff:ff:ff13: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
      link/ether 7e:95:3f:2f:29:78 brd ff:ff:ff:ff:ff:ffroot@tmp-n1:~# lspci|grep 8259904:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)04:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)04:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)root@tmp-n1:~# uname -aLinux tmp-n1 3.8.0-35-generic #52~precise1-Ubuntu SMP Thu Jan 30 17:24:40 UTC 2014 x86_64 x86_64 x86_64 GNU/Linuxroot@tmp-n1:~# ip -Vip utility, iproute2-ss111117root@tmp-n1:~# lsb_release -aNo LSB modules are available.Distributor ID: UbuntuDescription: Ubuntu 12.04.4 LTSRelease: 12.04Codename: preciseroot@tmp-n1:~# apt-get upgradeReading package lists... DoneBuilding dependency tree Reading state information... Done0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.root@tmp-n1:~#

      Here's the kind of thing I should see:

      [root@core-n5 network-scripts]# ip link show1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:002: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
      link/ether 00:14:d1:da:08:59 brd ff:ff:ff:ff:ff:ff3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
      link/ether 00:25:90:aa:16:ec brd ff:ff:ff:ff:ff:ff4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
      link/ether 00:25:90:aa:16:ed brd ff:ff:ff:ff:ff:ff8: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
      link/ether 00:25:90:cb:50:e1 brd ff:ff:ff:ff:ff:ff
      vf 0 MAC 00:00:00:00:00:a1, vlan 30, tx rate 4000 (Mbps)
      vf 1 MAC 00:00:00:00:00:a2, vlan 40, tx rate 6000 (Mbps)
      vf 2 MAC 00:00:00:00:00:00
      vf 3 MAC 00:00:00:00:00:009: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
      link/ether 00:00:00:00:00:a1 brd ff:ff:ff:ff:ff:ff10: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
      link/ether 00:00:00:00:00:a2 brd ff:ff:ff:ff:ff:ff11: eth6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
      link/ether 36:05:20:0c:64:c4 brd ff:ff:ff:ff:ff:ff12: eth7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
      link/ether aa:7e:2d:0d:3a:56 brd ff:ff:ff:ff:ff:ff[root@core-n5 network-scripts]#


      Sent from sourceforge.net because you indicated interest in
      https://sourceforge.net/p/e1000/bugs/403/

      To unsubscribe from further messages, please visit
      https://sourceforge.net/auth/subscriptions/

       

      Related

      Bugs: #403

  • Todd Fujinaka
    Todd Fujinaka
    2014-03-14

    • status: open --> closed