Menu
▾
▴
[E1000-devel] ixgbe routing performance
|
From: Aleksey C. <ale...@gm...> - 2011-11-24 10:16:20
|
Hello!
Recently set up two server with the following configuration:
HP ProLiant DL360 G7, 16 GB RAM, 2x Quad Core X5675 @ 3.07GHz
Linux Kernel 2.6.39.4, ixgbe 3.7.14-NAPI
Network interfaces
Eth0 - 09:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit
SFI/SFP+ Network Connection (rev 01)
Eth1 - 09:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit
SFI/SFP+ Network Connection (rev 01)
Eth2 - 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II
BCM5709 Gigabit Ethernet (rev 20)
Eth3 - 03:00.1 Ethernet controller: Broadcom Corporation NetXtreme II
BCM5709 Gigabit Ethernet (rev 20)
Eth4 - 04:00.0 Ethernet controller: Broadcom Corporation NetXtreme II
BCM5709 Gigabit Ethernet (rev 20)
Eth5 - 04:00.1 Ethernet controller: Broadcom Corporation NetXtreme II
BCM5709 Gigabit Ethernet (rev 20)
Servers are directly connected through eth5 NICs for sessions
synchronization.
eth0 and eth1 interfaces aggregated into a single 20G bond0 (802.3ad)
interface connected to Cisco 3750X switch stack
Network topology is one-armed router or router on a stick.
During testing with default iperf settings traffic reached 19,5 Gbit/s IN
without Discards and CPU usage was 60%.
Both servers are used as Load Balancers for LVS Tunnel (routing). So servers
handle only inbound traffic. GSO, GRO and LRO are disabled.
Currently we have approx. 3,5 Gbit/s inbound traffic, 1,2 M inbound
packets/sec and CPU Utilization 55-60% per single core.
With Six-Core CPUs and HT we got 24 TxRx queues and only 16 of them are
loaded.
Try to set bonding.tx_queues=24 and ixgbe.RSS=24,24 but got:
ixgbe: Invalid Receive-Side Scaling (RSS) specified (24), using default.
According to the documentation 82599-based network connections support up to
64 queues.
Is it possible to use more than TxRx 16 queues for routing?
Or maybe there are some other technologies useful for equally distributing
routing traffic to all 24 queues? Considering that clients can come from
various ip addresses.
Are there any advantages of disabling Flow Control while routing?
Ethtool output
# ethtool -i eth0
driver: ixgbe
version: 3.7.14-NAPI
firmware-version: 0x18bf0001
bus-info: 0000:09:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
# ethtool -c eth0
Coalesce parameters for eth0:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
rx-usecs: 1
rx-frames: 0
rx-usecs-irq: 0
rx-frames-irq: 0
tx-usecs: 0
tx-frames: 0
tx-usecs-irq: 0
tx-frames-irq: 256
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
# ethtool -S eth0
NIC statistics:
rx_packets: 8830720773
tx_packets: 8831192772
rx_bytes: 2772419463465
tx_bytes: 2916337716501
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 1729
collisions: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 115405
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_pkts_nic: 8830720806
tx_pkts_nic: 8831192792
rx_bytes_nic: 2807789549695
tx_bytes_nic: 2951760095381
lsc_int: 4
tx_busy: 0
non_eop_descs: 0
broadcast: 26209
rx_no_buffer_count: 0
tx_timeout_count: 0
tx_restart_queue: 19
rx_long_length_errors: 0
rx_short_length_errors: 0
tx_flow_control_xon: 19
rx_flow_control_xon: 0
tx_flow_control_xoff: 1203
rx_flow_control_xoff: 0
rx_csum_offload_errors: 131605
alloc_rx_page_failed: 0
alloc_rx_buff_failed: 0
lro_aggregated: 0
lro_flushed: 0
lro_recycled: 0
rx_no_dma_resources: 0
hw_rsc_aggregated: 0
hw_rsc_flushed: 0
fdir_match: 7944521
fdir_miss: 9050543560
fdir_overflow: 878
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
Best regards,
Aleksey
|