Menu
▾
▴
Re: [Scst-devel] 16Gbit not working
|
From: Lance G. <ei...@pa...> - 2014-12-18 23:42:08
|
Hello Quinn:
So I finally integrated your script into my script, and have some good news and bad news. The good news is this completely fixed the problem with Windows getting an “unknown error,” and the write performance is great at any connection rate. The bad news is I am getting an error at the BIOS level of the client, which reads “Adapter 2 mailbox error 0001”, and the read performance with a 16Gbit connection is still bad. Here’s the performance rates I got – I tested 4Gbit/8Gbit/16Gbit, with Iometer, 8MB transfer lengths with 16 outstanding I/Os:
Internal rates on the server W:3.1GB/sec, R:3.3GB/sec
Rates from Initiator:
4Gbit W:412MB/sec, R:406MB/sec
8Gbit W:823MB/sec, R:772MB/sec
16Gbit W:1577MB/sec, R:745MB/sec
Any ideas on how to fix the 16Gbit read rate?
Lance
From: Lance Gropper
Sent: Monday, December 15, 2014 2:40 PM
To: Quinn Tran ; scs...@li...
Subject: Re: [Scst-devel] 16Gbit not working
Hello Quinn:
Also doesn’t look like the –s option is working – after the command is executed with –s, the interrupts still seem to be funneled to CPU 0:
[root@QOS ~]# ./split_irq.sh -s
before: 34 : ff
after : 34 : 11
before: 37 : ff
after : 37 : 22
before: 35 : ff
after : 35 : 44
before: 43 : ff
after : 43 : 88
before: 36 : ff
after : 36 : 11
before: 44 : ff
after : 44 : 22
[root@QOS ~]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
0: 127 0 0 0 0 0 0 0 IO-APIC-edge timer
1: 2 0 0 0 0 0 0 0 IO-APIC-edge i8042
3: 2 0 0 0 0 0 0 0 IO-APIC-edge
4: 2 0 0 0 0 0 0 0 IO-APIC-edge
8: 1 0 0 0 0 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi
10: 2 0 0 0 0 0 0 0 IO-APIC-edge
12: 4 0 0 0 0 0 0 0 IO-APIC-edge i8042
16: 61 0 0 0 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1
23: 26 0 0 0 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2
24: 5 0 0 0 0 0 0 0 HPET_MSI-edge hpet2
25: 0 0 0 0 0 0 0 0 HPET_MSI-edge hpet3
26: 0 0 0 0 0 0 0 0 HPET_MSI-edge hpet4
27: 0 0 0 0 0 0 0 0 HPET_MSI-edge hpet5
28: 0 0 0 0 0 0 0 0 HPET_MSI-edge hpet6
32: 1367 0 0 0 0 0 0 0 PCI-MSI-edge mpt2sas0-msix0
33: 1178 0 0 0 0 0 0 0 PCI-MSI-edge mpt2sas1-msix0
34: 43 0 0 0 0 0 0 0 PCI-MSI-edge qla2xxx (default)
35: 0 0 0 0 0 0 0 0 PCI-MSI-edge qla2xxx (rsp_q)
36: 0 0 0 0 0 0 0 0 PCI-MSI-edge qla2xxx (atio_q)
37: 43 0 0 0 0 0 0 0 PCI-MSI-edge qla2xxx (default)
38: 5812 0 0 0 0 0 0 0 PCI-MSI-edge ahci
39: 17 0 0 0 0 796 0 0 PCI-MSI-edge eth0
40: 51 0 0 0 0 54 5960 0 PCI-MSI-edge eth1-rx-0
41: 29 0 0 0 0 0 15 898 PCI-MSI-edge eth1-tx-0
42: 2 0 0 0 0 0 0 0 PCI-MSI-edge eth1
43: 0 0 0 0 0 0 0 0 PCI-MSI-edge qla2xxx (rsp_q)
44: 0 0 0 0 0 0 0 0 PCI-MSI-edge qla2xxx (atio_q)
NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts
LOC: 27879 15758 8618 7856 17079 17767 19662 12957 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 0 0 0 0 IRQ work interrupts
RES: 256 254 521 696 408 219 344 382 Rescheduling interrupts
CAL: 275 418 417 385 324 414 345 409 Function call interrupts
TLB: 142 265 191 142 527 1241 901 604 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 7 7 7 7 7 7 7 7 Machine check polls
ERR: 0
MIS: 0
Lance
From: Quinn Tran
Sent: Friday, December 12, 2014 9:43 AM
To: Lance Gropper ; scs...@li...
Subject: Re: [Scst-devel] 16Gbit not working
Lance,
The interrupts on the 16G adapter are different than the 8G adapter. There's additional interrupts to handle the higher load. Some OS/kernel does a good job of spreading the Interrupts across different CPUs to spread the work load. Some don't spread it. This looks like a case of where all interrupt lines are funneled to CPU 0.
This require fine tuning for specific environment that you're in to get the best perf.
Attached is a script I created for this case:
a.. -l option: show where the interrupts are routing to by watching highlighted
b.. -s option: spread the interrupts across different cpu. Use this option before test start.
You're welcome to modify the script to fit your need. The setting does not persist across driver re-load or reboot.
Give this a try. Will include this script into the Qlogic git tree in the future.
Regards,
Quinn Tran
From: Lance Gropper <ei...@pa...>
Date: Thursday, December 11, 2014 4:42 PM
To: "scs...@li..." <scs...@li...>
Subject: [Scst-devel] 16Gbit not working
Hello All:
I got the latest Qlogic GIT for the 16Gbit target drivers – everything builds and loads with no problems. If I attach an 8Gbit Initiator, no problems, and I can saturate the pipes at about 760MB/sec on reads and writes. If, however, I use a 16Gbit initiator, I only get about 277MB/sec on reads and writes. What’s the deal?
Lance
--------------------------------------------------------------------------------
This message and any attached documents contain information from the sending company or its parent company(s), subsidiaries, divisions or branch offices that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.
--------------------------------------------------------------------------------
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
--------------------------------------------------------------------------------
_______________________________________________
Scst-devel mailing list
https://lists.sourceforge.net/lists/listinfo/scst-devel
|