[Scst-devel] Poor iSCSI Read Performance, Great Write Performance

[David L. <is...@gm...>]

Hey Guys,

I've been banging my head against this problem for quite awhile and feel
like I'm running in circles.  I'm working on configuring iSCSI using
iscsi-scst over 10Gig Ethernet.  Locally I can get roughly 950MB a sec
reads, however, over iSCSI I'm only seeing roughly half that.

With network testing tools I can get 9.5Gigabits out of 10, so I'm fairly
certain that the network card can push the data (confirming with iftop also
shows 9.5gigabits).
With various file system testing I can only get roughly half of that, which
equates to less than half of the 10gigabit pip (as checked with iftop).

Now, writes on the other hand are GREAT! I seem to get BETTER writes over
iSCSI then I get locally... (which actually is concerning I suppose!)
Soooo I'm kinda at a loss! Does anyone have any suggestions about what might
cause this?

The 10gig cards are directly connected to one another (no switch involved)
and MTU is set to 9000.

I'm using the open-iscsi initiator and I've set the bellow additional
settings:

node.session.timeo.replacement_timeout = 120
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.noop_out_interval = 10
node.conn[0].timeo.noop_out_timeout = 15
node.session.initial_login_retry_max = 4
node.session.cmds_max = 128
node.session.queue_depth = 128
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 16776192
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 16776192
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength =     16776192
node.conn[0].iscsi.HeaderDigest = None
node.session.iscsi.FastAbort = No

In the /etc/sysctl.conf I've set this:

net.core.rmem_max = 1073741824
net.core.wmem_max = 1073741824
net.ipv4.tcp_rmem = 1048576 16777216 1073741824
net.ipv4.tcp_wmem = 1048576 16770216 1073741824
net.ipv4.tcp_mem = 1048576 16770216 1073741824

I've adjusted down the rx usecs using ethtool, trying 50/25/10/0
I haven't changed any settings on the iscsi-scst side.

Hardware wise I'm working with a Dell R710 (initiator) talking with a Dell
R510 (target).
The R710 has dual 2..93GHz Quad Core Processors, 72GB of memory and a
Myricom 10Gig nic.
The R510 has dual 2.4GHz Quad Core Processors, 32GB of memory and a Myricom
10Gig nic.  It also has 12 300GB 15k SAS Drives in a hardware Raid0.

I feel like there may be something obvious that I'm missing or not thinking
about. Please let me know what additional information I can provide to help
figure out what's what.

Thanks in advance,
David