Re: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000 NIC

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

OK, Thanks John.

-Kumar

----- Original Message ----
From: "Ronciak, John" <joh...@in...>
To: Kumar Narayanan <kn...@ya...>; "Leech, Christopher" <chr...@in...>; "Brandeburg, Jesse" <jes...@in...>; e10...@li...
Cc: Vikram Venkataraghavan <vik...@ya...>
Sent: Thursday, June 19, 2008 9:28:27 AM
Subject: RE: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000 NIC

No way 
around having the rtnl_lock() lock set when using IOCTLs.

Yes 
you should be able to do line rate as long as you can supply the packets fast 
enough on the send side from user space.  The raw socket interface is fast 
enough and you aren't going though the regular TCP stack.  I assume your 
app will be doing some of this so that why I'm saying that you'll need to be 
able to supply the packets fast enough.

Cheers, 
John 
----------------------------------------------------------- 
"Those who would give up essential Liberty, to 
purchase a little temporary Safety, deserve neither Liberty nor Safety.", 
Benjamin Franklin 1755 

________________________________
 From: Kumar Narayanan [mailto:kn...@ya...] 
Sent: Thursday, June 19, 2008 9:03 AM
To: Leech, 
Christopher; Ronciak, John; Brandeburg, Jesse; 
e10...@li...
Cc: Vikram 
Venkataraghavan
Subject: Re: [E1000-devel] RSS - Receive-Side Scaling 
on Intel PRO/1000 NIC

Thanks 
for all the response..

Understood that the IOCTLs encounter rtnl_lock(). 
I am assuming that rtnl_lock() is absolutely
needed. I imagine that it's not 
recommended to bypass rtnl_lock(). (We don't plan to do this; I
was asking 
from an understanding perspective.)

We're running on an Intel Quad-Core. 
We'll look at Raw Socket, and get that to work. Do you
think that we can get 
the full line speed performance - with packet sizes >1000, with raw 
socket.

Thanks,
-Kumar

----- 
Original Message ----
From: "Leech, Christopher" 
<chr...@in...>
To: "Ronciak, John" 
<joh...@in...>; Kumar Narayanan <kn...@ya...>; 
"Brandeburg, Jesse" <jes...@in...>; 
e10...@li...
Cc: Vikram Venkataraghavan 
<vik...@ya...>
Sent: Wednesday, June 18, 2008 5:42:24 
PM
Subject: RE: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000 
NIC

IOCTLs to network interfaces are called under rtnl_lock(), so only 
one
IOCTL to any network interface in the system will be active at a 
time.
See dev_ioctl() in net/core/dev.c.

John's right, you most likely 
want a packet socket. See man packet.

- Chris

Ronciak, John 
wrote:
> An IOCTL to a driver is going to block until satisfied.  
This means
> that even with 4 interfaces running each IOCTL will be 
blocking in
> each instance of the driver.  You have not said how 
many processor
> you have running.  My guess is if you have  
each of the 4 interfaces
> running on a separate processor you'll get back 
to the 1 port
> performance.  I also think that you won't get much 
above some limit
> as the interface will block on each packet both sending 
and receiving.
> 
> Why can't you use a raw socket as the 
interface?  Seems like it would
> be a much better mechanism for your 
use.
> 
> 
> Cheers,
> John
> 
-----------------------------------------------------------
> "Those who 
would give up essential Liberty, to purchase a little
> temporary Safety, 
deserve neither Liberty nor Safety.", Benjamin
> Franklin 1755
> 
> 
> 
> ________________________________
> 
> 
From: Kumar Narayanan [mailto:kn...@ya...]
> Sent: Wednesday, 
June 18, 2008 4:10 PM
> To: Brandeburg, Jesse; Ronciak, John;
> e10...@li... Cc: Vikram Venkataraghavan
> Subject: Re: [E1000-devel] RSS - Receive-Side 
Scaling on Intel
> PRO/1000 NIC
> 
> 
> Hi,
> I am 
leaving the older message just for the sake of continuity in case
> you 
need to know any of the history.
> 
> We're using Intel Pro/1000 VT 
with RHEL 5.2 and trying to use IOCTL to
> send and receive data. 
We're
> using the igb driver but there's some change we put in to get 
the
> packet right after it's DMAed from the
> NIC, and use IOCTL to 
send the packet to the user space. This is the
> Rx path. In the Tx path 
we're calling
> IOCTL from user space to send a bunch of pointers to the 
driver and
> use Scatter-Gather to send the packet
> out.
> 
> We can't seem to get more than 50MByte/S per port, when we run on 
all
> 4 ports. With single port we can
> see 90MByte/S, occasionally 
reaching 105MBytes/S . With 2 ports it's
> somewhere at the rate of 
60MB/S,
> with 3 ports, it's 50+MB/S, and with 4 ports max. we're able to 
do
> 200MB/S aggregate. (4 ports at wire
> speed should give an 
aggregate of over 440MB/S, counting off TCP/IP
> overheads.)
> 
> We *believe* (not sure) that we're hitting serialization issues. 
We
> believe that even though we make 4
> IOCTL calls for packets to 
4 different ports all these calls are
> serialized. We can provide more 
info. as
> needed.
> 
> Please let us know if there's 
something fundamentally wrong. Any
> pointers to other's data would 
be
> helpful. Finally, if you need more data on the 
driver/kernel/debug
> commands/traces we can get that.
> 
> 
Thanks,
> -Kumar
> 
> 
> 
> ----- Original Message 
----
> From: "Brandeburg, Jesse" <jes...@in...>
> 
To: Kumar Narayanan <kn...@ya...>; "Ronciak, 
John"
> <joh...@in...>; Richard 
Scobie <ri...@sa...>;
> e10...@li...
> 
Sent: Tuesday, May 20, 2008 8:49:14 AM
> Subject: RE: [E1000-devel] RSS - 
Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> 
> 
yes, my reference to IGB *only* applies to 82575 based VT adapter, not
> 
to the 82571 based PT adapter.
> 
> Thanks for the dell 
reference!
> 
> Jesse
> 
> 
________________________________
> 
> From: Kumar Narayanan 
[mailto:kn...@ya...]
> Sent: Monday, May 
19, 2008 7:13 PM
> To: Brandeburg, Jesse; Ronciak, John; Richard 
Scobie;
> e10...@li...
> 
Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 
NIC
> 
> 
> Thanks for the quick response, Jesse.
> 
> I think that I'll use the e1000 on another server where we don't 
need
> RSS. Just that
> we spent time, and ended up probably costing 
all your time also.
> Moving on...
> 
> I suppose that your 
reference to IGB driver, MSI-X, 2.6.23 (or
> 2.6.25.4) etc. apply to 82575 
(Pro 1000 VT), right? Or does it apply
> to Pro 1000 PT also?
> 
> The NIC is orderable. Dell <http://www.dell.com/>  Part# : 430-2688.
> Go to 
"http://www.dell.com" 
and
> enter 430-2688 in the search box. It'll let you order from there. 
I
> just discovered
> it today - wasn't orderable till late 
lastweek.
> 
> -Kumar
> 
> 
> ----- Original 
Message ----
> From: "Brandeburg, Jesse" <jes...@in...>
> 
To: Kumar Narayanan <kn...@ya...>; "Ronciak, 
John"
> <joh...@in...>; Richard 
Scobie <ri...@sa...>;
> e10...@li...
> 
Sent: Monday, May 19, 2008 6:15:24 PM
> Subject: RE: [E1000-devel] RSS - 
Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> 
> 
First, you should be able to return your PT adapters, I'm sorry that
> you 
got a bad deal from the marketing documentation.
> 
> Second, the 
Intel provided IGB driver on sourceforge supports full tx
> and rx 
multiple queues with MSI-X (tx multiple queues requires a
> 2.6.23, 
recommend a 2.6.25.4 kernel) and RSS based steering to the
> multiple 
queues.  Right now the hash that steers the flows to a
> particular 
queue is random, but can easily be made static with a
> simple patch. 
> 
> I suggest you see if you can find an Intel distributor who 
will sample
> you an 82575 based board.  If that doesn't work for you 
then I suggest
> you buy one from dell (you'll probably have to call them, 
as we
> haven't found it on the website yet)
> 
> 
Jesse
> 
> ________________________________
> 
> From: 
Kumar Narayanan [mailto:kn...@ya...]
> Sent: Monday, May 
19, 2008 4:05 PM
> To: Ronciak, John; Brandeburg, Jesse; Richard 
Scobie;
> e10...@li...
> 
Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 
NIC
> 
> 
> Based on the email below and some more reading I 
am thinking of
> getting Intel
> Pro/1000 VT quad-port GbE 
NIC.  Please let me know if any of the
> following
> 
assumptions are incorrect.
> 1. It's based on 85275
> 2. It supports 
MSI-X
> 3. It has RSS support
> 4. Need kernel version 2.6.23 or 
RHEL 5.2
> 
> The reason for asking these questions is that looking 
at Intel's web
> site it appeared
> that Intel Pro/1000 PT quad-port 
GbE NIC supports RSS. While this may
> be of
> technical interest as 
a user I can't have RSS function with PT
> adapters with Linux. I
> 
didn't see *anything* in the Intel's product site that says that PT
> 
adapters has RSS
> feature but it's not usable with Linux. I am trying to 
prevent myself
> from falling into
> the same/similar trap 
now.
> 
> Thanks,
> -Kumar
> 
> 
> ----- 
Original Message ----
> From: "Ronciak, John" <joh...@in...>
> 
To: Kumar Narayanan <kn...@ya...>; "Brandeburg, 
Jesse"
> <jes...@in...>; 
Richard Scobie <ri...@sa...>;
> e10...@li...
> 
Sent: Friday, May 2, 2008 1:35:11 PM
> Subject: RE: [E1000-devel] RSS - 
Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
>> I am 
assuming that without RSS the interrupts are free to go to any
>> of 
the cores in a multi-core system. Is that a fair statement?
> Yes that is 
true assuming that user-space level IRQ balancer is being
> used.  
The in-kernel version doesn't work very well at all.
> 
>> With 
RSS we thought we might be able to bring in session stickiness -
> I.E. 
packets belonging to >the same session will find way to the same
> 
core. Is that a valid assumption?
> Well yes that is some what true but as 
Auke has said, not having MSI-X
> interrupts enabled on the 82571 adapters 
hurt the performance which
> could be gained.  You can turn off the 
IRQ balancer and the set the
> affinity of the connections to a single 
processor which might give you
> what you are after.
> 
> The 
real way to get all of this to work is to use the 82575 parts with
> the 
igb driver.  It supports MSI-X which might be better for what you
> 
are trying to do.
> 
> On the I/OAT front, please make sure that you 
try RHEL5.2 as there
> are a number of bug fixes and added features 
regarding I/OAT.  It
> matches what is in the 2.6.23 kernel regarding 
I/OAT.  It releases
> later this month but the final RCs are 
available today.
> 
> 
> Cheers,
> John
> 
-----------------------------------------------------------
> "Those who 
would give up essential Liberty, to purchase a little
> temporary Safety, 
deserve neither Liberty nor Safety.", Benjamin
> Franklin 1755
> 
> -----Original Message-----
> From: e10...@li...
> 
[mailto:e10...@li...] 
On Behalf Of Kumar
> Narayanan
> Sent: Friday, May 02, 2008 11:37 
AM
> To: Brandeburg, Jesse; Richard Scobie;
> e10...@li... Subject: Re: [E1000-devel] RSS -
> Receive-Side Scaling on Intel PRO/1000 
NIC
> 
> Hi Jesse, Thanks for the kind reply... I got a good amount 
of info.
> from your's and Auke's email.
> 
> We enabled IOAT 
with the hope that what's being said for WIN (that RSS
> is a
> 
necessary condition for IOAT) may be true for LINUX also, and that by
> 
enabling
> IOAT we could get RSS also going ...
> 
> I am 
assuming that without RSS the interrupts are free to go to any of
> the 
cores in
> a multi-core system. Is that a fair statement? With RSS we 
thought we
> might be
> able to bring in session stickiness - I.E 
packets belonging to the
> same session will
> find way to the same 
core. Is that a valid assumption?
> 
> What would be my options to 
achieve the session stickiness as I
> mentioned above?
> What's my 
next best option?
> 
> Thanks,
> -Kumar
> 
> 
> 
> ----- Original Message ----
> From: "Brandeburg, Jesse" 
<jes...@in...>
> 
To: Kumar Narayanan <kn...@ya...>; Richard Scobie
> 
<ri...@sa...>; e10...@li...
> 
Sent: Thursday, May 1, 2008 6:32:15 PM
> Subject: RE: [E1000-devel] RSS - 
Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> Hi Kumar, 
maybe look for "Crystal Beach" in bios for enabling IOAT.
> Some vendors 
left the code name in there.
> 
> yes, IOAT (copy offload) is not 
needed for supporting RSS.
> 
> while the hardware supports RSS, the 
Linux drivers don't currently do
> anything for it in e1000/e1000e.  
all the parts supported by e1000e
> have RSS and at least 2 queue support, 
but don't have MSI-X, so can
> only generate a ssingle interrupt vector, 
which kind of makes
> multiple queues not very useful on those 
parts.
> 
> Tell us what you were trying to achieve with RSS and let 
us help you
> get there, maybe without RSS.
> 
> Jesse
> 
> -----Original Message-----
> From: e10...@li...
> 
[mailto:e10...@li...] 
On Behalf Of Kumar
> Narayanan
> Sent: Thursday, May 01, 2008 6:28 
PM
> To: Richard Scobie; e10...@li...
> 
Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 
NIC
> 
> Hello Richard, Thanks for the email.
> 
> We 
don't need I/O AT per se. Somewhere we read in Intel doc. that RSS
> is 
a
> necessary condition for I/OAT to function. However, that seems to 
be
> relevant
> for WIN server OS only.
> 
> We want 
only RSS. We enabled I/O AT thinking that RSS *might* get
> 
enabled
> if I/O AT is also enabled. I wonder if anyone is using RSS in 
LINUX
> with Intel
> PRO/1000 GbE NIC. Intel technical support came 
back with no answer.
> Hence
> I am asking the e1000 
developers...
> 
> BTW, I checked. I/O AT mod is installed but it's 
not enabled. I read
> the URL
> you gave me. It appears that we need 
to enable it in BIOS. I am trying
> to understand
> what command in 
BIOS will enable I/O AT.
> 
> 
> 
> 
> ----- 
Original Message ----
> From: Richard Scobie <ri...@sa...>
> To: e10...@li...
> 
Sent: Thursday, May 1, 2008 5:15:13 PM
> Subject: Re: [E1000-devel] RSS - 
Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> Kumar 
Narayanan wrote:
>> Hello, I am using Intel Pro/1000 PT Quad Port 
Server NIC with RHEL 5.
>> I am trying to enable RSS - Receive-Side 
Scaling. The documentation
>> says that the above NIC supports LINUX 
I/O Scaling - RSS being part
>> of this I/O Scaling feature.
> 
> If you mean IOAT support, look here and read the README in the 
Linux
> download.
> 
> http://www.intel.com/support/network/adapter/pro100/sb/CS-023725.htm
> 
> Also look at this discussion for details of what you may need 
to
> enable in the BIOS:
> 
>
http://sourceforge.net/mailarchive/forum.php?thread_name=480FBB2F.709000
> 
9%40sauce.co.nz&forum_name=e1000-devel
> 
> Regards,
> 
> Richard
> 
> 
>
------------------------------------------------------------------------
> 
-
> This SF.net email is sponsored by the 2008 JavaOne(SM) 
Conference
> Don't miss this year's exciting event. There's still time to 
save
> $100. Use priority code J8TL2D2.
>
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j
> 
avaone
> _______________________________________________
> 
E1000-devel mailing list
> E10...@li...
> https://lists.sourceforge.net/lists/listinfo/e1000-devel
> 
> 
> 
> 
>
________________________________________________________________________
> 
____________
> Be a better friend, newshound, and
> know-it-all with 
Yahoo! Mobile <http://mobile.yahoo.com/> .  Try it
> now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
> 
> 
> 
> 
>
________________________________________________________________________
> 
____________
> Be a better friend, newshound, and
> know-it-all with 
Yahoo! Mobile.  Try it now.
> http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

Re: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000 NIC

Moved to github.com/intel

Re: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000 NIC