Re: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000 NIC

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

No way around having the rtnl_lock() lock set when using IOCTLs.

Yes you should be able to do line rate as long as you can supply the
packets fast enough on the send side from user space.  The raw socket
interface is fast enough and you aren't going though the regular TCP
stack.  I assume your app will be doing some of this so that why I'm
saying that you'll need to be able to supply the packets fast enough.

Cheers, 
John 
----------------------------------------------------------- 
"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.", Benjamin
Franklin 1755 

________________________________

From: Kumar Narayanan [mailto:kn...@ya...] 
Sent: Thursday, June 19, 2008 9:03 AM
To: Leech, Christopher; Ronciak, John; Brandeburg, Jesse;
e10...@li...
Cc: Vikram Venkataraghavan
Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000
NIC

Thanks for all the response..

Understood that the IOCTLs encounter rtnl_lock(). I am assuming that
rtnl_lock() is absolutely
needed. I imagine that it's not recommended to bypass rtnl_lock(). (We
don't plan to do this; I
was asking from an understanding perspective.)

We're running on an Intel Quad-Core. We'll look at Raw Socket, and get
that to work. Do you
think that we can get the full line speed performance - with packet
sizes >1000, with raw socket.

Thanks,
-Kumar

----- Original Message ----
From: "Leech, Christopher" <chr...@in...>
To: "Ronciak, John" <joh...@in...>; Kumar Narayanan
<kn...@ya...>; "Brandeburg, Jesse" <jes...@in...>;
e10...@li...
Cc: Vikram Venkataraghavan <vik...@ya...>
Sent: Wednesday, June 18, 2008 5:42:24 PM
Subject: RE: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000
NIC

IOCTLs to network interfaces are called under rtnl_lock(), so only one
IOCTL to any network interface in the system will be active at a time.
See dev_ioctl() in net/core/dev.c.

John's right, you most likely want a packet socket. See man packet.

- Chris

Ronciak, John wrote:
> An IOCTL to a driver is going to block until satisfied.  This means
> that even with 4 interfaces running each IOCTL will be blocking in
> each instance of the driver.  You have not said how many processor
> you have running.  My guess is if you have  each of the 4 interfaces
> running on a separate processor you'll get back to the 1 port
> performance.  I also think that you won't get much above some limit
> as the interface will block on each packet both sending and receiving.
> 
> Why can't you use a raw socket as the interface?  Seems like it would
> be a much better mechanism for your use.
> 
> 
> Cheers,
> John
> -----------------------------------------------------------
> "Those who would give up essential Liberty, to purchase a little
> temporary Safety, deserve neither Liberty nor Safety.", Benjamin
> Franklin 1755
> 
> 
> 
> ________________________________
> 
> From: Kumar Narayanan [mailto:kn...@ya...]
> Sent: Wednesday, June 18, 2008 4:10 PM
> To: Brandeburg, Jesse; Ronciak, John;
> e10...@li... Cc: Vikram Venkataraghavan
> Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> 
> Hi,
> I am leaving the older message just for the sake of continuity in case
> you need to know any of the history.
> 
> We're using Intel Pro/1000 VT with RHEL 5.2 and trying to use IOCTL to
> send and receive data. We're
> using the igb driver but there's some change we put in to get the
> packet right after it's DMAed from the
> NIC, and use IOCTL to send the packet to the user space. This is the
> Rx path. In the Tx path we're calling
> IOCTL from user space to send a bunch of pointers to the driver and
> use Scatter-Gather to send the packet
> out.
> 
> We can't seem to get more than 50MByte/S per port, when we run on all
> 4 ports. With single port we can
> see 90MByte/S, occasionally reaching 105MBytes/S . With 2 ports it's
> somewhere at the rate of 60MB/S,
> with 3 ports, it's 50+MB/S, and with 4 ports max. we're able to do
> 200MB/S aggregate. (4 ports at wire
> speed should give an aggregate of over 440MB/S, counting off TCP/IP
> overheads.)
> 
> We *believe* (not sure) that we're hitting serialization issues. We
> believe that even though we make 4
> IOCTL calls for packets to 4 different ports all these calls are
> serialized. We can provide more info. as
> needed.
> 
> Please let us know if there's something fundamentally wrong. Any
> pointers to other's data would be
> helpful. Finally, if you need more data on the driver/kernel/debug
> commands/traces we can get that.
> 
> Thanks,
> -Kumar
> 
> 
> 
> ----- Original Message ----
> From: "Brandeburg, Jesse" <jes...@in...>
> To: Kumar Narayanan <kn...@ya...>; "Ronciak, John"
> <joh...@in...>; Richard Scobie <ri...@sa...>;
> e10...@li...
> Sent: Tuesday, May 20, 2008 8:49:14 AM
> Subject: RE: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> 
> yes, my reference to IGB *only* applies to 82575 based VT adapter, not
> to the 82571 based PT adapter.
> 
> Thanks for the dell reference!
> 
> Jesse
> 
> ________________________________
> 
> From: Kumar Narayanan [mailto:kn...@ya...]
> Sent: Monday, May 19, 2008 7:13 PM
> To: Brandeburg, Jesse; Ronciak, John; Richard Scobie;
> e10...@li...
> Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> 
> Thanks for the quick response, Jesse.
> 
> I think that I'll use the e1000 on another server where we don't need
> RSS. Just that
> we spent time, and ended up probably costing all your time also.
> Moving on...
> 
> I suppose that your reference to IGB driver, MSI-X, 2.6.23 (or
> 2.6.25.4) etc. apply to 82575 (Pro 1000 VT), right? Or does it apply
> to Pro 1000 PT also?
> 
> The NIC is orderable. Dell <http://www.dell.com/>  Part# : 430-2688.
> Go to "http://www.dell.com" and
> enter 430-2688 in the search box. It'll let you order from there. I
> just discovered
> it today - wasn't orderable till late lastweek.
> 
> -Kumar
> 
> 
> ----- Original Message ----
> From: "Brandeburg, Jesse" <jes...@in...>
> To: Kumar Narayanan <kn...@ya...>; "Ronciak, John"
> <joh...@in...>; Richard Scobie <ri...@sa...>;
> e10...@li...
> Sent: Monday, May 19, 2008 6:15:24 PM
> Subject: RE: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> 
> First, you should be able to return your PT adapters, I'm sorry that
> you got a bad deal from the marketing documentation.
> 
> Second, the Intel provided IGB driver on sourceforge supports full tx
> and rx multiple queues with MSI-X (tx multiple queues requires a
> 2.6.23, recommend a 2.6.25.4 kernel) and RSS based steering to the
> multiple queues.  Right now the hash that steers the flows to a
> particular queue is random, but can easily be made static with a
> simple patch. 
> 
> I suggest you see if you can find an Intel distributor who will sample
> you an 82575 based board.  If that doesn't work for you then I suggest
> you buy one from dell (you'll probably have to call them, as we
> haven't found it on the website yet)
> 
> Jesse
> 
> ________________________________
> 
> From: Kumar Narayanan [mailto:kn...@ya...]
> Sent: Monday, May 19, 2008 4:05 PM
> To: Ronciak, John; Brandeburg, Jesse; Richard Scobie;
> e10...@li...
> Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> 
> Based on the email below and some more reading I am thinking of
> getting Intel
> Pro/1000 VT quad-port GbE NIC.  Please let me know if any of the
> following
> assumptions are incorrect.
> 1. It's based on 85275
> 2. It supports MSI-X
> 3. It has RSS support
> 4. Need kernel version 2.6.23 or RHEL 5.2
> 
> The reason for asking these questions is that looking at Intel's web
> site it appeared
> that Intel Pro/1000 PT quad-port GbE NIC supports RSS. While this may
> be of
> technical interest as a user I can't have RSS function with PT
> adapters with Linux. I
> didn't see *anything* in the Intel's product site that says that PT
> adapters has RSS
> feature but it's not usable with Linux. I am trying to prevent myself
> from falling into
> the same/similar trap now.
> 
> Thanks,
> -Kumar
> 
> 
> ----- Original Message ----
> From: "Ronciak, John" <joh...@in...>
> To: Kumar Narayanan <kn...@ya...>; "Brandeburg, Jesse"
> <jes...@in...>; Richard Scobie <ri...@sa...>;
> e10...@li...
> Sent: Friday, May 2, 2008 1:35:11 PM
> Subject: RE: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
>> I am assuming that without RSS the interrupts are free to go to any
>> of the cores in a multi-core system. Is that a fair statement?
> Yes that is true assuming that user-space level IRQ balancer is being
> used.  The in-kernel version doesn't work very well at all.
> 
>> With RSS we thought we might be able to bring in session stickiness -
> I.E. packets belonging to >the same session will find way to the same
> core. Is that a valid assumption?
> Well yes that is some what true but as Auke has said, not having MSI-X
> interrupts enabled on the 82571 adapters hurt the performance which
> could be gained.  You can turn off the IRQ balancer and the set the
> affinity of the connections to a single processor which might give you
> what you are after.
> 
> The real way to get all of this to work is to use the 82575 parts with
> the igb driver.  It supports MSI-X which might be better for what you
> are trying to do.
> 
> On the I/OAT front, please make sure that you try RHEL5.2 as there
> are a number of bug fixes and added features regarding I/OAT.  It
> matches what is in the 2.6.23 kernel regarding I/OAT.  It releases
> later this month but the final RCs are available today.
> 
> 
> Cheers,
> John
> -----------------------------------------------------------
> "Those who would give up essential Liberty, to purchase a little
> temporary Safety, deserve neither Liberty nor Safety.", Benjamin
> Franklin 1755
> 
> -----Original Message-----
> From: e10...@li...
> [mailto:e10...@li...] On Behalf Of Kumar
> Narayanan
> Sent: Friday, May 02, 2008 11:37 AM
> To: Brandeburg, Jesse; Richard Scobie;
> e10...@li... Subject: Re: [E1000-devel] RSS -
> Receive-Side Scaling on Intel PRO/1000 NIC
> 
> Hi Jesse, Thanks for the kind reply... I got a good amount of info.
> from your's and Auke's email.
> 
> We enabled IOAT with the hope that what's being said for WIN (that RSS
> is a
> necessary condition for IOAT) may be true for LINUX also, and that by
> enabling
> IOAT we could get RSS also going ...
> 
> I am assuming that without RSS the interrupts are free to go to any of
> the cores in
> a multi-core system. Is that a fair statement? With RSS we thought we
> might be
> able to bring in session stickiness - I.E packets belonging to the
> same session will
> find way to the same core. Is that a valid assumption?
> 
> What would be my options to achieve the session stickiness as I
> mentioned above?
> What's my next best option?
> 
> Thanks,
> -Kumar
> 
> 
> 
> ----- Original Message ----
> From: "Brandeburg, Jesse" <jes...@in...>
> To: Kumar Narayanan <kn...@ya...>; Richard Scobie
> <ri...@sa...>; e10...@li...
> Sent: Thursday, May 1, 2008 6:32:15 PM
> Subject: RE: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> Hi Kumar, maybe look for "Crystal Beach" in bios for enabling IOAT.
> Some vendors left the code name in there.
> 
> yes, IOAT (copy offload) is not needed for supporting RSS.
> 
> while the hardware supports RSS, the Linux drivers don't currently do
> anything for it in e1000/e1000e.  all the parts supported by e1000e
> have RSS and at least 2 queue support, but don't have MSI-X, so can
> only generate a ssingle interrupt vector, which kind of makes
> multiple queues not very useful on those parts.
> 
> Tell us what you were trying to achieve with RSS and let us help you
> get there, maybe without RSS.
> 
> Jesse
> 
> -----Original Message-----
> From: e10...@li...
> [mailto:e10...@li...] On Behalf Of Kumar
> Narayanan
> Sent: Thursday, May 01, 2008 6:28 PM
> To: Richard Scobie; e10...@li...
> Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> Hello Richard, Thanks for the email.
> 
> We don't need I/O AT per se. Somewhere we read in Intel doc. that RSS
> is a
> necessary condition for I/OAT to function. However, that seems to be
> relevant
> for WIN server OS only.
> 
> We want only RSS. We enabled I/O AT thinking that RSS *might* get
> enabled
> if I/O AT is also enabled. I wonder if anyone is using RSS in LINUX
> with Intel
> PRO/1000 GbE NIC. Intel technical support came back with no answer.
> Hence
> I am asking the e1000 developers...
> 
> BTW, I checked. I/O AT mod is installed but it's not enabled. I read
> the URL
> you gave me. It appears that we need to enable it in BIOS. I am trying
> to understand
> what command in BIOS will enable I/O AT.
> 
> 
> 
> 
> ----- Original Message ----
> From: Richard Scobie <ri...@sa...>
> To: e10...@li...
> Sent: Thursday, May 1, 2008 5:15:13 PM
> Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> Kumar Narayanan wrote:
>> Hello, I am using Intel Pro/1000 PT Quad Port Server NIC with RHEL 5.
>> I am trying to enable RSS - Receive-Side Scaling. The documentation
>> says that the above NIC supports LINUX I/O Scaling - RSS being part
>> of this I/O Scaling feature.
> 
> If you mean IOAT support, look here and read the README in the Linux
> download.
> 
> http://www.intel.com/support/network/adapter/pro100/sb/CS-023725.htm
> 
> Also look at this discussion for details of what you may need to
> enable in the BIOS:
> 
>
http://sourceforge.net/mailarchive/forum.php?thread_name=480FBB2F.709000
> 9%40sauce.co.nz&forum_name=e1000-devel
> 
> Regards,
> 
> Richard
> 
> 
>
------------------------------------------------------------------------
> -
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
> Don't miss this year's exciting event. There's still time to save
> $100. Use priority code J8TL2D2.
>
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j
> avaone
> _______________________________________________
> E1000-devel mailing list
> E10...@li...
> https://lists.sourceforge.net/lists/listinfo/e1000-devel
> 
> 
> 
> 
>
________________________________________________________________________
> ____________
> Be a better friend, newshound, and
> know-it-all with Yahoo! Mobile <http://mobile.yahoo.com/> .  Try it
> now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
> 
> 
> 
> 
>
________________________________________________________________________
> ____________
> Be a better friend, newshound, and
> know-it-all with Yahoo! Mobile.  Try it now.
> http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

Re: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000 NIC

Moved to github.com/intel

Re: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000 NIC