Re: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000 NIC

[Kumar N. <kn...@ya...>]

Thanks for all the response..

Understood that the IOCTLs encounter rtnl_lock(). I am assuming that rtnl_lock() is absolutely
needed. I imagine that it's not recommended to bypass rtnl_lock(). (We don't plan to do this; I
was asking from an understanding perspective.)

We're running on an Intel Quad-Core. We'll look at Raw Socket, and get that to work. Do you
think that we can get the full line speed performance - with packet sizes >1000, with raw socket.

Thanks,
-Kumar


----- Original Message ----
From: "Leech, Christopher" <chr...@in...>
To: "Ronciak, John" <joh...@in...>; Kumar Narayanan <kn...@ya...>; "Brandeburg, Jesse" <jes...@in...>; e10...@li...
Cc: Vikram Venkataraghavan <vik...@ya...>
Sent: Wednesday, June 18, 2008 5:42:24 PM
Subject: RE: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000 NIC

IOCTLs to network interfaces are called under rtnl_lock(), so only one
IOCTL to any network interface in the system will be active at a time.
See dev_ioctl() in net/core/dev.c.

John's right, you most likely want a packet socket. See man packet.

- Chris

Ronciak, John wrote:
> An IOCTL to a driver is going to block until satisfied.  This means
> that even with 4 interfaces running each IOCTL will be blocking in
> each instance of the driver.  You have not said how many processor
> you have running.  My guess is if you have  each of the 4 interfaces
> running on a separate processor you'll get back to the 1 port
> performance.  I also think that you won't get much above some limit
> as the interface will block on each packet both sending and receiving.
> 
> Why can't you use a raw socket as the interface?  Seems like it would
> be a much better mechanism for your use.
> 
> 
> Cheers,
> John
> -----------------------------------------------------------
> "Those who would give up essential Liberty, to purchase a little
> temporary Safety, deserve neither Liberty nor Safety.", Benjamin
> Franklin 1755
> 
> 
> 
> ________________________________
> 
> From: Kumar Narayanan [mailto:kn...@ya...]
> Sent: Wednesday, June 18, 2008 4:10 PM
> To: Brandeburg, Jesse; Ronciak, John;
> e10...@li... Cc: Vikram Venkataraghavan
> Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> 
> Hi,
> I am leaving the older message just for the sake of continuity in case
> you need to know any of the history.
> 
> We're using Intel Pro/1000 VT with RHEL 5.2 and trying to use IOCTL to
> send and receive data. We're
> using the igb driver but there's some change we put in to get the
> packet right after it's DMAed from the
> NIC, and use IOCTL to send the packet to the user space. This is the
> Rx path. In the Tx path we're calling
> IOCTL from user space to send a bunch of pointers to the driver and
> use Scatter-Gather to send the packet
> out.
> 
> We can't seem to get more than 50MByte/S per port, when we run on all
> 4 ports. With single port we can
> see 90MByte/S, occasionally reaching 105MBytes/S . With 2 ports it's
> somewhere at the rate of 60MB/S,
> with 3 ports, it's 50+MB/S, and with 4 ports max. we're able to do
> 200MB/S aggregate. (4 ports at wire
> speed should give an aggregate of over 440MB/S, counting off TCP/IP
> overheads.)
> 
> We *believe* (not sure) that we're hitting serialization issues. We
> believe that even though we make 4
> IOCTL calls for packets to 4 different ports all these calls are
> serialized. We can provide more info. as
> needed.
> 
> Please let us know if there's something fundamentally wrong. Any
> pointers to other's data would be
> helpful. Finally, if you need more data on the driver/kernel/debug
> commands/traces we can get that.
> 
> Thanks,
> -Kumar
> 
> 
> 
> ----- Original Message ----
> From: "Brandeburg, Jesse" <jes...@in...>
> To: Kumar Narayanan <kn...@ya...>; "Ronciak, John"
> <joh...@in...>; Richard Scobie <ri...@sa...>;
> e10...@li...
> Sent: Tuesday, May 20, 2008 8:49:14 AM
> Subject: RE: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> 
> yes, my reference to IGB *only* applies to 82575 based VT adapter, not
> to the 82571 based PT adapter.
> 
> Thanks for the dell reference!
> 
> Jesse
> 
> ________________________________
> 
> From: Kumar Narayanan [mailto:kn...@ya...]
> Sent: Monday, May 19, 2008 7:13 PM
> To: Brandeburg, Jesse; Ronciak, John; Richard Scobie;
> e10...@li...
> Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> 
> Thanks for the quick response, Jesse.
> 
> I think that I'll use the e1000 on another server where we don't need
> RSS. Just that
> we spent time, and ended up probably costing all your time also.
> Moving on...
> 
> I suppose that your reference to IGB driver, MSI-X, 2.6.23 (or
> 2.6.25.4) etc. apply to 82575 (Pro 1000 VT), right? Or does it apply
> to Pro 1000 PT also?
> 
> The NIC is orderable. Dell <http://www.dell.com/>  Part# : 430-2688.
> Go to "http://www.dell.com" and
> enter 430-2688 in the search box. It'll let you order from there. I
> just discovered
> it today - wasn't orderable till late lastweek.
> 
> -Kumar
> 
> 
> ----- Original Message ----
> From: "Brandeburg, Jesse" <jes...@in...>
> To: Kumar Narayanan <kn...@ya...>; "Ronciak, John"
> <joh...@in...>; Richard Scobie <ri...@sa...>;
> e10...@li...
> Sent: Monday, May 19, 2008 6:15:24 PM
> Subject: RE: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> 
> First, you should be able to return your PT adapters, I'm sorry that
> you got a bad deal from the marketing documentation.
> 
> Second, the Intel provided IGB driver on sourceforge supports full tx
> and rx multiple queues with MSI-X (tx multiple queues requires a
> 2.6.23, recommend a 2.6.25.4 kernel) and RSS based steering to the
> multiple queues.  Right now the hash that steers the flows to a
> particular queue is random, but can easily be made static with a
> simple patch. 
> 
> I suggest you see if you can find an Intel distributor who will sample
> you an 82575 based board.  If that doesn't work for you then I suggest
> you buy one from dell (you'll probably have to call them, as we
> haven't found it on the website yet)
> 
> Jesse
> 
> ________________________________
> 
> From: Kumar Narayanan [mailto:kn...@ya...]
> Sent: Monday, May 19, 2008 4:05 PM
> To: Ronciak, John; Brandeburg, Jesse; Richard Scobie;
> e10...@li...
> Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> 
> Based on the email below and some more reading I am thinking of
> getting Intel
> Pro/1000 VT quad-port GbE NIC.  Please let me know if any of the
> following
> assumptions are incorrect.
> 1. It's based on 85275
> 2. It supports MSI-X
> 3. It has RSS support
> 4. Need kernel version 2.6.23 or RHEL 5.2
> 
> The reason for asking these questions is that looking at Intel's web
> site it appeared
> that Intel Pro/1000 PT quad-port GbE NIC supports RSS. While this may
> be of
> technical interest as a user I can't have RSS function with PT
> adapters with Linux. I
> didn't see *anything* in the Intel's product site that says that PT
> adapters has RSS
> feature but it's not usable with Linux. I am trying to prevent myself
> from falling into
> the same/similar trap now.
> 
> Thanks,
> -Kumar
> 
> 
> ----- Original Message ----
> From: "Ronciak, John" <joh...@in...>
> To: Kumar Narayanan <kn...@ya...>; "Brandeburg, Jesse"
> <jes...@in...>; Richard Scobie <ri...@sa...>;
> e10...@li...
> Sent: Friday, May 2, 2008 1:35:11 PM
> Subject: RE: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
>> I am assuming that without RSS the interrupts are free to go to any
>> of the cores in a multi-core system. Is that a fair statement?
> Yes that is true assuming that user-space level IRQ balancer is being
> used.  The in-kernel version doesn't work very well at all.
> 
>> With RSS we thought we might be able to bring in session stickiness -
> I.E. packets belonging to >the same session will find way to the same
> core. Is that a valid assumption?
> Well yes that is some what true but as Auke has said, not having MSI-X
> interrupts enabled on the 82571 adapters hurt the performance which
> could be gained.  You can turn off the IRQ balancer and the set the
> affinity of the connections to a single processor which might give you
> what you are after.
> 
> The real way to get all of this to work is to use the 82575 parts with
> the igb driver.  It supports MSI-X which might be better for what you
> are trying to do.
> 
> On the I/OAT front, please make sure that you try RHEL5.2 as there
> are a number of bug fixes and added features regarding I/OAT.  It
> matches what is in the 2.6.23 kernel regarding I/OAT.  It releases
> later this month but the final RCs are available today.
> 
> 
> Cheers,
> John
> -----------------------------------------------------------
> "Those who would give up essential Liberty, to purchase a little
> temporary Safety, deserve neither Liberty nor Safety.", Benjamin
> Franklin 1755
> 
> -----Original Message-----
> From: e10...@li...
> [mailto:e10...@li...] On Behalf Of Kumar
> Narayanan
> Sent: Friday, May 02, 2008 11:37 AM
> To: Brandeburg, Jesse; Richard Scobie;
> e10...@li... Subject: Re: [E1000-devel] RSS -
> Receive-Side Scaling on Intel PRO/1000 NIC
> 
> Hi Jesse, Thanks for the kind reply... I got a good amount of info.
> from your's and Auke's email.
> 
> We enabled IOAT with the hope that what's being said for WIN (that RSS
> is a
> necessary condition for IOAT) may be true for LINUX also, and that by
> enabling
> IOAT we could get RSS also going ...
> 
> I am assuming that without RSS the interrupts are free to go to any of
> the cores in
> a multi-core system. Is that a fair statement? With RSS we thought we
> might be
> able to bring in session stickiness - I.E packets belonging to the
> same session will
> find way to the same core. Is that a valid assumption?
> 
> What would be my options to achieve the session stickiness as I
> mentioned above?
> What's my next best option?
> 
> Thanks,
> -Kumar
> 
> 
> 
> ----- Original Message ----
> From: "Brandeburg, Jesse" <jes...@in...>
> To: Kumar Narayanan <kn...@ya...>; Richard Scobie
> <ri...@sa...>; e10...@li...
> Sent: Thursday, May 1, 2008 6:32:15 PM
> Subject: RE: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> Hi Kumar, maybe look for "Crystal Beach" in bios for enabling IOAT.
> Some vendors left the code name in there.
> 
> yes, IOAT (copy offload) is not needed for supporting RSS.
> 
> while the hardware supports RSS, the Linux drivers don't currently do
> anything for it in e1000/e1000e.  all the parts supported by e1000e
> have RSS and at least 2 queue support, but don't have MSI-X, so can
> only generate a ssingle interrupt vector, which kind of makes
> multiple queues not very useful on those parts.
> 
> Tell us what you were trying to achieve with RSS and let us help you
> get there, maybe without RSS.
> 
> Jesse
> 
> -----Original Message-----
> From: e10...@li...
> [mailto:e10...@li...] On Behalf Of Kumar
> Narayanan
> Sent: Thursday, May 01, 2008 6:28 PM
> To: Richard Scobie; e10...@li...
> Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> Hello Richard, Thanks for the email.
> 
> We don't need I/O AT per se. Somewhere we read in Intel doc. that RSS
> is a
> necessary condition for I/OAT to function. However, that seems to be
> relevant
> for WIN server OS only.
> 
> We want only RSS. We enabled I/O AT thinking that RSS *might* get
> enabled
> if I/O AT is also enabled. I wonder if anyone is using RSS in LINUX
> with Intel
> PRO/1000 GbE NIC. Intel technical support came back with no answer.
> Hence
> I am asking the e1000 developers...
> 
> BTW, I checked. I/O AT mod is installed but it's not enabled. I read
> the URL
> you gave me. It appears that we need to enable it in BIOS. I am trying
> to understand
> what command in BIOS will enable I/O AT.
> 
> 
> 
> 
> ----- Original Message ----
> From: Richard Scobie <ri...@sa...>
> To: e10...@li...
> Sent: Thursday, May 1, 2008 5:15:13 PM
> Subject: Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
> PRO/1000 NIC
> 
> Kumar Narayanan wrote:
>> Hello, I am using Intel Pro/1000 PT Quad Port Server NIC with RHEL 5.
>> I am trying to enable RSS - Receive-Side Scaling. The documentation
>> says that the above NIC supports LINUX I/O Scaling - RSS being part
>> of this I/O Scaling feature.
> 
> If you mean IOAT support, look here and read the README in the Linux
> download.
> 
> http://www.intel.com/support/network/adapter/pro100/sb/CS-023725.htm
> 
> Also look at this discussion for details of what you may need to
> enable in the BIOS:
> 
>
http://sourceforge.net/mailarchive/forum.php?thread_name=480FBB2F.709000
> 9%40sauce.co.nz&forum_name=e1000-devel
> 
> Regards,
> 
> Richard
> 
> 
>
------------------------------------------------------------------------
> -
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
> Don't miss this year's exciting event. There's still time to save
> $100. Use priority code J8TL2D2.
>
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j
> avaone
> _______________________________________________
> E1000-devel mailing list
> E10...@li...
> https://lists.sourceforge.net/lists/listinfo/e1000-devel
> 
> 
> 
> 
>
________________________________________________________________________
> ____________
> Be a better friend, newshound, and
> know-it-all with Yahoo! Mobile <http://mobile.yahoo.com/> .  Try it
> now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
> 
> 
> 
> 
>
________________________________________________________________________
> ____________
> Be a better friend, newshound, and
> know-it-all with Yahoo! Mobile.  Try it now.
> http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ