Re: [E1000-devel] Small UDP packets routing performance...

[Xavier T. <xav...@si...>]

Hi Eric,

It's amazing how clear one may see things when somebody points in the right direction. We have been doing some tests and disabling IOMMU changed network performance completely ... And you were totally right, the moment I spent some time reading about IOMMU I've realized how much it affects the performance when you try to forward a big amount of packets per second. I'm amazed it's not one of the first recommendations for Linux routers/firewalls. 

And now hopefully now we will get clearance to buy an fast E5 CPU and some IXGBE cards to run some more tests using Libzero for DNA (Bonesi isn't fast enough for 10gbps) as we are really interested about routing/firewalling using Linux boxes and it finally seems possible :)

Thanks man, you did to us a really big favor telling us the source of the performance problem. IOMMU was in our list of suspects (With many other possibilities), but you saved us weeks of work. 

BTW, I know I'm probably abusing... But do you know any other extremely important kernel options like IOMMU that could affect performance that much? 

And well, I hope anybody in our same situation will find this post in the future.

Thanks again.

Saludos cordiales,
Xavier Trilla P.
Silicon Hosting

¿Todavía no conoces Bare Metal Cloud?
¡La evolución de los Servidores VPS ya ha llegado!

más información en: siliconhosting.com/cloud


-----Mensaje original-----
De: Eric Dumazet [mailto:eri...@gm...] 
Enviado el: jueves, 18 de abril de 2013 8:48
Para: Xavier Trilla
CC: e10...@li...; Arnau Marcé
Asunto: RE: [E1000-devel] Small UDP packets routing performance...

On Thu, 2013-04-18 at 02:42 +0000, Xavier Trilla wrote:
> Hi Eric,
> 
> We have been doing some more research, and finally we managed to get a 
> fast kernel with a new minimal configuration using the last release of 
> the 2.6.32 (2.6.32.60). We compiled what we think is a kernel with 
> options that could affect network performance removed. We used our 
> general networking/computing knowledge and reviewed every option, so 
> I'm sure it's not perfect but should be a better start point than the 
> one we had. Then we did a "make oldconfig" to adapt this ".config" to 
> a 3.4.41 kernel reviewing every new option.
> 
> But again we have the same situation, 2.6.32 kernel seems to be 
> extremely fast (we have successfully routed more than 2Mpps) and 3.4 
> is not able to route more than 500Kpps, and again when we do a "perf 
> top" we see _raw_spin_lock_irqsave consuming a lot of CPU.
> 
> BTW: Remember that we use for this test just one Xeon 5620 in a dual 
> Xeon environment, so 50% CPU usage means we are using the 100% of the 
> cores assigned to RSS queues.
> 
> Kernel 3.4.41 CLEAN
> 
> Packet generator: Two bonesi instances each one sending about 500Kpps 
> from 50K sources.
> Machine reaches 100% CPU usage on cores with RSS queues assigned, and 
> doesn't route more than 500kpps.
> 
> perf top output: http://pastebin.com/xByZnxL1 perf record output: 
> http://pastebin.com/2idhM3V1
> 
> Kernel 2.6.32.60 CLEAN
> 
> Packet Generator: Two bonesi instances each one sending about 500Kpps 
> 50K sources.
> Machine routes about 1Mpps (Bonesi doesn't generate exactly 500Kpps 
> per instance, so a small difference  is normal).
> 
> perf record output: http://pastebin.com/gfvcQNZv perf top output: 
> http://pastebin.com/8Qpp604p
> 
> Note: We use a CISCO 3750G to measure the pps in each port, so we can 
> see number of packets coming in and out (Flow control is disabled on 
> the switch ports).
> 
> As you may see, the main difference is that on kernel 3.4.41 
> _raw_spin_lock_irqsave is using the 46.4% of the CPU (So, 92.8% of the 
> Cores assigned to RSS queues).
> 
> Also, I'm starting to have doubts about this being related with IGB 
> driver... but anyway if you have any idea that could lead us in the 
> right direction it would be extremely useful :)
> 
> P.S.: I'm not familiar at all with lists, so I just did a "reply all"
> from my e-mail client but I see you'll get a direct copy of this 
> mail...  Should it work like that or I shall just reply to the 
> e10...@li... mail address? If I did wrong please 
> just let me know.
> 
> And... Thanks for your help :)
> 
> Saludos cordiales,
> Xavier Trilla P.
> Silicon Hosting
> 

Check on your .config files on both kernels :

grep IOMMU .config

Then try to boot your 3.4.41 machine adding "iommu=off" 

Or read the documentation about IOMMU in general...