kvm-devel

[kvm-devel] virtio_net backport performance

[Anthony L. <ali...@us...>]

Hey Rusty et al,

I've got automatic backports of the virtio modules[1] working back to 
2.6.18.  Everything seems okay except that for any kernel with the older 
NAPI api, performance is extremely bad.  I get about 1gbit on TX with 
2.6.24 but I get somewhere around 20mbit on 2.6.22.

I had to refactor the NAPI poll function heavily to get it working with 
the old api so my suspicion is that I'm doing something wrong.  I'd 
appreciate if ya'll could take a look at it.  I've included a patch that 
only has the differences of the backported version and the latest 
version.  Pay careful attention to the COMPAT_napi blocks as these are 
what would be used on older kernels.

It's a bit ugly because it's automatically generated by an awk script.

[1] http://hg.codemonkey.ws/virtio-ext-modules

Thanks,

Anthony Liguori

Re: [kvm-devel] virtio_net backport performance

[Rusty R. <ru...@ru...>]

T24gU2F0dXJkYXkgMDUgSmFudWFyeSAyMDA4IDA5OjI0OjQwIEFudGhvbnkgTGlndW9yaSB3cm90
ZToKPiBAQCAtMjE1LDE3ICsyMzEsNDAgQEAKPiDCoMKgwqDCoMKgwqDCoMKgc3RydWN0IHZpcnRu
ZXRfaW5mbyAqdmkgPSBydnEtPnZkZXYtPnByaXY7Cj4gwqDCoMKgwqDCoMKgwqDCoC8qIFN1cHBy
ZXNzIGZ1cnRoZXIgaW50ZXJydXB0cy4gKi8KPiDCoMKgwqDCoMKgwqDCoMKgcnZxLT52cV9vcHMt
PmRpc2FibGVfY2IocnZxKTsKPiArI2lmZGVmIENPTVBBVF9uYXBpCj4gK8KgwqDCoMKgwqDCoMKg
dmktPnJ2cS0+dnFfb3BzLT5lbmFibGVfY2IodmktPnJ2cSk7Cj4gK8KgwqDCoMKgwqDCoMKgaWYg
KG5ldGlmX3J4X3NjaGVkdWxlX3ByZXAodmktPmRldikpIHsKPiArwqDCoMKgwqDCoMKgwqDCoMKg
wqDCoMKgwqDCoMKgdmktPnJ2cS0+dnFfb3BzLT5kaXNhYmxlX2NiKHZpLT5ydnEpOwo+ICvCoMKg
wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqBfX25ldGlmX3J4X3NjaGVkdWxlKHZpLT5kZXYpOwo+
ICvCoMKgwqDCoMKgwqDCoH0gZWxzZQo+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqB2
aS0+cnZxLT52cV9vcHMtPmVuYWJsZV9jYih2aS0+cnZxKTsKPiArI2Vsc2UKPiDCoMKgwqDCoMKg
wqDCoMKgbmV0aWZfcnhfc2NoZWR1bGUodmktPmRldiwgJnZpLT5uYXBpKTsKPiArI2VuZGlmCj4g
wqB9CgpUaGlzIGxvb2tzIHdyb25nLiAgSXQgc2hvdWxkIGJlIHRoZSBzYW1lLCBleGNlcHQgbmV0
aWZfcnhfc2NoZWR1bGUgZG9lc24ndCAKdGFrZSB0aGUgbmFwaSBhcmcgKHRoZSBuYXBpIHN0cnVj
dCB1c2VkIHRvIGJlIGluIHRoZSBkZXZpY2UsIGJ1dCBtdWx0aXF1ZXVlIAptZWFucyBldmVyeW9u
ZSBuZWVkIHRvIHN1cHBseSB0aGVpciBvd24gbmFwaSBhcmcgbm93KS4KClJ1c3R5Lgo=

Re: [kvm-devel] virtio_net backport performance

[Rusty R. <ru...@ru...>]

On Saturday 05 January 2008 09:24:40 Anthony Liguori wrote:
> Hey Rusty et al,
>
> I've got automatic backports of the virtio modules[1] working back to
> 2.6.18.  Everything seems okay except that for any kernel with the older
> NAPI api, performance is extremely bad.  I get about 1gbit on TX with
> 2.6.24 but I get somewhere around 20mbit on 2.6.22.

OK, I tested this backport and immediately got oopses.  Revealed some 
interesting races in net driver (again, it's that damn callback disable 
causing problems).  New queue fixes these, but get awful performance:
'dd bs=1M count=1000 if=/dev/zero | nc 172.20.0.1 8889' takes almost 30 
seconds.

Found one bug in your code tho: if enable_cb returns false, it means the queue 
has *not* been enabled:

--- hack-module.awk.~1~	2008-01-06 10:49:16.000000000 +1100
+++ hack-module.awk	2008-01-07 19:08:40.000000000 +1100
@@ -49,7 +49,6 @@
     print "	netif_rx_complete(vi->dev);";
     print "";
     print "	if (!no_work && netif_rx_reschedule(vi->dev, received)) {";
-    print "		vi->rvq->vq_ops->disable_cb(vi->rvq);";
     print "		skb = NULL;";
     print "		goto again;";
     print "	}";

Will continue looking for performance regression tomorrow (actually, better 
check my changes haven't introduced it in non-backports first!).

Thanks!
Rusty.

Re: [kvm-devel] virtio_net backport performance

[Anthony L. <ali...@us...>]

Rusty Russell wrote:
> On Saturday 05 January 2008 09:24:40 Anthony Liguori wrote:
>   
>> Hey Rusty et al,
>>
>> I've got automatic backports of the virtio modules[1] working back to
>> 2.6.18.  Everything seems okay except that for any kernel with the older
>> NAPI api, performance is extremely bad.  I get about 1gbit on TX with
>> 2.6.24 but I get somewhere around 20mbit on 2.6.22.
>>     
>
> OK, I tested this backport and immediately got oopses.  Revealed some 
> interesting races in net driver (again, it's that damn callback disable 
> causing problems).  New queue fixes these, but get awful performance:
> 'dd bs=1M count=1000 if=/dev/zero | nc 172.20.0.1 8889' takes almost 30 
> seconds.
>   

rx performance is pretty abysmal but tx performance seems respectable.  
I'm getting strange results with netperf but my guess is that we're 
getting around 800mbit tx but only 25mbit rx.

tx is right around what it should be but rx is about an order of 
magnitude off.

> Found one bug in your code tho: if enable_cb returns false, it means the queue 
> has *not* been enabled:
>
> --- hack-module.awk.~1~	2008-01-06 10:49:16.000000000 +1100
> +++ hack-module.awk	2008-01-07 19:08:40.000000000 +1100
> @@ -49,7 +49,6 @@
>      print "	netif_rx_complete(vi->dev);";
>      print "";
>      print "	if (!no_work && netif_rx_reschedule(vi->dev, received)) {";
> -    print "		vi->rvq->vq_ops->disable_cb(vi->rvq);";
>      print "		skb = NULL;";
>      print "		goto again;";
>      print "	}";
>   

Thanks, I've applied this to the backport tree.

> Will continue looking for performance regression tomorrow (actually, better 
> check my changes haven't introduced it in non-backports first!).
>   

Excellent, thanks!

Regards,

Anthony Liguori

> Thanks!
> Rusty.
>