[Aoetools-discuss] Performance

[Michael G. <mg...@ne...>]

I know there have been several discussions of performance of AoE on  
the lists in the past.  I'm having slow performance that I just can't  
seem to get past.  Here's my setup:

Target:
Abit VA-20 Motherboard (KM400A chipset)
Athlon 2200+
256MB RAM
PATA UDMA/133 100GB drive
Onboard 10/100 Via Rhine (IP networking, system management)
Linksys 10/100/1000 ethernet adapter (RTL8169 based)
OS: Fedora 7

Initiator:
Same specs, except the drive is 80GB, and it has 1GB of RAM

The AoE network consists of the two GbE adapters connected via Cat6  
cabling and a Netgear GS105 5 port GbE switch, which supports jumbo  
frames (up to an MTU of 9000).

I'm using bonnie++ to do my tests, and I'm having serious difficulty  
getting block level read performance of anything better than about  
10MB/s.  As listed above, I'm using Fedora 7 with the AoE initiator  
module that comes with the kernel.  On the target, I'm currently using  
a locally-compiled vblade 15 with the O_DIRECT patch.  What really  
gets me is that I can't seem to track down where the bottleneck is.   
My benchmarks against the local hard drive on the target result in  
block level read performance as you'd expect, 60-65MB/s.  On the  
network side, due to a limitation in either the chipset or the module,  
I can only have an MTU on the GbE adapters of 7200, but both of them  
are set to that, so the MTU's match up on both sides (and /dev/etherd/ 
err isn't showing retransmissions except for once in a while).  I also  
increased the txqueuelen on each adapter to 100000.  The default 1000  
seemed too low.  My iperf tests on the GbE network with these settings  
show that the machines are able to communicate at about 850Mbps.

Now, I know that vblade isn't necessarily a high performance target,  
but I'd expect a little better performance than this.  I've seen  
various Internet-published trials that indicate that they're able to  
push over 100MB/s with vblade on a GbE network.  I'd be happy at this  
point if I could push even 40MB/s.  I can only assume from all of this  
that either I've done something wrong or my hardware is too old.  The  
VA-20 motherboard only has 32 bit 33MHz PCI slots, so the total bus  
throughput is only 1Gbps.  Given the shared bus architecture of PCI, I  
know that the max perceived throughput will be less since the read- 
from-disk operations and write-to-net operations share the same bus  
(slightly less when you account for bus overhead), but I'm not even  
getting close to that.

Any thoughts on what I might be doing wrong?  I can post the actual  
bonnie++ results from each of these tests if that would help.

Thanks.

--Mike