Re: [E1000-devel] Intel 10 Gigabit CX4 Dual Port Server Adaptor - Maximum Throughput?

[Matthew F. <m.f...@la...>]

2009/3/3 Brandeburg, Jesse <jes...@in...>:
> Matthew Faulkner wrote:
>> I have two Intel 10 Gigabit CX4 Dual Port Server Adaptors, which I
>> believe use the 82598EB chipset. I've put these cards in two machines
>> and connected them point-to-point with the CX4 cable.
>
> okay
>
>> I'm measuring the performance between these two machines.
>> I'm using Jumbo frames (9000MTU) and both ports.
>> The machines are both multi core machines with affinities set so that
>> the CPU is not the limiting the speed.
>
> okay, good
>
>> However, the absolute maximum speed I can achieve is ~1400MB/s, with
>> each port achieving equal bandwidth (~700MB/s). These machines are
>> also connected by two 1 gig cards. When i include these cards in the
>> performance test, the maximum bandwidth increases up to ~1645MB/s
>> (with both 1gig cards achieving ~123MB/s). So it seems doubtful to me
>> that the bus is limiting the speed.
>
> transmitting, receiving or both?

The numbers quoted here is the application goodput as seen on the
receiving side.
>
>> My question is what is limiting it? Am i misunderstanding the CX4 card
>> by thinking it's two 10GbE which would give me a theoretical maximum
>> of 2500MB/s. Does the card infact only give me a total of 10GB/S
>> (1250MB/s)? Is so then how do the two ports go faster than 10GbE?
>
> each port is capable at the wire of 20 Gb/s (10 Gb/s full duplex)
> aka each port is capable of 1.2GB/s (or 1200MB/s) in each direction.  Thats
> the easy part.
>
>> Or is there something I'm missing entirely?
>
> maybe a little.
>
>> I'll be grateful of any help you can provide. If you require the
>> settings of the driver i'm using (ixgbe) or the outputs of lspci,
>> dmesg etc please email. I've not included them in this email to keep
>> it short :)
>
> As it turns out a x8 PCIe slot is only capable of 40 Gb/s raw, and that is
> 8/10 bit encoded so that takes it down to 32Gb/s absolute maximum data
> payload rate (including all configuration transactions, etc.)
> this is for the slot only (so two ports share it)
>
> If you use two cards (using a single port each one) you'll probably see
> higher numbers.
>
> The increases you see with 1Gb/s cards implies that your system is not
> memory bandwidth limited.
>
> on a single port you should be able to see line rate bidirectional, the x8
> PCIe Gen 1 slot limits dual port throughput a bit.
>
> Hope this helps,
>  Jesse
>

So these numbers suggest that I should be able to get above the
1400MB/s (just above 11Gb/s) that i am getting?