|
From: Matthew F. <m.f...@la...> - 2009-03-03 22:12:24
|
2009/3/3 Brandeburg, Jesse <jes...@in...>: > Matthew Faulkner wrote: >> I have two Intel 10 Gigabit CX4 Dual Port Server Adaptors, which I >> believe use the 82598EB chipset. I've put these cards in two machines >> and connected them point-to-point with the CX4 cable. > > okay > >> I'm measuring the performance between these two machines. >> I'm using Jumbo frames (9000MTU) and both ports. >> The machines are both multi core machines with affinities set so that >> the CPU is not the limiting the speed. > > okay, good > >> However, the absolute maximum speed I can achieve is ~1400MB/s, with >> each port achieving equal bandwidth (~700MB/s). These machines are >> also connected by two 1 gig cards. When i include these cards in the >> performance test, the maximum bandwidth increases up to ~1645MB/s >> (with both 1gig cards achieving ~123MB/s). So it seems doubtful to me >> that the bus is limiting the speed. > > transmitting, receiving or both? The numbers quoted here is the application goodput as seen on the receiving side. > >> My question is what is limiting it? Am i misunderstanding the CX4 card >> by thinking it's two 10GbE which would give me a theoretical maximum >> of 2500MB/s. Does the card infact only give me a total of 10GB/S >> (1250MB/s)? Is so then how do the two ports go faster than 10GbE? > > each port is capable at the wire of 20 Gb/s (10 Gb/s full duplex) > aka each port is capable of 1.2GB/s (or 1200MB/s) in each direction. Thats > the easy part. > >> Or is there something I'm missing entirely? > > maybe a little. > >> I'll be grateful of any help you can provide. If you require the >> settings of the driver i'm using (ixgbe) or the outputs of lspci, >> dmesg etc please email. I've not included them in this email to keep >> it short :) > > As it turns out a x8 PCIe slot is only capable of 40 Gb/s raw, and that is > 8/10 bit encoded so that takes it down to 32Gb/s absolute maximum data > payload rate (including all configuration transactions, etc.) > this is for the slot only (so two ports share it) > > If you use two cards (using a single port each one) you'll probably see > higher numbers. > > The increases you see with 1Gb/s cards implies that your system is not > memory bandwidth limited. > > on a single port you should be able to see line rate bidirectional, the x8 > PCIe Gen 1 slot limits dual port throughput a bit. > > Hope this helps, > Jesse > So these numbers suggest that I should be able to get above the 1400MB/s (just above 11Gb/s) that i am getting? |