Gabor Gombas wrote:
> On Thu, Sep 11, 2008 at 09:13:00AM -0700, treed@... wrote:
>> On Thu, Sep 11, 2008 at 05:54:05PM +0200, Gabor Gombas spake thusly:
>>> Which kernel version? If it's not recent, you should try a recent one.
>>> We for example are stuck with 2.6.18 due to Xen and it also has problems
>>> with jumbo frames; 2.6.2 works just fine.
>> What sort of problems? I've been using 2.6.18 with CentOS 5.2 with Xen and
>> AoE quite heavily and not had any problems.
> It does not use jumbo frames no matter what I've tried. I've stared at
> the kernel source for some time and IMHO it _should_ work, but it does
> not. If I boot a recent kernel (obviously without Xen) then jumbo frames
> work perfectly.
This is an artifact of how the Xen networking works in the version of
Xen with CentOS 5.
When you create your domU, a vif will be created on the host dom0. The
problem is, MTU changes in the domU don't trickle down to the dom0.
You'll have to manually set the MTU on any created vif devices *and* the
bridge to which they're connected.
Consider the following situation:
A dom0 machine exists with two ethernet adapters, one (eth0) for
front-end traffic and the other (eth1) for AoE traffic.
At boot, the bridges br0 and stbr0 are created, and eth0 and eth1 are
added to them (respectively). eth1 is set to 9000 MTU, and stbr0 is also
set to 9000 MTU.
Each domU is created with two virtual ethernet adapters, one connected
to br0 (for front-end traffic) and another to stbr0 (for AoE traffic).
Upon instantiating the first domU, you would do something similar to:
box# xm create some-domu
box# ifconfig vif1.1 mtu 9000
(repeat until this command succeeds)
box# ifconfig stbr0 mtu 9000
Such a wonderful hack! I have to do something very similar to this each
time I start a domU. It's an endless source of trouble.
This has some beautiful side-effects. Because the vif is initially
created with a 1500 MTU, the system adds it to the bridge with a 1500
MTU. This causes the bridge device to throttle back to 1500 MTU. Any
frames received with >1500 MTU during the time it takes you to
successfully set the vif and bridge back to 9000 MTU are *dropped*,
effectively causing a momentary loss of storage connectivity. Sometimes,
the other domUs on the machine will throttle back to smaller frame sizes
in the meantime (forcing you to investigate all domUs after starting a
new one and manually resetting the frame size back to something
The fault of this lies in the network layer of Xen. It's got issues.
It's also impossible to force a vif creation with a higher MTU.
I've come to the conclusion that storage over ethernet is far from ideal
for this kind of set-up, given the shortcomings of Xen. One thing I
might want to investigate further is having the dom0 do all the storage
processing (e.g. exporting all the AoE devices to the dom0 and passing
Xen virtual block devices up to the domUs). Both methods have their
shortcomings and advantages. Most of the disadvantages/shortcomings are
due to using Ethernet as a storage fabric, however, and aren't specific
> Fortunately none of those Xen hosts are performance critical (yet) so I
> didn't do any serious kernel debugging. I've also tried compiling a new
> AoE module but the (then) current one from Coraid did not build with the
> Xen-patched 2.6.18 and I had no time to try older versions.
The performance will be absolutely abysmal without jumbo frame support.
Try doing this and report your results!