Re: [Aoetools-discuss] Problem with self built AoE Server

[kelsey h. <kh...@dr...>]

Gabor Gombas wrote:
> On Thu, Sep 11, 2008 at 09:13:00AM -0700, tr...@ul... wrote:
>> On Thu, Sep 11, 2008 at 05:54:05PM +0200, Gabor Gombas spake thusly:
>>> Which kernel version? If it's not recent, you should try a recent one.
>>> We for example are stuck with 2.6.18 due to Xen and it also has problems
>>> with jumbo frames; 2.6.2[456] works just fine.
>> What sort of problems? I've been using 2.6.18 with CentOS 5.2 with Xen and
>> AoE quite heavily and not had any problems.
> 
> It does not use jumbo frames no matter what I've tried. I've stared at
> the kernel source for some time and IMHO it _should_ work, but it does
> not. If I boot a recent kernel (obviously without Xen) then jumbo frames
> work perfectly.

This is an artifact of how the Xen networking works in the version of 
Xen with CentOS 5.

When you create your domU, a vif will be created on the host dom0. The 
problem is, MTU changes in the domU don't trickle down to the dom0. 
You'll have to manually set the MTU on any created vif devices *and* the 
bridge to which they're connected.

Consider the following situation:

A dom0 machine exists with two ethernet adapters, one (eth0) for 
front-end traffic and the other (eth1) for AoE traffic.

At boot, the bridges br0 and stbr0 are created, and eth0 and eth1 are 
added to them (respectively). eth1 is set to 9000 MTU, and stbr0 is also 
set to 9000 MTU.

Each domU is created with two virtual ethernet adapters, one connected 
to br0 (for front-end traffic) and another to stbr0 (for AoE traffic).

Upon instantiating the first domU, you would do something similar to:

box# xm create some-domu
box# ifconfig vif1.1 mtu 9000
(repeat until this command succeeds)
box# ifconfig stbr0 mtu 9000

Such a wonderful hack! I have to do something very similar to this each 
time I start a domU. It's an endless source of trouble.

This has some beautiful side-effects. Because the vif is initially 
created with a 1500 MTU, the system adds it to the bridge with a 1500 
MTU. This causes the bridge device to throttle back to 1500 MTU. Any 
frames received with >1500 MTU during the time it takes you to 
successfully set the vif and bridge back to 9000 MTU are *dropped*, 
effectively causing a momentary loss of storage connectivity. Sometimes, 
the other domUs on the machine will throttle back to smaller frame sizes 
in the meantime (forcing you to investigate all domUs after starting a 
new one and manually resetting the frame size back to something 
reasonable).

The fault of this lies in the network layer of Xen. It's got issues. 
It's also impossible to force a vif creation with a higher MTU.

I've come to the conclusion that storage over ethernet is far from ideal 
for this kind of set-up, given the shortcomings of Xen. One thing I 
might want to investigate further is having the dom0 do all the storage 
processing (e.g. exporting all the AoE devices to the dom0 and passing 
Xen virtual block devices up to the domUs). Both methods have their 
shortcomings and advantages. Most of the disadvantages/shortcomings are 
due to using Ethernet as a storage fabric, however, and aren't specific 
to AoE.

> Fortunately none of those Xen hosts are performance critical (yet) so I
> didn't do any serious kernel debugging. I've also tried compiling a new
> AoE module but the (then) current one from Coraid did not build with the
> Xen-patched 2.6.18 and I had no time to try older versions.

The performance will be absolutely abysmal without jumbo frame support.


Try doing this and report your results!

Good luck.
-kelsey