Thread: [Aoetools-discuss] Problem with self built AoE Server
Brought to you by:
ecashin,
elcapitansam
From: Philip K. <phi...@so...> - 2008-09-11 15:15:51
|
Hello, I built an AoE Server on an FPGA and a have problem with the Linux driver. Although my NIC supports jumboframes and I have inreased the mtu value, the driver does not send or request more than 2 sectors in a message. With the aoe-exercise tool I can set the packetsize manually and it works without any problems. Does the driver use any other information than the mtu value to set the packetsize, maybe any values that my server sends? Thanks, Philip |
From: Tracy R R. <tr...@ul...> - 2008-09-11 15:28:31
|
Philip Kirchhoff wrote: > I built an AoE Server on an FPGA and a have problem with the Linux > driver. Although my NIC supports jumboframes and I have inreased the mtu I'm confused: Are you saying you implemented AoE protocol on a Field Programmable Gate Array device? And it speaks to the network and everything as well? And the disk? I would love to hear more about this! > Does the driver use any other information than the mtu value to set the > packetsize, maybe any values that my server sends? Are both the target and initiator set up for 9000 (or whatever) MTU? What targets/initiators are you using? -- Tracy R Reed http://tracyreed.org |
From: Philip K. <phi...@so...> - 2008-09-11 15:49:20
|
Tracy R Reed schrieb: > Philip Kirchhoff wrote: >> I built an AoE Server on an FPGA and a have problem with the Linux >> driver. Although my NIC supports jumboframes and I have inreased the mtu > > I'm confused: Are you saying you implemented AoE protocol on a Field > Programmable Gate Array device? And it speaks to the network and > everything as well? And the disk? I would love to hear more about this! > >> Does the driver use any other information than the mtu value to set >> the packetsize, maybe any values that my server sends? > > Are both the target and initiator set up for 9000 (or whatever) MTU? > What targets/initiators are you using? > Yes, thats what FPGA means. :-) There's an external Ethernet Phy and and the MAC is a hardmacro on the FPGA. The disk is connected with a SATA interface. Both sides should be able to process jumboframes and, as I wrote, they do, when I use the aoe-exercise tool. |
From: Tracy R R. <tr...@ul...> - 2008-09-11 16:25:17
|
On Thu, Sep 11, 2008 at 05:49:21PM +0200, Philip Kirchhoff spake thusly: > Yes, thats what FPGA means. :-) So why an FPGA? Are you trying to make a small or cheap dedicated appliance? Or is it somehow for performance? Is this a proprietary product for some company that wants to sell it or is a hobby project for you? -- Tracy Reed http://tracyreed.org |
From: Philip K. <phi...@so...> - 2008-09-11 16:43:07
|
Tracy R Reed schrieb: > On Thu, Sep 11, 2008 at 05:49:21PM +0200, Philip Kirchhoff spake thusly: >> Yes, thats what FPGA means. :-) > > So why an FPGA? Are you trying to make a small or cheap dedicated > appliance? Or is it somehow for performance? Is this a proprietary product > for some company that wants to sell it or is a hobby project for you? > Its a proprietary product. FPAGA because its a very specific application which includes much more than the AoE Server. |
From: Gabor G. <go...@sz...> - 2008-09-11 15:54:02
|
Hi! On Thu, Sep 11, 2008 at 05:15:50PM +0200, Philip Kirchhoff wrote: > I built an AoE Server on an FPGA and a have problem with the Linux > driver. Although my NIC supports jumboframes and I have inreased the mtu > value, the driver does not send or request more than 2 sectors in a > message. With the aoe-exercise tool I can set the packetsize manually > and it works without any problems. Which kernel version? If it's not recent, you should try a recent one. We for example are stuck with 2.6.18 due to Xen and it also has problems with jumbo frames; 2.6.2[456] works just fine. Gabor -- --------------------------------------------------------- MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences --------------------------------------------------------- |
From: <tr...@ul...> - 2008-09-11 16:13:04
|
On Thu, Sep 11, 2008 at 05:54:05PM +0200, Gabor Gombas spake thusly: > Which kernel version? If it's not recent, you should try a recent one. > We for example are stuck with 2.6.18 due to Xen and it also has problems > with jumbo frames; 2.6.2[456] works just fine. What sort of problems? I've been using 2.6.18 with CentOS 5.2 with Xen and AoE quite heavily and not had any problems. -- Tracy Reed http://tracyreed.org |
From: Gabor G. <go...@sz...> - 2008-09-12 09:04:13
|
On Thu, Sep 11, 2008 at 09:13:00AM -0700, tr...@ul... wrote: > On Thu, Sep 11, 2008 at 05:54:05PM +0200, Gabor Gombas spake thusly: > > Which kernel version? If it's not recent, you should try a recent one. > > We for example are stuck with 2.6.18 due to Xen and it also has problems > > with jumbo frames; 2.6.2[456] works just fine. > > What sort of problems? I've been using 2.6.18 with CentOS 5.2 with Xen and > AoE quite heavily and not had any problems. It does not use jumbo frames no matter what I've tried. I've stared at the kernel source for some time and IMHO it _should_ work, but it does not. If I boot a recent kernel (obviously without Xen) then jumbo frames work perfectly. Fortunately none of those Xen hosts are performance critical (yet) so I didn't do any serious kernel debugging. I've also tried compiling a new AoE module but the (then) current one from Coraid did not build with the Xen-patched 2.6.18 and I had no time to try older versions. Gabor -- --------------------------------------------------------- MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences, Laboratory of Parallel and Distributed Systems Address : H-1132 Budapest Victor Hugo u. 18-22. Hungary Phone/Fax : +36 1 329-78-64 (secretary) W3 : http://www.lpds.sztaki.hu --------------------------------------------------------- |
From: Philip K. <phi...@so...> - 2008-09-11 15:59:47
|
Gabor Gombas schrieb: > Hi! > > On Thu, Sep 11, 2008 at 05:15:50PM +0200, Philip Kirchhoff wrote: > >> I built an AoE Server on an FPGA and a have problem with the Linux >> driver. Although my NIC supports jumboframes and I have inreased the mtu >> value, the driver does not send or request more than 2 sectors in a >> message. With the aoe-exercise tool I can set the packetsize manually >> and it works without any problems. > > Which kernel version? If it's not recent, you should try a recent one. > We for example are stuck with 2.6.18 due to Xen and it also has problems > with jumbo frames; 2.6.2[456] works just fine. > > Gabor > The kernel version is 2.6.24-19. But the kernels aoe driver did not work at all so I installed the newest release from coraid. |
From: Gabor G. <go...@sz...> - 2008-09-19 02:57:10
|
On Thu, Sep 11, 2008 at 05:59:52PM +0200, Philip Kirchhoff wrote: > The kernel version is 2.6.24-19. But the kernels aoe driver did not work > at all so I installed the newest release from coraid. One of my collegues managed to build the v62 AoE driver for this kernel and now jumbo frames seem to work now. Gabor -- --------------------------------------------------------- MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences --------------------------------------------------------- |
From: Ed C. <ec...@co...> - 2008-09-11 16:04:44
|
On Thu, Sep 11, 2008 at 05:15:50PM +0200, Philip Kirchhoff wrote: > > Hello, > > I built an AoE Server on an FPGA and a have problem with the Linux > driver. Although my NIC supports jumboframes and I have inreased the mtu > value, the driver does not send or request more than 2 sectors in a > message. With the aoe-exercise tool I can set the packetsize manually > and it works without any problems. > Does the driver use any other information than the mtu value to set the > packetsize, maybe any values that my server sends? The aoe driver also uses as a lower bound the smallest MTU of the local interfaces that can reach the target, and it also takes the "Sector Count" field of the AoE config query response into account. -- Ed Cashin <ec...@co...> |
From: Philip K. <phi...@so...> - 2008-09-11 16:25:15
|
Ed Cashin schrieb: > On Thu, Sep 11, 2008 at 05:15:50PM +0200, Philip Kirchhoff wrote: >> Hello, >> >> I built an AoE Server on an FPGA and a have problem with the Linux >> driver. Although my NIC supports jumboframes and I have inreased the mtu >> value, the driver does not send or request more than 2 sectors in a >> message. With the aoe-exercise tool I can set the packetsize manually >> and it works without any problems. >> Does the driver use any other information than the mtu value to set the >> packetsize, maybe any values that my server sends? > > The aoe driver also uses as a lower bound the smallest MTU of the > local interfaces that can reach the target, and it also takes the > "Sector Count" field of the AoE config query response into account. > Do you mean the "Buffer Count" value? But this value doesn't tell anything about the packet size, does it? Its the number of messages the server can queue. |
From: Ed C. <ec...@co...> - 2008-09-11 16:32:44
|
On Thu, Sep 11, 2008 at 06:25:19PM +0200, Philip Kirchhoff wrote: ... > Do you mean the "Buffer Count" value? But this value doesn't tell anything > about the packet size, does it? Its the number of messages the server can > queue. No, the Buffer Count and Sector Count are two different fields in the config query response. http://support.coraid.com/documents/AoEr10.txt -- Ed Cashin <ec...@co...> |
From: Philip K. <phi...@so...> - 2008-09-11 16:40:12
|
Ed Cashin schrieb: > On Thu, Sep 11, 2008 at 06:25:19PM +0200, Philip Kirchhoff wrote: > ... >> Do you mean the "Buffer Count" value? But this value doesn't tell anything >> about the packet size, does it? Its the number of messages the server can >> queue. > > No, the Buffer Count and Sector Count are two different fields in the > config query response. > > http://support.coraid.com/documents/AoEr10.txt > > Ok, I think thats it. In my version of this document (August 2004, I don't know why I did not found the actual one) there stands reserved instead of sector count so i set it to all zeros. Thanks!! |
From: Ed C. <ec...@co...> - 2008-09-12 13:16:11
|
On Fri, Sep 12, 2008 at 11:04:21AM +0200, Gabor Gombas wrote: > On Thu, Sep 11, 2008 at 09:13:00AM -0700, tr...@ul... wrote: ... > > What sort of problems? I've been using 2.6.18 with CentOS 5.2 with Xen and > > AoE quite heavily and not had any problems. > > It does not use jumbo frames no matter what I've tried. I've stared at > the kernel source for some time and IMHO it _should_ work, but it does > not. If I boot a recent kernel (obviously without Xen) then jumbo frames > work perfectly. It might just be the network driver changing. I noticed that jumbo support for the 82573 Intel PCIe chipsets appeared and disappeared a few times in the kernels 2.6.16 and up. -- Ed Cashin <ec...@co...> |
From: Gabor G. <go...@sz...> - 2008-09-15 11:53:22
|
On Fri, Sep 12, 2008 at 09:18:36AM -0400, Ed Cashin wrote: > It might just be the network driver changing. I noticed that jumbo > support for the 82573 Intel PCIe chipsets appeared and disappeared a > few times in the kernels 2.6.16 and up. No, it's Broadcom, and "ping -s 8000" works. Gabor -- --------------------------------------------------------- MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences --------------------------------------------------------- |
From: kelsey h. <kh...@dr...> - 2008-09-13 00:29:51
|
Gabor Gombas wrote: > On Thu, Sep 11, 2008 at 09:13:00AM -0700, tr...@ul... wrote: >> On Thu, Sep 11, 2008 at 05:54:05PM +0200, Gabor Gombas spake thusly: >>> Which kernel version? If it's not recent, you should try a recent one. >>> We for example are stuck with 2.6.18 due to Xen and it also has problems >>> with jumbo frames; 2.6.2[456] works just fine. >> What sort of problems? I've been using 2.6.18 with CentOS 5.2 with Xen and >> AoE quite heavily and not had any problems. > > It does not use jumbo frames no matter what I've tried. I've stared at > the kernel source for some time and IMHO it _should_ work, but it does > not. If I boot a recent kernel (obviously without Xen) then jumbo frames > work perfectly. This is an artifact of how the Xen networking works in the version of Xen with CentOS 5. When you create your domU, a vif will be created on the host dom0. The problem is, MTU changes in the domU don't trickle down to the dom0. You'll have to manually set the MTU on any created vif devices *and* the bridge to which they're connected. Consider the following situation: A dom0 machine exists with two ethernet adapters, one (eth0) for front-end traffic and the other (eth1) for AoE traffic. At boot, the bridges br0 and stbr0 are created, and eth0 and eth1 are added to them (respectively). eth1 is set to 9000 MTU, and stbr0 is also set to 9000 MTU. Each domU is created with two virtual ethernet adapters, one connected to br0 (for front-end traffic) and another to stbr0 (for AoE traffic). Upon instantiating the first domU, you would do something similar to: box# xm create some-domu box# ifconfig vif1.1 mtu 9000 (repeat until this command succeeds) box# ifconfig stbr0 mtu 9000 Such a wonderful hack! I have to do something very similar to this each time I start a domU. It's an endless source of trouble. This has some beautiful side-effects. Because the vif is initially created with a 1500 MTU, the system adds it to the bridge with a 1500 MTU. This causes the bridge device to throttle back to 1500 MTU. Any frames received with >1500 MTU during the time it takes you to successfully set the vif and bridge back to 9000 MTU are *dropped*, effectively causing a momentary loss of storage connectivity. Sometimes, the other domUs on the machine will throttle back to smaller frame sizes in the meantime (forcing you to investigate all domUs after starting a new one and manually resetting the frame size back to something reasonable). The fault of this lies in the network layer of Xen. It's got issues. It's also impossible to force a vif creation with a higher MTU. I've come to the conclusion that storage over ethernet is far from ideal for this kind of set-up, given the shortcomings of Xen. One thing I might want to investigate further is having the dom0 do all the storage processing (e.g. exporting all the AoE devices to the dom0 and passing Xen virtual block devices up to the domUs). Both methods have their shortcomings and advantages. Most of the disadvantages/shortcomings are due to using Ethernet as a storage fabric, however, and aren't specific to AoE. > Fortunately none of those Xen hosts are performance critical (yet) so I > didn't do any serious kernel debugging. I've also tried compiling a new > AoE module but the (then) current one from Coraid did not build with the > Xen-patched 2.6.18 and I had no time to try older versions. The performance will be absolutely abysmal without jumbo frame support. Try doing this and report your results! Good luck. -kelsey |
From: Gabor G. <go...@sz...> - 2008-09-15 12:00:13
|
On Fri, Sep 12, 2008 at 04:58:52PM -0700, kelsey hudson wrote: > This is an artifact of how the Xen networking works in the version of > Xen with CentOS 5. The affected machines run Mandriva. > Consider the following situation: > > A dom0 machine exists with two ethernet adapters, one (eth0) for > front-end traffic and the other (eth1) for AoE traffic. [...] We use a single NIC with multiple VLANs. It's only dom0 that can talk AoE, the domUs are not connected to the AoE VLAN. > The performance will be absolutely abysmal without jumbo frame support. On these machines we have enough memory for domUs so they can cache their whole working set and they do not produce much I/O, so right now it's not critical, but we have to do something sooner or later. Gabor -- --------------------------------------------------------- MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences --------------------------------------------------------- |