Thread: [Aoetools-discuss] AoE Target implementaion for LIO?
Brought to you by:
ecashin,
elcapitansam
From: Lars T. <ta...@bb...> - 2012-10-17 10:02:14
|
Hi there. Is there someone implementing an AoE target for Linux LIO? http://www.linux-iscsi.org/wiki/Main_Page I wonder if an AoE target in kernel space has some benefits speedwise. I'm using ggaoed quite a while. But it seems not to be developed further. Lars |
From: Tracy R. <tr...@ul...> - 2012-10-18 19:09:35
|
On Wed, Oct 17, 2012 at 02:45:06AM PDT, Lars Täuber spake thusly: > I wonder if an AoE target in kernel space has some benefits speedwise. Probably not. I've done a lot of AoE and never once have I seen CPU being the bottleneck. It is always disk first then network. I wouldn't want to potentially destabilize my kernel when it could be contained in userspace. > I'm using ggaoed quite a while. But it seems not to be developed further. All of the AoE stuff seems to be pretty dead these days. iSCSI has eaten AoE's lunch. I've given up on AoE (very sadly, since I have invested so much time into it over the last 5 years) and have been migrating all of my stuff to openiscsi. -- Tracy Reed |
From: Ed C. <ec...@co...> - 2012-10-19 13:34:42
|
AoE use is accelerating rapidly, and iSCSI persists because even though it's not the best fit for same-LAN data storage, folks can still get it to work if they are willing and able to deal with the complexity, sacrifice the performance, and don't need the kind of scaling that virtualization and cloud deployments require. Although a lot of the expansion of AoE use lately has been by users of the Coraid HBA, there are still a lot of first-time AoE users using the coraid.com-distributed Linux initiator and also the kernel.org-distributed initiator. Recently there have been a lot of patches from Coraid to the Linux Kernel Mailing List for bringing the kernel.org-distributed driver up to date. For example, http://thread.gmane.org/gmane.linux.kernel/1377362 We also fixed a regression in the Linux kernel's network layer that affected AoE performance: http://thread.gmane.org/gmane.linux.network/243626 I don't have much time to participate on the aoetools-discuss mailing list right now, partly so that I can keep generating those patches, but I think pessimism is especially inappropriate today, when the use of AoE is accelerating along with the development of its associated technologies, including open source technologies. -- Ed Cashin ec...@co... |
From: Torbjørn T. <tor...@tr...> - 2012-10-19 15:16:27
|
I'm happy to hear that there is now progress on getting the initiator updated in upstream, hopefully this will help to correct the assumption that AoE is dead or dying. Not that I'm very knowledgeable about the storage sector, but it seems to me that AoEs main problem as a technology is marketing. I've spoken to other sysadmins that merely go "Huh?" when I tell them we're running AoE, but many people seem to find the idea interesting when I explain what it is. For my part, I can say that I set up AoE on default Debian packages some time ago, and everything has more or less "Just Worked"(TM) since then. Not that my deployment is very fancy or has a high load on it, but still, AoE brings home the bacon for our part. On Fri, Oct 19, 2012 at 3:19 PM, Ed Cashin <ec...@co...> wrote: > AoE use is accelerating rapidly, and iSCSI persists because even though it's not the best fit for same-LAN data storage, folks can still get it to work if they are willing and able to deal with the complexity, sacrifice the performance, and don't need the kind of scaling that virtualization and cloud deployments require. > > Although a lot of the expansion of AoE use lately has been by users of the Coraid HBA, there are still a lot of first-time AoE users using the coraid.com-distributed Linux initiator and also the kernel.org-distributed initiator. Recently there have been a lot of patches from Coraid to the Linux Kernel Mailing List for bringing the kernel.org-distributed driver up to date. For example, > > http://thread.gmane.org/gmane.linux.kernel/1377362 > > We also fixed a regression in the Linux kernel's network layer that affected AoE performance: > > http://thread.gmane.org/gmane.linux.network/243626 > > I don't have much time to participate on the aoetools-discuss mailing list right now, partly so that I can keep generating those patches, but I think pessimism is especially inappropriate today, when the use of AoE is accelerating along with the development of its associated technologies, including open source technologies. > > -- > Ed Cashin > ec...@co... > > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_sfd2d_oct > _______________________________________________ > Aoetools-discuss mailing list > Aoe...@li... > https://lists.sourceforge.net/lists/listinfo/aoetools-discuss -- Vennlig hilsen Torbjørn Thorsen Utvikler / driftstekniker Trollweb Solutions AS - Professional Magento Partner www.trollweb.no Telefon dagtid: +47 51215300 Telefon kveld/helg: For kunder med Serviceavtale Besøksadresse: Luramyrveien 40, 4313 Sandnes Postadresse: Maurholen 57, 4316 Sandnes Husk at alle våre standard-vilkår alltid er gjeldende |
From: Alexandre <al...@gm...> - 2012-10-19 17:43:11
|
Hi, Happy to sse that Coraid is contributing to the kernel.org initiator! I have tryed for a while to convince customers of my former company to give AoE a try. After presenting them the technology they had no doubt about AoE superiority over iSCSI. However, they often chose iscsi as it appeared to be the de-facto standard. IMHO, this is, for a large part, due to the lack of activities in the various opensource implementation. That said this lack of activity is understandable as the specs didn't evolve, so I presume only few work has to be done... Targets like AoE are realy interresting project, but are now abandonned (partly) because of this I guess. In any case an activity on the client side is a good news! Question... is it possible to use Coraid HBA with a Linux target? If so, does it drop the need for the sofware initiator? Kind regards. 2012/10/19 Torbjørn Thorsen <tor...@tr...> > I'm happy to hear that there is now progress on getting the initiator > updated in upstream, hopefully this will help to correct the > assumption that AoE is dead or dying. > > Not that I'm very knowledgeable about the storage sector, but it seems > to me that AoEs main problem as a technology is marketing. > I've spoken to other sysadmins that merely go "Huh?" when I tell them > we're running AoE, but many people seem to find the idea interesting > when I explain what it is. > > For my part, I can say that I set up AoE on default Debian packages > some time ago, and everything has more or less "Just Worked"(TM) since > then. > Not that my deployment is very fancy or has a high load on it, but > still, AoE brings home the bacon for our part. > > On Fri, Oct 19, 2012 at 3:19 PM, Ed Cashin <ec...@co...> wrote: > > AoE use is accelerating rapidly, and iSCSI persists because even though > it's not the best fit for same-LAN data storage, folks can still get it to > work if they are willing and able to deal with the complexity, sacrifice > the performance, and don't need the kind of scaling that virtualization and > cloud deployments require. > > > > Although a lot of the expansion of AoE use lately has been by users of > the Coraid HBA, there are still a lot of first-time AoE users using the > coraid.com-distributed Linux initiator and also the kernel.org-distributed > initiator. Recently there have been a lot of patches from Coraid to the > Linux Kernel Mailing List for bringing the kernel.org-distributed driver up > to date. For example, > > > > http://thread.gmane.org/gmane.linux.kernel/1377362 > > > > We also fixed a regression in the Linux kernel's network layer that > affected AoE performance: > > > > http://thread.gmane.org/gmane.linux.network/243626 > > > > I don't have much time to participate on the aoetools-discuss mailing > list right now, partly so that I can keep generating those patches, but I > think pessimism is especially inappropriate today, when the use of AoE is > accelerating along with the development of its associated technologies, > including open source technologies. > > > > -- > > Ed Cashin > > ec...@co... > > > > > > > > > ------------------------------------------------------------------------------ > > Everyone hates slow websites. So do we. > > Make your web apps faster with AppDynamics > > Download AppDynamics Lite for free today: > > http://p.sf.net/sfu/appdyn_sfd2d_oct > > _______________________________________________ > > Aoetools-discuss mailing list > > Aoe...@li... > > https://lists.sourceforge.net/lists/listinfo/aoetools-discuss > > > > -- > Vennlig hilsen > Torbjørn Thorsen > Utvikler / driftstekniker > > Trollweb Solutions AS > - Professional Magento Partner > www.trollweb.no > > Telefon dagtid: +47 51215300 > Telefon kveld/helg: For kunder med Serviceavtale > > Besøksadresse: Luramyrveien 40, 4313 Sandnes > Postadresse: Maurholen 57, 4316 Sandnes > > Husk at alle våre standard-vilkår alltid er gjeldende > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_sfd2d_oct > _______________________________________________ > Aoetools-discuss mailing list > Aoe...@li... > https://lists.sourceforge.net/lists/listinfo/aoetools-discuss > |
From: Tracy R. <tr...@ul...> - 2012-10-19 18:47:43
|
My primary pain points in decreasing order of severity: 1. For years I've been battling alignment issues in virtual machines with no success. I can't take the performance hit or invest the time in twiddling alignment anymore. TCP overhead is nothing compared to extra writes due to disk alignment. iSCSI straight out of the box doesn't have this issue. 2. Zero distribution (CentOS/RedHat) integration. Having to maintain my own init scripts and making the system unload the aoe module before the bonding module on shutdown, start/stop the vblades, and various other bits of trickery has become a real hassle. 3. Despite claims that AoE use is accelerating I've never run into another person who uses it or even really knows what it is. I've never seen/heard about it in any trade magazine aside from a Linux Journal article years ago. Nobody stops in the AoE IRC channel except for disappointed "Age of Empires" players. Traffic on this list is nearly dead. The blog you set up is quiet. I've had an AoE google alert setup for years. There is nearly no chatter about it. I have had an RSS search feed for AoE on ServerFault for quite some time. It very rarely comes up. Currently, it hasn't been mentioned since July: http://serverfault.com/questions/412957/how-to-access-aoe-block-device-from-redhat-hypervisor And look at the comments. Ouch. There is a lot of support and training material for iSCSI and it is part of the RHCE exam. I have had to do all of my own in-house training for AoE and it scares customers that it is such a niche technology with so few people on earth knowing how to use it, despite its on-the-wire simplicity. It actually hurts my business for people to know I use AoE. At one point I thought I could sell it as a competitive advantage. Not only is it an operational hassle but it has turned into a competitive disadvantage. 4. While I have always advocated AoE on the basis of its simplicity, the actual number of steps needed to get iSCSI working on RedHat/CentOS are far fewer than AoE which in practice makes AoE not simple. I have now scripted/cookbooked the steps to getting iSCSI up and running and my junior admin can do it. Mounting an iSCSI volume is a minor task on the RHCE exam taken by relatively newbie Linux admins. The issues involved in getting AoE working reliably and especially the troubleshooting are beyond many administrators. I have puppet scripts to distribute kernel modules (have to distribute it and compile it per kernel if you aren't running the exact same version everywhere, and deal with a recompile when you yum update the kernel), manage overly complicated init scripts, tweak sysctls, manage vblade configs, etc. Using the builtin distro iSCSI none of this is necessary. I could perhaps resolve all of this myself by getting involved in Fedora and contributing patches to make AoE work as smoothly as iSCSI in RHEL. But so far I've been completely technically incapable of solving the alignment issue and as for the rest: Why should I invest so much time and effort duplicating the excellent work of the iSCSI people? Anne Hathaway of Coraid actually contacted me last year about working for Coraid. I was very tempted but simply unable to move away from San Diego. And the first thing I would have wanted to do would be to make AoE work well in RHEL as target and initiator which I really doubt Coraid would be willing to pay me to do... I have often had the feeling that AoE's lack of distro integration has been because Coraid would prefer we all spend tens of thousands on Coraid hardware and if the free version worked well enough there would be less incentive for people to do so. I understand that every business needs to make money. But I'm not sure I can get behind that as a business model. Last I looked (it's been a while) a Coraid box was all Supermicro hardware which I could assemble myself from Newegg. I seem to recall it was Plan 9 for the OS, a strange choice. I suspect it's something to do with licensing, which makes me somewhat uncomfortable. Again, they need to make money but I'm not interested in helping them to work against my own interest. I've got a business to run too, can't afford to purchase Coraid, and just wanted a decent ethernet SAN which, given today's level of commodity technology, is quite possible for a very reasonable price. I'm afraid I've hurt myself spending so much time on making AoE work. I wish AoE well and maybe some day I'll come back to it but for six years it has cost me way too much time (and therefore money) and now I'm a bit embarrassed by the whole thing. On Fri, Oct 19, 2012 at 06:19:13AM PDT, Ed Cashin spake thusly: > AoE use is accelerating rapidly, and iSCSI persists because even though it's not the best fit for same-LAN data storage, folks can still get it to work if they are willing and able to deal with the complexity, sacrifice the performance, and don't need the kind of scaling that virtualization and cloud deployments require. > > Although a lot of the expansion of AoE use lately has been by users of the Coraid HBA, there are still a lot of first-time AoE users using the coraid.com-distributed Linux initiator and also the kernel.org-distributed initiator. Recently there have been a lot of patches from Coraid to the Linux Kernel Mailing List for bringing the kernel.org-distributed driver up to date. For example, > > http://thread.gmane.org/gmane.linux.kernel/1377362 > > We also fixed a regression in the Linux kernel's network layer that affected AoE performance: > > http://thread.gmane.org/gmane.linux.network/243626 > > I don't have much time to participate on the aoetools-discuss mailing list right now, partly so that I can keep generating those patches, but I think pessimism is especially inappropriate today, when the use of AoE is accelerating along with the development of its associated technologies, including open source technologies. > > -- > Ed Cashin > ec...@co... > > -- Tracy Reed |
From: Gernot S. <ger...@gm...> - 2012-10-21 09:30:36
|
Hi Tracy, very thoughtful posting, I've arrived at similar conclusions and I am in a similar way from an early stage emotionally attached to a protocol of fine craftsmanship that has potential per se. In the beginning I've advocated the AoE protocol to be converted into an RFC, that would have helped in the first place and would have been an incentive for kernel maintainers to include it earlier and spread the word. Due but not only attributed to this fact AoE initiator support in the kernel materialized quite late. IMHO storage stuff belongs into the kernel, period. Besides, what benefit does an initiator have without a target implementation? The two user-space target implementations are dead for years and lagging behind the AoE specifications due to obvious reasons - conflict of interests. Right now we have a proprietary protocol and a single vendor that supports it with no incentive to have any alternative target implementations out there. Besides, the very important hypervisor vendors are not interested in anything beyond iSCSI and FCoE, so why should they bother to natively support AoE? Hence I consider it mission impossible to convince clients to bet their mission-critical storage future on an arguably fine nevertheless exotic single-vendor product and protocol only based on the price argument. I am happy that Coraid is doing fine as a niche player though and wish them all the best for the future. If you look at a success story how this could have worked out have a look at DRBD and Linbit. Regards, Gernot A m Freitag, den 19.10.2012, 11:47 -0700 schrieb Tracy Reed: > My primary pain points in decreasing order of severity: > > 1. For years I've been battling alignment issues in virtual machines with no > success. I can't take the performance hit or invest the time in twiddling > alignment anymore. TCP overhead is nothing compared to extra writes due to > disk alignment. iSCSI straight out of the box doesn't have this issue. > > 2. Zero distribution (CentOS/RedHat) integration. Having to maintain my own > init scripts and making the system unload the aoe module before the bonding > module on shutdown, start/stop the vblades, and various other bits of > trickery has become a real hassle. > > 3. Despite claims that AoE use is accelerating I've never run into another > person who uses it or even really knows what it is. I've never seen/heard > about it in any trade magazine aside from a Linux Journal article years ago. > Nobody stops in the AoE IRC channel except for disappointed "Age of Empires" > players. Traffic on this list is nearly dead. The blog you set up is quiet. > I've had an AoE google alert setup for years. There is nearly no chatter > about it. I have had an RSS search feed for AoE on ServerFault for quite > some time. It very rarely comes up. Currently, it hasn't been mentioned > since July: > > http://serverfault.com/questions/412957/how-to-access-aoe-block-device-from-redhat-hypervisor > > And look at the comments. Ouch. > > There is a lot of support and training material for iSCSI and it is part of > the RHCE exam. I have had to do all of my own in-house training for AoE and > it scares customers that it is such a niche technology with so few people on > earth knowing how to use it, despite its on-the-wire simplicity. It actually > hurts my business for people to know I use AoE. At one point I thought I > could sell it as a competitive advantage. Not only is it an operational > hassle but it has turned into a competitive disadvantage. > > 4. While I have always advocated AoE on the basis of its simplicity, the actual > number of steps needed to get iSCSI working on RedHat/CentOS are far fewer > than AoE which in practice makes AoE not simple. I have now > scripted/cookbooked the steps to getting iSCSI up and running and my junior > admin can do it. Mounting an iSCSI volume is a minor task on the RHCE exam > taken by relatively newbie Linux admins. The issues involved in getting AoE > working reliably and especially the troubleshooting are beyond many > administrators. I have puppet scripts to distribute kernel modules (have to > distribute it and compile it per kernel if you aren't running the exact same > version everywhere, and deal with a recompile when you yum update the > kernel), manage overly complicated init scripts, tweak sysctls, manage > vblade configs, etc. Using the builtin distro iSCSI none of this is > necessary. > > I could perhaps resolve all of this myself by getting involved in Fedora and > contributing patches to make AoE work as smoothly as iSCSI in RHEL. But so far > I've been completely technically incapable of solving the alignment issue and > as for the rest: Why should I invest so much time and effort duplicating the > excellent work of the iSCSI people? > > Anne Hathaway of Coraid actually contacted me last year about working for > Coraid. I was very tempted but simply unable to move away from San Diego. And > the first thing I would have wanted to do would be to make AoE work well in > RHEL as target and initiator which I really doubt Coraid would be willing to > pay me to do... > > I have often had the feeling that AoE's lack of distro integration has been > because Coraid would prefer we all spend tens of thousands on Coraid hardware > and if the free version worked well enough there would be less incentive for > people to do so. I understand that every business needs to make money. But I'm > not sure I can get behind that as a business model. Last I looked (it's been a > while) a Coraid box was all Supermicro hardware which I could assemble myself > from Newegg. I seem to recall it was Plan 9 for the OS, a strange choice. I > suspect it's something to do with licensing, which makes me somewhat > uncomfortable. Again, they need to make money but I'm not interested in helping > them to work against my own interest. > > I've got a business to run too, can't afford to purchase Coraid, and just > wanted a decent ethernet SAN which, given today's level of commodity > technology, is quite possible for a very reasonable price. I'm afraid I've hurt > myself spending so much time on making AoE work. I wish AoE well and maybe some > day I'll come back to it but for six years it has cost me way too much time > (and therefore money) and now I'm a bit embarrassed by the whole thing. > > On Fri, Oct 19, 2012 at 06:19:13AM PDT, Ed Cashin spake thusly: > > AoE use is accelerating rapidly, and iSCSI persists because even though it's not the best fit for same-LAN data storage, folks can still get it to work if they are willing and able to deal with the complexity, sacrifice the performance, and don't need the kind of scaling that virtualization and cloud deployments require. > > > > Although a lot of the expansion of AoE use lately has been by users of the Coraid HBA, there are still a lot of first-time AoE users using the coraid.com-distributed Linux initiator and also the kernel.org-distributed initiator. Recently there have been a lot of patches from Coraid to the Linux Kernel Mailing List for bringing the kernel.org-distributed driver up to date. For example, > > > > http://thread.gmane.org/gmane.linux.kernel/1377362 > > > > We also fixed a regression in the Linux kernel's network layer that affected AoE performance: > > > > http://thread.gmane.org/gmane.linux.network/243626 > > > > I don't have much time to participate on the aoetools-discuss mailing list right now, partly so that I can keep generating those patches, but I think pessimism is especially inappropriate today, when the use of AoE is accelerating along with the development of its associated technologies, including open source technologies. > > > > -- > > Ed Cashin > > ec...@co... > > > > > |
From: Nicolas J. <ni...@ju...> - 2012-10-21 13:19:20
|
On 2012-10-19 15:19, Ed Cashin wrote: > AoE use is accelerating rapidly, and iSCSI persists because even > though it's not the best fit for same-LAN data storage, folks can > still get it to work if they are willing and able to deal with the > complexity, sacrifice the performance, and don't need the kind of > scaling that virtualization and cloud deployments require. I don't think it's a realistic view. I think AoE missed its window of opportunity and is now dying. As Tracy said, it's ridiculously easy to setup an iSCSI system. It's a pain to get the AoE target up to date in any distro, and the initiator, while working, is far from being a well behaving component. I was never able to push AoE anywhere beside my projects. Blame me if you want, but most of the time free iSCSI was the obvious, variously documented, community supported, distro included, choice. Regards, N. > > Although a lot of the expansion of AoE use lately has been by users > of the Coraid HBA, there are still a lot of first-time AoE users > using the coraid.com-distributed Linux initiator and also the > kernel.org-distributed initiator. Recently there have been a lot of > patches from Coraid to the Linux Kernel Mailing List for bringing the > kernel.org-distributed driver up to date. For example, > > http://thread.gmane.org/gmane.linux.kernel/1377362 > > We also fixed a regression in the Linux kernel's network layer that > affected AoE performance: > > http://thread.gmane.org/gmane.linux.network/243626 > > I don't have much time to participate on the aoetools-discuss mailing > list right now, partly so that I can keep generating those patches, > but I think pessimism is especially inappropriate today, when the use > of AoE is accelerating along with the development of its associated > technologies, including open source technologies. > |
From: Ed C. <ec...@co...> - 2012-10-21 13:54:40
|
On Oct 21, 2012, at 9:03 AM, Nicolas Jungers wrote: > On 2012-10-19 15:19, Ed Cashin wrote: >> AoE use is accelerating rapidly, and iSCSI persists because even >> though it's not the best fit for same-LAN data storage, folks can >> still get it to work if they are willing and able to deal with the >> complexity, sacrifice the performance, and don't need the kind of >> scaling that virtualization and cloud deployments require. > > I don't think it's a realistic view. I think AoE missed its window of > opportunity and is now dying. As Tracy said, it's ridiculously easy to > setup an iSCSI system. It's a pain to get the AoE target up to date in > any distro, and the initiator, while working, is far from being a well > behaving component. Tracy had a good point in that there needs to be more work done to help Linux distros to integrate AoE technology, so that Linux admins have a smooth experience using it. AoE was probably too far ahead of its time in 2004 and is now coming into its time. A lot of the adoption lately has been very large scale special-purpose or enterprise installations, and not all of that has been on Linux. The need driving the CTOs of these highly competitive companies to adopt AoE-based technology is based on real-world trends that really can't be stopped, not based on emotion. The folks on the aoetools-discuss list won't necessarily know about these AoE-based deployments. But it's a good thing that Coraid is contributing actively to open source now, and it should help to ease its adoption on Linux-based architectures. It's open source, though, so although Coraid can be expected to take the lead, there are alternatives to complaining. ;) -- Ed Cashin ec...@co... |