Re: [OSR-devel] OpenVZ for OSR

[Gordan B. <go...@bo...>]

Marc Grimme wrote:

>> For the sake of education and diversity I have just finished the first
>>
>> attempt at this, purely to see if there is an inherent problem in
>> OpenVZ that might shoot this down.
>>
>> Well - I haven't found any such problems! :D
 >
> I don't see them either. As far as I understand them.

My big concern was that I didn't know what OpenVZ guests would do and 
how the OpenVZ host would handle a situation where another OpenVZ hosts 
(and it's OpenVZ guest) started modifying the directory trees 
concurrently. Thankfully, at least thus far, it hasn't blown up 
horribly, which is a good start. :)

>> The basic setup was this:
>>
>> Two identical host machines (virtual, because it was easier, but fully
>>
>> virtualized with KVM, CentOS 5.4). The host OS was stripped down to a
>>
>> bare minimum (a "mere" 850MB...) since I didn't feel applying the more
>>
>> sensibly sized OSR init root was vital for now and modding it would 
>> require extra work.
>>
>> Shared virtual disk (shared image, presented as an IDE device), with
>> GFS on top. So far, so standard.
>>
>> The shared GFS device was mounted under /vz/private (where OpenVZ
>> keeps the VM fs trees).
>>
>> CentOS 5.4 guest template was initialized in there. The guest config 
>> file was the same on both hosts, except for the IP address (the IP is
>>
>> configured on the host rather than the guest, but each guest can have
>>
>> independent iptables rules if required).
>>
>> Thus - the two guests were running on a GFS shared root. The one thing
>>
>> remaining would be to set up the entries in fstab to make sure that 
>> /cdsl.local gets bind mounted correctly at boot-up time, but other
>> than that, I'd say the preliminary prototype test has passed. :)
>>
>> The basic thing I wanted to achieve is to have a cleaner separation 
>> between the host provided shared rootfs and the guest so that there
>> are no issues during shutdown with unmounting file systems, etc. This 
>> prototype appears to have completely met those requirements.
>>
>> There is also an added bonus benefit that I hadn't thought of before -
>>
>> the guest can be cleanly rebooted without rebooting the host (which
>> also means without triggering fencing and suchlike).
>>
> Ok. Correct me if I'm wrong.
> My current understanding of your concept is as follows:
> 
> You want to create a OpenVZ Cluster that just does OpenVZ without anything else.
> 
> So no "Sharedroot" within OpenVZ. But it's small.
> 
> Then you'll boot the SharedRoot as a "Guest" for you apps.

I'm not sure if you're saying what I think you're saying or the reverse 
of it, so let me clarify.

Let's use the terms "init-root" for the non-shared-root bootstrap part 
(what the initrd does in OSR) and "real-root" for the actual shared root.

The init-root runs the OpenVZ patched kernel, and is responsible of 
setting up the clustering in order to bring up the shared root file 
systems. In essence, it's task is pretty similar to what OSR does at the 
moment.

The real-root is on the shared/replicated file system and an OpenVZ 
guest gets booted into it (OpenVZ uses a chroot on the host's fs rather 
than a disk image like all the other virtualization solutions).

So shared-root is obtained by the guest OpenVZ VMs, while the init-root 
provides all the usual things that OSR has to provide as usual 
(clustering start-up, shared-root fs setup).

> Advantages:
> * Better debugging (Console, ..)
> * Easier to maintain
> * Virtualizationperspective: Very low resource overhead. Thanks to OpenVZ Containers.

It also means that multiple SR clusters could be had on the same set of 
physical hosts with negligible overheads. This is a new feature. :)

> Disadvantages:
> * Tainted kernel (no support from RedHat, SAP etc.)

Not quite - IIRC OpenVZ is what Virtuozzo uses 
(http://www.parallels.com/uk/products/pvc45/) so commercial support is 
available. And OpenVZ project does provide patched versions of RHEL 
patched kernels.

> * Local installations for the "Host" Nodes needed.

Not necessarily - there is no reason why this couldn't fit into the 
initrd. My prototype ended up with an 850MB host install, but that was 
purely because I just dropped CentOS 5.4 on it, and pruned it down a 
bit, rather than gone the route of squeezing OpenVZ tools into the OSR 
initrd (which would yield a much more sensible footprint - although at 
the cost of not having two of the three advantages you listed above).

> Questions:
> * This sounds like the approach the ovirt people are doing. They are
> using stateless linux with RO-Root as image and then mount /var rw?!

Funny you should mention this. I was actually looking for some RHEV-H 
distro SRPMS so I could rebuild them and set up a RHEV-H host in some 
vague hope that it might be thinner than a full fat RHEL, but
1) I failed to find any and
2) Upon consideration I don't think it is likely to get that much 
thinner. OpenVZ have a "minimal" CentOS 5 template that comes in at 
about 46MB (not including the kernel, though!), so this _could_ be made 
pretty thin at a push. Of course, there is the trade-off between dieting 
and functionality.

How the host installs would be kept in (partial) sync is a different 
question - if the install can be trimmed down enough, it could be rolled 
into the initrd. If that ends up not being plausible it all rapidly 
turns into a can of worms because the next thing you know you'll end up 
needing shared root to boot-strap the shared root. :^)

> * Complexity: How to handle bonding/multipathing and other enterprise configurations?

That is really down to a question of whether OpenVZ can handle it. NIC 
bonding would be done on the host, and the guest would simply have a 
venet (virtual network) interface that magically has traffic routed via 
the multi-path or bonded host NICs.

The main point of this would be to separate the clustering aspect from 
the shared root aspect, and abstract it away from the guest. The guest 
would need minimal awareness of the fact that it is running in a shared 
root, over and above bind-mounting some paths to /cdsl.local and 
/cluster in it's fstab. The NIC configuration is done from the host 
(although this is a little dodgy in places WRT how it's implemented from 
the guest side, it's one of the things I'm looking into).

> Did I understand your concept right?

I hope so, but I re-explained above just to make sure. :)

> The biggest problem I see is the support one. This is a very nice
> concept for people who don't care about support.

As I mentioned above, commercial support for OpenVZ is available, and 
OpenVZ project does appear to also specifically provide RHEL kernels 
patched with OpenVZ. Parallels/Virtuozo do provide commercial support.

Gordan