I would tell you something about our openQRM solution for one of my customer in Germany.
At first, the hardware:
1x Dell PowerEdge 1950, 1x Intel QuadCore Xeon CPU, 2 GB memory, LSI 7104 Fibre-Channel 4 GBit
(for the openQRM Server)
3x Dell PowerEdge 1950, 2x Intel QuadCore Xeon CPU, 4 GB memory, LSI 7104 Fibre-Channel 4 GBit
(for the nodes)
1x Qlogic SANbox Express 1400 (4 Gbit)
1x Infortrend Eonstor A12F-G2422 SAN-Box
and now the software:
-Linux CentOS 4.4
My first Goal for my customer:
Three nodes as Xen Dom0 and in summary twelve Xen DomU on this. The Xen DomU boot over NFS from the QRM-server and mount the blockdevices direct from the SAN-Box. In the Xen DomU should run Oracle 9i, Oracle 10g, Oracle 10gR2, Lotus Domino Server, Samba 3, OpenLDAP, Apache and DNS.
The QRM server stored the filesystem images on the SAN-Box over FC.
I start with this project in November 2006, so I installed first openQRM 2.2.x and find the first problems, like only one NIC on the nodes and Dom0/DomU is supported. No support for FibreChannel or other blockdevices, no SMP and it was not possible to change the Xen Configuration over the openQRM GUI.
I must have two NIC's on den Xen Dom0 and DomU, because my customer have a management network and a production network run. And if the Xen Dom0 and DomU have only one NIC, all of the network traffic runs over the QRM-Server, wich will act as router in this case.
After a lot of emails and many phone-calls with Matt from Qlusters, he realised my wishes and changes very fast, so the result was the openQRM Xen-plugin 3.1.2.
Now it support more than one NIC, SMP, mounting blockdevices from the Fibre-Channel, changes in the Xen-Configuration on a running Xen-Partition from the openQRM GUI.
OpenQRM have no update function :-(
So I must deinstall openQRM 2.2.x from the QRM Server, install openQRM 3.1.2 and have to make a new configuration in the GUI.
After this I could do everything what I would do :-)
I add in the resource configuration tab in the field "Resource parameters":
XEN_PARTITION_NETWORK="00:00:09:09:00:01/xenbr1" XEN_PARTITION_HARDDISK="sda11/sda11/w, sda12/sda12w, sda13/sda13/w"
and stop the Xen-Partion and start it again. Important: A restart change nothing!
I create a few Virtual Enviroments and install the software:
-Lotus Domino: successful
-Oracle 9i: successful
-Oracle 10g: successful
-Oracle 10gR2: successful
-Samba, DNS, LDAP, Apache: successful
The cpu and network performance in the Xen DomU is very good.
But the performance in a Xen Dom0/DomU with the blockdevices in the SAN is very poor, remember: it is 4 Gbit FC.
A copy from one blockdevice to an other blockdevice is no problem and a 560 MB file is copied in a few seconds.
When I copy a 560MB file from a directory to an other directory on the same blockdevice it needs more than 90 seconds.
When I do the same thing on the QRM Server everything is fine, every copy with the 560 MB file needs a few seconds.
I search since January 2007 for a solution for this failure and find nothing :-(
The nodes have 4 GB memory, but after a boot with the Xen-Hypervisor I see only 3.2 GB, not more.
I look in the web and find the reason: the Xen-Hypervisor in openQRM was not compile with PAE and hughmem options.
After an email and a phone-call with Matt he told me how I can change this.
You find my little HowTo here:
My first problem could not be solved :-(
Now I realized this:
The nodes boot from the QRM Server over NFS a normal CentOS 4.4 and mount die blockdevices from the SAN. I install the software like Oracle and so on again and my customer can now work.
I mean openQRM is a great software solution and the developer from Qlusters do a great job.
Special Thanks to Matt from Qlusters for his endless endurance and great support work!!!
Phone +49 2131 15 34 98
Fax +49 2131 15 34 99
thank you so much for those detailed informations about your very interesting setup !
This is very useful for the other users of the Xen-plugin.
all the best,