Xen with HughMem > 4 GB

[Dietmar Simons]

Hello,

with openQRM 3.1.2, the Xen-Hypervisor doesn't support hughmem greater than 4 GB.

Example:
The nodes have 4 GB memory, but after a boot with the Xen-Hypervisor I see only 3.2 GB, not more.
I look in the web and find the reason: the Xen-Hypervisor in openQRM was not compile with PAE and hughmem options.

After an email and a phone-call with Matt he told me how I can change this.

Here is my little HowTo:
-Download the openQRM-3.1.2.tgz
-Download the openQRM-plugin-xen-3.1.2.tgz
-unpack openQRM-3.1.2.tgz to /usr/src
-unpack openQRM-plugin-xen-3.1.2.tgz to /usr/src/openqrm-3.1.2/src/plugins
-change to the directory /usr/src/openqrm-3.1.2/src/plugins/xen
-open the Makefile with your favorite editor
-change QRM_MAKE_OPTIONS="dist" \ to QRM_MAKE_OPTIONS="dist XEN_TARGET_X86_PAE=y" \ -save the changes
-install ncurses-devel and xorg-x11-devel, the xorg-x11 is needed for plugin xen to compile vnc
-follow the instruction from http://www.openqrm.org/howtos/howto-compile-and-build.html
until "make realclean", execude this command.
-change to the directory /usr/src/openqrm-3.1.2/src/plugins/xen
-execude make
and drink a coffee :-)
-make a backup from ~/qrm/tftpboot/boot
-change to /var/tmp/qrm/build/xen/xen-3.0.3-testing/dist/install/boot
-copy everything in this directory to ~/qrm/tftpboot/boot
-change to /var/tmp/qrm/build/xen/xen-3.0.3-testing/dist/install/lib/modules
-tar -cvzf modules-Xen-Hypervisor.tgz 2.6.16.29-xen
-copy modules-Xen-Hypervisor.tgz 2.6.16.29-xen to ~/qrm/tftpboot/boot
-change to /tmp
-copy ~/qrm/tftpboot/boot/initrd-Xen-Hypervisor.img to /tmp
-gunzip -S .img initrd-Xen-Hypervisor.img
-mount -oloop initrd-Xen-Hypervisor /mnt
-change to /mnt/lib/modules/2.6.16.29-xen
-copy: cp -av /var/tmp/qrm/build/xen/xen-3.0.3-testing/dist/install/lib/modules/2.6.16.29-xen/kernel/drivers/net/*.ko .
-change to /tmp
-umount /mnt
-gzip -S .img initrd-Xen-Hypervisor
-copy /tmp/initrd-Xen-Hypervisor.img to ~/opt/qrm/tftpboot/boot
-change to ~/qrm/tftpboot/boot
-rename vmlinuz-2.6.16.29-xen to vmlinuz-Xen-Hypervisor
-rename System.map-2.6.16.29-xen to System.map-Xen-Hypervisor
-change to ~/qrm/usr/tomcat/webapps/ROOT/unsecure/xen/
-make a backup from xen-install.tgz, initrd-Xen-Hypervisor.img and vmlinuz-Xen-Hypervisor
-copy ~/qrm/tftpboot/boot/initrd-Xen-Hypervisor.img in this directory
-copy ~/qrm/tftpboot/boot/vmlinuz-Xen-Hypervisor in this directory
-unpack xen-install.tgz
-change to ./xen/dist/install/lib/modules/2.6.16.29-xen
-copy: cp -av /var/tmp/qrm/build/xen/xen-3.0.3-testing/dist/install/lib/modules/2.6.16.29-xen/* .
-change back to ~/qrm/usr/tomcat/webapps/ROOT/unsecure/xen/
-tar -cvzf xen-install.tgz xen

Now we have successful install the new Xen-Hypervisor with hughmem support.
Restart your Xen Dom0 and it is active, you can check it with xm dmesg.

Special Thanks to Matt from Qlusters for his endless endurance and great support work!!!

Greetz
Dietmar Simons

________________________________________________

Simons IT-Service
Dietmar Simons
Matthiasstr. 18a
41468 Neuss
Germany

Phone +49 2131 15 34 98
Fax   +49 2131 15 34 99
________________________________________________

[Matt]

Hey Dietmar,

Great :) Many thanks for your useful tip !

Already picked it up for the next Xen-plugin version (soon come).
It will have + build the Xen-kernel with PAE+Highmen64g by default.

A lot of new and nice features went into the last Xen-plugin releases
because of your great feedback. Many thanks for that and stay tuned !

Maybe you could also tell us a bit more about your setup using
fibre-channel disks attached to the Xen-hosts and handed over
as lvm devices to the partitions ?

have a great day,

Matt

[Matt]

Woow, just saw your posted the info about your setup already ! BEST !

many thanks again + regards,

Matt