Xen with HughMem > 4 GB

  • Dietmar Simons

    Dietmar Simons - 2007-02-15


    with openQRM 3.1.2, the Xen-Hypervisor doesn't support hughmem greater than 4 GB.

    The nodes have 4 GB memory, but after a boot with the Xen-Hypervisor I see only 3.2 GB, not more.
    I look in the web and find the reason: the Xen-Hypervisor in openQRM was not compile with PAE and hughmem options.

    After an email and a phone-call with Matt he told me how I can change this.

    Here is my little HowTo:
    -Download the openQRM-3.1.2.tgz
    -Download the openQRM-plugin-xen-3.1.2.tgz
    -unpack openQRM-3.1.2.tgz to /usr/src
    -unpack openQRM-plugin-xen-3.1.2.tgz to /usr/src/openqrm-3.1.2/src/plugins
    -change to the directory /usr/src/openqrm-3.1.2/src/plugins/xen
    -open the Makefile with your favorite editor
    -change QRM_MAKE_OPTIONS="dist" \ to QRM_MAKE_OPTIONS="dist XEN_TARGET_X86_PAE=y" \ -save the changes
    -install ncurses-devel and xorg-x11-devel, the xorg-x11 is needed for plugin xen to compile vnc
    -follow the instruction from http://www.openqrm.org/howtos/howto-compile-and-build.html
    until "make realclean", execude this command.
    -change to the directory /usr/src/openqrm-3.1.2/src/plugins/xen
    -execude make
    and drink a coffee :-)
    -make a backup from ~/qrm/tftpboot/boot
    -change to /var/tmp/qrm/build/xen/xen-3.0.3-testing/dist/install/boot
    -copy everything in this directory to ~/qrm/tftpboot/boot
    -change to /var/tmp/qrm/build/xen/xen-3.0.3-testing/dist/install/lib/modules
    -tar -cvzf modules-Xen-Hypervisor.tgz
    -copy modules-Xen-Hypervisor.tgz to ~/qrm/tftpboot/boot
    -change to /tmp
    -copy ~/qrm/tftpboot/boot/initrd-Xen-Hypervisor.img to /tmp
    -gunzip -S .img initrd-Xen-Hypervisor.img
    -mount -oloop initrd-Xen-Hypervisor /mnt
    -change to /mnt/lib/modules/
    -copy: cp -av /var/tmp/qrm/build/xen/xen-3.0.3-testing/dist/install/lib/modules/*.ko .
    -change to /tmp
    -umount /mnt
    -gzip -S .img initrd-Xen-Hypervisor
    -copy /tmp/initrd-Xen-Hypervisor.img to ~/opt/qrm/tftpboot/boot
    -change to ~/qrm/tftpboot/boot
    -rename vmlinuz- to vmlinuz-Xen-Hypervisor
    -rename System.map- to System.map-Xen-Hypervisor
    -change to ~/qrm/usr/tomcat/webapps/ROOT/unsecure/xen/
    -make a backup from xen-install.tgz, initrd-Xen-Hypervisor.img and vmlinuz-Xen-Hypervisor
    -copy ~/qrm/tftpboot/boot/initrd-Xen-Hypervisor.img in this directory
    -copy ~/qrm/tftpboot/boot/vmlinuz-Xen-Hypervisor in this directory
    -unpack xen-install.tgz
    -change to ./xen/dist/install/lib/modules/
    -copy: cp -av /var/tmp/qrm/build/xen/xen-3.0.3-testing/dist/install/lib/modules/* .
    -change back to ~/qrm/usr/tomcat/webapps/ROOT/unsecure/xen/
    -tar -cvzf xen-install.tgz xen

    Now we have successful install the new Xen-Hypervisor with hughmem support.
    Restart your Xen Dom0 and it is active, you can check it with xm dmesg.

    Special Thanks to Matt from Qlusters for his endless endurance and great support work!!!

    Dietmar Simons


    Simons IT-Service
    Dietmar Simons
    Matthiasstr. 18a
    41468 Neuss

    Phone +49 2131 15 34 98
    Fax   +49 2131 15 34 99

    • Matt

      Matt - 2007-02-15

      Hey Dietmar,

      Great :) Many thanks for your useful tip !

      Already picked it up for the next Xen-plugin version (soon come).
      It will have + build the Xen-kernel with PAE+Highmen64g by default.

      A lot of new and nice features went into the last Xen-plugin releases
      because of your great feedback. Many thanks for that and stay tuned !

      Maybe you could also tell us a bit more about your setup using
      fibre-channel disks attached to the Xen-hosts and handed over
      as lvm devices to the partitions ?

      have a great day,


    • Matt

      Matt - 2007-02-15

      Woow, just saw your posted the info about your setup already ! BEST !

      many thanks again + regards,



Log in to post a comment.