ubuntu 9.10 server DomU on XenServer 5.6 beta

I’ve been trying to get ubuntu up an running as a DomU in PV mode under XenServer for a while.  The Citrix blog article is horribly old and some info is no longer accurate.  There are a few key things to getting this done that I discovered along they way.

Where’s Xen kernel?

The xen kernel is gone as all support is now built into the server kernel.  Check near the bottom of this ubuntu article.  You can look inside the kernel config files in /boot to see what support is compiled in and what is modules.

Pygrub, ext4, and Grub 2

Pygrub and xe-edit-bootloader cannot handle an ext4 partition.  Make sure the partition containing /boot is ext3.  xe-boot-loader cannot handle grub 2, but can be edited to allow the reading of grub.cfg.  Pygrub has some support for grub 2.  Out of the box it can handle ubuntu 9.10, but only if /boot is not a separate partition.  I’ll cover that a little later.

Note: I don’t see any support for grub.cfg in pygrub on XenServer 5.5 update 1.  I find it easier to just specify the kernel and ramdisk then to try and get grub 2 support into 5.5.  In this case altering xe-edit-bootloader is very helpful!

  1. First create a VM in XenCenter.  I used the other install media template.
  2. Install ubuntu to your liking. Just in case anything goes wrong with getting console output, it is recommended that you install the SSH server.  If you want to use something other than ext3, make a small first partition for /boot formatted as ext3.  Now you can make the remaining partitions whichever file system you like.  Once done, update the OS if you desire.
  3. The console for the DomU in PV mode will be hvc0.  You will need to tell getty to use this.  Copy the existing tty1.conf and edit it to replace “tty1” with “hvc0”.
    sudo cp /etc/init/tty1.conf /etc/init/hvc0.conf
    nano -w /etc/init/hvc0
  4. If you placed /boot on its own partition and do not want to alter any of the XenServer files, take note of the default menu entry in grub.cfg as you will need to place the options in XenServer.
  5. Shutdown the VM.
  6. Retrieve the UUID for the VM.  Assuming your VM is named “ubuntu”:
    xe vm-list name-label=ubuntu params=uuid
  7. Retrieve the VBD UUID for the storage attached to the VM.
    xe vm-disk-list uuid=<UUID>
  8. Clear out the HVM boot policy.
    xe vm-param-set uuid=<UUID> HVM-boot-policy=
  9. Set the PV bootloader to pygrub.
    xe vm-param-set uuid=<UUID> PV-bootloader=pygrub
  10. Lastly, set the VBD for the VM to be bootable.
    xe vbd-param-set uuid=<VBD UUID> bootable=true
  11. If /boot is a separate partition you must either modify pygrub on the XenServer, or specify your menuentry items now.  If /boot is not a separate partition, skip to step 14.
  12. To add support to pygrub for /boot to be a separate partition, edit /usr/bin/pygrub.  Go to line 395 and you should see the file list for grub 2.  It is only searching for /boot/grub/grub.cfg.  Like in the grub section, it needs to search for the file when /boot is the root of the partition also.  Add /grub/grub.cfg to that list.  Skip to step 14.
  13. If you are hesitant to edit the pygrub python script, simplify specify the arguments you need for the VM.   Specify the kernel, ramdisk, and other options that are present in the VMs grub.cfg.  Make sure you use the entries from your grub file, not these specific ones.
    xe vm-param-set uuid=<UUID> PV-bootloader-args="--kernel=vmlinuz-2.6.31-14-server \
    xe vm-param-set uuid=<UUID> PV-args="root=UUID=<UUID from grub root entry> ro quiet splash"
  14. Start the VM and login.  If your console fails to appear or seems to hang, try connecting to the VM using SSH.  If you missed they step about getting getty to output to hvc0, the console will appear to be hung.  In actuality it is just not displaying the login prompt on hvc0.
  15. Attach the xs-tools.iso to the VM and mount the image.  Mount will claim /dev/cdrom1 does not exist if it has not yet been attached to the VM in XenCenter.
    sudo mount /dev/cdrom1 /mnt
  16. Install the XenServer tools.  Make sure you use the proper file for your architecture (amd64 or i386).  A quick “ls” will confirm the version of the utilities in you xs-tools.iso.  My file name my differ.
    sudo dpkg -i /mnt/Linux/xe-guest-utilities_5.5.901-562_i386.deb
  17. Unmount the iso image and then detach it in XenCenter.
    sudo umount /mnt
  18. You are now up and running.  Update the system with aptitude if you like.  Keep in mind that if you update your kernel and you specified PV-bootloader-args, you will have to go back and update those accordingly.  I still have some issue like XenCenter not reporting the NIC IP and no performance data.  Hopefully that can be resolved soon so I can update this article.


I need to use xe-edit-bootloader.

To add grub2 support, edit /opt/xensource/bin/xe-edit-bootloader. On line 18 add the grub2 configuration files to the list.

default_file_list="/boot/grub/grub.cfg /grub/grub.cfg /boot/grub/menu.lst /grub/menu.lst"

After updating the kernel it continues to boot the old kernel.

If the new kernel is listed as the first and default entry in your grub file, make sure your PV-bootloader-args are not specified.  Retrieve the UUID of the VM using step 6 from the article.  Check the PV-bootloader-args parameter.

xe vm-param-list uuid=<UUID>

If you have specified PV-bootloader-args you need to update them to specify the new kernel.  You can use xe-edit-bootloader to find these values.  If you have decided to use pygrub to read the grub config at boot, then clear out the PV-bootloader-args and PV-args fields.

xe vm-param-set uuid=<UUID> PV-bootloader-args=
xe vm-param-set uuid=<UUID> PV-args=

Can’t I just use PV-kernel and PV-ramdisk?

These parameters on a VM are actually for when the kernel and ramdisk on on the XenServer (Dom0) filesystem.  The article here is for using the kernel inside the actual VM (DomU) itself.  For that, PV-bootloader-argsd is used.

Can’t type at login prompt.

I ran into one instance where I could see the login prompt but nothing I typed appeared.  First, make sure you click inside the console window.  If that does not resolve the issue, close and reopen XenCenter.  This fixed the issue when I encountered it.

PV is not working and I need to get the VM back up.

It is possible to get the VM back up if either PV mode will just not work or you need to go back and redo a step.  To do this simply shutdown the VM and respecify the HVM-boot-policy parameter. This parameter is case-sensitive!

xe vm-param-set uuid=<UUID> HVM-boot-policy="BIOS order"

To return to PV mode, clear the HVM-boot-policy parameter.

xe vm-param-set uuid=<UUID> HVM-boot-policy=

About Adam Oliver

Adam Oliver has been working in the IT field for over 10 years and is a Sales Engineer for Citrix Systems, Inc. Follow Adam on twitter at http://twitter.com/theadamoliver. Find out more about Citrix at www.citrix.com.
This entry was posted in Virtualization, XenServer and tagged , , . Bookmark the permalink.

7 Responses to ubuntu 9.10 server DomU on XenServer 5.6 beta

  1. Samuli Piela says:

    Big thank yous! This got me going with Ubuntu 10.04 x86_64 and XenServer 5.5.


  2. Greg says:

    Worked for me too, thanks. I imagine the performance data is a function of the version of the xs tools. We’ll have to see when 5.6 final comes out.

  3. Danny says:

    This saved my bacon, thank you for sharing your work.

  4. Roger says:

    At step 14 I get an error when I try to start the VM:
    The bootloader for this VM returned an error — did the VM installation succed? Error from bootloader: too many bootable disks (2 disks)

    Any chance you know what that’s about?

    • Adam Oliver says:

      See if xe-edit-bootloader says something similar. http://www.aikidokatech.com/?p=23

      • Roger says:

        Thanks for the quick response. That seemed to work but now I’m getting something else:
        Traceback (most recent call last): – File “/usr/bin/pygrub”, line 746, in ? – raise RuntimeError, “Unable to find partition containing kernel” – RuntimeError: Unable to find partition containing kernel

        You mentioned the /boot being on a different partition and I don’t think it is, but I’m not positive. That being said, I already confirmed that the grub/grub.cfg already existed in pygrub.

        Thanks again.

Leave a Reply

Your email address will not be published. Required fields are marked *