Xen - ParabolaWiki

# Xen

This document explains how to set up Xen 4.2 in Parabola. It uses the new oxenstored / xl toolstack (replaces the xend / xm toolstack which was deprecated in Xen 4.1).

## What is Xen?

According to the Xen development team:

  "The Xen hypervisor, the powerful free software industry standard for virtualization, offers a powerful, efficient, and secure feature set for virtualization of x86, x86_64, IA64, PowerPC, and other CPU architectures. It supports a wide range of guest operating systems including Windows, GNU/Linux, Solaris, and various versions of the BSD operating systems."

The Xen hypervisor is a thin layer of software which emulates a computer architecture. It is started by the boot loader and allows several operating systems to run simultaneously on top of it. Once the Xen hypervisor is loaded, it starts the "Dom0" (for "domain 0"), or privileged domain, which in our case runs a Linux-libre kernel. The hardware must, of course, be supported by this kernel to run Xen. Once the Dom0 has started, one or more "DomU" domains can be started and controlled from Dom0.

Xen.org provide a full overview

## Types of Virtualization Available with Xen

### Paravirtual (PV)

Paravirtualized guests require a kernel with support for Xen built in. This is default for all recent Linux kernels and some other Unix-like systems. Paravirtualized domUs usually run faster as they do not have to run in emulated hardware.

### Hardware Virtual (HVM)

For hardware virtualization in our domUs, the host system hardware must include either Intel VT-x or AMD-V (SVM) virtualization support. In order to verify this, run the following command on the host system:

    grep -E "(vmx|svm)" --color=always /proc/cpuinfo

If the above command does not produce output, then hardware virtualization support is unavailable and your hardware is unable to run Xen HVM guests. It is also possible that the host CPU supports one of these features, but that the functionality is disabled by default in the system BIOS. To verify this, access the host system's BIOS configuration menu during the boot process and look for an option related to virtualization support. If such an option exists and is disabled, then enable it, boot the system and repeat the above command.

### Paravirtual on Hardware (PV on HM)

There is a third mode which runs Xen on top of a HardwareVirtual guest.

### Recommended Practices

The current xen.org wiki has a section regarding best practices for running Xen. It includes information on allocating a fixed amount of memory dom0 and how to dedicate (pin) a CPU core for its own use.

## Obtaining Xen

Xen is available from kernels repo, it provides the means to create a Xen host. The Xen host maintains the tools and configuration files for creating and controlling Xen guests.

### After installation

The following steps are required after installation , most consist of adding a line or two to a configuration file, or commenting out others. All of these requirements are fully covered in this document.

The dom0 host requires

- systemd services to be started at boot time and additional configuration for
- a xenfs filesystem mount point,
- an entry in the bootloader configuration file
- additional networking configuration
- a configuration file for each guest
- a means of accessing each guest's kernel and initrd image.

Each domU guest needs

- to be installed (refer to the distro's installation guide)

Each paravirtualized (pv) guests needs

- a mount point corresponding to the virtual disk (specified in its etc/fstab or equivalent)
- tty1 replaced by a virtual console (specified in its etc/inittab or equivalant, or systemd service file).

To speed the introduction of 4.2, the maintainer during Xen 4.1 stepped aside; there are significant changes between 4.1 and 4.2, coupled with the transition of Arch from rc.d to systemd. It may take a short time for the new package to settle out so, for the moment, a section on building Xen from source is provided near the end.

## Bootloader Configuration

Xen requires that you boot a special xen kernel (xen.gz) which in turn boots your system's normal kernel. A new bootloader entry is needed. See #Bootloader Configuration. The menuentry for a Xen system starts a Xen kernel before starting the main host's kernel.

### grub2

To boot into the Xen system, we need a new menuentry in grub.cfg.

Example non-xen menuentry for LVM with gpt partition table

    menuentry 'Parabola GNU/linux-libre ' {
      insmod part_gpt
      insmod lvm
      insmod ext2
      set root='lvm/vg0-arch'
      linux /boot/vmlinuz-linux-libre root=/dev/mapper/vg0-arch ro init=/usr/lib/systemd/systemd quiet
      initrd /boot/initramfs-linux-libre.img
    }

The menuentry to boot the same parabola system after Xen has been installed. Get the UUID for lvm/vg0-arch by using blkid.

    menuentry 'Parabola GNU/linux-libre Xen 4.2' {
      insmod lvm
      insmod part_gpt
      insmod ext2
      set root='lvm/vg0-arch'
      search --no-floppy --fs-uuid --set=root 346de8aa-6150-4d7b-a8c2-1c43f5929f99
      multiboot /boot/xen.gz placeholder dom0_mem=1024M
      module /boot/vmlinuz-linux-libre placeholder root=/dev/mapper/vg0-arch ro init=/usr/lib/systemd/systemd quiet
      module  /boot/initramfs-linux-libre.img
    }

Example for a physical partition

    Parabola GNU/linux-libre(XEN)
    menuentry "Parabola GNU/linux-libre(XEN)" {
        set root=(hd0,X)
      search --no-floppy --fs-uuid --set=root 346de8aa-6150-4d7b-a8c2-1c43f5929f99
        multiboot /boot/xen.gz dom0_mem=1024M
        module /boot/vmlinuz-linux-libre-xen-dom0 root=/dev/sda ro
        module /boot/initramfs-linux-libre-xen-dom0.img
    }

More at Grub2

## Network Configuration

Previous versions of Xen provided a bridge connection whereas Xen 4.2 requires that network communications between the guest, the host (and beyond) is set up separately. Using dhcp throughout simplifies things while we get everything working, the guest. When fully working, the guest will normally benefit from a static network address.

Netcfg greatly simplifies network configuration and is now included as standard in the base package. Example configuration files are provided in etc/network.d/examples and Xen 4.2 provides scripts for various networking configurations in /etc/xen/scripts.

### Network Bridge

By default, Xen expects a bridge connection to be named xenbr0.

    # cd /etc/network.d
    # cp examples/bridge xenbridge-dhcp

make the following changes to xen-bridge:

    INTERFACE="xenbr0"
    BRIDGE_INTERFACE="eth0"
    DESCRIPTION="Xen bridge connection"

assuming your existing eth0 connection is called eth0-dhcp, edit /etc/conf.d/netcfg

    NETWORKS=(eth0-dhcp xenbridge-dhcp)

restart the network:

    systemctl restart netcfg.service

when the prompt returns, check all is well

    ip addr show
    brctl show

    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
       inet 127.0.0.1/8 scope host lo
       inet6 ::1/128 scope host
          valid_lft forever preferred_lft forever
    3: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
       link/ether 00:1a:92:06:c0:c0 brd ff:ff:ff:ff:ff:ff
       inet 192.168.1.3/24 brd 192.168.1.255 scope global xenbr0
       inet6 fe80::21a:92ff:fe06:c0c0/64 scope link
          valid_lft forever preferred_lft forever

    bridge name	bridge id		STP enabled	interfaces
    xenbr0		8000.001a9206c0c0	no		eth0

## Final preparations to start Xen at boot

Include in your /etc/fstab

     none /proc/xen xenfs defaults 0 0

Issue the following so that the services are started at bootup:

    # systemctl enable xenstored.service
    # systemctl enable xenconsoled.service
    # systemctl enable xendomains.service

Reboot into your new Arch Dom0.

## Creating guest domains (domU)

### Creating Paravirtualized (PV) Guests

The general procedure is: perform a normal or minimal installation of the distro that will become a guest; copy its kernel/initrd to a directory on the host; modify its /etc/fstab to use the virtual disk; modify its the way it sets up a terminal (getty); create a config file for xl.

## Running guest Domains

Start the guest domU and a console

    # xl create /etc/xen/pv-squeeze.cfg

Check all is well:

    # xl list

### Useful xl command examples

    # xl top
    # xl list
    # xl shutdown pv-squeeze
    # xl destroy pv-squeeze

## Worked example for gNewSense Metad as a guest

Install gNewSense 3.0 (do not bother with graphical interface, install as little as possible). Having installed it, boot into in your new Parabola Xen system and mount it.

The example has the guest gNewSense Metad installed onto /dev/vg0/pv_metad

    # mkdir /tmp/metad
    # mkdir -p /var/lib/xen/images/metad
    # mount -text4 /dev/vg0/pv_metad /tmp/metad/

Copy the kernel and initrd to a location available Xen. n.b. Metad has softlinks (vmlinuz and initrd.img) in its root directory to the current kernel, so check you have copied a real kernel, and not just a link!

    # cp /tmp/squeeze/vmlinuz /tmp/metad/initrd.img /var/lib/xen/images/metad

edit /tmp/metad/etc/fstab change its root entry to begin with /dev/xvda1

    # /dev/xvda1 / ext4 noatime,nodiratime,errors=remount-ro 0 1

gNewSense Metad uses /etc/inittab to configure its terminals. Other distros use other mechanisms. We need to replace the creation of terminals tty1, tty2 etc. with a single hvc0.

    Comment out any getty tty lines like these:
    1:2345:respawn:/sbin/getty 38400 tty1
    2:23:respawn:/sbin/getty 38400 tty2

    Replace with the single line
    hvc:2345:respawn:/sbin/getty 38400 hvc0

Create a guest configuration file by copying one of the given example files and editing as follows:

    # cp /etc/xen/xlexample.pvlinux /etc/xen/pv-metad.cfg

    edit /etc/xen/pv-metad.cfg with the following:
    name = "metad.pvlinux"
    kernel=/var/lib/xen/images/metad/vmlinuz
    ramdisk=/var/lib/xen/images/metad/initrd.img
    extra = "root=/dev/xvda1"
    memory = 256
    vcpus = 2
    disk = [ '/dev/vg0/pv_metad,raw,xvda1,rw' ]

Start the guest domU and a console

    # xl create /etc/xen/pv-metad.cfg

Check all is well:

    # xl list

    Name             ID   Mem VCPUs    State	Time(s)
    Domain-0         0   1024     2     r-----      26.2
    metad.pvlinux    1    123     2     -b----       1.5

Start a console:

    # xl console metad.pvlinux

    ( example output)
    [
       0.000000] Initializing cgroup subsys cpuset
    [    0.000000] Initializing cgroup subsys cpu
    [    0.000000] Linux version 2.6.32-5-xen-amd64 (gNewSense 2.6.32-46) ...UTC 2012
    [    0.000000] Command line: root=/dev/xvda1
    [    0.000000] KERNEL supported cpus:
    [    0.000000]   Intel GenuineIntel
    [    0.000000]   AMD AuthenticAMD
    [    0.000000]   Centaur CentaurHauls
    ...

## Installation notes for domU Paravirtualized guests

### Parabola

The default Parabola initramfs images lack essential xen modules. In the guest install, we need to add the following to mkinitcpio.conf

    MODULES = "xen-blkfront xen-fbfront xen-netfront xen-kbdfront"

and then rebuild its initramfs-linux-libre.img with mkinitcpio -p linux-libre.

## Common Errors

- 'xl list' complains about libxl

- Either you have not booted into the Xen system, or xen modules listed in xencommons script are not installed

- xl create fails

- check the guest's kernel is located correctly, check the pv-xxx.cfg file for spelling mistakes (like using initrd instead of ramdisk)

- Parabola guest hangs with a ctrl-d message

- press ctrl-d until you get back to a prompt, rebuild its initramfs described

- The domu guest hangs at 'crond'

- The guest's terminal needs to be set to hvc0 instead of tty1 See the gNewSense Metad example above.

- Error message "failed to execute '/usr/lib/udev/socket:/org/xen/xend/udev_event' 'socket:/org/xen/xend/udev_event': No such file or directory"

- caused by /etc/udev/rules.d/xend.rules; xend is (a) deprecated and (b) not used, so it is safe to remove xend.rules

## Building and Installing Xen Hypervisor and Dom0 Host from Source

Xen recommends that a Xen host (dom0) is 64-bit, guests may be either 32-bit or 64-bit. To build such a system requires a mixed 64/32-bit installation and packages from the the Community repository; the host uses a network bridge and a modified entry in the bootloader configuration file (for example, grub.cfg). These notes assume an installation using systemd is in use, as is the default for a new installation of Arch. For these reasons, you may prefer to make a fresh installation of Parabola on which to build and install Xen.

### Building Xen

Building and installing Xen significantly modifies your system. Xen is an established program, but Xen 4.2 is extremely new. Consider Xen 4.2 on an Parabola system to be untested. Consider yourself to be an alpha tester, perhaps make a throw-away Parabola system for the Xen installation.

It is best practise to backup and highly recommended to make a fresh installation of Parabola on which to build and install Xen.

- The build process installs additional source from git, so a working internet connection is required.
- systemd service files will be available soon. Until then we use (the currently still supported, but legacy) rc.d and rc.conf.

Edit /etc/pacman.conf to uncomment entries under repositries for multilib and community (three lines each). Prepare for and perform a full system upgrade (pacman -Syu). Install packages listed under 'Obtaining Xen'. Download Xen Hypervisor 4.2 tarball from http://xen.org/products/downloads.html. Unpack the tarball to a suitable location (tar xjf <path/to/tarball> location). Unfortunately, the build process also creates binaries and scripts for the old, deprecated xend/xm. The Xen documentation recommends building Xen as root.

    # cd xen-4.2.0
    # PYTHON=/usr/bin/python2
    # export PYTHON
    # ./configure
    # make world
    # make dist

The dist directory can be used to install Xen to any machine, but it

- sets the 'sticky' bit on all file permissions
- installs startup scripts to etc/init.d (equivalent of etc/rc.d)
- includes some udev rules for 'xend' which creates LOTS of error messages when booting up (xend is not used, having been replaced by xendomains)

The only script we need from etc/init.d is xendomains since the systemd service files given below replace etc/init.d/xencommons. The service files are based on those in Fedora 17 (which uses systemd and provides Xen 4.1). However, it places xendomains in /usr/libexec which is not present in Arch. The xendomain.service below uses /etc/xen/scripts as the location for xendomains.

Fix these problems with

    # cd dist
    # chmod -R -s install/
    # cp install/etc/init.d/xendomains install/etc/xen/scripts
    # rm install/etc/init.d/*
    # rmdir install/etc/init.d
    # rm install/etc/udev/rules.d/xend.rules

If installing to another Parabola system, make a tarball and copy it over:

    # cd ..
    # tar cjf ~/xen-dist-4.2.bz2 dist/

    copy the tarball to the other installation, boot into it
    use 'tar xjf xen-dist-4.2.bz2 .' to unpack
    then install packages listed under 'Packages required for host'

    Now change to the 'dist' directory and install
    # cd dist

Whether installing now, or to another installation, from the dist directory issue:

    # ./install.sh

### Enabling Xen under Systemd

Add the following files

/etc/modules/load/xen.conf

    xen-evtchn
    xen-gntdev
    xen-gntalloc
    xen-blkback
    xen-netback
    xen-pciback
    xen-acpi-processor

n.b The following were included in xencommons, but were not inserted by systemd evtchn gntdev netbk blkbk xen-scsibk usbbk pciback blktap2 blktap. xen-acpi-processor may not work on some machines. Remove this when getting error.

/usr/lib/systemd/system/xenstored.service

    [Unit]
    Description=Xenstored - daemon managing xenstore file system
    Before=libvirtd.service libvirt-guests.service
    After=dbus.service
    RefuseManualStop=true

    [Service]
    Type=forking
    PIDFile=/var/run/xenstored.pid
    ExecStart=/usr/sbin/xenstored --pid-file /var/run/xenstored.pid $XENSTORED_ARGS

    [Install]
    WantedBy=multi-user.target

/usr/lib/systemd/system/xenconsoled.service

    [Unit]
    Description=Xenconsoled - handles logging from guest consoles and hypervisor
    After=xenstored.service

    [Service]
    Type=simple
    PIDFile=/var/run/xenconsoled.pid
    ExecStart=/usr/sbin/xenconsoled

    [Install]
    WantedBy=multi-user.target

/usr/lib/systemd/system/xendomains.service

    [Unit]
    Description=Xendomains - start and stop guests on boot and shutdown
    Requires=proc-xen.mount xenstored.service
    After=proc-xen.mount xenstored.service xenconsoled.service
    ConditionPathExists=/proc/xen

    [Service]
    Type=oneshot
    RemainAfterExit=true
    ExecStartPre=/usr/bin/grep -q control_d /proc/xen/capabilities
    ExecStart=/etc/xen/scripts/xendomains start
    ExecStop=/etc/xen/scripts/xendomains stop

    [Install]
    WantedBy=multi-user.target

/usr/lib/systemd/system/proc-xen.mount

     [Unit]
     Description=Mount /proc/xen files
     ConditionPathExists=/proc/xen
     RefuseManualStop=true

     [Mount]
     What=xenfs
     Where=/proc/xen
     Type=xenfs

Issue the following so that the services are started at bootup:

    # systemctl enable xenstored.service
    # systemctl enable xenconsoled.service
    # systemctl enable xendomains.service

Add a new Xen menuentry in grub.cfg as described earlier and then reboot into the new Xen system and check all is well:

    # xl list

    Name                                        ID   Mem VCPUs	State	Time(s)
    Domain-0                                     0  1024     2     r-----       6.1

n.b. xencommons sets the name "Domain-0" in the xenstored database. The current systemd service files do not do this, so at the moment xl list displays "(null)" as the name for Dom0.

## Resources

- The homepage at xen.org
- The wiki at xen.org

## Acknowledgement

This wiki article is based on ArchWiki. We may have removed non-FSDG bits from it.
Categories:

- Virtualization and Emulation
- Kernel