Gentoo Logo
Gentoo Logo Side

Gentoo Xen Linux Guide

Contents:

1. Xen concepts

Introduction 

Xen is designed to allow you to run multiple operating systems simultaneously on a single computer with performance close to that of a native operating system .

In order to understand how this is possible some concepts need to be understood.

Basic Problems 

In order to run two or more operating systems simultaneously on the same machine two basic problems must be overcome.

  • Hardware isolation - how to avoid two operating systems accessing the same hardware at the same time.
  • Scheduling - deciding how to allocate CPU time to an OS

Probably the most difficult of these problems is Hardware isolation. There are two basic approaches to achieving hardware isolation, hardware emulation and hardware virtualization.

Emulation vs Virtualization 

Emulation involves presenting a fake representation of the hardware to the operating system. Emulation can allow you to run programs completely independently of the underlying hardware but it tends to be slow.

The other approach, Virtualization attempts to use the hardware directly and arbitrate its access at the hardware level. This approach tends to be much faster but requires hardware support which is not currently found in consumer hardware.

Xen takes an approach called para-virtualization that has performance similar to full virtualization, but that can be run on consumer level hardware.

Para-virtualization 

Para-virtualization uses a software layer below the OS to arbitrate hardware access and thus avoids the need for special hardware. OS's running under Xen need to be modified so that for operations requiring arbitration, they access the hypervisor rather then the hardware.

Xen terminology 

In Xen OS instances (file system and associated kernel) are referred to as domains. A domain that has direct access to hardware is referred to as a privileged domain. In the default setup the base operating system (the one everything is controlled from) is a privileged domain and all other domains are unprivileged.

Note: The first privileged domain is referred to as domain0 and is used to control all other domains

Note: domainN is used to refer to unprivileged domains

2. Gentoo in domain0

emerging the base components 

firstly you need to emerge the hypervisor and kernel source. The app-emulation/xen ebuild contains the hypervisor and a server for communicating with the hypervisor. The sys-kernel/xen-sources contains a Linux kernel modified to run under the Xen hypervisor.

Code listing 2.1: emerge xen

# emerge app-emulation/xen sys-kernel/xen-sources

you will need to configure your kernel for the hardware in your system. To do this change to the Xen sources directory then run make menuconfig.

Code listing 2.2: configure your kernel

# cd /usr/src/linux-2.6.11.10-xen
# make menuconfig

This is the same as any other kernel except that there will be some extra options for Xen related features. It is OK to remove drivers for hardware you don't have but do not change any Xen options.

Note: By default the required Xen settings are selected. You should only need to change settings for you specific hardware

The following Xen settings are required for an domain0 kernel.

Code listing 2.3: configure basic Xen options

(configure XEN settings)
XEN --->
  [*] Privileged Guest (domain 0)
  [*]  Block-device backend driver
  [*]  Network-device backend driver
  [*] Block-device frontend driver
  [*] Network-device frontend driver
  [ ]   Piplined transmitter (DANGEROUS)
  [*] Scrub memory before freeing it to Xen
      Processor Type (X86)  --->

Because we will be using LVM you need to have the Device mapper support selected. you should also select the appropriate drivers for you hard drive.

Code listing 2.4: configure required devices

(configure LVM support)
Device Drivers --->
  Multi-device support (RAID and LVM) --->
    <*>  Device mapper support

(if you are using ide drives at minimium set the following)
Device Drivers --->
  ATA/ATAPI/MFM/RLL support  --->
  <*>   Enhanced IDE/MFM/RLL disk/cdrom/tape/floppy support
  
  <*>   Include IDE/ATA-2 DISK support
  
  <*>   generic/default IDE chipset support
  

You need to make sure the /dev file system support is not selected and that all the file systems you are using are selected (i.e. ext3). You should also make sure Advanced partition selection is selected

Code listing 2.5: select required filesystems

(insure the /dev file system is not selected
File systems --->
  Pseudo Filesystems --->
    [*] /proc file system support
    [ ] /dev file system support (OBSOLETE)
    [*] Virtual memory file system support (former shm fs)
    
(Select one or more of the following options as needed by your system)
File system --->
  <*> Reiserfs support
  <*> Ext3 journalling file system support
  <*> JFS filesystem support
  <*> Second extended fs support
  <*> XFS filesystem support
  
(At least this should selected)  
File system --->
  Partition Types --->
    [*] Advanced partition selection

Now that your kernel is configured, it is time to compile and install it. Exit the menuconfig and start the compilation process:

Code listing 2.6: compile your kernel

# make && make modules_install

When the kernel has finished compiling, copy the kernel image to /boot.

Code listing 2.7: Installing the kernel

# cp vmlinux /boot/vmlinux-syms-2.6.11.10-xen
# cp vmlinuz /boot/vmlinuz-2.6.11.10-xen
# cp System.map /boot/System.map-2.6.11.10-xen
# cp .config /boot/config-2.6.11.10-xen

you will need to modify /boot/grub/grub.conf so the the hypervisor is run before the domain0 kernel is booted.

Code listing 2.8: Editing /boot/grub/grub.conf

# nano /boot/grub/grub.conf

add the following entry to the /boot/grub/grub.conf file.

Code listing 2.9: grub entry for xen

title=Xen 2.0.6 / XenLinux 2.6.11.10
# You need to set the root partition to your boot partition.
root (hd0,0)
kernel /xen.gz dom0_mem=131072
module /vmlinuz-2.6.11.10-xen root=/dev/hda6

You should now reboot your system and check that the privileged domain0 boots.

3. Gentoo in domainN

creating the root partition 

It is now time to setup the unprivileged domains. You will need to start by creating partitions for your new operating systems.

I have chosen the simplest partition scheme, a single root partition and a swap file. The choice is mostly arbitrary. You can check out the Gentoo Linux x86 Handbook for other possible partitioning schemes.

Code listing 3.1: creating logical xen partitions

# lvcreate -L2M -nxen1-root vg
# lvcreate -L100M -nxen1-swap vg

Note: If you haven't already setup lvm read Gentoo LVM2 installation

Note: vg will be whatever you call your volume group

Next we need to format the partitions. I have chosen to use ext3 but the choice is arbitrary.

Note: your xen kernel must be configured with support for the file system you choose

Code listing 3.2: Applying a file system on the root partition

# mkfs.ext3 /dev/vg/xen1-root

Code listing 3.3: Creating a Swap signature

# mkswap /dev/vg/xen1-swap

Installing the OS 

The final stage is installing your OS on the new partition. This is mostly the same as a normal install.

you will need to create the gentoo mount point and mount the root partition.

Code listing 3.4: Mounting partition

# mkdir /mnt/gentoo
# mount -t ext3 /dev/vg/xen1-root /mnt/gentoo

You will need to get a tarball with one of the stages and unpack it to your /mnt/gentoo directory. If you don't know how to do this read Gentoo Handbook chapter 5

Now unpack your downloaded stage onto your system. We use GNU's tar to proceed as it is the easiest method:

Code listing 3.5: Unpacking the stage

# cd /mnt/gentoo
# tar -xvjpf stage3-<subarch>-2005.0.tar.bz2

This will create the base folders. You will then need to bind the proc file system to the mount point on the new partition.

Code listing 3.6: Binding /proc

# mount --bind /proc /mnt/gentoo/proc

We will be using nfs to access portage so we will bind the portage partition from our main system. You will need to create the mount point then bind portage to it.

Code listing 3.7: Binding /usr/portage

# mkdir /mnt/gentoo/usr/portage
# mount --bind /usr/portage /mnt/gentoo/usr/portage

You should copy the /etc/resolve.conf file and the /etc/make.conf to /mnt/gentoo/etc

Code listing 3.8: Copying resolve.conf and make.conf

# cp /etc/resolve.conf /mnt/gentoo/etc/resolve.conf
# cp /etc/make.conf /mnt/gentoo/etc/make.conf

You should now chroot into the partition and continue the install process.

Code listing 3.9: Enter chroot

# chroot /mnt/gentoo /bin/bash
# env-update
# source /etc/profile

When you edit your fstab file it should contain the root file system and the swap file entries.

Code listing 3.10: Editing /etc/fstab

# nano /etc/fstab

Code listing 3.11: fstab entries

# there is no boot partition so don't include it
/dev/sda1   /     ext2    noatime      0 0
/dev/sda2   none  swap    sw           0 0

Note: You must have /dev/hdaN entries in you domain0 or the hypervisor will not be able to determine the major and minor for the device.

Set a static address because that is simple.

Code listing 3.12: Editting /etc/conf.d/net

# nano -w /etc/conf.d/net

Code listing 3.13: Setting static address

iface_eth0="192.168.0.4 broadcast 192.168.0.255 netmask 255.255.255.0"
gateway="eth0/192.168.0.1"

Next edit the hosts file and add the hostname.

Code listing 3.14: Editing /etc/hosts

# nano -e /etc/hosts

Code listing 3.15: Add hostname

127.0.0.1     localhost xen1

In order for the os to be started you need to create a configuration file as follows.

Code listing 3.16: Create startup file

# nano /etc/xen/auto/xen1

Code listing 3.17: Xen startup file

kernel = "/usr/src/linux-2.6.11.10-xen/vmlinuz"
memory = 128
name = "xen1"
vif = [ 'mac=aa:00:00:00:00:11, bridge=xen-br0' ]
disk = [ 'phy:0801,vg/xen1-root,hda1,w','phy:0802,vg/xen1-swap,hda2,w' ]
root = "/dev/hda1"

Start the hypervisor access daemon and add it to the default domain.

Code listing 3.18: Starting hypervisor daemon

# /etc/init.d/xend start

Code listing 3.19: Add xend to default

# rc-update add xend default

You can now start the new domain and display the domain boot sequence.

Code listing 3.20: starting domain

# xm create -c /etc/xen/auto/xen1

the xm command 

the xm allows you to control the hypervisor. You can use it to start and stop domains, to check domain status and number of other hypervisor related functions.

using the following command you can display the status of the guest domains.

Code listing 3.21: display domain status

# xm list

using the following command you can stop a domain synchronously

Code listing 3.22: stopping a domain

# xm shutdown -H -w xen1

4. Tips/Tricks/Thoughts/Hacks/etc

disable tls 

tls does not work directly with Xen and has to be emulated. This makes tls libraries very slow. It is recommended that you disable tls for domains running under Xen.

Code listing 4.1: Editing /etc/make.conf

# nano /etc/make.conf

Code listing 4.2: Unset tls

USE="-tls ... "

Code listing 4.3: Disable tls

# mv /lib/tls /lib/tls.disable

5. References


The contents of this document are licensed under the Creative Commons - Attribution / Share Alike license.
line
Updated 2005-06-16
line
Edward Middleton
Author

line
Summary:  This guide describes how to install Xen Linux over an existing gentoo system and how to create a second gentoo system to run in parallel with it.
line

Donate to support our development efforts.

line
The Gentoo Linux Store
line
php|architect

php|architect is the monthly magazine for PHP professionals, available worldwide in print and electronic format. A percentage of all the sales will be donated back into the Gentoo project.

line
Tek Alchemy

Tek Alchemy offers dedicated servers and other hosting solutions running Gentoo Linux.

line
DDR Memory at Crucial.com

Purchase RAM from Crucial.com and a percentage of your sale will go towards further Gentoo Linux development.

line
Win4Lin at NeTraverse

Win4Lin from NeTraverse lets you run Windows applications under Gentoo Linux at native speeds.

line
Copyright 2001-2003 Gentoo Technologies, Inc. Questions, Comments, Corrections? Email www@gentoo.org.