README for the xen packages =========================== Please read the documentation of Xen in one of the xen-doc-* packages. About ----- Xen allows you to run multiple virtual machines on a single physical machine. See the Xen homepage for more info: http://www.cl.cam.ac.uk/Research/SRG/netos/xen/ If you want to use xen, you need to install the xen package, the needed python packages and bridge-utils and a kernel-xen package. You then need to reboot your machine. Instead of booting a normal Linux kernel, you boot the Xen microkernel and a slightly changed Linux kernel; this kernel runs in domain0 and will drive most of your hardware. The fact that the kernel needs to be changed makes this approach called paravirtualization. It results in very good performance http://www.cl.cam.ac.uk/Research/SRG/netos/xen/performance.html but has the downside of unchanged guests not being supported. However, upcoming hardware features (e.g. intel's VT) will help overcome this limitation on the long term. Xen kernels ----------- Xen supports two kinds of kernels: A Domain 0 kernel, the one that you use to boot your machine after Xen has initialized the CPU and the memory. This kernel contains all the hardware drivers. Its often called xen0 kernel. It normally controls the other domains. For other virtual machines, you don't need the hardware drivers; these kernels can not access the hardware directly anyway. (You can change this and assign some PCI devices to secondary domains, but we'll keep it simple for now.) Instead you use virtual network and block device drivers to access the network and storage. This kernel is often call xenU kernel. (U = Unprivileged). For simplicity, we only ship one kernel for now. It can be used both as a domain 0 and an unprivileged kernel. As most of the hardware drivers are modules anyway, the additional footprint your kernel for the unprivileged domains gets this way is small. The kernel is contained in the kernel-xen package which you need to install to use xen. Booting ------- To get Xen running enter something like this into your grub config title Xen2 kernel (hd0,5)/xen-2.0.gz dom0_mem=458752 module (hd0,5)/vmlinuz-2.6.11.4-21.8-xen parameters module (hd0,5)/initrd-2.6.11.4-21.8-xen.gz Replace (hd0,5) with the partition that holds your /boot directory in grub-speak, i.e. hda1 -> (hd0,0) and sda5 -> (sd0,4). Replace parameters with the kernel parameters that you want to pass to your kernel. These should be very similar to a normal kernel you boot on bare iron. Replace dom0_mem by the amount of memory in kB you have in your machine minus 65536. (64MB is reserved by Xen.) You can pass lower numbers as well. This will be useful after you have stripped down domain0 and want to run several additional domains. Xen2 supports max 3.5GB. If you are coming from SUSE Linux prior to 9.3 Professional, please be aware that the IDE drivers have been modularized for 9.3. Please make sure you have the needed drivers listed in the INITRD_MODULES variable in /etc/sysconfig/kernel prior to calling mkinitrd. You'll normally need ide-disk, and a chipset specific driver such as e.g. amd74xx, piix, or via82cxxx. To have DMA, better _not_ include ide-generic in the initrd! There's one complication with mkinitrd and grub: mkinitrd does attach the bootsplash picture to the gzipped initrd; grub barfs on unzipping the junk after the image and then does not pass it on to xen. Please pass -s off to mkinitrd when calling it manually to avoid this problem. (The postinstall script of kernel-xen that does call mkinitrd knows about this peculiarity and passes -s off.) Once you have booted this configuration successfully, you are running Xen with a domain 0 kernel on top of it. Use xm list to see the list of Xen domains and xm dmesg to see the boot log from Xen. xm only works if xend is running. Start scripts ------------- First you need to activate xend; an init script is part of the package, it should be activated at installation time; you can (de)activate it using insserv (or chkconfig). You can also start it manually with rcxend start. There's more start scripts in /etc/init.d/: boot.xen does move away /lib/tls or put it back into place depending on whether a xen hypervisor is running or not. See below (Thread-Local Storage section) for more information on this. xendomains can be used to start multiple domains on boot. Note that I have not adapted this script to the SUSE look'n'feel. Installing root filesystems or fs images ----------------------------------------- You need to have a root FS with an unprivileged kernel to boot a secondary xen virtual machine. The root filesystem can live on a block device (e.g. a hard disk partition or an LVM2 or EVMS volume) or in a file that holds the filesystem image. Note that every domain needs its own root filesystem. Domains can share filesystems that are mounted read-only from _all_ domains such as e.g. /usr or /opt. Never share an FS that's mounted read-write somehow. For sharing writable data between VMs, use NFS or other networked or cluster filesystems. Below, we'll shortly depict three possible ways to get a root FS installed for xen. 1) Create FS image based on the Novell SUSE rescue image 2) Create FS using the uml-install-suse script from uml-utilities 3) Use the YaST tool to install into a directory 4) Reuse an existing installation Ad 1) Rescue image based root FS To get started quickly, you can use the resuce image from the Novell SUSE installation CD/DVD. It's on the first CD/DVD in the boot/ directory with the name rescue. Copy it to the hard disk, loop mount it, remove the kernel modules and copy your xen kernel (and a selection of needed kernel modules) there. umount it again. To boot it, use a xen-rescue from /etc/xen/ as a template and enter the location of the image file as hda1. There's a script mk-xen-rescue-img.sh in /usr/share/doc/packages/xen/ which can be used to create this image for you. The use of this script is highly recommended. It also creates a config file for you. The disadvantage of using the rescue way of constructing a root FS is that the result does not have an RPM database, so you can't easily add packages using rpm. On the positive side, the result is relatively small yet has most what's needed to get started with networking. Ad 2) uml-install-suse TBW Ad 3) YaST dirinstall a1) Create a root FS image - create disc image file (dd if=/dev/zero of=img ...) - create file system (mkXYZfs img) - mount image (mount -o loop img ...) a2) OR create a root FS on a partition - create FS (mkXYZfs /dev/LVMvolume) - mount it (mount /dev/LVMvolume ...) b) start yast2 dirinstall module, select mounted directory as a target, select required packages (don't forget to select XEN kernel, you need some of the kernel modules in the root FS) c) create /etc/fstab in the root FS d) umount image/FS e) configure and start XEN (use the image/FS as storage) f) In the running new domain (see below how to start it): - manually configure network (yast lan) - disable unneeded services (powermanagement, hwclock, etc.) - optionally have a look at /usr/share/doc/packages/xen/boot.local.xenU and copy the boot command line parsing bits into the new domain Ad 4) Reuse exisintg installation Straight forward; it is advisable to install kernel-xen in there, so you have the matching kernel modules. Booting secondary domains -------------------------- Before you can boot, you need to have some memory available for your secondary domains; as you have given everything to the domain 0 initially, you need to take some away from your domain0 kernel by using xm balloon. Look at your domains with xm list. Now use xm balloon 0 SOMENUMBER with SOMENUMBER being the amount of megabytes that you want to leave to your domain0, so use the current number and subtract 64 from it. Then use xm create /etc/xen/config vmid=3 to boot a virtual machine. Have a look at running sessions with xm list again. Attach to the booted session with xm console. Or pass -c during creation. Most conveniently done with the screen terminal multiplexer. Have a look at the other xm commands. You can start other secondary domains the same way, just remember that every domain needs its own root FS. Note that the secondary domains don't need to have the same kind of system. You can just as well boot a 2.4 xen kernel or a Debian distribution or even NetBSD guests. Don't forget to make the set of kernel modules matching the booted guest kernel available in the guest, though. If you generate the kernels and the initrd from domain0, you need to manually keep the modules in the secondary domains and the ones that are used to produce the initrd in sync. Networking ---------- Your virtual domains become much more useful if your can reach them via network. The default xen setup sets up a virtual bridge (xen-br0) in domain 0 when you start xend. Your eth0 device is enslaved to it. The secondary domains get a virtual network interface eth0, that's visible to domain0 as vifN.0 and connected to the bridge. This means that if you set up an IP address in the secondary domains belonging to the same subnet as eth0 from your domain0, you'll be able to communicate not only with the other secondary domains, but also with domain0 and with the external network. If you have a dhcp server running in your network, your secondary domains should succeed getting an IP address. Be aware that this may have unwanted security implications; you may want to opt for routing instead of bridging, so you can set up firewalling rules in domain0. Please read about the network configuration in the xen manual. You can set up bridging or routing also for other interfaces. The network setup is done via the scripts in /etc/xen/scripts. They do not support ipv6 at this moment, but this is just a limitation of the scripts. If you use SUSEfirewall2, you'll probably want to add xen-br0 to the list of interfaces; as eth0 is on there, you'll probably want to add xen-br0 to the same class as eth0. If you use the rescue images created by above mentioned script, you'll have a boot script inside that parses the ip=.... boot parameter, which you can pass from the xen config file, so you can have it all work automatically. Edit the xen-rescue config files accordingly. Thread-Local Storage -------------------- Since some time, the glibc thread library (NPTL) uses a shortcut to access thread-local variables at a negative segment offset from the segment selector gs instead of reading the linear address from the TDB (offset 0). Unfortunately, this is optimization has been made the default by the glibc and gcc maintainers, as it saves one indirection. For xen this is bad: The access to these variables will trap and xen will need to use some tricks to make the access work. It does work, but it's very slow. One way to work around is to move the libc in /lib/tls away, so the pre-i686 version gets used, where no such tricks are done. The inot script boot.xen does that (if activated) depending on whether you are running under Xen or not. Long-term, there's hope the glibc maintainers will change their mind and allow one more indirection in favour of not screwing up Xen. SUSE Linux 9.1 and SLES9 are not affected by this; we hope to have a solution in time for SUSE Linux 10.0. Security -------- The xen control daemon listens on port 8000; if you're concerned about security you may want to allow it only to be controlled from localhost and not bind to other IP addresses. localhost only is default in Novell SUSE Linux installations b/c security considerations. This may prevent successful migration of virtual machines. We changed the xend to observe the xend-privileged-port setting (in xend-config.sxp). If it's set to 1, xend will only accept configuration commands from ports below 1024. Together with only binding to localhost, this should provide a minimum of security against local users to change virtual machines. Note that the protection is not complete; a normal user in domain 0 can still connect to the consoles of the domains. We also did not fully audit all code paths, so a malevolent user in domain 0 may still get more control over other domains than intended. So it's a good idea not to run xend unless you have certain trust over all domain 0 users. We hope to provide better protection against non-root users in domain 0 with future updates. The upstream xen (at the moment of this writing) does not have the port check; this corresponds to xend-privileged-port being set to 0, which allows all domain 0 users full control over Xen. Note that these considerations do not affect unprivileged domains. Misc ---- xsrv allows to perform a subset of the operations that you can do from domain 0 with xm by using a web interface (localhost:8080). The web interface is not very mature. The xend control interface at http://localhost:8000/xend/ is also accessible and can be accessed by a web browser directly instead of using the xm tool. Limitations ----------- Some hardware may not be supported by the domain0 kernel under Xen. ACPI is used to set up the memory map, interrupt routing etc. in Xen, but is not supported for the domain0 kernel, so you can't read the battery status of your laptop or use the cpufreq CPU speed control. The ACPI and APIC code in Xen is not as powerful as the one from newer versions of Linux; thus some hardware may just refuse to boot. At this moment, xen does not yet support the 64bit mode of x86-64 (AMD64/EM64T) CPUs. You can run these CPUs in 32bit though. And there's work in progress to support the 64bit mode as well. There's no support for PAE36 (read: Xen does support max ~3.5GB RAM). There's currently no support for hugetlbfs either. Xen does support multiple CPUs; however the domains will only see one CPU each at this moment. SMP domains are being worked on. Some agpgart drivers may not have been fixed yet and thus may crash your machine (dom0 kernel). Remove them in such a case. In general booting into runlevel 3 may be a good idea in case your machine crashes on strating X11. Devices on the ISA or PCMCIA bus tend not to work. Disclaimer ---------- Xen did perform amazingly well in our tests and proved very stable. Still, you should be careful when using it; just like you'd be careful if you boot an experimental kernel. Expect that it may not boot and be prepared to have a fallback solution for that scenario. Be prepared that it does not support all of your hardware. And for the worst of all cases, have your most valuable data backed up. (This is always a good idea, BTW.) Feedback -------- In case you have remarks aboyt, problems with, ideas for, praise for, or ... for xen, please report it back to the xen-devel list . If you find issues with the packaging/setup/... done by Novell/SUSE etc., please report to http://www.suse.de/feedback/ ENJOY! Your Novell SUSE Team.