In the Autumn, I wrote about some experiments I did using KVM and virtual machines to build and test MariaDB binary packages on a number of different platforms. In the period since then I added some polish and refinements, and the system is now running well for some time. We build and test packages for Debian (4 and 5), Ubuntu (8.04 to 10.04), Centos 5, and generic Linux; amd64 and i386 architectures.
To better control the startup and shutdown of the virtual machines, I created
a small wrapper script around KVM called
runvm. This wrapper
encapsulates the steps needed to boot up a virtual machine, run a series of
commands inside it, and shut it down gracefully afterwards. Some special care
is taken in the script to ensure that the virtual machine is always shut down
after use (gracefully if possible), even in case of various failures or the
loss of the parent process or controlling TTY. And if a conflicting virtual
machine somehow manages to escape shutdown,
runvm automatically attempts to
terminate it before starting a new one. This extra robustness is important for
fully automated testing as
in our Buildbot setup, to
ensure that the system can run unattended for longer periods of time.
So for example, here is how to run a build inside a virtual machine using
runvm --port=2222 ubuntu-hardy-i386.qcow2 \ "= scp -P 2222 mariadb-5.1.41-rc.tar.gz localhost:" \ "tar zxf mariadb-5.1.41-rc.tar.gz" \ "cd mariadb-5.1.41-rc && ./configure" \ "cd mariadb-5.1.41-rc && make"
ubuntu-hardy-amd64.qcow2 is a KVM image already installed
with compilers and set up for password-less ssh access (using public key
authentication). Port 2222 on the host side is forwarded to the ssh service
(port 22) on the guest side (so by specifying different
options it is easy to run multiple
runvm invocations in parallel;
in our Buildbot setup we run 3 builds in parallel this way).
Note the use of the
scp command, prefixed with an equals sign
=". Commands prefixed in this way are run on the host side
rather than the guest side; this is a convenient way to copy data in or
results out of the virtual machine while the
runvm session is
runvm in this way we are able to easily and flexibly manage
a large number of virtual machines for automated builds with very little
overhead and complexity. In fact we have around 70 distinct virtual machines!
The only resource they take is a little disk space (around 37 GByte). And the
virtual machines images are also simple to set up, requiring only a minimal
install; no need to set up networking bridges or IP addresses, or to install a
Buildbot client. All the complex logic runs on the host system, which only
needs to be installed once.
By keeping the virtual images simple, we also achieve that builds and tests run in a minimal environment, which is useful to detect any missing dependencies or other problems that do not show themselves on normal developer machines with a full desktop install (we even do install testing on a separate virtual machine from the one used to build, with compilers etc. not installed on the one used to test installation).
A further refinement of this is to create a new temporary virtual machine image before each step as a copy of a reference image, run the build, and throw away the temporary image after the build. This avoids any possibility of a previous build influencing a following build in any way (and thus also simplifies the build setup, as we can install stuff freely without any need to do cleanup). It also avoids having to fix a broken image, like needing to manually run fsck after a crash or similar. We use this technique for most of our binary package builds in Buildbot.
To use this copy-and-discard technique with
--base-image option is useful:
runvm --port=2222 --base-image=ubuntu-hardy-i386.qcow2 tmp.qcow2 \ "= scp -P 2222 mariadb-5.1.41-rc.tar.gz localhost:" \ "tar zxf mariadb-5.1.41-rc.tar.gz" \ "cd mariadb-5.1.41-rc && ./configure" \ "cd mariadb-5.1.41-rc && make"
This will run the build in a temporary copy
tmp.qcow2 of the
ubuntu-hardy-i386.qcow2, without modifying the
reference image in any way. This uses the copy-on-write feature of
qcow2 image format (see
qemu-img(1)), so it even
takes only very little time (fraction of a second) and minimal space (only
changed blocks are written to the new image).
This is basically how the package testing in our Buildbot setup is done. There
are some further details of course, like more options for the build commmands
and extra care to get logfiles out to debug problems; the full details are
configuration file. But the basic principle is just a number
runvm commands like the example above.
runvm tool is available under GPL on Lauchpad in the project
Tools for MariaDB. In the
bzr repository it is found
someone finds it useful or has suggestions for improvements, please drop us a
Here is the output of
Usage: /home/knielsen/devel/maria/my/mariadb-tools/buildbot/runvm <options> image.qcow2 [command ...] Boot the given KVM virtual machine image and wait for it to come up. Run the list of commands one at a time, aborting on receiving an error. When all commands are run (or one of them failed), shutdown the virtual machine and exit. Commands are by default run inside the virtual machine using ssh(1). By prefixing a command with an equals sign '=', it will instead be run on the host system (for example to copy files into or out of the virtual machine using scp(1)). Some care is taken to ensure that the virtual machine is shutdown gracefully and not left running even in case the controlling tty is closed or the parent process killed. If a previous virtual machine is already running on a conflicting port, an attempt is made to shut it down first. For this purpose, a PID file is created in $HOME/.runvm/ Available options: -p, --port=N Forward this port on the host side to the ssh port (port 22) on the guest side. Must be different for each runvm instance running in parallel to avoid conflicts. The default is 2222. To copy files in/out of the guest use a command prefixed with '=' calling scp(1) with the -P option using the port specified here, like this: runvm img.qcow2 "=scp -P 2222 file.txt localhost:" -u, --user=USER Name of the account to ssh into in the guest. Defaults to the name of the user invoking runvm on the host. -m, --memory=N Amount of memory (in megabytes) to allocate to the guest. Defaults to 2047. --smp=N Number of CPU cores to allocate to the guest. Defaults to 2. -c, --cpu=NAME Type of CPU to emulate for KVM, see qemu(1) for details. For example: --cpu=qemu64 For 64-bit amd64 emulation --cpu=qemu32 For 32-bit x86 emulation --cpu=qemu32,-nx 32-bit and disable "no-execute" The default is qemu32,-nx --netdev=NAME Network device to emulate. The 'virtio' device has good performance but may not have driver support in all operating systems, if so another can be specified. The default is virtio. --kvm=OPT Pass additional option OPT to kvm. Specify multiple times to pass more than one option. For example runvm --kvm=-cdrom --kvm=mycd.iso img.qcow2 ... --initial-sleep=SECS Wait this many seconds before starting to poll the guest ssh port for it to be up. Default 15. --startup-timeout=SECS Wait at most this many seconds for the guest OS to respond to ssh. If this time is exceeded assume it has failed to boot correctly. Default 300. --shutdown-timeout=SECS Wait at most this many seconds for the guest OS to shutdown gracefully after sending a shutdown command. If this time is exceeded, assume the guest has failed to shutdown gracefully and kill it forcibly. Default 120. --kvm-retries=N If the guest fails to come up, retry the boot this many times before giving up. This helps if the virtual machine sometimes crashes during boot. Default 3. -l, --logfile=FILE File to redirect the output from kvm into. This includes any (error) messages from kvm, and also includes anything the guest writes to the kvm emulated serial port (it can be useful to set the guest to send boot loader and kernel messages to the serial console and log them with this option). Default is to not log this output anywhere. -b, --base-image=IMG Instead of booting an existing image, create a new copy-on-write image based on IMG. This uses the -b option of qemu-img(1). IMG is not modified in any way. This way, the booted image can be discarded after use, so that each use of IMG is using the same reference image with no risk of "polution" between different invocations. Note that this DELETES any existing image of the same name as the one specified on the command line to boot! It will be replaced with the image created as a copy of IMG, with any modifications done during the runvm session.