In the Autumn, I wrote about some experiments I did using KVM and virtual machines to build and test MariaDB binary packages on a number of different platforms. In the period since then I added some polish and refinements, and the system is now running well for some time. We build and test packages for Debian (4 and 5), Ubuntu (8.04 to 10.04), Centos 5, and generic Linux; amd64 and i386 architectures.
To better control the startup and shutdown of the virtual machines, I created a small wrapper script around KVM called runvm. This wrapper encapsulates the steps needed to boot up a virtual machine, run a series of commands inside it, and shut it down gracefully afterwards. Some special care is taken in the script to ensure that the virtual machine is always shut down after use (gracefully if possible), even in case of various failures or the loss of the parent process or controlling TTY. And if a conflicting virtual machine somehow manages to escape shutdown, runvm automatically attempts to terminate it before starting a new one. This extra robustness is important for fully automated testing as in our Buildbot setup, to ensure that the system can run unattended for longer periods of time.
So for example, here is how to run a build inside a virtual machine using runvm:
runvm --port=2222 ubuntu-hardy-i386.qcow2 \
"= scp -P 2222 mariadb-5.1.41-rc.tar.gz localhost:" \
"tar zxf mariadb-5.1.41-rc.tar.gz" \
"cd mariadb-5.1.41-rc && ./configure" \
"cd mariadb-5.1.41-rc && make"
Here, ubuntu-hardy-amd64.qcow2 is a KVM image already installed with compilers and set up for password-less ssh access (using public key authentication). Port 2222 on the host side is forwarded to the ssh service (port 22) on the guest side (so by specifying different --port options it is easy to run multiple runvm invocations in parallel; in our Buildbot setup we run 3 builds in parallel this way).
Note the use of the scp command, prefixed with an equals sign “=“. Commands prefixed in this way are run on the host side rather than the guest side; this is a convenient way to copy data in or results out of the virtual machine while the runvm session is running.
Using runvm in this way we are able to easily and flexibly manage a large number of virtual machines for automated builds with very little overhead and complexity. In fact we have around 70 distinct virtual machines! The only resource they take is a little disk space (around 37 GByte). And the virtual machines images are also simple to set up, requiring only a minimal install; no need to set up networking bridges or IP addresses, or to install a Buildbot client. All the complex logic runs on the host system, which only needs to be installed once.
By keeping the virtual images simple, we also achieve that builds and tests run in a minimal environment, which is useful to detect any missing dependencies or other problems that do not show themselves on normal developer machines with a full desktop install (we even do install testing on a separate virtual machine from the one used to build, with compilers etc. not installed on the one used to test installation).
A further refinement of this is to create a new temporary virtual machine image before each step as a copy of a reference image, run the build, and throw away the temporary image after the build. This avoids any possibility of a previous build influencing a following build in any way (and thus also simplifies the build setup, as we can install stuff freely without any need to do cleanup). It also avoids having to fix a broken image, like needing to manually run fsck after a crash or similar. We use this technique for most of our binary package builds in Buildbot.
To use this copy-and-discard technique with runvm, the --base-image option is useful:
runvm --port=2222 --base-image=ubuntu-hardy-i386.qcow2 tmp.qcow2 \
"= scp -P 2222 mariadb-5.1.41-rc.tar.gz localhost:" \
"tar zxf mariadb-5.1.41-rc.tar.gz" \
"cd mariadb-5.1.41-rc && ./configure" \
"cd mariadb-5.1.41-rc && make"
This will run the build in a temporary copy tmp.qcow2 of the reference imageubuntu-hardy-i386.qcow2, without modifying the reference image in any way. This uses the copy-on-write feature of the qcow2 image format (see qemu-img(1)), so it even takes only very little time (fraction of a second) and minimal space (only changed blocks are written to the new image).
This is basically how the package testing in our Buildbot setup is done. There are some further details of course, like more options for the build commmands and extra care to get logfiles out to debug problems; the full details are available in our Buildbot configuration file. But the basic principle is just a number of runvm commands like the example above.
The runvm tool is available under GPL on Lauchpad in the project Tools for MariaDB. In the bzr repository it is found as buildbot/runvm. If someone finds it useful or has suggestions for improvements, please drop us a line on the maria-developers@lists.launchpad.net mailing list.
Here is the output of runvm --help:
Usage: /home/knielsen/devel/maria/my/mariadb-tools/buildbot/runvm <options> image.qcow2 [command ...]
Boot the given KVM virtual machine image and wait for it to come up.
Run the list of commands one at a time, aborting on receiving an error.
When all commands are run (or one of them failed), shutdown the virtual
machine and exit.
Commands are by default run inside the virtual machine using ssh(1). By
prefixing a command with an equals sign '=', it will instead be run on the
host system (for example to copy files into or out of the virtual machine
using scp(1)).
Some care is taken to ensure that the virtual machine is shutdown
gracefully and not left running even in case the controlling tty is
closed or the parent process killed. If a previous virtual machine is
already running on a conflicting port, an attempt is made to shut it
down first. For this purpose, a PID file is created in $HOME/.runvm/
Available options:
-p, --port=N Forward this port on the host side to the ssh port (port
22) on the guest side. Must be different for each runvm
instance running in parallel to avoid conflicts. The
default is 2222.
To copy files in/out of the guest use a command prefixed
with '=' calling scp(1) with the -P option using the port
specified here, like this:
runvm img.qcow2 "=scp -P 2222 file.txt localhost:"
-u, --user=USER Name of the account to ssh into in the guest. Defaults to
the name of the user invoking runvm on the host.
-m, --memory=N Amount of memory (in megabytes) to allocate to the guest.
Defaults to 2047.
--smp=N Number of CPU cores to allocate to the guest.
Defaults to 2.
-c, --cpu=NAME Type of CPU to emulate for KVM, see qemu(1) for details.
For example:
--cpu=qemu64 For 64-bit amd64 emulation
--cpu=qemu32 For 32-bit x86 emulation
--cpu=qemu32,-nx 32-bit and disable "no-execute"
The default is qemu32,-nx
--netdev=NAME Network device to emulate. The 'virtio' device has good
performance but may not have driver support in all
operating systems, if so another can be specified.
The default is virtio.
--kvm=OPT Pass additional option OPT to kvm. Specify multiple times
to pass more than one option. For example
runvm --kvm=-cdrom --kvm=mycd.iso img.qcow2 ...
--initial-sleep=SECS
Wait this many seconds before starting to poll the guest
ssh port for it to be up. Default 15.
--startup-timeout=SECS
Wait at most this many seconds for the guest OS to respond
to ssh. If this time is exceeded assume it has failed to
boot correctly. Default 300.
--shutdown-timeout=SECS
Wait at most this many seconds for the guest OS to
shutdown gracefully after sending a shutdown command. If
this time is exceeded, assume the guest has failed to
shutdown gracefully and kill it forcibly. Default 120.
--kvm-retries=N If the guest fails to come up, retry the boot this many
times before giving up. This helps if the virtual machine
sometimes crashes during boot. Default 3.
-l, --logfile=FILE File to redirect the output from kvm into. This includes
any (error) messages from kvm, and also includes anything
the guest writes to the kvm emulated serial port (it can
be useful to set the guest to send boot loader and kernel
messages to the serial console and log them with this
option). Default is to not log this output anywhere.
-b, --base-image=IMG
Instead of booting an existing image, create a new
copy-on-write image based on IMG. This uses the -b option
of qemu-img(1). IMG is not modified in any way. This way,
the booted image can be discarded after use, so that each
use of IMG is using the same reference image with no risk
of "polution" between different invocations.
Note that this DELETES any existing image of the same
name as the one specified on the command line to boot! It
will be replaced with the image created as a copy of IMG,
with any modifications done during the runvm session.