Building MariaDB/MySQL with Buildbot and KVM

Testing and automation. These two are key to ensuring high quality of software releases.

Ever since I worked briefly in the team at MySQL AB that is responsible for creating the binary (and source) packages of MySQL releases, I have had the vision of a fully automated release procedure. Whenever someone pushes a new commit to the release branch revision control tree, the continuous integration test framework should kick in and do all the steps needed for producing release packages:

  • Checkout the new revision.
  • Build a source tarball, and save it.
  • For each platform, build a binary package from the source tarball. The build should be done in a freshly installed machine without any revision control checkouts, previous build trees, or extra installed software, to ensure that no unwanted dependencies or stray references to other files or packages are introduced.
  • For each platform, install the binary packages, this time on freshly installed machines with also no build tools (compilers, development packages, etc.) installed, to check that they install correctly without unexpected dependencies. Run tests of the installed server, including starting the server and running basic queries and test suites.
  • Upgrade testing, installing the new packages on machines prepared with earlier installations, and testing that the upgrade procedure works and preserves old data.

To do this efficiently, clearly the use of virtual machines is needed. This weekend I played with KVM and Buildbot, and managed to set up a proof-of-concept of this that I am really pleased with.

KVM

There are lots of options for virtualisation these days, including KVM, Xen, VirtualBox, and Vmware. I use KVM, and I really like it. The integration into the distributions is excellent (sudo apt-get install kvm and you’re up and running). The interface is powerful and flexible, and at the same time really simple to learn and use. Just a couple of commands with man pages, like it should be in a Unix system. Basically, it just works!

I started by installing a basic ubuntu Jaunty server in a virtual machine:

  qemu-img create -f qcow2 vm-jaunty-i386-base.qcow2 8G
  kvm -m 2047 -hda vm-jaunty-i386-base.qcow2 -cdrom ubuntu-9.04-server-i386.iso \
    -boot d -smp 2 -cpu qemu32,-nx -net nic,model=virtio -net user -redir tcp:2222::22

I use the user mode network stack with port forwarding for ssh access. This allows to run kvm without root privileges, avoids any need to manage different MAC addresses, avoids the need for routing or configuring interfaces, etc.

Using the virtio network driver greatly improved throughput for me when copying things into and out of the virtual machine. The -cpu qemu32,-nx (disable “No eXecute” support) is needed in this case due to some bug or incompatibility, or the installation hangs upon reboot. As usual Google is your friend in cases like this:

Incidentally, I did this using remote X over an SSH connection. This works fine, no need for physical access to the host server. After installation we will run the virtual machine without a graphic console, but it was just easier to use the stock Ubuntu installer than trying to find a way to install over the emulated serial port.

Initial setup

Next I did some basic preparation to make the installed virtual machine work well for command line and script usage. However, the amount of extra packages installed is kept to a minimum to get proper testing against unwanted dependencies.

I Installed ssh server for remote access. I then set it up to use the serial console (as we will be running kvm in -nographic mode). To get a login prompt on serial port 0, create /etc/event.d/ttyS0:

    start on stopped rc2
    start on stopped rc3
    start on stopped rc4
    start on stopped rc5
    stop on runlevel 0
    stop on runlevel 1
    stop on runlevel 6
    respawn
    exec /sbin/getty 115200 ttyS0

To get the kernel to output its boot log to the serial port, edit the kernel line in /boot/grub/menu.lst, removing quiet splash and adding console=ttyS0,115200n8 console=tty0. To get Grub to use the serial port, add these lines to /boot/grub/menu.lst:

    serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
    terminal --timeout=3 serial console

Next, we need a user account inside the virtual machine that we can use from the outside with passwordless login and sudo access. Inside the guest, create the account and grant passwordless sudo:

    sudo adduser --disabled-password buildbot
    sudo adduser buildbot sudo
    sudo visudo
    # uncomment `%sudo ALL=NOPASSWD: ALL'

Then, in the host create an SSH public/private key pair without passphrase:

    ssh-keygen -t dsa

Copy the resulting ~/.ssh/id_dsa.pub from the host into ~/.ssh/authorized_keys in the guest.

Now we should be able to test that things work:

    kvm_pid_2222' ; exec kvm -m 2047 -hda /kvm/vms/vm-jaunty-i386-makedist.qcow2 \
        -redir 'tcp:2222::22' -boot c -smp 2 -cpu qemu32,-nx -nographic \
        -net nic,model=virtio -net user
    # We should get a login prompt in the terminal window
    ssh -p 2222 buildbot@127.0.0.1 'sudo id'
    # We should get root access without login or sudo asking for password.

We now have the basis for scripting actions against the virtual machine: We can start up the guest from the command line (and shutdown with kill from the host or sudo shutdown -h now from the guest). And we can run commands inside the guest using ssh -p 2222 buildbot@127.0.0.1. The next step is to create variants of this base virtual installation for the different purposes we need.

qemu-img create -b base_image.qcow2

The qcow2 virtual hard disk image format used by qemu (and kvm) has a very powerful feature, activated with the -b option of qemu-img create:

    qemu-img create -b vm-jaunty-i386-base.qcow2 -f qcow2 vm-jaunty-i386-makedist.qcow2

This creates a new image vm-jaunty-i386-makedist.qcow2, which is initially a clone of the base image vm-jaunty-i386-base.qcow2 that takes up (almost) no extra space. But as we use this new image, changes are added in the new image (copy-on-write), without modifying the original base image. This allows painless mass cloning and modification of virtual machines without having to re-install, and without taking up unnecessary extra disk space and I/O for copying images.

We use this to create a virtual machine that we will use to produce the source tarball from bzr sources. This needs installing bzr and some development packages (compilers etc).

    sudo apt-get install bzr
    sudo apt-get build-dep mysql-5.1-server

Note the very nice build-dep feature of apt-get, it actually installs a ton of packages needed to build the MySQL server (and MariaDB has the save dependencies). I also copied in an existing shared bzr repository; this is not strictly necessary, but saves a very painful initial cloing of the entire MariaDB repository from Launchpad (bzr is just painfully slow on source trees of the size of MariaDB/MySQL):

    scp -rp -P 2222 .bzr buildbot@127.0.0.1:

Another virtual machine image is set up for building the binary packages (this does not need bzr):

    qemu-img create -b vm-jaunty-i386-base.qcow2 -f qcow2 vm-jaunty-i386-build.qcow2

(with a bit more planning, I could have cloned -makedist from -build; now I just repeated the install of mysql-server-5.1 dependencies, but not the bzr install. Also, a refinement wouldbe to setup the -build guest without autotools and bison, to check that build is possible without those installed).

Finally, a third image for testing installation:

    qemu-img create -b vm-jaunty-i386-base.qcow2 -f qcow2 vm-jaunty-i386-install.qcow2

I will be testing a bintar package install, so create the mysql user and group:

     sudo adduser --system --group mysql

With these preparations, we should be ready to put the pieces together:

Buildbot

For MariaDB, we use Buildbot for continuous integration testing. The Pushbuild system I developed at MySQL was never released publicly, and in any case it is better to use a general tool like Buildbot that is widely used and maintained by a large community.

I have been very satisfied with Buildbot. It has its quirks and bugs, but we can fix those over time (and have fixed a number of them already, as well as added extra features we needed). I think Buildbot has all of the right ideas for doing serious continuous integration testing. As I read in some presentation, running the builds and tests is the easy part. The hard part is providing the information and tools needed by developers to fix problems that are found by testing. Fixing these problems is what it is all about, after all, not just producing pretty status reports.

First, I installed a buildbot slave on the host machine:

    sudo apt-get install buildbot
    sudo addgroup buildbot kvm  # To allow buildbot to run kvm
    sudo -u buildbot buildbot create-slave --usepty=0 /var/lib/buildbot/maria-slave \
        hasky.askmonty.org:9989 knielsen-kvm-x86 <password>

Then I set up an account for this in the Buildbot master, and configured the builder.

With the above preparation, configuring the build is just setting up the proper shell commands to be run against the slave, although it is of course a bit more involved than for a normal configure+make. I really like the simplicity of this. Basically, after initial preparation of the KVM images, there is very little setup required on the buildbot slave host, it is all just normal shell commands configured on the master. Of course going forward we can refine some of this and maybe put some of it into generic scripts called from the main config, but for a proof-of-concept I think it is brilliant that one can see exactly which commands are run.

I included the complete config in all detail at the end of this post, but here are the main points.

f_kvm_jaunty_x86.addStep(Compile(
        logfiles={"kernel": "kernel_2222.log"},
        command=["sh", "-c", """
kill -9 "$(cat kvm_pid_2222)"
(exec sh -c "echo \$\$ > 'kvm_pid_2222' ; exec kvm -m 2047 \
    -hda /kvm/vms/vm-jaunty-i386-makedist.qcow2 -redir 'tcp:2222::22' -boot c \
    -smp 2 -cpu qemu32,-nx -nographic -net nic,model=virtio -net user" \
    </dev/null >kernel_2222.log 2>&1) &
sleep 15
while : ; do ssh -o ConnectTimeout=4 -p 2222 buildbot@127.0.0.1 true && break; sleep 2; done
ssh -p 2222 buildbot@127.0.0.1 'mkdir -p buildbot && cd buildbot && \
    rm -Rf build && bzr co "lp:~maria-captains/maria/mariadb-5.1-knielsen" build && \
    cd build && BUILD/compile-dist && make dist && \\
    mv "$(make show-dist-name).tar.gz" ..'
"""]))

The kill command removes any previous left-over kvm process (better safe than sorry). We run kvm in the backgroud, getting the console output through a log file. Note that redirecting the kvm output is necessary, as the buildstep will wait for all processes to close the stdout before considering the buildstep done.

After starting the virtual machine, we wait for boot to have completed by
checking for successful ssh connection in the while loop. Once it
is ready, we send the commands to build the source tarball into the guest
using ssh.

f_kvm_jaunty_x86.addStep(SetProperty(
        property="distname",
        command=["ssh", "-p", "2222", "buildbot@127.0.0.1", "cd buildbot/build && make show-dist-name"],
        ))

This gets the base name of the source tarball into a Buildbot build property, an essential feature of Buildbot for more advanced usage. We will need this name in the following build steps (the name depends on the version of the MariaDB server code).

f_kvm_jaunty_x86.addStep(ShellCommand(
        command=["sh", "-c", WithProperties("""
scp -P 2222 buildbot@127.0.0.1:buildbot/%(distname)s.tar.gz .
ssh -p 2222 buildbot@127.0.0.1 'sudo shutdown -h now'
while : ; do sleep 5; kill -0 "$(cat kvm_pid_2222)" || break; done
rm -f kvm_pid_2222
""")],))

We copy out the generated source tarball (we will need it in the next buildstep, which runs in a different virtual machine). We then shutdown this guest, and wait for it to finish with another while loop. Note the use of WithProperties to interpolate the source tarball name obtained in the previous build step.

f_kvm_jaunty_x86.addStep(Compile(
        command=["sh", "-c", WithProperties("""
qemu-img create -b /kvm/vms/vm-jaunty-i386-build.qcow2 -f qcow2 vm-tmp-2222.qcow2
kill -9 "$(cat kvm_pid_2222)"
(exec sh -c "echo \$\$ > 'kvm_pid_2222' ; exec kvm -m 2047 -hda vm-tmp-2222.qcow2 \
    -redir 'tcp:2222::22' -boot c -smp 2 -cpu qemu32,-nx -nographic -net nic,model=virtio \
    -net user" </dev/null >>kernel_2222.log 2>&1) &
# ...
ssh -p 2222 buildbot@127.0.0.1 'rm -Rf buildbot && mkdir buildbot'
scp -P 2222 %(distname)s.tar.gz buildbot@127.0.0.1:buildbot/
ssh -p 2222 buildbot@127.0.0.1 'cd buildbot && tar zxf %(distname)s.tar.gz && \
    cd %(distname)s && ./configure ...'
# ...
""")],))

Here (and in the following install step), we use qemu-img create -b to create a new, temporary image to work in. This ensures that each build will run in a clean, fresh install, without any risk of contamination from previous builds. (The reason we did not do this for the initial step is that we want to save the bzr revisions pulled from Launchpad so we do not have to keep repeatedly pulling the old ones over for each new build. An alternative would be to keep the permanent shared repository on the host machine and export from that inside the virtual machine).

And that’s it! Full config details below, but it is basically the same, just with different commands run in the different steps. The result is a builder that fully automatically tests build and install on real machines with the correct setup, 100% repeatable between builds.


The results from this can be seen on the MariaDB Buildbot pages. Things are likely to to shuffle around as we extend
and refine this, but for now an example build can be seen here:

This is just a quick proof of concept, but I think all of the essential ingredients are in there. I am hoping that in the not too distant future we will be using something like this regularly to check MariaDB release builds, which should be very good for getting ensuring both the quality and efficiency of future MariaDB releases!

Full config

f_kvm_jaunty_x86= factory.BuildFactory()
f_kvm_jaunty_x86.addStep(Compile(
        description=["making", "dist"],
        descriptionDone=["make", "dist"],
        logfiles={"kernel": "kernel_2222.log"},
        command=["sh", "-c", """
kill -9 "$(cat kvm_pid_2222)"
(exec sh -c "echo \$\$ > 'kvm_pid_2222' ; exec kvm -m 2047 \
    -hda /kvm/vms/vm-jaunty-i386-makedist.qcow2 -redir 'tcp:2222::22' -boot c -smp 2 \
    -cpu qemu32,-nx -nographic -net nic,model=virtio -net user" \
    </dev/null >kernel_2222.log 2>&1) &
sleep 15
while : ; do ssh -o ConnectTimeout=4 -p 2222 buildbot@127.0.0.1 true && break; sleep 2; done
ssh -p 2222 buildbot@127.0.0.1 'mkdir -p buildbot && cd buildbot && \
    rm -Rf build && bzr co "lp:~maria-captains/maria/mariadb-5.1-knielsen" build && \
    cd build && BUILD/compile-dist && make dist && \
    mv "$(make show-dist-name).tar.gz" ..'
"""
                 ],
        ))
f_kvm_jaunty_x86.addStep(SetProperty(
        property="distname",
        command=["ssh", "-p", "2222", "buildbot@127.0.0.1",
                 "cd buildbot/build && make show-dist-name"],
        ))
f_kvm_jaunty_x86.addStep(ShellCommand(
        description=["copying", "tarball"],
        descriptionDone=["copying", "tarball"],
        logfiles={"kernel": "kernel_2222.log"},
        command=["sh", "-c", WithProperties("""
scp -P 2222 buildbot@127.0.0.1:buildbot/%(distname)s.tar.gz .
ssh -p 2222 buildbot@127.0.0.1 'sudo shutdown -h now'
while : ; do sleep 5; kill -0 "$(cat kvm_pid_2222)" || break; done
rm -f kvm_pid_2222
""")],
        ))
f_kvm_jaunty_x86.addStep(Compile(
        description=["making", "bintar"],
        descriptionDone=["make", "bintar"],
        logfiles={"kernel": "kernel_2222.log"},
        command=["sh", "-c", WithProperties("""
qemu-img create -b /kvm/vms/vm-jaunty-i386-build.qcow2 -f qcow2 vm-tmp-2222.qcow2
kill -9 "$(cat kvm_pid_2222)"
(exec sh -c "echo \$\$ > 'kvm_pid_2222' ; exec kvm -m 2047 -hda vm-tmp-2222.qcow2 \
    -redir 'tcp:2222::22' -boot c -smp 2 -cpu qemu32,-nx -nographic -net nic,model=virtio \
    -net user" </dev/null >>kernel_2222.log 2>&1) &
sleep 15
while : ; do ssh -o ConnectTimeout=4 -p 2222 buildbot@127.0.0.1 true && break; sleep 2; done
ssh -p 2222 buildbot@127.0.0.1 'rm -Rf buildbot && mkdir buildbot'
scp -P 2222 %(distname)s.tar.gz buildbot@127.0.0.1:buildbot/
ssh -p 2222 buildbot@127.0.0.1 'cd buildbot && tar zxf %(distname)s.tar.gz && \
    cd %(distname)s && CC="gcc -static-libgcc" CXX="gcc -static-libgcc" \
    CFLAGS="-O2 -fno-omit-frame-pointer -g" CXXFLAGS="-O2 -fno-omit-frame-pointer -g" \
    ./configure --prefix=/usr/local/mysql --exec-prefix=/usr/local/mysql \
    --libexecdir=/usr/local/mysql/bin --localstatedir=/usr/local/mysql/data \
    --with-server-suffix=1 --with-comment="(MariaDB - http://askmonty.org/)" \
    --with-system-type=linux-gnu --enable-shared --enable-static --enable-thread-safe-client \
    --enable-local-infile --with-big-tables --with-libwrap --with-ssl --without-docs \
    --with-readline --with-extra-charsets=all --with-embedded-server --with-libevent \
    --with-partition --with-zlib-dir=bundled --with-plugins=max-no-ndb && make -j3 && \
     sudo rm -Rf /usr/local/mysql && sudo make install && \
     sudo mv /usr/local/mysql /usr/local/%(distname)s-Linux-x386 && \
    tar zcf ../%(distname)s-Linux-x386.tar.gz -C /usr/local %(distname)s-Linux-x386/'
scp -P 2222 buildbot@127.0.0.1:buildbot/%(distname)s-Linux-x386.tar.gz .
ssh -p 2222 buildbot@127.0.0.1 'sudo shutdown -h now'
while : ; do sleep 5; kill -0 "$(cat kvm_pid_2222)" || break; done
rm -f kvm_pid_2222
""")],
        ))
f_kvm_jaunty_x86.addStep(Test(
        description=["testing", "bintar"],
        descriptionDone=["test", "bintar"],
        logfiles={"kernel": "kernel_2222.log"},
        command=["sh", "-c", WithProperties("""
qemu-img create -b /kvm/vms/vm-jaunty-i386-install.qcow2 -f qcow2 vm-tmp-2222.qcow2
kill -9 "$(cat kvm_pid_2222)"
(exec sh -c "echo \$\$ > 'kvm_pid_2222' ; exec kvm -m 2047 -hda vm-tmp-2222.qcow2 \
    -redir 'tcp:2222::22' -boot c -smp 2 -cpu qemu32,-nx -nographic -net nic,model=virtio \
    -net user" </dev/null >>kernel_2222.log 2>&1) &
sleep 15
while : ; do ssh -o ConnectTimeout=4 -p 2222 buildbot@127.0.0.1 true && break; sleep 2; done
ssh -p 2222 buildbot@127.0.0.1 'rm -Rf buildbot && mkdir buildbot'
scp -P 2222 %(distname)s-Linux-x386.tar.gz buildbot@127.0.0.1:buildbot/
ssh -p 2222 buildbot@127.0.0.1 'cd buildbot && \
    sudo rm -Rf /usr/local/mysql /usr/local/%(distname)s-Linux-x386 && \
    sudo tar zxf %(distname)s-Linux-x386.tar.gz -C /usr/local/ && \
    sudo ln -s %(distname)s-Linux-x386 /usr/local/mysql && cd /usr/local/mysql && \
    sudo sudo chown -R mysql . && sudo chgrp -R mysql . && \
    sudo bin/mysql_install_db --user=mysql && sudo chown -R root . && \
    sudo chown -R mysql data mysql-test && \
    cd mysql-test && sudo su -s /bin/sh -c "perl mysql-test-run.pl alias" mysql'
ssh -p 2222 buildbot@127.0.0.1 'sudo shutdown -h now'
while : ; do sleep 5; kill -0 "$(cat kvm_pid_2222)" || break; done
rm -f kvm_pid_2222
""")],
        ))

bld_kvm_jaunty_x86 = {'name': 'kvm-jaunty-x86',
                      'slavename': 'knielsen-kvm-x86',
                      'builddir': 'kvm-jaunty-x86',
                      'factory': f_kvm_jaunty_x86,
                      }

c['builders'].append(bld_kvm_jaunty_x86)

Leave a comment

Your email address will not be published. Required fields are marked *