Fixing a MariaDB package bug

One of the things that I am really happy about in MariaDB is that we have our releases available as apt (and yum for Centos) repositories. This is largely thanks to being able to build this on the OurDelta package build infrastructure (which again builds on things like the Debian packaging scripts for MySQL).

Something like the Debian apt-get package system (which is also used by Ubuntu) is one of the major innovations in the Free Software world in my opinion. Debian has spent many years refining this system to where it is today. Want to run the mysql client, but it isn’t installed? Just try to run it on your local Ubuntu host:

    $ mysql
    The program 'mysql' can be found in the following packages:
     * mysql-client-5.0
     * mysql-client-5.1
    Try: sudo apt-get install <selected package>
    -bash: mysql: command not found

Installing software does not get much easier than that!

Now, MariaDB is not in the distributions yet. However, it is easy to add external repositories like OurDelta into your system, after which packages from the external repositories are available fully integrated with the package system:

    wget -O- http://ourdelta.org/deb/ourdelta.gpg | sudo apt-key add -
    sudo wget http://ourdelta.org/deb/sources/karmic-mariadb-ourdelta.list \
        -O /etc/apt/sources.list.d/ourdelta.list

(there are also GUI ways to do this of course, for those who prefer that). After this is done, installation is just a sudo apt-get install mariadb-server-5.1 away, security updates will appear automatically for MariaDB just like any other package, etc. It will even upgrade an existing MySQL 5.0 installation automatically (but do take a backup first).

In order to make all this work, there is a lot of work going on behind the scenes in the scripts that make up the .deb packaging. I think most people underestimate the amount of work and clever engineering that goes into making a well-working .deb package. It is easy to laugh at how behind the latest stable Debian release is on software versions (and I am happy to do this on occasion as well). But Debian is still unique in the sheer amount of software it contains and the level of integration of each package in the whole system. And it is this work through the last more than 15 years that allows something like Ubuntu to exists, with an upgrade system that allows it to do 6-month release cycles to provide up-to-date software to its users.

As an example of the kind of details that needs to be dealt with, I wanted to explain a tricky packaging bug I fixed recently.

The bug was with installing MariaDB 5.1.39 on top of an existing MySQL 5.0 installation in Debian and Ubuntu. This is supposed to automatically run the mysql_upgrade program to upgrade all tables from the 5.0 format to the 5.1 format. The symptoms were that the upgrade was not performed correctly, for example the system tables in the mysql database were missing some of the columns added in the 5.1 version.

What made this tricky was that the bug was quite elusive. It did not happen always, and some platforms were ok while others (eg. Ubuntu Hardy amd64) seemed to have it more or less repeatedly. Even worse, even when it did occur, it went away by itself (this is because the .deb MariaDB and MySQL packages actually check for the need to upgrade the database on every server start, and the upgrade procedure always worked when the server was restarted after installation).

After poking for this for some hours, I managed to get it reproducibly on a KVM virtual machine containing Ubuntu Hardy. I then traced the problem into the upgrade procedure, which happens in the /etc/mysql/msyql_upgrade script which is run from /etc/init.d/mysql start. Comparing the log from this (in syslog) with the corresponding log from a successful installation showed that the upgrade procedure seemed to be aborted half-way through, but with absolutely no output (like an error message) to indicate any reason for this.

Now the fact that running the upgrade procedure (or just server start) manually did not exhibit any problems made it quite puzzling to figure out what was going on. Looking through the source code of mysql_upgrade and mysqlcheck (which is called from mysql_upgrade) did not reveal anything that would indicate a problem like this. One step would be to start instrumenting the code with printouts, but this would require building a full set of packages and installing them inside a virtual machine for every iteration, which would have been quite time-consuming.

A better approach turned out to be to install the package running under strace -f. This generates a log (45 MByte of it!) of all system calls made by the installation process, including child processes like the mysql_upgrade invocation. Digging through this for a while, I finally discovered that the upgrade process was being terminated because it received a SIGHUP (hangup) signal!

Now why would it be killed with SIGHUP? Turns out it is due to this snippet from the /etc/mysql/debian-start bash script:

    (
      upgrade_system_tables_if_necessary;
      check_root_accounts;
      check_for_crashed_tables;
    ) <&2 &

The upgrade procedure is run in the background (as it may take a long time, and since this is run on every server restart, it could also happen eg. during host boot, which we do not want to block for long periods of time). But there is no protection against the controlling terminal going away! My guess is that apt-get in some cases will allocate a pseudo-tty to deal with package configuration input, and this is closed when installation is done, causing the background upgrade procedure to be killed with SIGHUP.

Now finally the problem is understood, and the fix is a one-liner: just add this before starting the background job:

    trap "" SIGHUP

Problem solved!

[Incidentally, if you installed MariaDB 5.1.39 .debs and experienced this problem, there is an easy workaround for this bug: just restart the MariaDB server once after installation (sudo /etc/init.d/mysql restart). This will make the upgrade procedure go through if it did not already. This fix will be included in the upcoming MariaDB 5.1.41 release.]

Leave a comment

Your email address will not be published. Required fields are marked *