Fixing a MariaDB package bug|
One of the things that I am really happy about in MariaDB is that we have our
releases available as apt (and yum for Centos) repositories. This is
largely thanks to being able to build this on
the OurDelta package build infrastructure
(which again builds on things like the Debian packaging scripts for MySQL).
Something like the Debian
apt-get package system (which is also
used by Ubuntu) is one of the major innovations in the Free Software world in
my opinion. Debian has spent many years refining this system to where it is
today. Want to run the mysql client, but it isn't installed? Just try to run
it on your local Ubuntu host:
The program 'mysql' can be found in the following packages:
Try: sudo apt-get install <selected package>
-bash: mysql: command not found
Installing software does not get much easier than that!
Now, MariaDB is not in the distributions yet. However, it is easy to add
external repositories like OurDelta into your system, after which packages
from the external repositories are available fully integrated with the package
wget -O- http://ourdelta.org/deb/ourdelta.gpg | sudo apt-key add -
sudo wget http://ourdelta.org/deb/sources/karmic-mariadb-ourdelta.list \
(there are also GUI ways to do this of course, for those who prefer that).
After this is done, installation is just a
sudo apt-get install mariadb-server-5.1
away, security updates will appear automatically for MariaDB just like any
other package, etc. It will even upgrade an existing MySQL 5.0 installation
automatically (but do take a backup first).
In order to make all this work, there is a lot of work going on behind the
scenes in the scripts that make up the .deb packaging. I think most people
underestimate the amount of work and clever engineering that goes into making
a well-working .deb package. It is easy to laugh at how behind the latest
stable Debian release is on software versions (and I am happy to do this on
occasion as well). But Debian is still unique in the sheer amount of software
it contains and the level of integration of each package in the whole
system. And it is this work through the last more than 15 years that allows
something like Ubuntu to exists, with an upgrade system that allows it to do
6-month release cycles to provide up-to-date software to its users.
As an example of the kind of details that needs to be dealt with, I wanted to
explain a tricky packaging bug I fixed recently.
The bug was with installing MariaDB 5.1.39 on top of an existing MySQL 5.0
installation in Debian and Ubuntu. This is supposed to automatically run the
mysql_upgrade program to upgrade all tables from the 5.0 format to the 5.1
format. The symptoms were that the upgrade was not performed correctly, for
example the system tables in the
mysql database were missing some
of the columns added in the 5.1 version.
What made this tricky was that the bug was quite elusive. It did not happen
always, and some platforms were ok while others (eg. Ubuntu Hardy amd64) seemed to
have it more or less repeatedly. Even worse, even when it did occur, it went
away by itself (this is because the .deb MariaDB and MySQL packages actually
check for the need to upgrade the database on every server start, and the
upgrade procedure always worked when the server was restarted after
After poking for this for some hours, I managed to get it reproducibly on a
KVM virtual machine containing Ubuntu Hardy. I then traced the problem into
the upgrade procedure, which happens in
/etc/mysql/msyql_upgrade script which is run
/etc/init.d/mysql start. Comparing the log from this (in
syslog) with the corresponding log from a successful installation showed that
the upgrade procedure seemed to be aborted half-way through, but with
absolutely no output (like an error message) to indicate any reason for this.
Now the fact that running the upgrade procedure (or just server start)
manually did not exhibit any problems made it quite puzzling to
figure out what was going on. Looking through the source code
mysqlcheck (which is called
mysql_upgrade) did not reveal anything that would indicate a
problem like this. One step would be to start instrumenting the code with
printouts, but this would require building a full set of packages and
installing them inside a virtual machine for every iteration, which would have
been quite time-consuming.
A better approach turned out to be to install the package running
strace -f. This generates a log (45 MByte of it!) of all
system calls made by the installation process, including child processes
mysql_upgrade invocation. Digging through this for a
while, I finally discovered that the upgrade process was being terminated
because it received a SIGHUP (hangup) signal!
Now why would it be killed with SIGHUP? Turns out it is due to this snippet
/etc/mysql/debian-start bash script:
) <&2 &
The upgrade procedure is run in the background (as it may take a long time,
and since this is run on every server restart, it could also happen eg. during
host boot, which we do not want to block for long periods of time). But there
is no protection against the controlling terminal going away! My guess is
apt-get in some cases will allocate a pseudo-tty to deal
with package configuration input, and this is closed when installation is
done, causing the background upgrade procedure to be killed with SIGHUP.
Now finally the problem is understood, and the fix is a one-liner: just add
this before starting the background job:
trap "" SIGHUP
[Incidentally, if you installed MariaDB 5.1.39 .debs and experienced this
problem, there is an easy workaround for this bug: just restart the MariaDB
server once after installation (
sudo /etc/init.d/mysql restart).
This will make the upgrade procedure go through if it did not already. This
fix will be included in the upcoming MariaDB 5.1.41 release.]
Tags: debian, debugging, freesoftware, mariadb, mysql, programming, ubuntu