Building TokuMX and TokuDB for Production

Recently, we’ve seen a few people ask us about building TokuMX from scratch. While it’s best if you just use the binaries you can get from us (they have all the right optimizations, we’ve tested them, and we can interpret coredumps they generate), we recognize there are other reasons you might need to do a custom build.

Since we actually build six distinct products all using the Fractal Tree indexing® library (community and enterprise versions of TokuDB for MySQL, TokuDB for MariaDB, and TokuMX), our build process is pretty complicated, compared to software packages that might, for example, just involve one source repository and link against a few standard libraries. Our TokuMX builds involve four git repositories, three separate build stages, two different build tools, and three different compiler spec requirements, and TokuDB is even worse. We want to make sure that if you have to build from scratch, that you get all the optimizations we want you to have, and that you have the fewest steps in between you and a tarball.

I got pretty excited about Vagrant yesterday as I was building out our test infrastructure a little larger, so today I tried something new.

I created a Vagrant config that has all the dependencies you need to build any of the community products, and simplified build scripts that you can use to build TokuMX or TokuDB. Check it out here: Tokutek/vagrant-tokutek-builder.

To get started, clone the repo and provision a machine:

 $ git clone http://github.com/Tokutek/vagrant-tokutek-builder
 $ cd vagrant-tokutek-builder
 $ vagrant up
 $ vagrant ssh

Now, to build TokuMX 1.0.3, just check out the proper revisions of the source, and run the build script:

 $ (cd ft-index; git checkout tokumx-1.0.3)
 $ (cd mongo; git checkout 1.0.3)
 $ ./build-tokumx.sh

This builds a tarball in the mongo directory that you can copy to another machine and use.

To build TokuDB 7.0.4, just use the make.mysql.bash script in the ft-engine repo.

 $ ~/ft-engine/scripts/make.mysql.bash --cc=gcc-4.7 --cxx=g++-4.7 \
       --mysqlbuild=mysql-5.5.30-tokudb-7.0.4-linux-x86_64

This builds a tarball in mysql or mariadb, depending on your choice. There isn’t anything special about this script in the VM, but the VM is set up with all the right dependencies already installed.

Pull requests are welcome. Happy building, and as always, feedback is welcome!