Continuous Integration and Testing for Simulation Codes

Back to Essay List

Modified: 23 Jul 2020

BibTeX Entry

RIS Citation

Print

This week I added all of my simulation projects to the Travis CI continuous integration system. Travis is a hosted service, running a Ruby-based project which will check out your code from Github, run it on a clean virtual machine preloaded with common build tools and pieces of infrastructure (e.g., databases, libraries), and perform your unit or functional tests.

Travis reports by email and in their web interface, of course, but more importantly, each project has a “badge” or icon which can be included in the project’s README.md on Github, and which shows whether the build is broken, or passing all tests. This is an improvement to simply having unit or functional tests in one’s project, because the public can understand whether a project is working without downloading and building it.

Here, for example, is the status of my current projects:

• CTPy:
• axelrod-ct:
• ctmixtures:

and several of their common code dependencies:

• pytransmission:
• mmadsenr:

Improving Ease of Installation

I also found that Travis drastically improved the installability (and thus reproducibility) of my projects, especially across platforms. All of the projects listed here are based on Python 2.7, but they rely on a variety of dependencies, some of which link to common (but very large) C++ libraries and others linked Python to my own C code via the SWIG code generator. In one’s own work, none of this infrastructure ever gets built all at once – instead, it evolves in one’s development environment as capabilities are added. So it is easy to forget what a complete build process looks like.

Travis forced me to iterate on build and test scripts until every dependency was accounted for, and could be added (via apt) or built from scratch, on a clean Ubuntu 12.04 LTS system. So I am sure that on a stock Ubuntu system, anybody can install any of the projects listed here (although I still need to add a manual install script to each project to replicate what Travis does).

I do hope that the Travis folks add options for more build environments. It was very hard, for example, to get a modern version of SciPy working on Ubuntu 12.04 LTS. The stock version is too old to have the special mathematical functions library, so you have to build it from scratch. Which takes forever. The projects above take from 6 seconds to 3 minutes to build and test. Except for axelrod-ct, which takes 24 minutes. 23 of which are simply building scipy – every time I make a change. That inefficiency could be solved by having a more recent binary package for 12.04, or test environments based upon something more recent in the Ubuntu lineup (say, 13.10).

Improving the Travis Experience Itself

Iterating on Travis was time consuming, especially for the first project (to be expected), and for a particularly tricky build which used SWIG to generate glue code between python and a C library.

I found a good solution by simulating the Travis build environment with an EC2 instance running Ubuntu 12.04 LTS, with build-essentials and a host of other basics installed (a subset of packages described in the Travis docs themselves, including mongodb and git). The resulting AMI is useful for running your code through various build processes, to make sure that you’re not missing something in the Travis environment that (say) your desktop environment has by default, etc.

I’ve made the AMI public on EC2: ami-1edd2176, image name is Travis Tests - Everything Installed. You should be able to boot this image (and choose an instance size as appropriate, I used a relatively large instance for compiling scipy, otherwise it would have taken hours on a free t1.micro instance. But the free tier should be adequate for many, many projects especially pure R or python projects.)¹

This is a decent solution, but it would be terrific if the Travis CI team released their own official AMIs and perhaps a VMware image, so that folks have an official, clean environment to work against, to prevent slow iteration against the full system. This is particularly important when your code or a dependency might have a very long build – a good example is a dependency on a particular version of the Boost C++ libraries. If a binary version isn’t available for Ubuntu 12.04 LTS, it’s a looong compile on the Travis environment. Heck, it’s a long build even when you throw all of the cores on a quad i7 at it…

---

1. Although, looking at my bill, I still spent less than a dollar on computing time to get the long scipy build completed. So, maybe splurge beyond t1.micro and tear less of your hair out.↩