Testing Across Node.js Versions Using Docker
The Problem: Testing
NAN is a project designed to assist in building native (C++) Node.js add-ons while maintaining compatibility with Node and V8 from Node versions 0.8 onwards. V8 is undergoing major internal changes which is making add-on development very difficult. NAN’s purpose is to abstract that pain. Instead of having to keep your code compatible across Node/V8 versions, NAN does it for you, and this is no simple task. This means that we have to be sure to keep NAN tested and compatible with all of the versions it claims to support. This is not a trivial exercise!
Travis CI can help a little with this. It’s possible to use nvm to test across different versions of Node.js even beyond the officialy supported versions. We’ve tried this with NAN, without a whole lot of success. Ideally, you’d have better choice of Node versions, but Travis has had some difficulty keeping up. Also, historical npm bugs that ship with older versions of Node.js tend to cause a high failure rate due to npm install problems. For this reason, we don’t even publish the Travis badge on the NAN README because it just doesn’t work.
The other problem with Travis is that it’s a CI solution, not a proper testing solution. Even if it worked well, it’s not really that helpful in the development process, as you need rapid feedback that your code is working on your target platforms (this is one reason why I love back-end development more than front-end development!)
The Solution: Docker
Enter Docker and DNT. Docker is a tool that simplifies the use of Linux containers to create lightweight, isolated compute “instances”. Solaris and its variants have had this functionality for years in the form of “zones”, but it’s a relatively new concept for Linux and Docker makes the whole process a lot more friendly. Dockers relative simplicity has meant an amazing amout of activity in the Linux container space in recent months, it’s become a huge ecosystem almost overnight.
DNT: Docker Node Tester
Docker Node Test, or DNT, is a very simple utility that contains two tools for working with Docker and Node.js. One tool helps to setup containers for testing, and the other runs your project’s tests in those containers.
DNT includes a setup-dnt script that sets up the most basic Docker images required to run Node.js applications, and nothing extra. It first creates an image called dev_base that uses the default Docker “ubuntu” image, and adds the build tools required to compile and install Node.js
Next it creates a node_dev image that contains a complete copy of the Node.js source repository. Finally, it creates a series of images that are required for the tests you want to run. For each Node version, it creates an image with Node installed and ready to use.
Setting up a project is a matter of creating a .dntrc file in the root directory of the project. This configuration file sets a NODE_VERSIONS variable with a list of all of the versions of Node you want to test against. This list can include “master” to test the latest code from the Node repository. You also set a TEST_CMD variable with a series of commands required to set up, compile, and execute your tests. The setup-dnt command can be run against a .dntrc file to make sure that the appropriate Docker images are ready. The dnt command can then be used to execute the tests against all of the Node versions you specified.
Since Docker containers are completely isolated, DNT can run tests in parallel as long as the machine has the resources. The default is to use the number of cores on the computer as the concurrency level, but this can be configured if this isn’t appropriate for the kinds of tests you want to run.
It’s also possible to customise the base test image to include other external tools and libraries required by your project, although this is a manual step in the set-up process.
Currently DNT is designed to parse TAP test output by reading the final line as either “ok” or “not ok” to report test status back on the command-line. It is configurable, but you need to supply a command that will transform test output to either an “ok” or “not ok” (sed to the rescue?). The non-standard output of the Mocha TAP reporter is also supported out of the box.
Current Uses
My primary use case is for testing NAN. Being able to test against all the different V8 and Node APIs while coding is super helpful, particularly when tests run so quickly! My NAN .dntrc file tests against master, many of the 0.11 releases since 0.11.4 (0.11.0 to 0.11.3 are explicitly not supported by NAN and 0.11.11 and 0.11.12 are completely broken for native addons), and the last five releases of the 0.10 and 0.8 series. At the moment that’s 18 versions of Node in all, and on my computer the test suite takes approximately 20 seconds to complete across all of these releases. The NAN .dntrc file is shown below.
NODE_VERSIONS="\
master \
v0.11.10 \
v0.11.9 \
v0.11.8 \
v0.11.7 \
v0.11.6 \
v0.11.5 \
v0.11.4 \
v0.10.26 \
v0.10.25 \
v0.10.24 \
v0.10.23 \
v0.10.22 \
v0.8.26 \
v0.8.25 \
v0.8.24 \
v0.8.23 \
v0.8.22 \
"
OUTPUT_PREFIX="nan-"
TEST_CMD="\
cd /dnt/test/ && \
npm install && \
node_modules/.bin/node-gyp --nodedir /usr/src/node/ rebuild && \
node_modules/.bin/tap js/*-test.js; \
"Next, I configured LevelDOWN for DNT. LevelDOWN is a raw C++ binding that exposes LevelDB to Node.js. Its main use is the backend for LevelUP. The needs are much simpler, as the tests only need to do a compile and run a lot of node-tap tests. The LevelDOWN .dntrc is shown in the following code sample.
NODE_VERSIONS="\
master \
v0.11.10 \
v0.11.9 \
v0.10.26 \
v0.10.25 \
v0.8.26 \
"
OUTPUT_PREFIX="leveldown-"
TEST_CMD="\
cd /dnt/ && \
npm install && \
node_modules/.bin/node-gyp --nodedir /usr/src/node/ rebuild && \
node_modules/.bin/tap test/*-test.js; \
"Another native Node add-on that I’ve set up with DNT is my libssh Node.js bindings. This one is a little more complicated because you need to have some non-standard libraries installed before compile. My .dntrc adds some extra apt-get sauce to fetch and install those packages. It means the tests take a little longer but it’s not prohibitive. An alternative would be to configure the node_dev base image to add these packages to all of my versioned images. The node-libssh .dntrc is shown below.
NODE_VERSIONS="master v0.11.10 v0.10.26"
OUTPUT_PREFIX="libssh-"
TEST_CMD="\
apt-get install -y libkrb5-dev libssl-dev && \
cd /dnt/ && \
npm install && \
node_modules/.bin/node-gyp --nodedir /usr/src/node/ rebuild --debug && \
node_modules/.bin/tap test/*-test.js --stderr; \
"LevelUP isn’t a native add-on, but it does use LevelDOWN, which requires compiling. For the DNT config I’m removing node_modules/leveldown/ prior to npm install so it gets rebuilt each time for each new version of Node. The LevelUP .dntrc is shown below:
NODE_VERSIONS="\
master \
v0.11.10 \
v0.11.9 \
v0.10.26 \
v0.10.25 \
v0.8.26 \
"
OUTPUT_PREFIX="levelup-"
TEST_CMD="\
cd /dnt/ && \
rm -rf node_modules/leveldown/ && \
npm install --nodedir=/usr/src/node && \
node_modules/.bin/tap test/*-test.js --stderr; \
#"Future Work
It’s not hard to imagine this forming the basis of a local CI system as well as a general testing tool. The speed even makes it tempting to run the tests on every git commit, or perhaps even every save. Already, the New Relic Node.js agent team is using an internal fork of DNT for the very complicated job of testing their agent against many versions of Node.js combined with tests for various common server frameworks.
I’m always keen to have contributors, if you have particular needs and the skills to implement new functionality then I’d love to hear from you. I’m generally very open with my open source projects and happy to add contributors who add something valuable.
See the DNT GitHub repo for installation and detailed usage instructions.
Rod is one of the speakers at this year’s Web Directions Code, happening in Melbourne on May 1st and 2nd. Use discount code SITEPOINT to get the lowest price on Web Directions Code tickets!