Posted on Fri 10 January 2014

Testing Django Apps with Docker

When building a Django app with many dependencies it can be easy to loose track of what is needed in order to install the application on a new system. Developer machines can have a lot of non-isolated dependencies installed from other projects. New developers trying to install an application can find it won't run properly due to undocumented dependencies. Deployments to live can result in services ceasing to function due to missing libraries.

When we build our Django apps there are two dependency file lists we keep: packages.txt, where we keep all our system dependences and requirements.txt, for all of our python dependences.

When we do a deployment the following command will install our system dependencies on live:

➫ cat ~/app/packages.txt | xargs apt-get -y --force-yes install

Here's the contents of packages.txt for one of our applications:

➫ cat ~/app/packages.txt
autoconf
bc
build-essential
checkinstall
curl
git-core
jpegoptim
libevent-2.0-5
libevent-dev
libjpeg-dev
libmagic-dev
libmagickwand-dev
libmysqlclient-dev
libpcre3-dev
libssl-dev
libtool
libxslt-dev
linux-kernel-headers
lsof
mysql-server
pngcrush
python-dev
rabbitmq-server
redis-server
ruby-compass
ruby-sass
supervisor
unzip
uuid-dev
zip
zlib1g-dev

The next command we run installs all of our python dependencies:

➫ grep -v distribute ~/app/requirements.txt |  xargs pip install -q

I've removed distribute from the list of modules that would be installed. pip was including it when we ran pip dump >> requirements.txt and it would break pip itself mid-install. Here's a section of requirements.txt for one of our projects:

➫ cat requirements.txt
Django==1.5.5
Markdown==2.3.1
PIL==1.1.7
Pygments==1.6
South==0.8.4
Wand==0.3.5
amqp==1.0.13
anyjson==0.3.3
argparse==1.2.1
billiard==3.3.0.0
...

That works pretty well but what would help give us confidence that these files contain every dependency and will run properly is to run them on a fresh install of Ubuntu 13.

In the past I'd use vagrant for this but it required having Ruby, RubyGems and VirtualBox running on each developer's machine. Making sure RubyGems and the vagrant gem install properly with all their dependencies was always moving target. To top that off, VirtualBox would reserve all the memory allocated for it and not use deltas when writing to the hard drive.

Enter Docker. Docker is a wrapper for Linux Virtual Containers (abbreviated as LXC). They only allocate as much memory as they currently need, they use AUFS to perform copy-on-write which effectively is a delta and means they don't use any more disk space than they need and this all installs with dependences that come with more stable installation instructions.

In the root of our application we have a Dockerfile. Once the developer has installed docker all they need to do is run sudo docker build . and it will create a fresh Ubuntu 13 container, install our application with their configuration and run manage.py test. Here's what our Dockerfile looks like:

➫ cat Dockerfile
# Fresh installation test
#
# To run this: sudo docker build .

FROM stackbrew/ubuntu:saucy
MAINTAINER Stickyworld Tech Team

RUN apt-get update
RUN apt-get install -y python-software-properties python-pip python-setuptools python-dev build-essential python

ADD ./ /code/

ENV DEBIAN_FRONTEND noninteractive
RUN echo mysql-server-5.5 mysql-server/root_password password root | debconf-set-selections
RUN echo mysql-server-5.5 mysql-server/root_password_again password root | debconf-set-selections
RUN cat /code/packages.txt | xargs apt-get -y --force-yes install

RUN ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib
RUN ln -s /usr/lib/x86_64-linux-gnu/libfreetype.so /usr/lib
RUN ln -s /usr/lib/x86_64-linux-gnu/libz.so /usr/lib
RUN ldconfig

RUN grep -v distribute /code/requirements.txt | xargs pip install

RUN cd /code/src/ && python manage.py test

When we run this file we want to make sure everything in packages.txt and requirements.txt will install properly. If there is a package in either which isn't found an error will be raised. If a library is missing it should eventually come up when we run through all of our tests.

There are two hacks in the Dockerfile, the first is to make sure MySQL installs properly as it's listed in packages.txt even though our tests will run with an in-memory SQLite3 database. The second is making sure libjpeg, libfreetype and libz can be found by PIL so we can process JPEGs and other images properly in our code.

I'm happy Docker has done such a good job of wrapping LXC and making the format of Dockerfiles easy to work with. I also like the added confidence we get when doing deployments. If you're interested in working with Docker and/or LXC and helping us build more reliant code please do drop me a line.

Signup for our low-traffic newsletter:

Powered by MailChimp

© Giulio Fidente. Built using Pelican. You can fork the theme on github. .