When building a Django app with many dependencies it can be easy to loose track of what is needed in order to install the application on a new system. Developer machines can have a lot of non-isolated dependencies installed from other projects. New developers trying to install an application can find it won't run properly due to undocumented dependencies. Deployments to live can result in services ceasing to function due to missing libraries.
When we build our Django apps there are two dependency file lists we keep: packages.txt, where we keep all our system dependences and requirements.txt, for all of our python dependences.
When we do a deployment the following command will install our system dependencies on live:
➫ cat ~/app/packages.txt | xargs apt-get -y --force-yes install
Here's the contents of packages.txt for one of our applications:
➫ cat ~/app/packages.txt autoconf bc build-essential checkinstall curl git-core jpegoptim libevent-2.0-5 libevent-dev libjpeg-dev libmagic-dev libmagickwand-dev libmysqlclient-dev libpcre3-dev libssl-dev libtool libxslt-dev linux-kernel-headers lsof mysql-server pngcrush python-dev rabbitmq-server redis-server ruby-compass ruby-sass supervisor unzip uuid-dev zip zlib1g-dev
The next command we run installs all of our python dependencies:
➫ grep -v distribute ~/app/requirements.txt | xargs pip install -q
I've removed distribute from the list of modules that would be installed. pip was including it when we ran pip dump >> requirements.txt and it would break pip itself mid-install. Here's a section of requirements.txt for one of our projects:
➫ cat requirements.txt Django==1.5.5 Markdown==2.3.1 PIL==1.1.7 Pygments==1.6 South==0.8.4 Wand==0.3.5 amqp==1.0.13 anyjson==0.3.3 argparse==1.2.1 billiard==126.96.36.199 ...
That works pretty well but what would help give us confidence that these files contain every dependency and will run properly is to run them on a fresh install of Ubuntu 13.
In the past I'd use vagrant for this but it required having Ruby, RubyGems and VirtualBox running on each developer's machine. Making sure RubyGems and the vagrant gem install properly with all their dependencies was always moving target. To top that off, VirtualBox would reserve all the memory allocated for it and not use deltas when writing to the hard drive.
Enter Docker. Docker is a wrapper for Linux Virtual Containers (abbreviated as LXC). They only allocate as much memory as they currently need, they use AUFS to perform copy-on-write which effectively is a delta and means they don't use any more disk space than they need and this all installs with dependences that come with more stable installation instructions.
In the root of our application we have a Dockerfile. Once the developer has installed docker all they need to do is run sudo docker build . and it will create a fresh Ubuntu 13 container, install our application with their configuration and run manage.py test. Here's what our Dockerfile looks like:
➫ cat Dockerfile # Fresh installation test # # To run this: sudo docker build . FROM stackbrew/ubuntu:saucy MAINTAINER Stickyworld Tech Team RUN apt-get update RUN apt-get install -y python-software-properties python-pip python-setuptools python-dev build-essential python ADD ./ /code/ ENV DEBIAN_FRONTEND noninteractive RUN echo mysql-server-5.5 mysql-server/root_password password root | debconf-set-selections RUN echo mysql-server-5.5 mysql-server/root_password_again password root | debconf-set-selections RUN cat /code/packages.txt | xargs apt-get -y --force-yes install RUN ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib RUN ln -s /usr/lib/x86_64-linux-gnu/libfreetype.so /usr/lib RUN ln -s /usr/lib/x86_64-linux-gnu/libz.so /usr/lib RUN ldconfig RUN grep -v distribute /code/requirements.txt | xargs pip install RUN cd /code/src/ && python manage.py test
When we run this file we want to make sure everything in packages.txt and requirements.txt will install properly. If there is a package in either which isn't found an error will be raised. If a library is missing it should eventually come up when we run through all of our tests.
There are two hacks in the Dockerfile, the first is to make sure MySQL installs properly as it's listed in packages.txt even though our tests will run with an in-memory SQLite3 database. The second is making sure libjpeg, libfreetype and libz can be found by PIL so we can process JPEGs and other images properly in our code.
I'm happy Docker has done such a good job of wrapping LXC and making the format of Dockerfiles easy to work with. I also like the added confidence we get when doing deployments. If you're interested in working with Docker and/or LXC and helping us build more reliant code please do drop me a line.