Comparing 3 Continuous Intergration Services For a Docker Compose Project

In this post, I’m going to compare 3 different hosted continuous integration services (Codeship, CircleCI and Travis) for a Docker Compose project.

What I’m looking for

I’ll be judging the options based on the 3 main things I look for from a CI.

• Speed. I want something that will build and run my tests as fast as possible. I want to minimize the time spent waiting for my pull request’s checks to pass or fail.
• Cost. How much does it cost to use and how does that scale as your project/team grows. I’ll also compare the costs of public and open source project vs a private one.
• Setup complexity and ease. The less time spent to get everything up and running the better. Is the documentation useful? Are there example projects for a quickstart?.

This post doesn’t focus on pushing, publishing Docker images or deploying them as services.

If you have experiences with those aspects and you’d like to share, feel free to drop a comments below.

Our sample project

To compare our CIs, I have set up a Python-based Django + Django Rest Framework application that uses Docker Compose for it’s local development. It uses pytest as its test runner and needs to run both unit tests and integrations tests against a Postgres database. The project has purposefully resource-heavy dummy tests setup to test the performance speed of the CI solutions. to test the performance speed of each CI solution.

Codeship Pro

Codeship has two offerings, Basic and Pro. Pro includes Docker support, so we will only be looking at that one.

Setup

Codeship requires two key files to setup your Docker containers and then run the required tests.

Codeship Services File codeship-services.yml

Here you outline what Docker services your test suite needs. This file is very similar to a docker-compose.yml, in fact, in its absence, Codeship will automatically search for a docker-compose.yml file to use in its place.

# codeship-services.yml
api:
  build: .
  depends_on:
    - db
  cached: true
  environment:
    - DATABASE_URL=postgresql://user:password@db:5432/db
    - DEBUG=true
    - ALLOWED_HOSTS=*
    - SECRET_KEY=foo
db:
  image: healthcheck/postgres
  environment:
    - POSTGRES_PASSWORD=password
    - POSTGRES_USER=user
    - POSTGRES_DB=db

However, a few of the configuration options differ from docker-compose.yml. For example:

• encrypted_env_file is added. This lets you securely store encrypted environment variables in your source control to be used by your services. Using Codeship’s CLI jet environment variables can be added, edited and removed.
• cached is added. This enables Codeship to cache a built image in their own secure registry after a build is finished. Allowing the CI to pull the image, rather than re-building each time. When setup correctly, this can help save time.

Not all configuration keys are available, including privileged.

Codeship Steps File codeship-steps.yml

After your services are built/pulled, this file defines what commands the services need to run for their test suite.

# codeship-steps.yml
- name: api tests
service: api
command: pytest tests/

For our project, this is fairly simple. However, more advanced usages allow you too:

• Parallelize steps. This spins up separate containers to run simultaneously. If setup correctly this can make better use of the host machine’s resources to run tests faster.
• Limit specific commands to specific branches.

Pricing

Open source projects on Codeship are free.

They have a free plan, that includes:

• 100 builds a month
• 1 concurrent build
• 1 parallel test pipeline
• Unlimited projects/users/teams

Their pricing starts at $75/month. This gives you 1 concurrent build on 1 small instance (2 VCPU, 3.75 GB Memory). Pricing scales linearly by the number of instances and their size.

Codeship Pro builds run on individual EC2 instances running in the us-east-1 region. They all use a fixed version of Docker (18.03, the latest stable release when this was written).

CircleCI 2.0

For Docker-based projects, CircleCI enables you to run jobs in one of two virtual environments:

• In Docker images (executor_type=docker).
• In a dedicated Linux VM (executor_type=machine).

I prefer using the machine executor, as the configuration is simpler and seems easier to integrate with a Docker Compose project. However, CircleCI documentation’s notes that the use of the machine executor may require an additional fee in a future pricing update.

I’ve reached out to CircleCI’s support to clarify what that may mean, and will update this post with that information if they reply.

The docker executor requires a few workarounds (documented here) that add to the complexity. It still a feasible option, but the few gotchas sway my favour towards the machine executor

Setup

# .circleci/config.yml
version: 2
jobs:
  build:
    machine: true
    steps:
      - checkout
      - run:
          name: Create .env file
          command: touch .env
      - run:
          name: Build Containers
          command: docker-compose build
      - run:
          name: Run Postgres
          command: docker-compose up -d db
      - run:
          name: Run Tests
          command: docker-compose run api pytest tests/

CircleCI’s configuration YML lets you define multiple jobs, that can have multiple steps (commands).

In this case, we have kept things simple and set up a single job, that builds our containers, starts the database then runs the tests.

You can control the jobs orchestration using a separate option called workflows. This can allow you to run things in parallel (on the same machine), and potentially use resources better to speed up your test suite. You can also run tests in parallel across multiple machines, however, this is defined for each job.

Pricing

Open source projects are free on CircleCI. In their FAQ they state they offer up to 4 containers for public projects.

They have a free plan, that includes:

• 1 container limited to 1500 build minutes per month.
• Unlimited projects/users

After which, you pay $50 per container (and have unlimited build minutes per month). Each container allows you to run 1 concurrent job. Multiple containers allow you to either run multiple builds at the same time or single builds with parallel jobs.

Each CircleCI build container, when run as an machine executor has 2 CPU @ 2.3 GHz and 8GB Memory.

You can also pay extra (how much is currently private) to enable Docker Layer Caching which could speed up large image builds. I’ve reached out to CircleCI’s support about how much this costs and will update this post if I receive a reply.

Currently, the default Docker version used is 17.09.0-ce (compared to the latest stable release of 18.03.1-ce). To pin a specific Docker version, you can do so my using year-month versioned image, e.g. circleci/classic:201808-01.

TravisCI

Setup

Travis uses a .travis.yml file to configure any build/test/deploy steps that run on your virtual environment. Unlike the previous two options, Travis doesn’t come with Docker installed. Instead, as part of the build, we need to apt-get Docker, and curl install Docker Compose.

sudo: required
? addons
apt:
  packages:
    - docker-ce
services:
  - docker

? env
DOCKER_COMPOSE_VERSION: 1.22.0

before_install:
  - sudo rm /usr/local/bin/docker-compose
  - curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
  - chmod +x docker-compose
  - sudo mv docker-compose /usr/local/bin

  - touch .env
  # Setup your application stack. You may need to tweak these commands if you
  # doing out-of-the-ordinary docker-compose builds.

install:
  - docker-compose pull
  - docker-compose build
  - docker-compose up -d db

script:
  - docker-compose run api pytest tests/

After that, it’s just a simple matter of running the same commands we use locally to execute the tests.

To run commands in parallel, you need to employ your bash-foo. For example you can run a command in the background by appending an &.
i.e.

install:
  - docker-compose pull&

If you are after machine parallelism, Travis offers a build matrix, that allows you to split up your jobs across workers.

Pricing

Open source projects are free on Travis CI.

For private projects, they have a free trial that gives you 100 builds (in total, doesn’t reset each month).

After that, pricing starts at $69 per month, and gives you:

• 1 concurrent job
• Unlimited build minutes
• Unlimited repos/collaborators

Pricing scales by job concurrency and discounts as the plan grows, e.g. 1 concurrent job is $69, 10 concurrent jobs is $489 ($48.90 each).

Based on your configuration your worker machine’s specs can differ. In our case, the virtual environment was run on GCE, with 2 cores and 7.5GB Memory.

Speed Test

To compare the speed of each offering, we’ll look at two things:

• How long the test-suite took to run, reported by pytest
• How long the overall build took, reported by each service.

From testing, Codeship built and ran the fastest, followed closely by CircleCI. Travis lagged behind. It seems to be slower to spin up a instance and the added installation needed for Docker/Docker Compose costs additional time.

Conclusion

From our 3 options, it’s a close call between Codeship and CircleCI. Codeship built and ran the fastest, was easy to cache built images and had good clear documentation and example projects. However, CircleCI’s pricing ($20 cheaper on the first tier) could justify it’s ever so slightly slower build time.

Subjectively, I liked CircleCI’s configuration the most, as it didn’t require me to change my local Docker Compose configuration. [However, one future cause for concern might be their note that machine executor pricing may change.

CircleCI is a narrow winner for me. If you have complex and build intensive Docker images, Codeship’s caching might tip it over the edge.

What do you think? I’d love to hear from others if they have any additional experiences! Share them in the comments!