Continuous Integration (CI) lets you define tests to run before any code is merged - in practice, this means that you can make sure that nothing breaks in response to a change. A lesser-known benefit of CI is that it helps other developers set up the software with any required dependencies.

I've used CI for every new project I've made since 2016. Even the projects that I only work on myself have used CI -  If I return to an older project of mine, I can quickly make new changes by relying on CI to tell me if I broke an old feature I'd forgotten about.

Another great feature of CI is that it reminds me how to set up the software - I can remember how I set up the database, how to start the requisite docker images, and what setup scripts to run in which order by looking at my acceptance test configurations. All I have to do is read the relevant configuration carefully to be able to deploy the software again.

You could almost consider CI for acceptance tests to be "GitOps" - the configuration describes exactly what it takes to deploy software to the point that you can run tests on it. It works so well for my projects that I don't bother writing any documentation for how to deploy, since the deployment documentation will almost certainly be more out of date  than the acceptance test configuration.

This approach works very well for my own repositories, but in trying to do the same in other repositories I've had a problem: Almost all of the CI providers I've tried have been black boxes. That is, they look more like configuration files than test scripts. Consider this CI configuration for the Ghost blogging platform that I use for webapp.io's blog:

dist: xenial
language: node_js
node_js: ['12', '10', '8']
cache: yarn
services: ['mysql']

if: NOT head_branch =~ ^renovate
env:
  matrix:
  - DB=sqlite3 NODE_ENV=testing
  - DB=mysql NODE_ENV=testing-mysql
matrix:
  include:
  - node_js: '10'
    env: TEST_SUITE=lint
  allow_failures:
  - node_js: '12'
install:
- if [ "$TRAVIS_NODE_VERSION" == "12" ]; then yarn --ignore-engines; else yarn; fi
before_script:
- if [ $DB == "mysql" ]; then mysql -e 'create database ghost_testing'; fi
- if [ "$DB" == "sqlite3" ]; then yarn add --ignore-engines --force sqlite3; fi
script: |
  if [ "$TEST_SUITE" == "lint" ]; then
    yarn lint
  elif [[ "$TRAVIS_PULL_REQUEST_BRANCH" =~ ^renovate || "$TRAVIS_EVENT_TYPE" == "cron" ]]; then
    yarn ci:regression
  else
    yarn ci
  fi

How do I deploy the software and run the tests myself? I have to carefully parse lines of slightly indented YAML to understand the sequence of actions I need to take. It's especially hard for beginners - this configuration doesn't even tell you how to install node or yarn.

YAML seems like a really poor choice for configurations like this, and there's a file format that's much more suited to the task: Dockerfiles! Dockerfiles tell you exactly where everything goes, how to install libraries, and what commands to run. If this test could be written in Dockerfile syntax, they would be much easier to follow.

The problem with writing tests in Dockerfiles is that the format isn't really up to it - It's difficult to run multiple services concurrently (everything shuts down between every "layer" of the build), and it's especially difficult to run acceptance tests (start a database, a webserver, call some API endpoints.) Folks usually end up using Docker for unit tests, and write acceptance tests in some obscure yaml syntax, where the acceptance tests are the ones that I want to read the most!

The solution for writing human-readable acceptance tests would be a file format half way between a Dockerfile and a CI configuration blob. Here's an example that I would personally prefer for the ghost configuration above:

FROM ubuntu:16.04

RUN apt-get update
PERMUTE (
  RUN apt-get -y install nodejs=='1.10.*' yarn
  RUN apt-get -y install nodejs=='1.12.*' yarn && \
      alias yarn='yarn --ignore-engines'
  RUN apt-get -y install nodejs=='1.14.*' yarn
)

CHECKPOINT # save the state to avoid re-downloading things on every test run

RUN yarn

PERMUTE (
  RUN apt-get -y install mariadb && \
      systemctl start mariadb && \
      mysql -e 'create database ghost_testing'
  RUN apt-get -y install sqlite3-dev && \
      yarn add --force sqlite3
)

CHECKPOINT # hibernate & save the test runner with mysql already running to avoid re-downloading it and starting it on every test run

RUN if [ "$TEST_SUITE" == "lint" ]; then \
      yarn lint \
    elif [[ "$BRANCH" =~ ^renovate || "$EVENT_TYPE" == "cron" ]]; then \
      yarn ci:regression \
    else \
      yarn ci \
    fi

This configuration is very close to a Dockerfile, with the biggest change being a PERMUTE directive to repeat actions for different versions of libraries. You could even make a deployment documentation generator for files like this, where each "PERMUTE" would be a choice (Which version of nodejs? mysql or sqlite3?) - If developers wanted to set up your software, they could just follow the documentation docs to get *something* running. The docs would never be out of date as long as the tests pass -- self documenting CI!