End-to-end tests and CI

When you make changes to code in a repository, you usually want to make sure that you haven't broken core user flows like the sign up button.

To automatically check nothing is broken, people usually start a copy of their app within their CI pipeline and then run test browsers to click around. That way you can make sure your app still builds, starts, and mostly works after every new change.

For a typical full-stack webapp, these are the sorts of steps that run after every new change:

Explanation of CI pipeline: A build step where "docker-compose build" runs, then a step where "docker-compose up" runs, then a final step where browser tests run

This pipeline works fine, but it's significantly more expensive and slow than it could be:

  • Images have to be built from scratch, or constantly pushed/pulled from a registry somewhere
  • The database has to be re-migrated and re-seeded every time
  • It's hard to parallelize things, because the setup needs to re-run for every independent copy of the environment.

Faster and cheaper e2e tests using base images

Since the build steps, database, and deployment definitions rarely change between code changes, it'd be nice if we could have a "base image" of a machine which already had everything started.

This base image would have:

  • A migrated and seeded database already running
  • Build caches populated with files from an earlier build
  • Images and large files pre-downloaded
  • The repository itself pre-loaded

If we had such a base image, we could pull in our new change and build/start things in seconds.

For a long time this was technically infeasible, but there's an exciting new technology that facilitates this exact use-case.

Firecracker

Firecracker is the framework that powers AWS Lambda. It allows you to make "snapshots" of a running VM, which contain the entire disk/memory of the vm. When you restore one, you also restore all of the VM's files and processes.

Diagram of a VM turning into a file, then back into a VM
Firecracker turns VMs into files and back into VMs.

We could create a script that created a firecracker VM & set it up to be an ideal base image. Then we could copy the file that contained the VM's files and processes,

Getting started with firecracker

It's actually relatively simple to use firecracker because of their quickstart guide and other amazing resources.

Using firecracker on Linux can be as simple as a few commands:

# Step 1. download firecracker
curl -fSsL https://github.com/firecracker-microvm/firecracker/releases/download/v1.0.0/firecracker-v1.0.0-x86_64.tgz | tar -xz
mv firecracker-* firecracker

# Step 2. Download files required to start the VM
image_bucket_url="https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/$arch"
wget "${image_bucket_url}/kernels/vmlinux.bin"
wget "${image_bucket_url}/rootfs/bionic.rootfs.ext4"

# Step 3. Create a config file
cat <<EOF > vmconfig.json
{
  "boot-source": {
    "kernel_image_path": "vmlinux.bin",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "bionic.rootfs.ext4",
      "is_root_device": true,
      "is_read_only": false
    }
  ]
}
EOF

# Step 4. Start the VM
./firecracker --unix-socket /tmp/firecracker.socket --config-file vmconfig.json

Great, we have a VM, now what?

We wanted to use firecracker to create a "zygote snapshot" which contained all of our dependencies. If you're following along, use Julia Evans' tutorial to set up networking, and you should be able to ssh directly into your VM!

From there, run your setup commands as necessary:

$ apt-get install docker postgresql
$ docker pull ubuntu:18.04
$ wget http://example.com/my-large-file.txt ~/my-large-file.txt
Some example commands that set up a VM with the dependencies for running brower tests

Finally, exit the VM and take a snapshot! (see Firecracker docs)

curl --unix-socket /tmp/firecracker.socket -i \
    -X PATCH 'http://localhost/vm'
    -H  'Accept: application/json' \
    -H  'Content-Type: application/json' \
    -d '{"state": "Paused"}'

curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT 'http://localhost/snapshot/create' \
    -H  'Accept: application/json' \
    -H  'Content-Type: application/json' \
    -d '{
            "snapshot_type": "Full",
            "snapshot_path": "./snapshot_file",
            "mem_file_path": "./mem_file",
            "version": "0.23.0"
    }'

How it all fits together

Now that we have our disk and memory snapshot files, we can just run a simple script to prepare the VM to run our CI pipeline:

  1. Copy the zygote files:  bionic.rootfs.ext4, snapshot_file, and mem_file to another directory
  2. Restore the snapshot files using the firecracker API
  3. Run the build for your pipeline within the restored VM

Benchmark

Enough about theory, let's look at a benchmark!

This benchmark uses an open source Slack clone, which uses Docker Compose, Go, PostgreSQL, and React.

Firecracker-based CI is 10.5x faster

build / start / test for a full-stack webapp

This benchmark shows a drastic 10x speedup due to having a warm cache. The firecracker VM came up almost instantly, and ran the build steps in just a few seconds due to the reused caches. This meant that the whole pipeline could run in only 30s.

In comparison, the "traditional CI" VMs took over a minute just to boot and clone the code, and then had to re-build the images from scratch, before ever being able to run the tests.

See these links for methodology:

Want to try firecracker-based CI yourself?

Webapp.io offers firecracker-based VMs for CI even on our free plan. If you'd like to try out the approach mentioned in this post without setting it up yourself, you can create a webapp.io account and try a sample in just a few minutes at https://webapp.io.

Discussions: