Running tests in parallel

Running end-to-end tests is an integral part of many workflows, ensuring that the end user does not catch any errors that may have slipped into production. Commonly, end-to-end tests are run using a tool like Cypress. An issue that arises, however, is that end-to-end tests can take some time to run and end up limiting productivity for developers.

A solution to this is to run Cypress tests in parallel to reduce the amount of time that a developer waits for tests to complete. This can be done using webapp.io's SPLIT directive, which allows the runner to duplicate its entire state a number of times at a specific point. A graphic depicting the use of the SPLIT directive to duplicate the state of the runner 5 times is shown below.

In this example, we demonstrate how to set up Cypress with a Docker Compose application, how to run docker-compose cypress on webapp.io, and how to run tests in parallel. 10 tests were run each time. When measured for time differences, the tests that were run in parallel (using a larger SPLIT) were completed in significantly less time than those that were not run in parallel (using a smaller or no SPLIT). The tests themselves are arbitrary, in this case the tests chose a random number between 10 and 100 and waited that many seconds to run.

How to set up Cypress with Docker Compose:

The first step that is required to set up Cypress with Docker Compose is to make the Cypress service. Some key details to note when creating the service are:

• Make the service depend on the service of webapp you wish to test
• Use docker-compose profile to not start service unless told to
• Specify command to run Cypress tests

How to run docker-compose Cypress on webapp.io:

Next, create a base Layerfile that starts your application. It'll look something like the following example:

FROM vm/ubuntu:22.04

# install docker-ce (from tutorial for ubuntu)
RUN apt-get update && \
    apt-get install apt-transport-https ca-certificates curl software-properties-common && \
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - && \
    add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable" && \
    apt-get update && \
    apt install docker-ce

# install docker compose
RUN curl -L "https://github.com/docker/compose/releases/download/1.28.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose && \
    chmod +x /usr/local/bin/docker-compose

# copy the root (i.e., repository root) to /root in the runner
COPY / /root

RUN REPEATABLE docker-compose up -d --build --force-recreate --remove-orphans && sleep 5

EXPOSE WEBSITE localhost:8000

In a second Layerfile, inherit the base and start your cypress service using the --profile flag and the --exit-code-from flag to get the exit code of your tests. This allows the base or cypress to be snapshotted if neither change. That’ll look something like this:

FROM ../base

# CYPRESS_RECORD_KEY is added in the "secrets" tab of the webapp.io dashboard
SECRET ENV CYPRESS_RECORD_KEY

SPLIT 10
RUN docker-compose --profile cypress up --exit-code-from cypress cypress

How to run Cypress tests in parallel:

Finally, to run the Cypress tests in parallel, the following steps are taken:

• Create a project on dashboard.cypress.io
• Add your record key as a secret to webapp.io
• Add the secret and SPLIT to your Layerfile
• Add the record and parallel options to your cypress command in your cypress service definition
• Use the webapp env vars JOB_ID and SPLIT_NUM to identify the parallel runs

Once the record and parallel options are added, the Cypress service will look like the following:

 cypress:
    image: cypress:latest
    profiles:
      - cypress
    depends_on:
      - web
    environment:
    # pass base url to test pointing at the web application
      - CYPRESS_baseUrl=http://traefik:80
    command:  "npx cypress run --record --key '${CYPRESS_RECORD_KEY}' --parallel --ci-build-id ${JOB_ID}-${SPLIT_NUM}"
    ipc: host
    build: ./services/cypress
    ports:
      - "9000:80"
    volumes:
      - "./services/cypress/tests:/app/tests:ro"

The runs are visible on the Cypress dashboard, as seen below:

They are also visible on the webapp.io dashboard, where "Level 1" runs consist of base Layerfiles and "Level 2" runs of Cypress tests. In the image below, the time it takes to run the Cypress tests is visible on each run:

Test Times

Now that the tests are set up, let’s look at our results. Our arbitrary test was tested with a number of SPLIT amounts: none, 2, 5, and 10. The numbers were as expected: no SPLIT took 3x as long as SPLIT 10 to run.

To break it down further, no split took 12 minutes, SPLIT 2 took 7 minutes, SPLIT 5 took 5 minutes, and SPLIT 10 took 4 minutes.

Want to try to run Cypress tests in parallel using the SPLIT directive yourself?

Webapp.io's team tier allows users to use up to 12 concurrent runs, which means up to 12 SPLITS can be run at one time. Our free tier does not come with use of the SPLIT directive, but feel free to reach out if you'd like to give it a try at no cost. If you have any questions about setting up webapp.io, using the SPLIT directive to run your Cypress tests in parallel, or our pricing tiers, feel free to contact us at [email protected].