Running tests in parallel
Running end-to-end tests is an integral part of many workflows, ensuring that the end user does not catch any errors that may have slipped into production. Commonly, end-to-end tests are run using a tool like Cypress. An issue that arises, however, is that end-to-end tests can take some time to run and end up limiting productivity for developers.
A solution to this is to run Cypress tests in parallel to reduce the amount of time that a developer waits for tests to complete. This can be done using webapp.io's SPLIT directive, which allows the runner to duplicate its entire state a number of times at a specific point. A graphic depicting the use of the SPLIT directive to duplicate the state of the runner 5 times is shown below.
In this example, we demonstrate how to set up Cypress with a Docker Compose application, how to run docker-compose cypress on webapp.io, and how to run tests in parallel. 10 tests were run each time. When measured for time differences, the tests that were run in parallel (using a larger SPLIT) were completed in significantly less time than those that were not run in parallel (using a smaller or no SPLIT). The tests themselves are arbitrary, in this case the tests chose a random number between 10 and 100 and waited that many seconds to run.
How to set up Cypress with Docker Compose:
The first step that is required to set up Cypress with Docker Compose is to make the Cypress service. Some key details to note when creating the service are:
- Make the service depend on the service of webapp you wish to test
- Use docker-compose profile to not start service unless told to
- Specify command to run Cypress tests
How to run docker-compose Cypress on webapp.io:
Next, create a base Layerfile that starts your application. It'll look something like the following example:
In a second Layerfile, inherit the base and start your cypress service using the --profile flag and the --exit-code-from flag to get the exit code of your tests. This allows the base or cypress to be snapshotted if neither change. That’ll look something like this:
How to run Cypress tests in parallel:
Finally, to run the Cypress tests in parallel, the following steps are taken:
- Create a project on dashboard.cypress.io
- Add your record key as a secret to webapp.io
- Add the secret and SPLIT to your Layerfile
- Add the record and parallel options to your cypress command in your cypress service definition
- Use the webapp env vars JOB_ID and SPLIT_NUM to identify the parallel runs
Once the record and parallel options are added, the Cypress service will look like the following:
The runs are visible on the Cypress dashboard, as seen below:
They are also visible on the webapp.io dashboard, where "Level 1" runs consist of base Layerfiles and "Level 2" runs of Cypress tests. In the image below, the time it takes to run the Cypress tests is visible on each run:
Now that the tests are set up, let’s look at our results. Our arbitrary test was tested with a number of SPLIT amounts: none, 2, 5, and 10. The numbers were as expected: no SPLIT took 3x as long as SPLIT 10 to run.
To break it down further, no split took 12 minutes, SPLIT 2 took 7 minutes, SPLIT 5 took 5 minutes, and SPLIT 10 took 4 minutes.
Want to try to run Cypress tests in parallel using the SPLIT directive yourself?
Webapp.io's team tier allows users to use up to 12 concurrent runs, which means up to 12 SPLITS can be run at one time. Our free tier does not come with use of the SPLIT directive, but feel free to reach out if you'd like to give it a try at no cost. If you have any questions about setting up webapp.io, using the SPLIT directive to run your Cypress tests in parallel, or our pricing tiers, feel free to contact us at [email protected].