Sometimes, automated end-to-end (e2e) tests fail and it’s difficult to tell why. Screenshots and stack traces are useful, but video of the actual test running in the CI environment would also be helpful. That’s why we recently added video to the automated tests we run in CircleCI. We are now able to see everything that occurred during the test and can quickly debug issues that would previously require more investigation.
How we’re using it
Any time we run our full automated e2e test suite, we make a video recording of every test. If the test passes, we delete the video. If it fails, we keep the video, and it’s returned as an artifact in CircleCI. We also output the path of the video file so it’s easier to find the file that corresponds to the test that you’re looking at.
We are currently only making recordings when running the full suite of tests; we don’t record the “canaries” to avoid adding additional overhead to simple test runs.
How it works
The builds we use in CircleCI are running in Linux docker images. To get video of the automated tests, we install FFmpeg in the container and use Xvfb for the display. For each test we:
- Start an Xvfb session on a display that is not already in use.
- Start a recording of the display using FFmpeg.
- Run the automated test in Chrome. There is a
--displayparameter that allows you to specify which display to run the test on. - Stop the FFmpeg recording when the test is completed or has failed.
- Stop the Xvfb session.
Get involved!
Feel free to check out our e2e tests repository, make a fork, and give us your feedback or suggestions. Pull requests are always welcome.
***
Brent Sessions is an Excellence Wrangler for Automattic working to provide tools and technology to help developers improve product testing.
