At WordPress.com we strive to continuously deliver new functionality whilst ensuring that we provide a consistent and reliable user experience.

One key element to all of this is developing self-testing code; that is, creating and maintaining a comprehensive suite of automated tests that are developed in conjunction with our functional code.

The WordPress Calypso User Interface supports React unit and component testing. Because the WordPress.com user experience depends on many other factors beyond just the precision of our Calypso components, we have also been building a suite of automated end-to-end (e2e) tests that verify our critical user flows are working as intended.

Now, the WordPress.com team is proud to announce that we have released our automated e2e test suite as open-source. You can find it here on GitHub.

Why open source is so important

You could say our automated testing pyramid looks like this:

The quality of our code and our product depend on the amount of feedback we get and on the amount of people who use them. If we’re developing behind closed doors, we are putting artificial limits to both.

We have done our best work in the open, let’s continue working this way.

~ Matt Mullenweg, Automattic CEO

The technical details

Our tests use the ES2015 syntax on Node, which should make it it easy for anyone familiar with Calypso to contribute.

We are utilizing the open-source selenium-webdriver project to drive real user browsers. We write our e2e tests in the same test specifications library as Calypso (Mocha) and we run these on the same CI platform that being on headless CircleCI Linux containers.

Since our platform is fully responsive, we run the same tests (with slightly different logic) against three different screen resolutions to ensure our customer experience functions across different device types.

We’ve also done some neat things like developing real-time Slack notifications (with screenshots) when tests are failing so we can investigate quickly.

It’s still early days, and we’re still striving to make these tests as deterministic and reliable as possible.

You can run these tests against a local WordPress Calypso development environment. We automatically run these tests on every WordPress.com deployment, as well as every six hours, to ensure end-to-end flows are working as intended for our users. We are planning to run these, or a subset of these, against pull requests as they are developed to prevent bugs being introduced into our platform.

Automated tests have already been successful in making us aware of issues sooner so we can fix them faster.

Get involved!

Feel free to check out our repository, make a fork, and provide us with feedback or suggestions. Pull requests are always welcomed.

***

Alister Scott is an Excellence Wrangler for Automattic and blogs regularly about software testing at WatirMelon.