Why Use Automated Testing Kit? - Part 1

End-to-end automated testing has become significantly easier in recent years. This shift began with the release of the Cypress testing framework nearly a decade ago. With features like time travel, an intuitive UI, and simple setup, Cypress made writing tests enjoyable for developers. While Cypress demonstrated how modern tools could streamline test creation, Microsoft later introduced Playwright, a strong competitor.

Despite these user-friendly tools, I noticed after several Cypress implementations that teams were repeatedly writing similar core tests (often referred to as smoke tests) and helper functions. That's how the idea of Automated Testing Kit was born.

Automated Testing Kit is designed to save you time by including these core pieces:

• a set of essential smoke tests written in Cypress and Playwright specifically designed for Drupal
• test framework helper functions, such as logging a user in, communicating with a remote server via Drush and more
• Drupal hooks that make targeting selectors easier
• pre-flight checks that ensure the target server is ready for testing and thorough documentation.

The Automated Testing Kit library now includes over two dozen tests and continues to expand. To see it in action, follow this Drupal recipe (text instructions, video instructions) to install it on a fresh Drupal installation. If you'd like a preview first, check out:

• the project page on Drupal.org
• the introduction video (42 min)
• or watch it in action on Drupal Forge.

When installing Automated Testing Kit on an existing site, the setup script will prompt you to choose between Cypress or Playwright tests. Each testing framework has unique strengths but the tests are identical between versions. You can explore the differences between the testing frameworks on our Learning Resources documentation page. (By the way, do not install the demo recipe on an existing site as it will overwrite the configuration. When you are ready to install Automated Testing Kit on your site, refer to the documentation.)

In this series, we’ll dive into each component of Automated Testing Kit. First, let’s explore why every site should incorporate automated end-to-end testing.

Why Automated Testing Matters

Websites serve as the primary entry point for most organizations, enabling visitors to complete tasks like purchasing products or learning about services. Preventing bugs that lead to downtime is critical for business operations.

According to Oxford Economics' report, The Hidden Costs of Downtime, downtime costs major companies $9,000 per minute, or $540,000 per hour. These high costs stem from lost sales, brand damage, declining stock prices, and sometimes even regulatory fines.

Think about your own experience when a website malfunctions or becomes unavailable: are you more or less likely to return to it in the future?

Here are compelling reasons to implement an end-to-end automated testing program:

• Fast-paced, modern release schedules can introduce undetected bugs; automated testing identifies these issues.
• Refactoring code for major new features is more dependable with automated testing.
• Maximizing development time on new features, rather than bug fixes, relies on automated testing.
• Fixing bugs reliably and with less frustration—without introducing new bugs—is simpler with automated testing.

Repairing Bugs Gets More Expensive Over Time

Top-performing teams identify bugs as early as possible during website development. This is because fixing defects becomes significantly more costly in terms of money, time and peace of mind later.

Catching bugs early is often an order of magnitude less expensive. Beyond financial costs, discovering bugs late can disrupt business operations and divert effort from productive tasks, like developing new features and improvements.

Figure 1. Add testing early in the website building process, including by specifying how testing will be used in the requirements phase. 

Credit: adapted from Shawn DeArmond and Mark Miller

"Shifting-left," that is, moving the testing as early as possible in the implementation process, is part of moving a team's mindset from having quality assurance to quality engineering.

When Should You Run Tests?

Automated testing can be integrated into nearly every stage of the development process. A solid principle for any team is "test early, test often."  

While teams new to end-to-end testing may not implement tests at every stage initially, the goal should be to eventually include testing at all the points outlined below:

Table 1. Opportunities for Automated Testing.

Does Automated Testing Kit Replace Other Testing?

It can—depending on the exact test. Generally, though, end-to-end automated testing complements your existing testing procedures. If you have unit, integration, or other tests, usually it's best to retain them, as they often provide detailed insights that end-to-end tests may not easily capture.  

However, since many projects lack any testing, it’s common for new end-to-end testing programs to cover everything. Starting with end-to-end tests is a great approach, as they verify whether key user journeys function correctly, which often reveals significant system issues. Over time, incorporate more granular tests or specialized tools to address specific problems if you find they are needed.

Figure 2. How end-to-end testing often works in the real world.

Credit: adapted from Shawn DeArmond and Mark Miller

Conclusion

The advent of tools like Cypress and Playwright has greatly simplified end-to-end automated testing but they don't come with tests and helper functions designed for Drupal. That's where Automated Testing Kit comes in. Automated Testing Kit speeds up the setup of an end-to-end automated testing program.

Automated testing is essential for detecting bugs early, minimizing costly downtime, and giving you peace of mind.

In the next part of this series, we’ll explore the contents of the Kit and how to apply it to your project. Ready to get started? Install the Demo Recipe, try Automated Testing Kit on Drupal Forge, and join #automated_testing_kit in the Drupal Slack workspace to have your questions answered.

1. The Hidden Costs of Downtime, Oxford Economics, 2024.