Advanced Features of the Automated Testing Kit - Part 3

In Part 1 of the series, we introduced the Automated Testing Kit and its benefits. In part 2, we introduced typical setups and tools you might want to use. 

Once the basics of automated testing are established and the team is efficiently adding new tests and resolving issues, it’s time to consider the next steps. Here are some key strategies and capabilities you may want to add.

Expand Test Coverage

Teams often start with a core set of tests and expand them over time. Test coverage typically grows in two ways:

New Features Require New Tests

Each time a new feature is developed, it’s ideal to create corresponding tests simultaneously. These tests are often written by QA team members who specialize in test creation, typically joining near the end of feature development to focus on testing the new functionality. However, this approach has a challenge: waiting until the end of development to involve QA can strain resources. The QA team, responsible for designing tests and writing scripts, is often preoccupied with hands-on testing—exploring features, identifying bugs, and reporting issues to developers. When a feature has many bugs, QA may report a new issue only to be immediately asked to retest a different bug fixed moments earlier. This leaves little time for writing new tests exactly when they’re needed. Two solutions can address this.

• Shift Test Writing to Developers: In many cases, having developers write tests is effective, as they’re deeply familiar with the code.
• Bring in Temporary Test Writers: Alternatively, consider hiring external specialists who focus solely on writing tests. This ensures better test coverage at launch, though they’ll need some developer guidance to understand the feature’s functionality.

Add Tests from the Backlog Each Sprint

When new feature development isn’t consuming the QA team’s time, have them write tests for items in the backlog. Incrementally, this approach steadily increases test coverage.

Enhancing Accessibility Testing

Many excellent accessibility assessment tools are available that crawl a website and generate reports detailing identified issues. However, these reports are often delivered after the public has already begun interacting with the site. Can this process be shifted earlier?

Absolutely. If you’re using Cypress, their new accessibility feature enables earlier testing. For teams not using Cypress or those without enterprise-scale resources, you can integrate the axe-core accessibility library into your test suite.

Setting up the Automated Testing Kit is straightforward with the kit. Our configuration file allows you to specify the types of tests to run (or exclude) and provide a list of URLs. The configuration looks like this:

# Rules that are disabled.
disable:
 - color-contrast
 - landmark-no-duplicate-contentinfo
 - landmark-contentinfo-is-top-level
 - page-has-heading-one
 - region
# List of URL to check. Can specify additional rules to disable.
URL:
 /:
   disable:
     - heading-order
 /services:
 /how-we-do-it:
 /contact-us:

Listing 1. A typical atk_accessibility.yml configuration file. At the beginning, we disable certain test features. Following that are the URLs to test with specific accessibility tests we don't want on a page-by-page basis.

This approach enables you to identify and resolve accessibility issues before releasing new features.

Performance Testing/Audit

Similar to accessibility testing, there are robust tools available to evaluate your website’s performance. These tools crawl your site and generate reports detailing its performance metrics. However, as with accessibility tools, these reports often arrive after the site is live. Can performance issues be identified earlier?

By using the Google Lighthouse library with the Automated Testing Kit, they can. In the Kit, configure performance tests and specify a list of URLs. with a configuration file that looks like this:

thresholds:
 performance: 70
 accessibility: 90
 seo: 70
 best-practices: 70
URL:
 /:
 /services:
 /how-we-do-it:
 /contact-us:

Listing 1. A typical atk_audit.yml configuration file. Initially, we set thresholds; when test scores fall below these, the test fails. After that are the URLs to test. 

This allows you to address performance issues before releasing new features, just as you would with accessibility issues.

Visual Regression Testing

Both Cypress and Playwright offer visual testing capabilities, capturing snapshots of your website, comparing them, and flagging any differences. This straightforward approach can be effective for identifying graphical rather than functional issues.

Although both frameworks provide this sort of visual regression testing, it's often more convenient and easier to use a tool such as Diffy. Diffy makes it easy to compare snapshots, minimize false-positives and set up scheduling.

Use end-to-end testing to test user journeys and ensure that functionality is working correctly. Use visual regression testing when you want to spot layout issues, theme issues and other visual discrepancies. Teams can implement either one first.

Conclusion

Thanks for your interest in the Automated Testing Kit and for reading to the end! We've a wealth of resources in the Learning Resources section of the documentation, including videos of presentations we've given that demonstrate how the Kit is applied to real projects. If you have questions, ask on the #automated_testing_kit channel of the Drupal Slack workspace.