Born out of a vision that every Drupal should have a suite of tests that ensure the basic operation works, version 1.0 of the Automated Testing Kit for Drupal showcases 18 beneficial tests and 15 utility functions, with further developments on the horizon.

You May Like

Article

Elevated Third's Free Webinar: Harnessing the Power of AI in Drupal

Article

Elevated Third's Free Webinar: Harnessing the Power of AI in Drupal

"My vision is for every Drupal site to provide the highest quality experience for visitors. A great way to achieve this is by regularly running a set of end-to-end tests that ensure key features are always working."

notes André Angelantoni, the project lead of the Automated Testing Kit, Layout Builder Kit, and Campaign Kit, in an email correspondence with Alka Elizabeth, a sub-editor with The DropTimes.

Automated Testing Kit contains useful utilities for end-to-end testing using Cypress.io and Playwright, allowing for transition between frameworks and the simultaneous execution of tests in both. 

André wrote 18 smoke tests and 15 functions in both Cypress and Playwright, a team runs these tests, and functions that are configured to run consistently across various stages, including local development machines, pull requests (utilizing a preview tool like Tugboat.qa), deployments, and nightly runs on their production site. The goal is to detect issues at their earliest stages, enabling quicker and more cost-effective resolution.

"The first 18 tests validate core features, including registration, editing nodes, sitemap regeneration and more. I’d love people to contribute more tests so that we get to 50 fundamental tests."

If asked about why each team should independently write these tests when they can create a central library of tests, André replies,

" The Automated Testing Kit will gain more tests over time. And, of course, teams will add their own tests unique to their site. Gradually, test coverage will expand as both test sets grow. The result? Fewer last-minute hotfixes need to be pushed to production, and a happy team that sleeps better at night."

Some of the most interesting aspects of the Automated Testing Kit include the use of 'Ethereal.email,' a free virtual email server that enables capturing test emails using unique tokens, ensuring accurate email confirmation during testing. Integrating Drush for server communication and utilising an SSH tunnel for Pantheon targets further exemplify the kit's adaptability to various server environments, including local, containerized, and remote setups. Moreover, the kit employs preprocess hooks to facilitate the confirmation of displayed nodes, terms, and images, enhancing the precision and reliability of the testing process.

"It was fun figuring out how to ensure that ATK would work with any architecture. There are two parts to that, the technical part and the design part."

André continued.

From a technical perspective, the Automated Testing Kit needs to be able to run the tests on any machine and target any machine. The tests can run on a developer machine directly on the host OS (macOS or Linux), inside a container, on GitHub, or in a test grid such as Testery.io. The tests can target any machine, whether on a developer machine or remote. They had to set up an SSH tunnel to target Pantheon because the typical Drush alias methods don’t work for Pantheon.

From a design perspective, the key is to set up proper tagging from the start. Each test has a category tag, such as @register-login and @user, along with one or more property tags, such as @alters-db and @smoke. With careful tagging, precisely the right tests can run on each target. This is important, for instance, when a test that modifies the database should not run on production.

"The other fun part was writing the tests in both frameworks; it’s an excellent way to learn about the strengths and drawbacks of each one. If a team is in the process of deciding which framework to use, they should take a look at the Learning Resources chapter of the Automated Testing Kit documentation. It contains a large collection of videos and blog posts comparing the two tools."

As mentioned, the kit is designed for end-to-end testing using Cypress.io and Playwright, introducing a wide range of tests and utilities, including logging in, logging out, user registration, resetting passwords, and Contact Us form testing. The Automated Testing Kit module also features tests for 404 and 403 errors, entity creation, update and deletion, sitemap testing, and a myriad of convenient functions such as retrieving usernames and Drupal UIDs.

The choice of Cypress and Playwright was one after enough probing. In the world of open-source end-to-end testing, Cypress currently has the most market share. Andre remembers it as a very good tool and the one he started with. However, one of its design choices causes new developers to learn non-standard programming techniques. This, unfortunately, slows down the learning process unnecessarily. Cypress also cannot control multiple browsers, which is a requirement for some testing projects. There is often a workaround for this last deficiency, but it’s one more thing to handle.

On the plus side for Cypress, certain highly technical tests are easier to write since it runs in the browser. Also, there is an astounding number of community resources. Almost everything has been figured out at this point. Resolving issues usually requires knowing where to find the solution (hint: https://cypress.tips) or selecting from the many available courses.

Playwright, which doesn’t have the impediments mentioned above, runs tests at least 3x faster, integrates excellently with Visual Studio Code, and responds faster in the issue queue, will take over the lead position in a few years. 

"Automated Testing Kit can be added at any point in the lifecycle of any project. Remember that the target server needs access to Drush for certain tests to work and for post-test cleanup to occur."

At this point in its development, the Automated Testing Kit consists of a module with hooks, a custom Drush command, a library of tests written in Cypress and Playwright, two custom Javascript utility libraries and extensive documentation.

André shares that the best time to start adding end-to-end tests is at the beginning of the project. It will help quickly catch breaking changes during development. Even better, developers can add tests to validate new features as the features are developed. Keeping the test suite always up-to-date reduces the chance of lurking bugs appearing just days before the launch.

Future enhancements are in the pipeline for the Automated Testing Kit Module, including confirming each entry in sitemap.xml, access to the translation dictionary t() from within a test, object cache tests, CKEditor tests, and search tests. The ultimate goal is to establish a standard test for each crucial aspect of Drupal, simplifying testing processes and providing modifiable examples for testers.

Those intrigued by the idea of starting an end-to-end testing initiative should watch Setting up a Comprehensive Automated Testing Regime for Drupal using Cypress and Playwright from the 2023 Stanford Web Camp.

Then, watch the demo of the Automated Testing Kit presented for Drupal Pupils.

"Additionally, when teams find themselves in a time or resource crunch, our team at Performant Labs can set up the testing project in the CI/CD pipeline using the Automated Testing Kit and write additional custom tests using one of the engagement packages offered. For some clients, it will be ideal for the client's team to take over test suite maintenance at the end of the engagement. For other clients, it will make more sense for us to maintain the tests for them."

concludes André Angelantoni, the project lead of the Automated Testing Kit.

If you are curious about end-to-end Automated Testing, Oliver Davies has released a free email course on how to write automated tests and do test-driven development in Drupal. Access more information here.