Faceted Search Design Fails Under Automated Crawling in Drupal Implementations

Performance issues on Drupal search pages are being traced to how faceted navigation interacts with automated crawlers. In a DrupalCon Chicago lightning talk, Stephen Musgrave presented case studies showing how crawler traffic can trigger large volumes of uncached requests, leading to degraded performance and operational instability.

In an accompanying article published by Capellic, Musgrave describes recurring incidents across multiple sites rather than isolated spikes. In one case, daily visits approached 2 million while cache hit ratios dropped below 20%, indicating that most requests were bypassing caching layers and hitting the database directly. He attributes this pattern to bots systematically traversing every possible combination of search facets.

Faceted search in Drupal, commonly implemented using the Facets module, generates a distinct URL for each filter combination. These URLs typically include query parameters such as f[0], f[1], and related keys representing selected taxonomy terms or filters. When facets are rendered as links, each parameter combination becomes a crawlable endpoint. Automated systems can iterate through these combinations at scale, producing a large number of uncached requests that must be resolved at the database level.

Short-term mitigation strategies focus on controlling incoming requests. Web application firewalls (WAFs) and CDN-level protections can block or challenge traffic matching facet query patterns, for example by targeting requests containing f%5B0%5D in the query string. Additional approaches include contributed modules such as Bot Blocker, which filters requests based on user-agent signals, and facet-specific controls that limit parameter combinations. However, these methods depend on detectable patterns and may be less effective against distributed or spoofed traffic.

The longer-term solution outlined in both the talk and article shifts focus from traffic control to application design. Recent versions of the Facets ecosystem, including the Facets Exposed Filters sub-module, allow facets to be rendered as form elements instead of links. In this configuration, facet selections are integrated into the Views exposed filter form rather than generating standalone URLs. This changes the interaction model from link traversal to form submission, reducing the number of crawlable endpoints.

This approach typically involves reconfiguring facet blocks to use exposed filters and, in many implementations, coordinating with modules such as Better Exposed Filters to control rendering and user interface behaviour. While the refactor can require front-end adjustments and careful alignment with existing Views configurations, it alters how query parameters are generated and processed. Because automated crawlers generally do not submit forms, the change limits systematic traversal of filter combinations without relying on traffic filtering alone.

In the cases presented, this architectural shift improved cache hit ratios to above 90% and stabilised traffic patterns. The findings suggest that performance issues in these scenarios stem less from traffic volume alone and more from how search interfaces are exposed to automated access, particularly when URL-based filtering creates an effectively unbounded crawl surface.