How AI Crawlers Are Disrupting Government Websites

Staff Reporter

Matt West of Forum One highlights a growing issue affecting government websites: aggressive AI crawlers from platforms like OpenAI, Anthropic, and Perplexity are overloading .gov infrastructure. These bots often bypass robots.txt, exploit sitemap and search pages, and trigger excessive dynamic page requests, leading to cache misses, timeouts, and inflated server costs.

West outlines detection methods using tools like New Relic, Cloudflare, and Drupal logs. He offers mitigation strategies such as edge-level blocking, cache tuning, and rate limiting. For Drupal sites, he recommends modules like Bot Blocker, cache configuration updates, and reduced reliance on uncacheable views and filters.

The article promotes West’s Drupal GovCon 2025 session on this topic, emphasizing the need to balance AI discoverability with infrastructure protection. Published by Forum One, the post serves as a practical guide for public-sector teams managing Drupal performance under AI crawler pressure.

Reference: AI Crawlers Are Breaking .Gov Websites — And What You Can Do About It by Matt West, Forum One (12 August 2025)

Government Websites

Drupal

Disclosure: This content is produced with the assistance of AI.

Disclaimer: The opinions expressed in this story do not necessarily represent that of TheDropTimes. We regularly share third-party blog posts that feature Drupal in good faith. TDT recommends Reader's discretion while consuming such content, as the veracity/authenticity of the story depends on the blogger and their motives.

Note: The vision of this web portal is to help promote news and stories around the Drupal community and promote and celebrate the people and organizations in the community. We strive to create and distribute our content based on these content policy. If you see any omission/variation on this please reach out to us at #thedroptimes channel on Drupal Slack and we will try to address the issue as best we can.