AI Crawlers Are Crushing Your Website: Here's What You Can Do About It

24 Mar 2026, 6:30 pm (UTC)
50 Minutes

AI-powered crawlers scrape public government websites to feed large language models. They can hit sites quickly, often ignore robots.txt, and overwhelm infrastructure that is not built for that traffic. The session examines how these crawlers target sites, what patterns to watch for, and what happens when a site cannot keep up.

It covers ways to detect, throttle, or block crawlers using CDNs, WAFs, and server-level protections, as well as Drupal settings for caching, permissions, and crawl behavior. The session also addresses when blocking is appropriate, what public content should be exposed, and how to discuss risks with stakeholders, alongside a checklist of protective actions.

Other Sessions