Dries Buytaert’s Analysis Shows AI Crawlers Read Far More Than They Send Back

Recent analysis by Dries Buytaert examines how AI crawlers interact with web content after he made every page on his personal site available in Markdown format. The experiment aimed to determine whether offering a lightweight, AI-friendly format would change how AI systems crawl or reference content.

Reviewing a month of Cloudflare logs, Buytaert found that AI crawlers fetched large volumes of pages while sending almost no traffic back in return. For every citation generated by AI answer engines, their bots had crawled roughly 1,241 pages. The experiment also showed that providing Markdown versions did not reduce crawler traffic. Instead, bots typically fetched both HTML and Markdown versions, slightly increasing overall crawl activity.

The analysis also examined emerging conventions intended to guide AI systems, including the proposed llms.txt file. According to the data, requests for this file came almost entirely from SEO auditing tools rather than AI crawlers. Buytaert concludes that while Markdown may be useful for experimentation, there is currently little evidence that it improves how AI systems reference content, suggesting that clear writing and authoritative publishing remain the most reliable strategy.