robots.txt blanket-blocked all AI crawlers, including on-demand answer engines
robots.txt Disallow: / applies uniformly to both bulk-training scrapers and on-demand answer-engine crawlers
QueryKey Cases's robots.txt previously treated all AI-related crawlers identically with a blanket Disallow, lumping bulk-training scrapers together with on-demand answer-engine crawlers that fetch a specific page at query time on a user's behalf rather than for bulk training. This blocked the platform from being citable by answer engines even though the product markets directly to AI/automation teams.
Root Cause
The crawler policy did not distinguish between crawler categories with fundamentally different behavior and licensing implications - bulk training scrapers ingest content wholesale for model training, while on-demand crawlers fetch a single page per user query and do not retain it for training.
Solution Steps
- Keep bulk-training scrapers (GPTBot, CCBot, anthropic-ai, cohere-ai) blocked to protect CC-BY-4.0 content from uncredited training reuse.
- Allow on-demand/answer-engine crawlers (ChatGPT-User, Claude-Web, PerplexityBot) since they fetch a specific page at query time on a user's behalf, and add a comment explaining the distinction so it isn't accidentally reverted to a blanket block.
- Confirm the on-demand crawler group is genuinely allowed.
Validation
[object Object]
Rollback
[object Object]
Ecosystem: nodejs
License: CC-BY-4.0
Last updated: Jul 6, 2026