Paste a robots.txt (yours or anyone's — it's at
/robots.txt on every site) and see exactly what it asks of each AI crawler:
training bots that feed models, retrieval bots that fetch to answer a live prompt, and AI
search indexers. Blocking each has a different consequence — the verdicts say which is which.
100% client-side — nothing is fetched or sent · works offline
Generate a policy instead
Pick a stance per crawler. A common posture:
block training, allow retrieval and search — your content stops feeding models but keeps
appearing (and being cited) in answers.
robots.txt is a request, not a wall. Compliant operators honor it; scrapers spoofing
those user agents do not. Enforcement means verifying published IP ranges or TLS fingerprints at your edge —
the method is in the AI crawler guide, and
ad.rip does it as a service.