RFC 9309 crawl decisions for a fixed, versioned sample of 24 public origins, each computed from that origin's own robots.txt by our own engine, refreshed on demand and cached. The sample is a fixed, versioned list, not a ranking or an endorsement.
24 of 24 origins analysed; claudebot disallowed on 11, gptbot disallowed on 10, perplexitybot disallowed on 10, ccbot disallowed on 10, * disallowed on 4, googlebot disallowed on 1.
Full dataset: /v1/agents/policy (JSON) or /v1/agents/policy?format=csv (CSV). See the same analysis for one URL at /check. Want more than the sample? GET /v1/agents/policy/origin?url=... - paid, $0.03 per call: every group, every rule with its line number, which group governs each well-known agent and why, sitemaps and two digests.