Origin: https://www.theguardian.com
robots.txt: https://www.theguardian.com/robots.txt (status 200, present True)
Bytes: 2873 · Crawl-delay: None · Sitemaps: 2
State: present
| Agent | Decision |
|---|---|
| * | allowed |
| gptbot | allowed |
| claudebot | disallowed |
| googlebot | allowed |
| perplexitybot | disallowed |
| ccbot | disallowed |
A robots.txt that could not be read is reported unknown, never allowed. Generated at 2026-10-02T08:18:18Z, dataset schema 1.0.1, seed version 1.
Full dataset: /v1/agents/policy (JSON) or /v1/agents/policy?format=csv (CSV). /check · every sampled origin: /policy.
Want more than this page? https://wodan-posture.app.goedvps.com/v1/agents/policy/origin?url=https%3A%2F%2Fwww.theguardian.com - paid, $0.03 per call: every group, every rule with its line number, which group governs each well-known agent and why, sitemaps and two digests.