# Global Metro Power Rankings - robots.txt
# Permissive crawl policy: the dataset is meant to be read and cited.
# AI policy (ruled by Ashwin, 2026-11-03): no AI training, answer engines welcome.
# Plain-words statement: https://rankings.citizenofnowhere.org/ai-access

User-agent: *
Allow: /
Disallow: /admin
Disallow: /admin/
Disallow: /api/admin
Disallow: /api/admin/

# -------------------------------------------------------------------
# Search and answer engines: allowed. These are the agents that power
# real-time citation in ChatGPT Search, Claude web search, Perplexity,
# Google AI Overviews, Bing Copilot, etc. They read a page to answer a
# question and credit the source.
# -------------------------------------------------------------------

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: GoogleOther
Allow: /

User-agent: Applebot
Allow: /

User-agent: Meta-ExternalFetcher
Allow: /

User-agent: MistralAI-User
Allow: /

User-agent: YouBot
Allow: /

# -------------------------------------------------------------------
# AI training crawlers: opted out. Until 2026-11-03 these were on the
# allowlist; the ruling is that the site's data is read and cited, not
# used to train models. robots.txt is advisory, so this states policy.
# -------------------------------------------------------------------

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: cohere-ai
Disallow: /

# -------------------------------------------------------------------
# Denied. The agents below feed nothing that credits the source. They
# surfaced in the 2026-08-05/06 crawl (ShapBot 23.8k requests, SleepBot
# 3.3k) and the permissive "User-agent: *" rule at the top was letting
# them through.
#
# robots.txt is advisory, so this is a statement of policy, not an
# enforcement mechanism. Enforcement is a Cloudflare WAF custom rule
# matching the same two agents. Keep the two in sync: an agent added here
# and not there is decorative.
# -------------------------------------------------------------------

User-agent: ShapBot
Disallow: /

User-agent: SleepBot
Disallow: /

Sitemap: https://rankings.citizenofnowhere.org/sitemap.xml