# StatShep robots.txt (policy 2026-10-03) # Full policy: https://statshep.com/ai-policy ยท For agents: https://statshep.com/llms.txt # # Search engines and AI search/answer engines: welcome. Index, quote and link. # Agents fetching a page because a person asked: welcome, including /api/v1/ and the MCP server at /mcp. # AI training: not permitted. Do not use this site's content to train or fine-tune AI models. # # Content signals (contentsignals.org): # search = building a search index and returning links and short excerpts: YES # ai-input = feeding content into AI models at answer time (retrieval, grounding, cited answers): YES # ai-train = training or fine-tuning AI models: NO # Under Article 4 of EU Directive 2019/790 we expressly reserve the right to text and data mining for # AI training; see the policy URL above (also /.well-known/tdmrep.json). User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Content-Usage: train-ai=n, ai-use=y, search=y Allow: / Disallow: /admin Disallow: /lookup/ Disallow: /r/ Disallow: /account Disallow: /workspace Disallow: /signin Disallow: /auth/ Disallow: /api/ Disallow: /mcp User-agent: Googlebot User-agent: Bingbot User-agent: Applebot User-agent: DuckDuckBot User-agent: YouBot User-agent: Amzn-SearchBot User-agent: Diffbot User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: DuckAssistBot User-agent: meta-webindexer User-agent: MistralAI-Index Content-Signal: search=yes, ai-input=yes, ai-train=no Content-Usage: train-ai=n, ai-use=y, search=y Allow: / Disallow: /admin Disallow: /lookup/ Disallow: /r/ Disallow: /account Disallow: /workspace Disallow: /signin Disallow: /auth/ Disallow: /api/ Disallow: /mcp User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: MistralAI-User User-agent: Amzn-User User-agent: meta-externalfetcher User-agent: Google-Agent User-agent: Diffbot-User Content-Signal: search=yes, ai-input=yes, ai-train=no Content-Usage: train-ai=n, ai-use=y, search=y Allow: / Allow: /api/v1/ Allow: /mcp Disallow: /admin Disallow: /lookup/ Disallow: /r/ Disallow: /account Disallow: /workspace Disallow: /signin Disallow: /auth/ User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Claude-Web User-agent: Google-Extended User-agent: GoogleOther User-agent: Google-CloudVertexBot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: FacebookBot User-agent: Amazonbot User-agent: MistralAI-Training User-agent: CCBot User-agent: Bytespider User-agent: cohere-ai User-agent: cohere-training-data-crawler User-agent: AI2Bot User-agent: Ai2Bot-Dolma User-agent: GrokBot User-agent: xAI-Grok User-agent: omgili User-agent: omgilibot User-agent: ImagesiftBot Content-Signal: search=yes, ai-input=yes, ai-train=no Content-Usage: train-ai=n, ai-use=y, search=y Disallow: / Sitemap: https://statshep.com/sitemap.xml