# Search engines and AI search: allowed User-agent: Googlebot User-agent: Bingbot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / # AI training and bulk scrapers: blocked User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: meta-externalagent User-agent: Amazonbot Disallow: / # SEO and competitive intelligence tools: blocked User-agent: AhrefsBot User-agent: AhrefsSiteAudit User-agent: SemrushBot User-agent: SiteAuditBot User-agent: SplitSignalBot User-agent: MJ12bot User-agent: DotBot User-agent: rogerbot User-agent: BLEXBot User-agent: DataForSeoBot User-agent: Screaming Frog SEO Spider User-agent: ia_archiver Disallow: / User-agent: * Allow: / Disallow: /api/ Crawl-delay: 10 Sitemap: https://news.longevitydocs.org/sitemap.xml