# robots.txt for JohnsonForSenate.us # Campaign site for Brendan J. Johnson — Michigan State Senate District 26 # Last updated: April 20, 2026 # ----------------------------------------------------------------------------- # Content Signals # ----------------------------------------------------------------------------- # Content-Signal directives indicate how the site owner prefers content be # used. Well-behaved crawlers should honor these preferences alongside the # standard Allow/Disallow directives below. Content-Signal: search=yes, ai-input=yes, ai-train=yes # ----------------------------------------------------------------------------- # Allowed: Traditional search engines # ----------------------------------------------------------------------------- User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-News Allow: / User-agent: Bingbot Allow: / User-agent: DuckDuckBot Allow: / User-agent: Slurp Allow: / User-agent: YandexBot Allow: / # ----------------------------------------------------------------------------- # Allowed: AI/LLM crawlers (Western-origin) # ----------------------------------------------------------------------------- User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-Web Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: FacebookExternalHit Allow: / User-agent: Amazonbot Allow: / User-agent: MistralAI-User Allow: / User-agent: CCBot Allow: / User-agent: cohere-ai Allow: / User-agent: Diffbot Allow: / User-agent: TwitterBot Allow: / # ----------------------------------------------------------------------------- # Blocked: Chinese-origin AI/LLM # ----------------------------------------------------------------------------- # Note that some crawlers listed here (notably Bytespider # and DeepSeek) have been reported to ignore robots.txt # directives. Enforcement of those will require server-level or # firewall rules beyond this file. User-agent: Manus-User Disallow: / User-agent: Bytespider Disallow: / User-agent: TikTokSpider Disallow: / User-agent: PanguBot Disallow: / User-agent: PetalBot Disallow: / User-agent: DeepSeekBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: Baiduspider-render Disallow: / User-agent: Baiduspider-image Disallow: / User-agent: 360Spider Disallow: / User-agent: Yisouspider Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Sogou inst spider Disallow: / # ----------------------------------------------------------------------------- # Default: allow all other crawlers # ----------------------------------------------------------------------------- User-agent: * Allow: / # ----------------------------------------------------------------------------- # Sitemap # ----------------------------------------------------------------------------- Sitemap: https://johnsonforsenate.us/sitemap.xml