# robots.txt for https://projectcode.dev # Strategy: welcome search + AI/answer-engine crawlers on public marketing pages # (SEO + AEO + GEO), block private app, account and internal routes everywhere. # Private routes are repeated per agent because most crawlers only obey the # FIRST matching User-agent group — global "*" rules do NOT inherit. # This Disallow list is hand-maintained (only the AI Discovery block below is # auto-patched by generate-ai-discovery.mts) — keep it in sync with that script's # PRIVATE_ROUTE_PREFIXES, which is the source of truth for what's excluded from # sitemap.xml. # --------------------------------------------------------------------------- # Traditional search engines # --------------------------------------------------------------------------- User-agent: Googlebot Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ User-agent: Googlebot-Image Allow: / User-agent: Bingbot Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ User-agent: DuckDuckBot Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ User-agent: Applebot Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ # --------------------------------------------------------------------------- # AI answer engines & assistants (AEO / GEO) # Allow so projectcode.dev can be cited in AI answers; private routes stay blocked. # --------------------------------------------------------------------------- # OpenAI — ChatGPT search index, user-triggered fetch, model training User-agent: OAI-SearchBot Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ User-agent: ChatGPT-User Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ User-agent: GPTBot Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ # Anthropic — Claude search, user fetch, crawler User-agent: ClaudeBot Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ User-agent: Claude-SearchBot Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ User-agent: Claude-User Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ User-agent: anthropic-ai Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ # Perplexity — answer engine + user fetch User-agent: PerplexityBot Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ User-agent: Perplexity-User Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ # Google Gemini / Vertex AI generative training User-agent: Google-Extended Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ # Apple Intelligence User-agent: Applebot-Extended Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ # Microsoft Copilot User-agent: CopilotBot Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ # DuckDuckGo AI (DuckAssist) User-agent: DuckAssistBot Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ # Amazon (Alexa / Rufus) User-agent: Amazonbot Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ # Meta AI User-agent: meta-externalagent Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ # Mistral AI user fetch User-agent: MistralAI-User Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ # Cohere User-agent: cohere-ai Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ # Common Crawl — feeds many open LLM datasets User-agent: CCBot Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ # --------------------------------------------------------------------------- # Social / link unfurl bots — allow so shared links render rich previews # --------------------------------------------------------------------------- User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / User-agent: LinkedInBot Allow: / User-agent: Slackbot Allow: / User-agent: WhatsApp Allow: / User-agent: TelegramBot Allow: / User-agent: Discordbot Allow: / # --------------------------------------------------------------------------- # Aggressive scraper bots with no SEO/AEO value — block entirely # --------------------------------------------------------------------------- User-agent: Bytespider Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Diffbot Disallow: / User-agent: PetalBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / # --------------------------------------------------------------------------- # Everyone else — public pages allowed, private routes blocked # --------------------------------------------------------------------------- User-agent: * Allow: / Disallow: /dashboard Disallow: /editor Disallow: /admin Disallow: /internal Disallow: /application-details Disallow: /api/ Disallow: /payment Disallow: /project/access Disallow: /editor-auth Disallow: /accept-invite Disallow: /tools/ Sitemap: https://projectcode.dev/sitemap.xml # --------------------------------------------------------------------------- # AI Discovery Layer - auto-appended by generate-ai-discovery.mts # --------------------------------------------------------------------------- LLMs-Txt: https://projectcode.dev/llms.txt LLMs-Full-Txt: https://projectcode.dev/llms-full.txt AI-Manifest: https://projectcode.dev/ai/manifest.json Schema: https://projectcode.dev/schema.json Knowledge: https://projectcode.dev/knowledge/index.json