# Client decks (/deck/*) and prospect demos are deliberately NOT listed here.
#
# They are kept out of search by `X-Robots-Tag: noindex` in public/_headers,
# which is the only mechanism that actually guarantees deindexing. Listing them
# here would be worse on two counts:
#
#   1. A Disallow blocks crawling, not indexing. Google can still index a
#      blocked URL that someone links to and show it as a bare result. Because
#      the page is never fetched, the noindex header is never read — so the
#      Disallow would defeat the very control that protects these pages.
#   2. robots.txt is world-readable. Naming every demo path here published a
#      list of client and prospect names to anyone who asked for the file.
#
# Do not re-add Disallow rules for those paths. To make a deck genuinely
# private rather than merely unlisted, put it behind Cloudflare Access — a
# noindex is a request to search engines, not an access control.

User-agent: *
Allow: /

# Block AI training scrapers (not discovery/answer bots — GPTBot, ClaudeBot,
# PerplexityBot and OAI-SearchBot are intentionally allowed so the site stays
# citable in answer engines).
User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

Sitemap: https://neothread.ai/sitemap.xml