robots.txt & Sitemap
A robots.txt with AI Content Signals, plus a matching sitemap.xml. Declare how crawlers and AI may use your site.
Content Signals declare whether your pages may appear in search, be used as AI input (e.g. answering a user's question), or train AI models.
Adds Disallow: / for GPTBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent
robots.txt
User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Enter your site URL to generate the sitemap