Free AI robots.txt generator
Pick which AI crawlers may read your site and copy the robots.txt rules. Block model training while staying visible in AI answers. No account required.
-
:
Your robots.txt rules
Everything runs in your browser, so nothing you type is sent anywhere. Add the rules to the robots.txt file at your site root.
Block training, not search.
AI companies run separate crawlers for separate jobs.
Training crawlers such as GPTBot, ClaudeBot, and Google-Extended collect content to build models. Blocking them keeps your pages out of future training data.
Search and retrieval crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot index pages so assistants can cite them in live answers. If you block them, the assistant cannot read your page, so you drop out of those answers. That is the cost of "Block all" for any brand that wants to be recommended.
User-initiated fetchers such as ChatGPT-User and Claude-User load a page when a person asks about it. Some operators treat these as user requests rather than crawling, so robots.txt may not bind them. Allow them unless you have a reason not to.
Check the result with the AI crawler access checker, and point allowed crawlers at your best pages with an LLMs.txt generator file.
See whether AI answers actually mention you.
Pineprompt tracks your visibility, sentiment, and share of voice across six live AI platforms, refreshed daily.