What is an AI crawler access test?

It reads your robots.txt and evaluates each AI crawler user-agent against its rules, so you can see which crawlers are allowed to read your site and which are blocked.

Does blocking GPTBot remove my site from ChatGPT answers?

Not by itself. GPTBot relates to training. ChatGPT search uses OAI-SearchBot, and live fetches use ChatGPT-User, so each needs its own rule.

Does blocking Google-Extended affect Google Search or AI Overviews?

No. Google-Extended only controls use of your content for Gemini and Vertex AI. Search and AI Overviews follow Googlebot.

Does robots.txt guarantee a crawler stays out?

No. Robots.txt is a request that well-behaved crawlers honor. Some user-initiated fetchers may treat it differently, so use server-level controls if you need enforcement.

Free tool

AI crawler access checker: test your robots.txt

One line in your robots.txt decides whether AI platforms read your site at all. Enter your domain. Pineprompt reads your robots.txt and reports, crawler by crawler, whether the crawlers behind ChatGPT, Claude, Perplexity, and Gemini are allowed in.

allowed, blocked

Read from No robots.txt found, so everything is allowed by default

Free to use, up to 3 times a day. The result reads robots.txt only; it does not guarantee a given platform honors the file.

Definition

AI crawlers read the open web before they answer.

AI crawlers are the bots that AI companies run to read the open web. Some gather pages to train a model; others fetch pages live to ground an answer. Each one identifies itself with a user-agent your robots.txt allows or blocks.

If your robots.txt disallows GPTBot, OpenAI's crawler will not read your site, so your pages are less likely to inform a ChatGPT answer. The same logic holds for ClaudeBot, PerplexityBot, and Google-Extended. A block you added years ago, or one a plugin added for you, can quietly keep you out of AI answers today.

The checker resolves your domain, fetches /robots.txt, and evaluates each crawler against the rules it finds, reporting whether each one may read your site root.

Which AI crawlers exist, and what each one does.

Different crawlers do different jobs, and each has its own robots.txt rule. Blocking one does not block the others.

User-agent	Operator	What it does
`GPTBot`	OpenAI	Collects content to train OpenAI models.
`OAI-SearchBot`	OpenAI	Powers ChatGPT Search results.
`ChatGPT-User`	OpenAI	Fetches pages when ChatGPT browses on request.
`ClaudeBot`	Anthropic	Collects content to train Claude.
`anthropic-ai`	Anthropic	Anthropic content crawler.
`Claude-SearchBot`	Anthropic	Improves Claude search result quality.
`Claude-User`	Anthropic	Fetches pages when a Claude user asks.
`PerplexityBot`	Perplexity	Indexes pages for Perplexity answers.
`Perplexity-User`	Perplexity	Fetches pages when a Perplexity user asks.
`Google-Extended`	Google	Controls use in Gemini and Vertex AI.
`Applebot-Extended`	Apple	Controls use in Apple Intelligence.
`Amazonbot`	Amazon	Improves Amazon services and may train Amazon AI models.
`Bytespider`	ByteDance	Trains ByteDance and TikTok models.
`CCBot`	Common Crawl	Open crawl that feeds many models.
`Meta-ExternalAgent`	Meta	Collects content to train Meta AI.
`cohere-ai`	Cohere	Fetches pages for Cohere assistant requests.
`DuckAssistBot`	DuckDuckGo	Retrieves pages for DuckAssist answers.
`MistralAI-User`	Mistral	Fetches pages when a Le Chat user asks.

How to read your results

Allowed means your robots.txt does not stop that user-agent from reading your site root. Blocked means a rule disallows it. A missing robots.txt allows everything. Training crawlers and search or live-fetch crawlers are separate decisions: you can block GPTBot and ClaudeBot while allowing OAI-SearchBot, ChatGPT-User, and PerplexityBot, so your pages can still be cited in answers.

Frequently asked

What is an AI crawler access test?: It reads your robots.txt and evaluates each AI crawler user-agent against its rules, so you can see which crawlers are allowed to read your site and which are blocked.
Does blocking GPTBot remove my site from ChatGPT answers?: Not by itself. GPTBot relates to training. ChatGPT search uses OAI-SearchBot, and live fetches use ChatGPT-User, so each needs its own rule.
Does blocking Google-Extended affect Google Search or AI Overviews?: No. Google-Extended only controls use of your content for Gemini and Vertex AI. Search and AI Overviews follow Googlebot.
Does robots.txt guarantee a crawler stays out?: No. Robots.txt is a request that well-behaved crawlers honor. Some user-initiated fetchers may treat it differently, so use server-level controls if you need enforcement.

Setup

Let the AI crawlers in.

Pages an AI crawler cannot read never inform an AI answer. Allow the major crawlers and your content stays in the running. The following robots.txt block allows the most common ones explicitly.

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Allowing a crawler is a choice. Some teams deliberately block training crawlers while allowing the live retrieval ones. The checker shows you exactly where you stand before you decide.

More free GEO tools.

Open the door to the AI crawlers, then check the rest of your AI search readiness with the other free tools.

GEO readiness checker LLMs.txt generator Schema markup generator What is GEO? Citation tracking All free tools

Allowed in, but are you cited?

Letting the crawlers in is the first step. Pineprompt tracks whether the platforms then actually name and cite your brand, refreshed daily across six live AI platforms.

See pricing Get started