Skip to main content
Citegrade

Free tool

Is your robots.txt blocking the AI crawlers you want citing you?

An assistant can only cite a page it is allowed to fetch. This reads your robots.txt the way a crawler does and tells you which AI user agents it lets through — including the ones people block by accident while trying to stay out of training data.

No account, no email. One request to the site's robots.txt.

Blocking training and blocking citations are different decisions

Most advice about AI and robots.txt treats every bot as one thing. They are not. OpenAI, Anthropic and Perplexity each run separate crawlers for separate jobs, and the rule you write affects only the ones you name.

Training crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot — collect pages that future models learn from. Blocking them is a reasonable position to hold about your writing being used as training data, and it costs you nothing in citations today.

Search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot — fetch pages so an assistant can cite them in an answer it is composing right now. Blocking one of these removes you from consideration entirely, no matter how good the page is.

The common mistake is wanting the first and getting the second by accident: a broad rule written to keep work out of training data takes the live citations with it. If you want both, name them separately.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Stay out of OpenAI's training data while remaining citable in ChatGPT search, Claude and Perplexity.

The crawlers this checks

Each of these publishes its user agent and follows the standard robots.txt rules. Tokens are matched exactly, so a rule forGPTBot does not apply to OAI-SearchBot.

AI search

These fetch pages so an assistant can cite them in an answer. Blocking one removes you from consideration for citations being generated right now.

  • OAI-SearchBot · OpenAI

    ChatGPT search cannot retrieve this page when answering a question.

  • PerplexityBot · Perplexity

    Perplexity cannot index the page, so it cannot appear as a source.

  • Claude-SearchBot · Anthropic

    Claude cannot retrieve this page when it searches the web.

User-requested

These fetch a page because someone asked the assistant about that specific URL. Blocking one means the assistant cannot read your page even when a reader hands it the link.

  • ChatGPT-User · OpenAI

    ChatGPT cannot open this page even when a user pastes the link.

  • Claude-User · Anthropic

    Claude cannot open this page even when a user pastes the link.

  • Perplexity-User · Perplexity

    Perplexity cannot open this page on a user's request.

Model training

These collect pages that future models learn from. Blocking one is a legitimate choice about your work being used as training data — it just does not affect whether you get cited today.

  • GPTBot · OpenAI

    The page is excluded from OpenAI model training data.

  • ClaudeBot · Anthropic

    The page is excluded from Anthropic model training data.

  • Google-Extended · Google

    The page is excluded from Gemini training and grounding.

  • meta-externalagent · Meta

    The page is excluded from Meta AI training data.

  • CCBot · Common Crawl

    The page is left out of Common Crawl, which feeds many public datasets.

  • anthropic-ai · Anthropic

    Legacy Anthropic token, still honoured by some tooling.

Things that catch people out

A rule for one bot does not cover the others

Robots.txt matches the user-agent token exactly. Blocking GPTBot leaves OAI-SearchBot, ChatGPT-User and every other token untouched — which is usually what you want, but only if you meant it.

Allow beats Disallow when it is more specific

Disallow: / followed by Allow: /blog/ does not block your blog. The longest matching rule wins, so the blog stays crawlable. Plenty of checkers get this wrong and report a total block; this one follows the actual rule.

A group without a blank line before it is still a new group

A User-agent: line that follows a rule starts a fresh group. Rules do not carry over from the block above, even when the file has no blank line separating them.

Rules can differ by path

Checking a domain checks its root. If you disallow /docs/ for a crawler, paste a docs URL to see that.

Not every crawler obeys the file

Robots.txt is a request, not a control. The crawlers listed here document their tokens and follow the rules; others do not, and no robots.txt will stop those. Blocking at the edge is a different job from this one.

Access is the floor, not the goal

Letting a crawler in is necessary and nowhere near sufficient. Most pages that go uncited are perfectly crawlable — the answer is buried, the claims are hedged, or nothing on the page can be lifted out and quoted on its own. That is what a readiness audit measures.

Audit a page freearrow_forward

Two audits a month on the free plan. See how scoring works.