Citevana

Free tool · no sign-up

Which AI crawlers can access your website?

Enter your domain and we read your robots.txt the way crawlers do (RFC 9309), then show whether each major AI user agent is allowed, partly blocked or blocked. AI search and user-requested fetchers need access for your pages to be cited; training crawlers are a separate choice you can make.

Last reviewed September 16, 2026

AI crawler checker

Public websites only. We fetch the page once as CitevanaBot and don't store its content.

How it works

  1. We fetch /robots.txt from your domain and parse its user-agent groups, allow and disallow rules, wildcards and sitemap lines.

  2. For each AI user agent we apply the most specific matching group (or the * group) and the longest matching rule, for your homepage and a few common paths.

  3. You get a plain-language summary, a ranked to-do list, and a table with each bot's operator, purpose and status, linked to the operator's own documentation.

Frequently asked questions

What is an AI crawler?

It is an automated client run by an AI company. Some collect content for model training (for example GPTBot or ClaudeBot), some build search indexes for AI answers (for example OAI-SearchBot, Claude-SearchBot or PerplexityBot), and some fetch a page only when a user asks (for example ChatGPT-User or Claude-User).

Should I block AI crawlers?

It depends on your goals. If you want to be cited in AI answers, allow the search and user-requested agents. Blocking training crawlers is a separate business decision; operators such as OpenAI, Anthropic and Google document separate tokens so you can choose.

How do I block only AI training?

Add a group for each training token, for example 'User-agent: GPTBot' followed by 'Disallow: /'. Google documents that its Google-Extended token controls use for Gemini training and grounding and does not affect Google Search ranking.

Does robots.txt guarantee a bot stays out?

No. robots.txt is a voluntary standard. The operators listed here say they respect it, but it is not access control; use authentication or firewall rules for content that must stay private.

Is the Citevana AI crawler checker free?

Yes. It is free to use without an account, with a generous daily limit to prevent abuse.