New: Use Eli in ChatGPT and Claude
ProductAI SearchWebsite ConversionAI AgentsPricingResearchSign inCheck my website

Free tool · No signup

Can AI crawlers and agents actually reach your site?

We read your robots.txt, response headers, redirects and rendering, then request your homepage as each crawler. You get a simple matrix: allowed, blocked or unclear, with the evidence.

Usually 5–15 seconds. No AI calls, no signup.

What it checks

A crawler-by-crawler access matrix with the evidence line for each.

01

Nine crawlers and agents

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot and Bingbot.

02

Two kinds of evidence

What your robots.txt says for each crawler, and what your server actually returned when asked with that crawler’s user agent.

03

Beyond robots.txt

HTTP status, redirect chain, robots meta, X-Robots-Tag, canonical, JavaScript dependency and firewall or CDN challenge signals.

When to use it

Example use cases

  • After enabling a CDN bot-protection setting: confirm you did not block AI search crawlers by accident.
  • When AI answers describe an outdated version of your product: check whether crawlers can still read the site.
  • Before deciding a crawler policy: see exactly what is allowed today.

Methodology

How the result is produced

Analysis version 1.0.0 · Scoring methodology 2026-10-04.1 · Updated 2026-10-04

  1. robots.txt is parsed per RFC 9309: the group naming the crawler wins, otherwise the “*” group; the longest matching Allow or Disallow rule decides.
  2. Each crawler’s user agent is used for one homepage request. A block status that a normal browser request does not receive is reported as a live block.
  3. This is a user-agent test from Eli’s servers. Sites that verify crawlers by IP address may treat the real crawler differently, so a live result that disagrees with robots.txt is reported as “Unclear”, not as a verdict.
  4. llms.txt is detected and reported for information only. It is not an established ranking or inclusion requirement and carries no score weight.

If something cannot be tested, the report says “Not tested” or “Unable to verify”. Uncertainty is never turned into a score.

FAQ

Questions about this tool

Should I allow every AI crawler?

That is a business decision. Training crawlers (GPTBot, ClaudeBot, Google-Extended) and search or user-triggered agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) do different jobs. Blocking the search agents removes you from answers buyers see.

Do I need an llms.txt file?

It is optional. No major AI provider has confirmed it as a ranking or inclusion requirement. We report whether it exists and do not score it.

Why does a crawler show “Unclear”?

Either robots.txt and the live response disagree, or the live request could not be verified. Real crawlers come from known IP ranges that a third-party test cannot use.

The suite

Related free tools

All free tools
AI Business ReadinessTest whether buyers, and the AI systems working for them, can find, understand, evaluate and act on your website.Open tool →AEO / GEO Page GraderGrade one page as an answer and recommendation source for AI-assisted buying research, with transparent rules.Open tool →

A free check finds the problem. Eli keeps fixing it.

Run the free check first. If the gaps matter, connect Eli and let it keep working on them.

Connect Eli