AI Visibility Checker

Publishers keep asking whether a page is visible to the new crawlers, and the honest answer is a set of files you can open yourself. AI Visibility Checker fetches the URL you typed, then requests robots.txt, llms.txt, and llms-full.txt on that same origin. This check does not query a chatbot. Nothing in the result is a reply from a model, a citation rank, or a guess about how often a bot will visit. If a row is red, the evidence is a header, a meta tag, or a line from a file, and you can compare that line with the file on the server.

The crawler list is fixed so the math stays visible. The eleven names are GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, Amazonbot, CCBot, Applebot-Extended, Bytespider, and Meta-ExternalAgent. For each name the parser selects the most specific User-agent group, falling back to the star group when that bot has no group of its own. A rule matches from the start of the path. An asterisk in a pattern matches across characters, and a dollar sign at the end means the path must end there. The longest matching pattern wins. When an Allow and a Disallow are the same length, Allow wins. If nothing matches, the default is allow. The summary then says how many of those eleven are blocked, for example one of eleven blocked and ten of eleven allowed, so you can see the fraction without a second number hiding it.

A missing robots.txt is not treated as a refusal. When the host answers 404 or 410, every named crawler is marked allowed and the row is a warning. The warning exists because you may have meant to publish a file and the server is serving a soft gap, not because the crawlers are blocked. An empty 200 is different: that file was published and it simply has no rules, so the default allow stands as a pass when nobody is blocked. If the response is HTML, a browser challenge, or a status that is not a successful text file, the row fails and says the file could not be read. In that case the summary does not pretend to count blocks. Gzip is decoded before any group is read, using the same decoder the sitemap tools use for a compressed robots.txt.

llms.txt is a convention, not a ranking factor this page invents. The check requests /llms.txt on the origin of the page that was actually fetched. It passes when the body, after a trim, starts with a markdown H1 such as a line that reads # Name, and the body is not an HTML document. The first heading is printed so you can see which title the file is offering. A 404 or 410 is a warning. Missing is not a fail, because many useful sites have not published the file yet and a missing guide is not the same as a block. A file that comes back as a full HTML page, including a challenge page that happens to use that path, fails the row. The failure is about the shape of the body, not about a model refusing the site.

llms-full.txt is the longer companion and it is optional here. The same H1 test is used, and the first heading is shown when the file matches. If the path 404s, the row is a note. A note is recorded so you can see that the optional file is absent, and it is not counted as a failure of the page. An HTML body at that path still fails, because a challenge page should not be mistaken for the longer markdown file. You can publish the short file without the long one. The checker will not scold you for that choice.

The page itself is read for the robots meta element and for the X-Robots-Tag header when the fetch exposes one. noindex fails that check, including the token none, which means the same thing for indexing. nosnippet, noai, and noimageai are warnings. They limit what a consumer may quote or reuse, but they are not the same decision as noindex, so they do not flip the summary into the noindex sentence. If both a hard noindex and a softer token are present, the row fails and the evidence still lists every token that was found. When neither the meta element nor the header is present, the row passes and says so. A page that never came back, because the status was not successful, fails this check with the status rather than claiming the page is indexable.

Extractable content is a bundle of signals taken from the HTML after the fetch. You see the title element, the exact text of the H1, the meta description, a rough word count of the main text after script and style elements are removed, and whether a JSON-LD script is present. The H1 row warns when the heading is missing and when more than one H1 is present, because a crawler that is building a title from the page then has to choose. The word count is a count of words left in the body, not a quality grade and not a promise that a model will quote them. JSON-LD is recorded as present or not found. It is not validated against a rich-result gallery. If you need a structural check of a script you already wrote, that work belongs on a schema tool, not inside this visibility list.

The canonical link is reported as present or absent, and the check says whether the absolute address points at another host. A same-host canonical passes. A missing canonical is a warning, because the page can still be fetched but you have not named the preferred URL. A canonical on a different host is also a warning, and the evidence prints that absolute URL so you can see the host yourself. Relative hrefs are resolved against the fetched page before the hosts are compared. The sitemap hint is smaller than a sitemap audit. If robots.txt contains one or more Sitemap lines, those URLs are printed and explicitly not fetched. This page does not walk child sitemaps, does not count the URLs inside them, and does not decide whether the page you typed appears in the file. When robots.txt is missing, the sitemap row is a note. When the robots file could not be read, the sitemap row says that too.

The summary line is the only total on the page, and it is a sentence rather than a badge out of one hundred. It states how many of the eleven crawlers are blocked, whether llms.txt exists, and whether the page is noindex. When robots.txt cannot be read, the sentence says the crawler rules were not counted, instead of printing a zero that would look like a clean allow. There is no AI score, no ChatGPT rank, and no traffic estimate. If you want a different cut of the same host, the Robots.txt Sitemap Checker compares Sitemap lines with a sitemap address, the LLMs.txt Checker + Validator reads markdown you paste, and the Sitemap Checker looks at one sitemap URL. None of those pages queries a chatbot either.

The fetch has the same walls as the other tools that download a public file on this site. The URL has to be http or https, with no username or password in it. Loopback, private ranges, link-local addresses, and the cloud metadata hosts are rejected by the existing guard, including 127.0.0.1 and 169.254.169.254, and a redirect that lands on one of those addresses is rejected again. Responses stop at five megabytes. Redirects stop after a few hops. A timeout ends the attempt. The HTML that is kept is the body that already fit under that cap. The checker does not open a second client of its own, so a rule you rely on for sitemap fetches is the same rule here. If the host answers with a challenge page, you will see that the file could not be read, which is the useful outcome: you learn that the check did not silently treat a block page as permission.