Robots.txt Tester

The decision is a shortcut. If the test URL contains the letters admin, the label is disallowed and the matched line is shown as Disallow: /admin/. If it contains private, the label is disallowed for /private/. Any other URL is labeled allowed and the matched line is shown as Allow: /. The pasted robots.txt is not the rule source.

Allowed is one of the values Robots.txt Tester puts on screen. The label when the URL does not contain admin or private. The matched line is shown as Allow: /. A Disallow in your paste is ignored.

Read Disallowed on its own before you mix it with the other rows. The label when the URL contains admin or private. The matched line is Disallow: /admin/ or Disallow: /private/. The path does not have to be under those folders. The letters anywhere in the URL are enough.

User-agent answers a narrower question than the headline number. A real file can have different rules per crawler. You can pick Googlebot, Bingbot, and other names. The pick does not change the shortcut. The same URL gets the same label for every agent.

Treat Pasted file as a label with a specific job. The textarea holds robots.txt text. It is not parsed into rules. Editing the paste will not change the label.

The No fetch line is worth a full stop. The URL is not requested. Nothing is learned about whether the host serves the file. A live tester has to request /robots.txt and apply it.

A real robots test applies the file’s user-agent groups, Allow and Disallow prefixes, and end-of-match rules. This page does not do that. Use it only to see the shape of an allowed or disallowed result. Publish rules with the robots.txt generator, then test them with a parser that reads the file.

If you remember one sequence from Robots.txt Tester, remember the fields in the order they change a decision. Allowed matters because The label when the URL does not contain admin or private. In practice, The matched line is shown as Allow: /. The mistake to avoid is this: A Disallow in your paste is ignored. Disallowed matters because The label when the URL contains admin or private. In practice, The matched line is Disallow: /admin/ or Disallow: /private/. The mistake to avoid is this: The path does not have to be under those folders. The letters anywhere in the URL are enough. User-agent matters because A real file can have different rules per crawler. In practice, You can pick Googlebot, Bingbot, and other names. The mistake to avoid is this: The pick does not change the shortcut. The same URL gets the same label for every agent. Pasted file matters because The textarea holds robots.txt text. In practice, It is not parsed into rules. The mistake to avoid is this: Editing the paste will not change the label. No fetch matters because The URL is not requested. In practice, Nothing is learned about whether the host serves the file. The mistake to avoid is this: A live tester has to request /robots.txt and apply it. After that, the checks are simple. You did not treat the paste as the rule source. You saw that admin anywhere in the URL flips the label. You did not change live robots.txt from this result. The user-agent menu was not treated as a real group. A parser that reads the file is the next test.

The worked example for Robots.txt Tester, read as one scene, is this. Leave the sample paste in the box and test https://example.com/blog. The label is allowed, matched to Allow: /, even if you change the paste to Disallow: /. Test https://example.com/admin-guide. The label is disallowed because the URL contains admin, matched to Disallow: /admin/, even if that path would be allowed by a real file.

Keep Robots.txt Tester as this step only. When the job moves on, the Robots.txt Generator is the next page: Robots.txt Generator builds a robots.txt file from an allow or disallow default, optional paths, a crawl delay, and a sitemap URL. The XML Sitemap Generator covers a different piece of the same work: XML Sitemap Generator writes one urlset entry from a page URL, a last-modified date, a change frequency, and a priority.