SearchOrbit AI
Check a site’s crawl directives and the sitemap they point at, and ask whether a named crawler may fetch a particular URL. 10 free checks a day, no account needed.
SearchOrbit AI reads the file itself and reports what is in it — the groups, the rules, the declared sitemaps — using the same matching its own crawler obeys.
Point it at a website and it reads /robots.txt, reports every group and how many rules each one holds, lists the sitemaps the file declares, and validates the first of them. Point it at a robots.txt or a sitemap directly and it checks that file alone.
In the sitemap it looks at the root element and its namespace, whether every <loc> is the complete URL the protocol requires, duplicate entries, entries on another host, and whether <lastmod>, <changefreq> and <priority> hold values a consumer will actually read rather than silently drop. A sitemap index is followed to its children.
The eligibility test is not a second opinion. It runs SearchOrbit AI’s own RFC 9309 implementation — exact product-token group selection, most-specific rule wins, percent-octet normalisation — so the answer is a statement about what this crawler will actually do, not about how some other parser reads the file.
Where the file cannot be understood in full, it says so instead of guessing. The crawler stops in exactly that situation, and a validator that answered confidently where the crawler refuses would be describing a different product.
A robots.txt controls whether a crawler may fetch a URL. It does not decide whether Google indexes it: a disallowed URL can still appear in results if other pages link to it, and an allowed URL is under no obligation to appear at all. Nothing here reports on what is in anyone’s index — Search Console does that.
To see what a crawler makes of a page once it is allowed to fetch it, run a free SEO audit or check its meta tags.