Tools / SEO
robots.txt Tester
Test whether a URL is crawlable for a given User-Agent based on robots.txt rules. Fetches your site's robots.txt, tests whether a given path is crawlable for the selected User-Agent and shows exactly which Allow/Disallow rule matched.
What is robots.txt?
robots.txt is a plain-text file at the root of your site (e.g. example.com/robots.txt) that tells search-engine bots which URLs they are allowed to crawl. It uses User-agent, Allow, Disallow, Crawl-delay and Sitemap directives. A bot downloads the file, picks the most specific group for its user-agent and behaves accordingly.
robots.txt is different from the robots meta tag. robots.txt blocks crawling but a URL that is already known can still appear in search results — Google can learn the URL from external links even if it can\'t fetch the page. To fully keep a page out of the index the page must include <meta name="robots" content="noindex">, and Google can only see that directive if the page is crawlable. See Google\'s robots.txt documentation for details.
About robots.txt
Disallow prevents the bot from downloading the page; noindex tells Google not to include it in the index. A page can't effectively use both — if Google can't crawl the page it can't see the noindex tag. To truly keep a page out of the index: remove the Disallow and add a noindex meta.
Crawl-delay tells the bot how many seconds to wait between requests. Bing, Yandex and Baidu honor it; Google does not — in Google you adjust crawl rate via Search Console instead. It's useful for small servers to avoid overload.
If robots.txt returns 404 (or any 4xx), Google assumes no restrictions and treats the whole site as crawlable. If it returns 5xx, Google treats the entire site as temporarily un-crawlable — that's dangerous and can cause big ranking drops. Make sure robots.txt reliably returns 200 and is cacheable.
Google and Bing: the longest matching rule wins (most specific). On a tie Allow takes precedence. Example: if /private/ is Disallowed but /private/public.html is Allowed, the second rule is longer so public.html can be crawled. This tool applies exactly the same matching logic.
Full technical SEO audit
robots.txt, sitemap, canonicals, Core Web Vitals, mobile usability and structured data — all in one report.