Tools / SEO
robots.txt Tester
Test whether a URL is crawlable for a given User-Agent based on robots.txt rules. Fetches your site's robots.txt, tests whether a given path is crawlable for the selected User-Agent and shows exactly which Allow/Disallow rule matched.
What is robots.txt?
robots.txt is a plain-text file at the root of your site (e.g. example.com/robots.txt) that tells search-engine bots which URLs they are allowed to crawl. It uses User-agent, Allow, Disallow, Crawl-delay and Sitemap directives. A bot downloads the file, picks the most specific group for its user-agent and behaves accordingly.
robots.txt is different from the robots meta tag. robots.txt blocks crawling but a URL that is already known can still appear in search results — Google can learn the URL from external links even if it can\'t fetch the page. To fully keep a page out of the index the page must include <meta name="robots" content="noindex">, and Google can only see that directive if the page is crawlable. See Google\'s robots.txt documentation for details.
About robots.txt
Disallow prevents the bot from downloading the page; noindex tells Google not to include it in the index. A page can't effectively use both — if Google can't crawl the page it can't see the noindex tag. To truly keep a page out of the index: remove the Disallow and add a noindex meta.
Crawl-delay tells the bot how many seconds to wait between requests. Bing, Yandex and Baidu honor it; Google does not — in Google you adjust crawl rate via Search Console instead. It's useful for small servers to avoid overload.
If robots.txt returns 404 (or any 4xx), Google assumes no restrictions and treats the whole site as crawlable. If it returns 5xx, Google treats the entire site as temporarily un-crawlable — that's dangerous and can cause big ranking drops. Make sure robots.txt reliably returns 200 and is cacheable.
Google and Bing: the longest matching rule wins (most specific). On a tie Allow takes precedence. Example: if /private/ is Disallowed but /private/public.html is Allowed, the second rule is longer so public.html can be crawled. This tool applies exactly the same matching logic.
Full technical SEO audit
robots.txt, sitemap, canonicals, Core Web Vitals, mobile usability and structured data — all in one report.
When should you use robots.txt Tester?
The KEYDAL robots.txt Tester tool is a browser-based utility that developers, system administrators, SEO specialists and enterprise technology teams use in their daily operations. It requires no installation, is free, and produces results instantly. It is designed so local teams can run audits without connecting to server environments and run analyses without touching production.
Typical scenarios include: post-migration verification, comparing domain or hosting providers, diagnosing customer issues, security auditing (pre-pentest reconnaissance), root-cause analysis of email deliverability problems, validating CDN or proxy configuration, surfacing technical audit data for SEO teams, and rapid information gathering during incident response. You can copy results as text and share them or paste them into internal documentation.
The KEYDAL infrastructure team provides web hosting, VPS, dedicated server management, server hardening, DNS configuration and SSL/TLS deployment services from Türkiye. Beyond these tools, we deliver server setup and operations support across Hetzner, OVHcloud, Contabo, DigitalOcean and Turkish providers.
Your queries are never stored on our servers
KEYDAL tools run stateless: domain names, IPs, URLs or other inputs are not persisted to any database. Logs are kept only for security purposes (rate limiting, abuse detection) and deleted within 30 days. For tools that handle sensitive data (tokens, API keys, JWTs), processing happens entirely in your browser — nothing is sent over the network. See our Privacy Policy for details.
All tools run over HTTPS with TLS 1.3 support. KEYDAL is a Türkiye-based technology company and complies fully with local data-protection regulations (KVKK) and GDPR principles.
You may also be looking for
The KEYDAL free tools collection includes DNS lookup, WHOIS lookup, SSL certificate checker, HTTP headers analyzer, IP geolocation, uptime checker, JSON formatter, JWT decoder, Base64 encode/decode, QR code generator, meta tag analyzer and robots.txt tester. All browser-based, free, no installation.
If you are comparing server prices, see our web hosting, VPS, VDS, cloud hosting, dedicated server and storage pages. See all tools →