What to know when building a robots.txt
For a robots.txt file to be recognized it must be published at your site's root — that is, https://example.com/robots.txt. A file placed in a subdirectory such as https://example.com/blog/robots.txt is never read by search engines, and every subdomain needs its own robots.txt. Having no Disallow line under a User-agent block, or leaving Disallow empty, is valid syntax and means "crawl everything" for that bot — which is exactly what the "Allow everything" preset produces.
Disallow does not guarantee a page stays out of the index. Blocking a path with Disallow stops the bot from crawling that page, but if the page is linked from elsewhere Google can still list it in search results (without seeing its content). To keep a page out of the index for certain, add a <meta name="robots" content="noindex"> tag or an X-Robots-Tag HTTP header to the page — and for that to work the page must actually be crawlable (i.e. not Disallowed). This is a commonly confused but important nuance.
Google completely ignores the Crawl-delay directive; in Google you manage crawl speed via the crawl rate settings in Search Console instead. Bing and Yandex, on the other hand, do respect it, letting you set the minimum number of seconds a bot waits between requests — useful for avoiding overload on smaller servers. Finally, adding a Sitemap: line to robots.txt is a widely recommended practice that makes your sitemap easier for search engines to discover.
When should you use robots.txt Generator?
The KEYDAL robots.txt Generator tool is a browser-based utility that developers, system administrators, SEO specialists and enterprise technology teams use in their daily operations. It requires no installation, is free, and produces results instantly. It is designed so local teams can run audits without connecting to server environments and run analyses without touching production.
Typical scenarios include: post-migration verification, comparing domain or hosting providers, diagnosing customer issues, security auditing (pre-pentest reconnaissance), root-cause analysis of email deliverability problems, validating CDN or proxy configuration, surfacing technical audit data for SEO teams, and rapid information gathering during incident response. You can copy results as text and share them or paste them into internal documentation.
The KEYDAL infrastructure team provides web hosting, VPS, dedicated server management, server hardening, DNS configuration and SSL/TLS deployment services from Türkiye. Beyond these tools, we deliver server setup and operations support across Hetzner, OVHcloud, Contabo, DigitalOcean and Turkish providers.
Your queries are never stored on our servers
KEYDAL tools run stateless: domain names, IPs, URLs or other inputs are not persisted to any database. Logs are kept only for security purposes (rate limiting, abuse detection) and deleted within 30 days. For tools that handle sensitive data (tokens, API keys, JWTs), processing happens entirely in your browser — nothing is sent over the network. See our Privacy Policy for details.
All tools run over HTTPS with TLS 1.3 support. KEYDAL is a Türkiye-based technology company and complies fully with local data-protection regulations (KVKK) and GDPR principles.
You may also be looking for
The KEYDAL free tools collection includes DNS lookup, WHOIS lookup, SSL certificate checker, HTTP headers analyzer, IP geolocation, uptime checker, JSON formatter, JWT decoder, Base64 encode/decode, QR code generator, meta tag analyzer and robots.txt tester. All browser-based, free, no installation.
If you are comparing server prices, see our web hosting, VPS, VDS, cloud hosting, dedicated server and storage pages. See all tools →