What Is robots.txt and How Do You Create One Correctly?

What Is robots.txt?

robots.txt is a plain text file published in the root directory of a website that tells search engine bots (crawlers) which parts of the site they may crawl and which they should leave alone. Search engines like Google and Bing check this file before they start crawling a site and act according to the rules inside it. The file does not block access; it merely offers a directive to well-behaved bots — malicious bots can ignore these rules entirely.

If you are dealing with a small site, robots.txt is usually not a critical need; but a site with many pages can use this file to stop unnecessary crawling from burning through a limited crawl budget. Closing off areas that gain nothing from bot traffic — admin panels, search result pages, filter URLs — with Disallow lets bots visit the pages that actually matter more often.

The Correct Location for the robots.txt File

For a robots.txt file to be recognised by search engines, it has to be published in the root directory of the site: https://example.com/robots.txt. If the file is placed in a subdirectory such as https://example.com/blog/robots.txt, no search engine will read it. Each subdomain also needs its own robots.txt file; example.com/robots.txt is not valid for blog.example.com, which needs blog.example.com/robots.txt.

The Core Directives: User-agent, Disallow, Allow, Sitemap

robots.txt syntax consists of a handful of simple directives. Every rule block starts with a User-agent line stating which bot the block applies to; * covers all bots.

• User-agent: States the bot the rule applies to (e.g. Googlebot, or * for all bots).
• Disallow: Defines a path the bot should not crawl; for example Disallow: /wp-admin/.
• Allow: Used to create an exception inside a broader Disallow rule.
• Sitemap: States the full address of the sitemap.xml file, and can appear anywhere in the file.

A User-agent block with no Disallow line at all, or with an empty Disallow value, is valid syntax and means "crawl everything" for that bot.

Wildcards, Comments and Multiple User-agent Blocks

In Disallow and Allow paths the * character can be used as a wildcard standing in for zero or more characters, while $ indicates that the URL ends exactly at that point. Lines beginning with # are comments and are ignored by the bot — handy for documenting your rules.

If you want to treat different bots differently, you can define more than one User-agent block in the same file. It is possible, for example, to keep general bots entirely out of a folder while defining an exception inside it for one specific bot:

Paths in robots.txt are also case-sensitive; /Private/ and /private/ are treated as different paths. This detail matters particularly on Linux-based hosting, where the server file system itself is case-sensitive.

Disallow Does Not Mean the Page Will Not Be Indexed

There is a common mistake here: Disallow does not guarantee non-indexing. Blocking a path with Disallow prevents the bot from crawling that page; but if the page is linked from another site, Google may still list it in search results without ever seeing its content.

Crawl-delay: The Difference Between Google, Bing and Yandex

The Crawl-delay directive states the minimum number of seconds a bot should wait between requests. Google, however, ignores this directive entirely; for Google, crawl speed is managed through the crawl rate settings in Search Console. Bing and Yandex do honour Crawl-delay — useful for sites on small or shared servers that want to cap sudden bot load.

Common robots.txt Mistakes

• Putting the file in a subfolder instead of the root directory — in which case it is never read at all.
• Confusing Disallow with noindex and assuming a page will disappear from search results entirely.
• Accidentally closing off the whole site with Disallow: / and not noticing.
• Forgetting to add the Sitemap line; it is a simple but effective addition that makes the sitemap easier to discover.
• Thinking Crawl-delay will work on Google too.

Example: robots.txt for WordPress Sites

On WordPress installations, a typical structure that closes off the admin and system files while allowing the necessary AJAX endpoint looks like this:

Every line in this structure has a purpose: /wp-admin/ and /wp-includes/ hide the admin and core files, /wp-content/plugins/ the plugin files, /wp-content/cache/ the cache output, and /trackback/ and xmlrpc.php the system endpoints that produce no content. admin-ajax.php, meanwhile, is left open with a separate Allow, because many themes use that endpoint on the front end as well.

Do Not Block CSS and JS Files

Another common mistake when preparing a robots.txt is closing off the folders that hold stylesheet (CSS) and script (JS) files with Disallow. Search engines evaluate pages by rendering them much as a modern browser would; blocking access to CSS and JS files makes it harder for the bot to see the page the way users actually see it. So when you close off theme and plugin folders, make sure the .css and .js files inside them do not accidentally become unreachable.

Build It Quickly and Correctly With a robots.txt Generator

Working with an interface that guarantees correct syntax lowers the risk of error compared with writing the rules by hand. KEYDAL's robots.txt generator collects the User-agent, Disallow/Allow paths, Sitemap address and Crawl-delay value in a form and produces a valid robots.txt; ready-made presets such as WordPress and "allow everything" let you start quickly. The tool runs entirely in your browser, and nothing you enter is sent to a server.

Once you have published the file, all you need to do is confirm that it is reachable in your site's root directory by visiting the /robots.txt path in your browser.