Robots.txt Guide: Syntax, Rules and Examples

Quick answer: A robots.txt file is a plain text file at the root of your site that tells crawlers which paths they may fetch. It controls crawling, not indexing, and it is not access control. Keep it short: block only areas that waste crawl effort, and add a Sitemap line.

What robots.txt does and does not do

The Robots Exclusion Protocol is standardized in RFC 9309. Well-behaved crawlers, including Googlebot and Bingbot, read the file before crawling and follow its rules. The standard itself says the rules are not a form of access authorization.

• It controls crawling, not indexing. Google Search Central notes that a URL blocked by robots.txt can still appear in search results, without a description, if other pages link to it.
• To keep a page out of Google's index, use a noindex robots meta tag or X-Robots-Tag header, and leave the page crawlable so Google can see that instruction.
• It is public. Anyone can read it, so listing secret paths only advertises them. Protect private areas with authentication.

Where the file lives

The file must be at /robots.txt on each host, for example https://example.com/robots.txt. It applies only to that protocol, host and port, so shop.example.com and http://example.com need their own files. A robots.txt file in a subfolder is ignored.

Save it as UTF-8 plain text. RFC 9309 requires crawlers to parse at least 500 kibibytes, and Google documents the same 500 KiB limit, which is far more than most sites need. If the file returns a 4xx status such as 404, crawlers may crawl everything. If the server returns a 5xx error or cannot be reached, RFC 9309 says crawlers must assume everything is disallowed, so a broken robots.txt URL can pause crawling of the whole site.

Syntax and matching rules

A file is made of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules:

• A crawler uses the group whose User-agent matches its name, and the * group only if no named group matches.
• Paths start with / and are case-sensitive. Disallow: /Admin/ does not block /admin/.
• The most specific rule wins, meaning the one with the longest matching path. If an Allow and a Disallow rule are equally specific, Allow should be used.
• * matches any sequence of characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks URLs ending in .pdf.
• Disallow: with an empty value blocks nothing, while Disallow: / blocks the whole site.
• Sitemap: takes a full URL and is not tied to any group.
• Lines starting with # are comments.

Google ignores Crawl-delay and does not support noindex inside robots.txt.

Worked example: a WordPress site

For a WordPress site at example.com, you want to keep crawlers out of the admin area and internal search results while still allowing admin-ajax.php, which some themes call from public pages:

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /?s=

Sitemap: https://example.com/sitemap.xml

How these rules apply:

• /wp-admin/admin-ajax.php matches both the Allow rule (24 characters) and /wp-admin/ (10 characters). The longer Allow rule wins, so it may be crawled.
• /wp-admin/options.php matches only the Disallow rule and is blocked.
• /?s=shoes is blocked by /?s=.
• /blog/hello-world/ matches no rule, so it may be crawled.

How to use the free Robots.txt Generator

1. Leave User-agent as * for all crawlers, or enter one crawler name such as Googlebot.
2. In Allow paths, one per line, enter any exceptions, such as /wp-admin/admin-ajax.php.
3. In Disallow paths, one per line, enter the paths to block, such as /wp-admin/ and /?s=. Start each path with /.
4. Enter the full Sitemap URL, such as https://example.com/sitemap.xml, or leave it empty.
5. Select Generate robots.txt. The output lists the User-agent line, then the Allow lines, then the Disallow lines, then a blank line and the Sitemap line, exactly as in the worked example.
6. Select Copy result, save the text as robots.txt and upload it to your site's root.

The generator writes one group per run and does not check your paths. To give different crawlers different rules, generate each group and combine them by hand with a blank line between groups. If both path boxes are empty, the output is only User-agent: *, which allows everything. After uploading, open /robots.txt in a browser and check the robots.txt report in Google Search Console.

Common mistakes

• Leaving Disallow: / from a staging site in place after launch, which blocks the entire site.
• Using robots.txt to remove pages from search results. Use noindex and allow crawling instead.
• Blocking CSS and JavaScript files that search engines need to render your pages.
• Expecting a parent domain's file to cover a subdomain.
• Writing a relative sitemap path such as /sitemap.xml instead of the full URL.