Short answer
- Put one or more User-agent lines before their Allow and Disallow rules.
- Use Disallow: / to ask the selected crawlers not to crawl any path on that host.
- Use an empty Disallow value when the selected crawlers may crawl every path.
- Match URL paths from the first character and treat path casing as significant.
- Use the most specific matching path when more than one rule applies.
- When equally specific Allow and Disallow rules match, Allow wins under RFC 9309.
Common robots.txt Disallow and Allow examples
| Goal | Rule | What it does |
|---|---|---|
| Block one directory | Disallow: /private/ | Blocks matching paths that begin with /private/. |
| Block all paths | Disallow: / | Blocks every path for the selected user-agent group. |
| Allow all paths | Disallow: | Leaves the Disallow value empty, so no path is blocked by that rule. |
| Allow one exception | Allow: /private/preview/ | Allows the more-specific path inside a broader blocked directory. |
How to Disallow all crawlers
Use the wildcard user-agent and a single slash to ask compliant crawlers not to request any path on the host. This rule controls crawling; it does not make content private and does not guarantee that already discovered URLs disappear from search results.
Test the rule before publishing it, especially on a production site, because the slash matches the entire host.
User-agent: * Disallow: /
User-agent groups
A group starts with one or more User-agent lines followed by rules. A later User-agent line starts a new group after rules have appeared.
The wildcard user-agent applies when no more specific group matches the crawler. Test the named crawler you care about rather than assuming every group is combined.
User-agent: * Disallow: /private/ User-agent: Googlebot Disallow: /search/
Most-specific path matching
If Disallow blocks a broad directory and Allow names a longer path inside it, the longer matching rule is more specific.
The tester should show both the result and the rule responsible for it so you can review the decision before deployment.
| Rule | Test path | Result |
|---|---|---|
| Disallow: /private/ | /private/report | Blocked |
| Allow: /private/preview/ | /private/preview/report | Allowed |
| No matching rule | /public/page | Allowed |
Allow one path inside a Disallow rule
A longer Allow path can create a crawlable exception inside a broader blocked directory. In this example, /private/report stays blocked while /private/preview/report is allowed.
Write the path exactly as it appears after the host name. A robots.txt path normally begins with a slash and is matched case-sensitively.
User-agent: * Disallow: /private/ Allow: /private/preview/
Paths, query parameters, wildcards, and end anchors
Robots.txt rules match the URL path from its first character and can include a query string. A narrow query-parameter rule should include enough of the path to avoid blocking unrelated URLs.
RFC 9309 defines the core prefix behavior. Some major crawlers also support * wildcards and a trailing $ end anchor, but extension behavior should be checked against the documentation for the crawler you target.
User-agent: * Disallow: /search?sort= Disallow: /*.pdf$
Syntax and crawler limits
RFC 9309 defines the core group and prefix-matching behavior. Wildcards, end anchors, crawl-delay, and other extensions can vary by crawler.
Keep rules explicit, test representative paths, and verify important crawler-specific behavior against that crawler's documentation.
Test Allow and Disallow rules before publishing
- Test one path that should be blocked and one path that should stay allowed.
- Test the exact crawler group you expect to match, including the wildcard fallback.
- Review the matched rule, not only the allowed or blocked result.
- Keep robots.txt at the host root and verify the live file after deployment.
FAQ
How do I Disallow all pages in robots.txt?
Use User-agent: * followed by Disallow: /. This asks compliant crawlers not to crawl any path on that host, but it is not access control and does not guarantee removal from search results.
How do I allow all crawlers in robots.txt?
Use User-agent: * with an empty Disallow value, or omit a blocking group. Test the published file to confirm no other matching group blocks the path.
Does the first matching robots.txt rule win?
No. Under RFC 9309, the most specific matching path wins. Rule order does not replace the specificity check.
Are robots.txt paths case-sensitive?
They should be matched case-sensitively, so /Admin/ and /admin/ can produce different results.
What happens when no rule matches?
The path is allowed. An empty Disallow rule also does not block a path.
Can robots.txt Disallow a URL with query parameters?
A rule can include the path and query string, such as /search?sort=. Keep the prefix narrow and test representative URLs so the rule does not block more than intended.
Does robots.txt Disallow remove a page from Google?
Not reliably. Disallow manages crawler access. A blocked URL can still be discovered or indexed, so use an index-control method when removal from search is the goal.