Short answer
- Put one or more User-agent lines before their Allow and Disallow rules.
- Match URL paths from the first character and treat path casing as significant.
- Use the most specific matching path when more than one rule applies.
- When equally specific Allow and Disallow rules match, Allow wins under RFC 9309.
User-agent groups
A group starts with one or more User-agent lines followed by rules. A later User-agent line starts a new group after rules have appeared.
The wildcard user-agent applies when no more specific group matches the crawler. Test the named crawler you care about rather than assuming every group is combined.
User-agent: * Disallow: /private/ User-agent: Googlebot Disallow: /search/
Most-specific path matching
If Disallow blocks a broad directory and Allow names a longer path inside it, the longer matching rule is more specific.
The tester should show both the result and the rule responsible for it so you can review the decision before deployment.
| Rule | Test path | Result |
|---|---|---|
| Disallow: /private/ | /private/report | Blocked |
| Allow: /private/preview/ | /private/preview/report | Allowed |
| No matching rule | /public/page | Allowed |
Syntax and crawler limits
RFC 9309 defines the core group and prefix-matching behavior. Wildcards, end anchors, crawl-delay, and other extensions can vary by crawler.
Keep rules explicit, test representative paths, and verify important crawler-specific behavior against that crawler's documentation.
FAQ
Does the first matching robots.txt rule win?
No. Under RFC 9309, the most specific matching path wins. Rule order does not replace the specificity check.
Are robots.txt paths case-sensitive?
They should be matched case-sensitively, so /Admin/ and /admin/ can produce different results.
What happens when no rule matches?
The path is allowed. An empty Disallow rule also does not block a path.