SEO
Robots.txt Generator and Tester
Generate robots.txt rules, check core syntax, and test crawler access against URL paths in your browser.
No core robots.txt syntax problems found.
This local tester follows RFC-style prefix and most-specific-rule matching. Crawler-specific extensions may behave differently.
How to use Robots.txt Generator and Tester
Best for
- Creating a small robots.txt file before launching or changing a website.
- Testing whether a crawler user-agent is allowed to request specific URL paths.
- Finding the most specific Allow or Disallow rule that matches a path.
Steps
- Choose Generate to build user-agent, path, and sitemap directives.
- Review or edit the generated robots.txt text directly.
- Choose Test, enter a user-agent and URL paths, then review each matched rule before deployment.
Notes
- Robots.txt manages crawling; it does not reliably prevent indexing or protect private content.
- Crawler-specific wildcard, crawl-delay, and extension behavior can differ from the RFC-oriented local test.
FAQ
Does robots.txt prevent a page from appearing in search results?
Not reliably. A blocked URL can still be discovered from links. Use an appropriate noindex method or access control when the goal is indexing or privacy.
How does the tester choose between Allow and Disallow rules?
It uses the most specific matching path. When equally specific Allow and Disallow rules match, Allow wins.
Does this tool fetch my live robots.txt file?
No. Paste or generate the rules you want to test. All parsing and path tests run locally in your browser.