Short answer
| Goal | Typical control | Important limit |
|---|---|---|
| Manage crawler access | robots.txt | The URL may still be discovered or indexed |
| Prevent indexing | noindex | The crawler must access the page to see it |
| Protect private content | Authentication | Neither robots.txt nor noindex is access control |
What robots.txt controls
Robots.txt tells compliant crawlers which URLs they may request. It is mainly a crawl-management mechanism.
Because blocked URLs can still be discovered through links, robots.txt is not a reliable way to remove a URL from search results.
What noindex controls
A noindex directive tells a supporting search engine not to include a page in its index after the crawler reads the directive.
Do not block the same page in robots.txt when the crawler needs to fetch it and discover the noindex directive.
Neither option protects private content
Robots.txt is public, and noindex does not stop someone from opening a URL. Use authentication or another real access control for confidential material.
Review the goal first: crawler management, search indexing, and content security are different problems.
FAQ
Can a robots.txt-blocked URL appear in Google?
Yes. Google explains that a blocked URL can still appear when it is discovered through links, even if Google cannot crawl the page content.
Should I use robots.txt and noindex on the same page?
Usually not when you need the noindex directive processed. The crawler must be able to fetch the page to see that directive.
Can noindex protect a private document?
No. Use authentication or another access-control method for private content.