A text file at the root of a site that tells crawlers which URLs they may access. Google says it is mainly for managing crawler traffic, and is not a way to keep a page out of Google; use noindex or a password for that.
Example
Meera's developer blocks crawling of her cart and checkout URLs in robots.txt. For an old sale page she wants out of results, he uses a noindex tag instead.
How you'll hear it
“Someone disallowed the whole /products/ folder in robots.txt. That is why the catalogue vanished.”
Don't confuse it with
- noindex: a page-level instruction to keep a page out of the index. A page blocked by robots.txt can still be indexed if other sites link to it, because Google cannot see a noindex it is not allowed to crawl.
Related terms
Want to use this at work? AI-led SEO Excellence is aFactor's 16-week program for SEO work.
Sources
- Introduction to robots.txt, Google Search Central