What Is a Robots.txt File?

Robots.txt is a tiny text file with an outsized ability to ruin your week. Used right, it guides crawlers politely. Used wrong, it can quietly tell Google to ignore your entire website.

The short version

A robots.txt file is a plain text file at the root of your site that tells search engine crawlers which areas they are allowed or not allowed to crawl.

What it is good for

• Keeping crawlers out of admin or staging areas

• Reducing crawl waste on low value sections

• Pointing crawlers to your sitemap

The danger

One overly broad disallow rule can block your whole site from being crawled. It also does not reliably keep pages out of the index, that is what other controls are for. Handle it carefully and double check before deploying.

FAQ

Does robots.txt remove a page from Google?

Not reliably. It controls crawling, not indexing. To keep a page out of results you generally need a different method, such as a noindex directive.