robots.txt: Definition, Structure, & Example

What does robots.txt mean?

The robots.txt is a small text file located in a website’s root directory that tells search engine crawlers which areas they are allowed to visit and which they are not. This allows website operators to specify that certain directories or individual pages should not be crawled.

Typical examples include technical sections, certain filter pages, or other URLs that are irrelevant for crawling.

A common misconception about the robots.txt file: The text file controls when the crawler visits a page, but it does not determine whether a page will still appear in search results. To reliably keep a page out of the index, other measures are required.

Furthermore, the robots.txt file is not an access control mechanism. Reputable, standards-compliant crawlers follow its instructions, whereas malicious crawlers may ignore them.

Where does the robots.txt file come from?

The robots.txt file traces its origins to an informal agreement, the “Robots Exclusion Standard” from 1994, which allowed website operators to control the behavior of crawlers.

Despite the robots.txt file’s considerable age, it remains the standard to this day and is supported by the major search engines.

Why is the robots.txt file important?

Not every page belongs in search results. Furthermore, a website’s limited crawl budget shouldn’t be consumed by unimportant sections.

With the help of the robots.txt file, website owners can exclude unimportant sections from crawling, thereby reducing unnecessary crawling efforts—especially for large websites.

Advantages

The robots.txt file gives you control over which sections of a website crawlers visit.
It helps large websites focus their crawl budget on the important sections.
It has a simple structure and is supported by major search engines.

Challenges

The text file is deceptively simple and, at the same time, carries a risk. A single incorrect entry can exclude entire websites from crawling and affect their visibility.

It also does not reliably prevent pages from appearing in search results. Malicious programs can easily ignore the robots.txt file, which is why it is not suitable for protecting sensitive content.

Example

A company has countless filter and sort options and wants to prevent crawlers from using up its limited crawl budget on them.

By including specific instructions in the robots.txt file, it excludes these variations from crawling. This prevents the crawler from wasting unnecessary resources on these URLs, while ensuring that important product and category pages remain accessible for crawling.