Fold 1What is robots.txt?

Fold 2How do you read it?

A robots.txt file groups rules by bot:

User-agent: *
Disallow: /drafts/

User-agent specifies which bot the rules apply to. Here, * means all bots.

Disallow specifies a path those bots should avoid. It matches any path that starts with /drafts/, such as /drafts/first-post.

Paths without a matching restriction are allowed by default.

Fold 3See it on a real website

Open our robots.txt. It includes these rules:

User-agent: *
Disallow: /robots-txt-demo

Now open the demo page. It still loads in your browser. robots.txt asks bots to avoid the page; it doesn’t block access.

Fold 4When would you use it?

Use robots.txt to ask bots to skip pages they don’t need to visit, such as internal search results or multiple filtered versions of the same product list. This can reduce unnecessary requests to your website.

A disallowed page can still appear in search results if other pages link to it. To keep a page out of search, use a noindex instruction. Allow bots to access the page so they can read that instruction. To keep content private, require a login.

Fold 5Is robots.txt blocking Google?

If Google can’t crawl a page, look in your robots.txt for a Disallow rule that matches the page’s path. These two lines block your whole website:

User-agent: *
Disallow: /

If your file has a User-agent: Googlebot group, Google follows that group and ignores the * group. Remove or narrow the rule, then use URL Inspection in Search Console to ask Google to crawl the page again.