Crawl control issue

How to fix robots.txt rules blocking important pages

A robots.txt rule can tell crawlers not to request a path. If it covers a page, asset or directory that should be crawled, the rule can hide useful content from a normal crawl.

Run a free website audit Check robots.txt

What this issue means

robots.txt is a site-root text file that gives well-behaved crawlers crawl instructions. It controls crawling, not guaranteed indexing: a blocked URL can still be known from links or other signals.

Why it matters

  • Search engines may not fetch important pages or the assets needed to understand them.
  • A blocked page can remain indexed without its current content if the URL is discovered elsewhere.
  • An overly broad rule such as Disallow: / can prevent a whole site from being crawled, while an accidental narrow rule can block a key section.

How to fix it

  1. 1

    Identify the exact matching rule

    Check the site-root robots.txt response, user-agent group and path. Review the most specific Allow and Disallow rules rather than assuming a line applies to every crawler.

  2. 2

    Decide whether the path should be crawled

    Keep private, duplicate or administrative areas restricted when appropriate. Remove or narrow a rule that covers public pages, important assets or a whole site by mistake.

  3. 3

    Publish a valid root file

    Serve robots.txt at the site root with a successful text response. Keep directives readable and include the correct XML sitemap URL when you have one.

  4. 4

    Test after deployment

    Recheck the file and sample affected URLs. Use Search Console’s robots testing tools where available, and remember that removing a block does not instantly guarantee a crawl or index change.

Check the page, then audit the site

Use the focused checker to inspect one URL, then run a free SeeSpect audit to find related issues across up to five pages. The free plan includes two scans a day and no credit card is required.

Check robots.txt → Start the free audit →

Frequently asked questions

Does robots.txt remove a URL from Google?

No. robots.txt primarily controls crawling. A blocked URL can still be indexed if its address is discovered through other signals, so use appropriate indexability controls for pages that should stay out of search.

Where must robots.txt live?

It belongs at the root of the host, such as https://example.com/robots.txt. A file in a subdirectory does not control the whole host.