Technical SEO · 17 Nov 2026 · 6 min read
Robots.txt explained: what to allow and block
A robots.txt file tells crawlers which parts of your site they may request. It manages crawling. It does not remove pages from Google and it is not a lock.
The short answer
Robots.txt is a plain text file at the root of your site that tells crawlers which paths they may fetch. It controls crawling, not indexing, so a blocked page can still appear in results if other sites link to it. To keep a page out of search, use noindex or a password.
What robots.txt does
The robots.txt file sits at the root of your domain, at an address like yoursite.com/robots.txt. Before a well behaved crawler fetches pages, it asks for this file and reads the rules written for it. Each rule names a crawler, using User-agent, and then lists paths it may or may not request, using Allow and Disallow. You can also add a line that points to your XML sitemap.
Think of it as a sign at the entrance of a building, not a locked door. Reputable crawlers from the major search engines follow it. Other bots may ignore it entirely, and nothing in the file stops a person from opening any address by hand. That has an important consequence: robots.txt is not a way to protect private material. Anything confidential belongs behind a login.
The file is also public. Anyone can read it, so avoid listing the names of private folders in it. A line saying Disallow for a hidden admin folder tells a curious reader exactly where to look.
For a clear picture of how fetching differs from storing and ranking, see our overview of crawling, indexing and ranking. Robots.txt only touches the first of those steps.
Can robots.txt remove a page from Google?
No, and this is the most common misunderstanding. Blocking a page stops crawlers reading it. It does not tell the search engine to forget the address. If other pages link to a blocked address, Google can still list the bare address in results, usually without a description, because it knows the page exists but has not been allowed to look inside.
Worse, blocking can work against you. If you add noindex to a page and also block it in robots.txt, the crawler never fetches the page and never sees the noindex instruction. The page stays listed. The same logic applies to canonical tags, as we explain in our guide to canonical tags.
Google also stopped supporting a noindex rule written inside robots.txt itself some years ago, so do not rely on advice that suggests it. Check the current documentation on Google Search Central if you meet an old tutorial that uses it.
| You want to | Use | Why |
|---|---|---|
| Keep a page out of search results | A noindex tag, left crawlable | The crawler must read the page to see the instruction |
| Keep a page private | A login or password | Robots.txt is public and optional |
| Save crawling effort on low value paths | Disallow rules | Crawlers skip those paths |
| Merge duplicate addresses | Canonical tag or redirect | Crawlers need to read the pages |
What is usually worth blocking
Most small business sites need very little in this file. Blocking is for paths where crawling adds no value and may waste effort. Typical candidates include:
- Internal search result pages, which can produce endless address variations.
- Cart, checkout and account paths on a store, since these have nothing for a searcher.
- Admin and login areas, as a courtesy, while keeping in mind that this is not security.
- Filter and sort combinations that create huge numbers of near identical addresses.
- Staging or test folders, although it is much safer to keep a test site behind a password.
For a site with a few dozen pages, crawl effort is rarely a problem, and an almost empty robots.txt is fine. The topic matters more for large stores and sites that generate addresses automatically. Our guide to measuring SEO results shows how to tell whether crawling really is holding you back before you start editing rules.
A short example in plain words: the file names all crawlers with User-agent set to an asterisk, then disallows the search results path and the cart path, then gives the address of the sitemap. That is a complete and sensible file for many stores.
What you should not block
Keep open anything that a search engine needs to understand and display your pages. That includes the stylesheets and scripts that build the page. Google renders pages much as a browser does, and if the files that control layout are blocked, it may see a broken or empty page and judge it accordingly. Images you want to appear in image search should also stay crawlable.
Do not block pages you want to rank. That sounds obvious, but it happens when a rule is written too broadly. A rule that blocks a folder name also blocks every address that begins with it. If a folder called guides is blocked, so is every article inside it.
Do not block pages in order to hide duplicate content. Use the right tool: a redirect or canonical tag, as the earlier table showed. And do not block the sitemap address itself.
Finally, remember that a page blocked from crawling cannot pass on what it knows. If a blocked page is the only route to other pages, those pages can become hard to find, which connects to the problem described in our guide to orphan pages.
Mistakes that block a whole site
The worst single error is one line: Disallow with a lone slash under User-agent set to all crawlers. It tells every crawler to stay out of the entire site. It is usually a leftover from development, when the owner wanted the site hidden while it was built. If it goes live with the launch, search traffic can fall away within days or weeks.
This is why launch checklists should include opening the live robots.txt in a browser. Do it on launch day and again after any redesign, platform change or plugin update. Some content management systems have a setting that discourages search engines from indexing the site; check that it is off once you go live.
- A leftover blanket block copied from a staging site.
- A wildcard rule that matches more addresses than intended.
- Blocking style and script folders.
- Case mistakes. Paths in the file are case sensitive, so a rule written in capitals may not match your lowercase addresses.
- A file placed in a subfolder. Crawlers look only at the root, so a file anywhere else is ignored.
- Two files in conflict after a migration, with a platform and a plugin each writing its own.
If you suspect a block, the robots.txt report and URL Inspection in Search Console show whether Google can fetch a given page. Our beginner guide to Search Console explains where to find them.
Testing a change before you save it
Edit carefully, because one extra character can change a rule's meaning. A safe routine has four steps. Copy the current file and keep the copy. Make one change at a time. Test a handful of important addresses, such as the home page, a service page, an article and a product, to confirm each is still allowed. Then watch Search Console for a week or two for new crawl errors or pages reported as blocked.
When you plan a new page, check that it is not caught by an old rule. A page that is crawlable but never linked is a separate problem, and good basic on-page work, set out in our one page SEO checklist, covers titles, headings and links. Clear navigation, including breadcrumb trails, helps crawlers move through the allowed parts of the site.
If the file is on a larger site and you are unsure of the impact, the search engine optimisation service from Liftup Technologies includes a technical review, and a developer can test rules against your real address list before they go live.
Related guides
- SEO services
- Google search console for beginners
- How to measure seo results
- On page seo checklist for one page
- Orphan Pages: How to Find Them and Link Them In
- Breadcrumbs for SEO: Navigation and Structured Data
Questions and answers
What does robots.txt do?
It is a public text file at the root of a site that tells crawlers which paths they may or may not request. Well behaved crawlers follow it. It manages crawling, so it is not a security tool.
Can robots.txt remove a page from Google?
No. A blocked page can still be listed if other sites link to it. To keep a page out of results, use a noindex tag and leave the page crawlable so the instruction can be read, or protect it with a login.
What mistakes block a whole site?
The usual cause is a Disallow rule for the whole site left over from development. Others are a wildcard that matches too much, blocking style and script folders, and a file placed outside the root. Open the live file after every launch or migration.
Do I need a robots.txt file at all?
Not strictly. If the file is missing, crawlers assume they may fetch everything. Many small sites still keep a short file so they can point to the sitemap and block paths such as the cart.
Where does the robots.txt file go?
At the root of the host, for example yoursite.com/robots.txt. A file in a subfolder is ignored. Each subdomain needs its own file if you want rules there.
Tell us what you are building.
A short call, a written plan and a fixed quote. No obligation.