Robots.txt is a plain text file of a few lines, and it is one of the few files on a website that can wipe out years of search traffic in an afternoon. It rarely happens through bad intent. It happens because a staging rule rode along with a launch, a plugin rewrote the file, or someone used robots.txt for a job it was never designed to do.
The good news: almost every robots.txt disaster is easy to diagnose and quick to reverse once you know what to look for. Recovery is usually days to a few weeks, not months, provided you fix the right thing first.
What robots.txt actually controls
Robots.txt lives at the root of a host (https://example.com/robots.txt) and tells crawlers which URLs they may fetch. That is all. It does not decide what appears in search results.
This distinction causes most of the damage:
- Blocking a URL does not remove it from Google. If other pages link to it, Google can still index the bare URL without ever reading it. Search Console reports these as "Indexed, though blocked by robots.txt", usually with no title or description in the result.
- Blocking a URL hides its noindex tag. A
noindexmeta tag or header only works if Google can crawl the page and see it. Block the page in robots.txt and the noindex is invisible. - Each host has its own file.
example.com,www.example.com,shop.example.comand thehttp://versions are separate hosts as far as crawlers are concerned. A rule in one file does not apply to the others.
Keep that model in mind (robots.txt controls crawling, meta robots and X-Robots-Tag control indexing) and most of the mistakes below become obvious.
Mistake 1: Shipping the staging Disallow: /
The classic. A developer blocks crawlers on the staging site so unfinished pages stay out of Google:
User-agent: *
Disallow: /Then the whole site, robots.txt included, is copied to the live server. Within a few days, Google stops refreshing pages, new pages never get picked up, and rankings slide as cached copies go stale. Because the effect is gradual, people often blame an algorithm update rather than a two-line file.
WordPress has a cousin of this trap. Settings, Reading, Discourage search engines from indexing this site is ticked on most staging installs. On current versions it adds a noindex, nofollow robots meta tag to every page (older versions changed the virtual robots.txt instead), and migration plugins happily carry the setting across to the live site.
Fix: make "check robots.txt and the WordPress visibility setting" a named step on your launch list, not something you hope to remember. Better still, protect staging with a password (HTTP basic auth) instead of robots.txt. Crawlers cannot index what they cannot open, and there is nothing to forget to undo.
Mistake 2: Using robots.txt to remove pages from Google
A thank-you page, an old landing page or a duplicate tag archive shows up in search, so someone adds Disallow: /thank-you/. Weeks later it is still there, now with "No information is available for this page" under it.
That is the crawl versus index split at work. Google is no longer allowed to read the page, so it cannot see any noindex tag you add, and it keeps the URL because links still point at it.
Fix: do it in this order.
- Remove the Disallow rule so Google can crawl the page again.
- Add
<meta name="robots" content="noindex">to the page, or send anX-Robots-Tag: noindexheader (the only option for PDFs and images). - Wait for Google to recrawl it and drop it. You can speed that up with URL Inspection in Search Console.
- If it is urgent (personal data, a leaked file), use the Removals tool in Search Console as well. It hides the URL for about six months while the noindex does the permanent work.
If the page should simply not exist any more, delete it and let it return a 404 or 410, or 301 redirect it to the closest live page. Only add a Disallow afterwards if crawl volume on those URLs is genuinely a problem, which on a small business site it almost never is.
Mistake 3: Blocking CSS, JavaScript and images
Older robots.txt advice told you to block folders like /wp-includes/, /assets/ or /js/ to "save crawl budget". Google now renders pages much like a browser does. If it cannot load your stylesheets and scripts, it sees a broken, unstyled page: menus that never open, content injected by JavaScript that never appears, a layout that looks unusable on a phone.
The result is weaker rankings for pages that look fine to every human visitor. Blocked images also drop out of Google Images, which matters for restaurants, trades, venues and anyone whose work sells visually.
Fix: remove Disallow rules for any folder that holds CSS, JavaScript, fonts or images. On WordPress the only folder worth blocking is /wp-admin/, with an Allow for /wp-admin/admin-ajax.php because some themes load front-end content through it. Then use URL Inspection, Test live URL, View tested page to confirm the screenshot looks like your real site.
Mistake 4: Misreading how paths and wildcards match
Robots.txt rules are prefix matches, and that catches people out in both directions.
| Rule | Blocks | Also blocks (often by accident) |
|---|---|---|
Disallow: /blog | /blog, /blog/ | /blog-tips/, /blogroll, /blog.html |
Disallow: /blog/ | Everything inside /blog/ | Nothing else, but not /blog itself |
Disallow: /*? | Any URL with a query string | Paginated, filtered and tracked URLs you may want crawled |
Disallow: /*.pdf$ | URLs ending in .pdf | Nothing; $ anchors the end of the URL |
Disallow: /Private/ | /Private/ | Not /private/: paths are case sensitive |
When an Allow and a Disallow both match a URL, Google applies the most specific rule, meaning the one with the longest path. If they are exactly the same length, the less restrictive Allow wins. Other crawlers are not always that careful, so avoid relying on ties.
The other trap is an empty value. Disallow: with nothing after it means "block nothing", while Disallow: / means "block everything". One character is the difference between a healthy site and an invisible one, so read those lines twice.
Mistake 5: Getting user-agent groups wrong
Rules belong to the User-agent group above them, and a crawler follows only the single most specific group that names it. It does not combine groups. That leads to a subtle failure:
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search/
User-agent: Googlebot
Disallow: /old-promo/Here Googlebot obeys only its own group. It will happily crawl /cart/, /checkout/ and every internal search URL, because those rules sit in the * group it now ignores. The reverse also happens: someone adds a Googlebot group with Disallow: / while "testing", and only Google disappears.
Fix: if you give any crawler its own group, repeat every rule that should apply to it. Keep the file short enough that this is easy to see. Also check that every rule sits below a User-agent line; rules at the top of the file before any user-agent are ignored.
Mistake 6: Blocking Google while trying to block AI
Since 2023 many site owners have tried to keep their content out of AI training, and a panicked edit can easily take out search at the same time. The tokens matter:
Googlebotis Google Search. Block it and you leave Google.Google-Extendedis a control token for Google's AI models (Gemini and Vertex AI). Blocking it does not affect crawling or ranking in Search.GPTBot,ClaudeBot,CCBotand similar names are AI companies' training crawlers. Some companies run separate user-triggered or search agents with different names, so check each one's documentation before blocking.
Fix: block AI training crawlers by their exact names in their own groups, and never touch Googlebot or Bingbot unless you genuinely want out of search. Our generator lists the training, search and user-fetch crawlers separately so you can decide each one deliberately.
Mistake 7: A robots.txt that errors, redirects or is not really there
What your server returns for /robots.txt matters as much as what the file says. Google documents how it treats each case:
| Response | What Google does |
|---|---|
| 200 with valid rules | Follows the rules |
| 404, 410 or another 4xx (except 429) | Treats the site as having no restrictions and crawls everything |
| 5xx or 429 | Treats the whole site as disallowed at first, then falls back to its last cached copy. If the error lasts for weeks, it eventually stops trusting the cache. |
| Redirect | Follows up to five hops, then treats it as a 404 |
| Over 500 KiB | Ignores everything past the limit |
The dangerous row is 5xx. A firewall rule, a security plugin or an overloaded host that answers /robots.txt with a 500 or 503 tells Google to stop crawling your entire site. We have also seen single-page apps and some page builders return the home page HTML with a 200 status for any unknown path, robots.txt included. That HTML is not valid robots.txt, so the result is unpredictable.
The second trap lives in the headers. A staging server set up with Header set X-Robots-Tag "noindex" in .htaccess or the server config will noindex every page and file it serves, and that header is invisible when you view the page in a browser. When the config is copied to production, the site drops out of the index even though robots.txt and the page source look perfect.
Mistake 8: Trusting rules Google does not support
Robots.txt files collect folklore. These lines do nothing in Google, so a site relying on them is not protected the way its owner thinks:
Noindex:inside robots.txt. Google stopped honouring it in September 2019. Use the meta tag or header instead.Crawl-delay:. Google ignores it (Bing and some others still read it). Google adjusts its crawl rate automatically based on how quickly your server responds.Host:. A legacy Yandex directive that has no effect elsewhere. Use 301 redirects and canonical tags to pick your preferred host.- Comments or rules in a file saved as
Robots.txtorrobots.TXT. On Linux hosts such as Hostinger the filename is case sensitive, so crawlers requesting/robots.txtget a 404 and assume no rules at all.
A related slip is the sitemap. The Sitemap: line must be a full absolute URL (https://example.com/sitemap.xml, not /sitemap.xml), and the sitemap should only list URLs you allow and want indexed. A sitemap full of URLs that robots.txt blocks, or that carry noindex, sends Google mixed signals and wastes its attention on pages you do not want.
A safe default for most small business sites
Most brochure sites, service businesses and portfolios need almost nothing in robots.txt. This is a sensible starting point for a WordPress site:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://example.com/sitemap_index.xmlAnd for a static or hand-built site, even less:
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlBlock internal search results (they create endless thin URLs), admin areas and genuinely private tools. Leave everything else open, and use noindex for pages that should be crawlable but not listed, such as thank-you pages and account screens.
Tip: keep a copy of your production robots.txt in your project and compare it after every deploy, plugin update or host migration. A two-second diff catches the mistakes that otherwise take two weeks to notice in traffic.
How to check and recover
If traffic has dropped and you suspect robots.txt, work through this in order:
- Open the live file at
https://yourdomain/robots.txtand at thewwwandhttp://variants. Look forDisallow: /, unexpected user-agent groups and blocked asset folders. - Check the status code and headers of both robots.txt and your home page. You want a 200 for robots.txt and no
X-Robots-Tag: noindexon pages. - Check the page source of a few key pages (view source, then search for
robots) for a leftovernoindexmeta tag. - Open Search Console. Settings, robots.txt shows which robots.txt files Google found, when it last fetched them and any parse problems. The Pages report lists URLs "Blocked by robots.txt" and "Excluded by noindex tag".
- Fix, then ask for a recrawl. In the robots.txt report you can request a recrawl of the file. Then run URL Inspection on your most important pages and click Request indexing, and resubmit your sitemap.
Google usually refetches robots.txt within a day. Important pages tend to return within days, and the full site over a few weeks as Google works back through it. Rankings normally recover with them, because nothing about the content changed. If traffic is not back after a month, look for a second cause rather than waiting longer.
Launching or relaunching a site? Work through our website launch checklist, which includes robots.txt, sitemaps and the other launch-day checks.
Frequently asked questions
Why is a page I blocked in robots.txt still showing in Google?
Because Disallow only stops crawling. If other pages link to the URL, Google can still list it without a description. To remove a page, let Google crawl it and add a noindex meta tag or X-Robots-Tag header, or delete it so it returns a 404 or 410.
How long does it take to recover after removing Disallow: /?
Google usually refetches robots.txt within about a day, and you can request a recrawl in the Search Console robots.txt report. Key pages often return within days once you request indexing, and the rest of the site over a few weeks.
Should I block CSS and JavaScript files in robots.txt?
No. Google renders pages like a browser, so blocking stylesheets, scripts, fonts or images makes your pages look broken to it and can hurt rankings. Only block admin areas, internal search results and genuinely private paths.
Will blocking AI crawlers in robots.txt hurt my Google rankings?
Not if you block them by their own names, such as GPTBot or CCBot, and use the Google-Extended token for Google's AI models. Google-Extended does not affect Search. Never block Googlebot unless you want to leave Google Search entirely.
What happens if my site has no robots.txt file?
Nothing bad. If /robots.txt returns a 404, crawlers assume they may crawl everything. A missing file is far safer than one that returns a server error, which can make Google pause crawling of the whole site.