Why noindex needs a crawlable page
noindex controls indexing, not crawling: the rule works only if robots.txt does not block the URL and the crawler can reach it.
Two carriers, one rule
In HTML: <meta name="robots" content="noindex">. In a response header: X-Robots-Tag: noindex. Google treats the two forms the same; the header value can be noindex or none. In meta-tag form, the rule is set per page, so it works for site owners without root access to the server.
How long removal takes
A noindexed page stays in results until Googlebot recrawls it, which can take months for less important pages.
Site owners can ask Google to recrawl a noindexed page with the URL Inspection tool.
Search Console's Page Indexing report lists the pages where Googlebot found a noindex rule.
Other search engines may interpret noindex differently, so a page can still appear in their results.
When Google says noindex is the wrong tool
Google points to noindex (or password protection), not robots.txt, as the way to keep a page out of Google. Google does not support a noindex line inside robots.txt. For other jobs, Google points elsewhere.
Duplicates. Google advises against using noindex to choose between duplicates on one site;
rel="canonical"is the preferred tool.Crawl budget. noindex saves no crawl budget: Google still has to request the page to see the rule, and advises against using it that way.
Sensitive content. Google calls deleting or updating the content the most secure option: noindex keeps the page out of Google only, and users and search engines that don't support noindex can still access it.
Permanent removal. When a page must leave Google Search for good, Google puts deleting or changing the content first, before password protection and noindex.
Site moves
If the new site carries noindex during development, Google suggests listing those URLs in advance so the rules come off when the move starts. Once redirects are live, Google says to update any noindex rules that were added to the new site to prevent early indexing. For tooling and timing, see the best tools for a website migration and how long a website migration takes.
noindex on Pagepro's launch checklist
At cutover, Pagepro tests a short list of items that must work on launch day. Turning noindex off is one of them.
Sources
- Block Search Indexing with noindexGoogle · developers.google.com
- Robots meta tag, data-nosnippet, and X-Robots-Tag specificationsGoogle · developers.google.com
- Robots.txt Introduction and GuideGoogle · developers.google.com
- How to specify a canonical URL with rel="canonical" and other methodsGoogle · developers.google.com
- Optimize your crawl budgetGoogle · developers.google.com
- Remove Your Site Info from GoogleGoogle · developers.google.com
- Site Moves and MigrationsGoogle · developers.google.com