The five checks that make a URL indexable
A URL is indexable when it passes these five checks.
Findable. Google needs a link to the page from a known page or a sitemap entry.
Crawlable. A page that Googlebot cannot crawl cannot be indexed from its content. Google lists server problems, network issues and robots.txt rules as common access issues.
No
noindex. The rule sits in a meta robots tag or in an X-Robots-Tag HTTP header, and Google treats the two the same. Once Googlebot reads it, Google drops the page from Search entirely, even if other sites link to it.Canonical to itself. Google picks one representative URL from a set of duplicates, and the URL must be that one.
A status that keeps it in. In Google's indexing pipeline, any 4xx status except 429 removes a previously indexed URL from the index. URLs that keep returning server errors (5xx) are eventually dropped, even though Google keeps them at first. A 2xx status code alone is not enough to get a page indexed.
A robots.txt block does not keep a page out of Google. It stops crawling, and a blocked URL can still be indexed without a description if other pages link to it. With a robots.txt block in place, Google never reads the noindex rule, and the URL can still appear in results.
When 'not indexed' is the right result
Google states the goal is to get the canonical version of every important page indexed, not every URL. Google says a non-indexed URL is fine when it is excluded for the right reason, such as an intended robots.txt rule. The same goes for a duplicate or a deliberate noindex.
In Search Console:
The Page indexing report lists the pages where Googlebot found a noindex rule.
The status 'Crawled - currently not indexed' means Google crawled the page but did not index it, and resubmitting it is not needed.
You can ask Google to recrawl a noindexed page with the URL Inspection tool.
Google names low content quality, meta robots rules that disallow indexing, and site design as common indexing issues. A replatform can trip several checks at once; see the headless CMS SEO migration guide.
The indexability check Pagepro runs at launch
At cutover, Pagepro tests a short list of items that must work on launch day, and turning noindex off is one of them. That way a staging rule does not block the new site.
For indexing problems on a live Next.js site, see Next.js SEO optimization.
Sources
- In-Depth Guide to How Google Search WorksGoogle · developers.google.com
- Page indexing report - Search Console HelpGoogle · support.google.com
- Block Search Indexing with noindexGoogle · developers.google.com
- What is URL CanonicalizationGoogle · developers.google.com
- How HTTP status codes, and network and DNS errors affect Google SearchGoogle · developers.google.com
- Robots.txt Introduction and GuideGoogle · developers.google.com