What Google does when a URL returns 404
Google handles a 404 in three ways. Google does not process newly discovered URLs that return 404. It removes already indexed URLs from its index once they return a 4xx code such as 404. For URLs returning 404 and other non-429 4xx codes, Google gradually crawls them less often.
Google says that, in general, 404 errors do not affect a site's search performance. Outside Google, caches may store a 404 response without explicit cache headers, because a 404 is heuristically cacheable.
Custom 404 pages, soft 404s and when to redirect
MDN names mistyped URLs and pages moved or deleted without a redirect as the common causes of 404s. MDN advises keeping the number of broken internal and external links low because 404s hurt user experience. Regular link checks are one of the web development best practices.
Custom 404 pages and soft 404s
Google recommends a custom 404 page that keeps the site's look and navigation. Google says the page must still return a 404 status code so the error page itself is not indexed. A soft 404 is a "page not found" page that returns 200 (success) instead of a 404 status. Google also warns that redirecting many old URLs to one irrelevant page, such as the homepage, might be treated as a soft 404.
Which status to return
Moved content. Where a replacement page exists, Google prefers a 301 to it over a 404.
Removed for good. When the server knows the absence is likely permanent, RFC 9110 prefers 410 Gone over 404. Google currently treats 410 (Gone) the same as 404 (Not Found).
No robots file. If a site has no
robots.txtfile, Google expects therobots.txtURL to return a proper 404.
Unpublish, don't postpone
When pages are not content-ready on launch day, Pagepro takes them out, so they return 404, and releases them as content lands. The launch date holds, and late client content no longer delays it. For 404s on a live Next.js site, see Next.js support.
Sources
- RFC 9110: HTTP SemanticsIETF · rfc-editor.org
- 404 Not Found - HTTP | MDNMozilla (MDN) · developer.mozilla.org
- How HTTP status codes, and network and DNS errors affect Google SearchGoogle · developers.google.com
- 404 (Page Not Found) errors - Search Console HelpGoogle · support.google.com
- Troubleshoot Google Search crawling errorsGoogle · developers.google.com
- Site Moves and MigrationsGoogle · developers.google.com