How Google reads meta robots rules
Placement and crawler tokens
Google also honors a meta robots tag placed in the body, although the head is the documented location.
Google accepts only two crawler tokens in the name attribute,
googlebotandgooglebot-news; other values are ignored.name="robots"covers search crawlers only. Non-search crawlers such asAdsBot-Googlemay need their own tag.
Values and defaults
The default is all (no restrictions), so writing index, follow or all explicitly has no effect. The value none is shorthand for noindex, nofollow. The nofollow value tells Google not to follow the links on the page.
Conflicts and access
When robots rules conflict, Google applies the more restrictive one. With separate tags for robots and googlebot, Googlebot obeys the sum of all negative rules from both.
Crawlers can read and follow the tag only if they are allowed to access the page. Where phones get different HTML, Google wants the meta robots tags on both versions to match.
Snippet controls, AI Overviews and retired values
nosnippet. In Google, nosnippet also keeps the page content from being used as direct input for AI Overviews and AI Mode.max-snippet:[number]. Sets the maximum number of characters Google may use as a text snippet.0is equivalent tonosnippet, and-1lets Google choose the length. In Google, max-snippet also limits how much content may be used as direct input for AI Overviews and AI Mode.max-image-preview:[setting]. Sets the maximum size of an image preview in search results:none,standardorlarge. Withlarge, Google may show a preview up to the width of the viewport.max-video-preview:[number]. Sets the maximum number of seconds Google may use as a video snippet.0allows at most a static image, and-1sets no limit.unavailable_after:[date/time]. Tells Google not to show the page in search results after that date and time. The date must use a widely adopted format such as RFC 822, RFC 850 or ISO 8601. After that date, Googlebot considerably decreases the crawl rate of the URL.data-nosnippet. An HTML attribute that keeps parts of a page out of Google snippets. Google supports it only onspan,divandsectionelements. It is boolean, sodata-nosnippet="false"still excludes the text.noarchive. Google no longer uses noarchive, because the cached-page link no longer exists.
PDFs, videos and images have no HTML head, so Google points to the X-Robots-Tag header for them.
For robots metadata in a Next.js app, see Next.js SEO. For an audit of indexing rules across a site, see Next.js SEO optimization.
Sources
- Robots meta tag, data-nosnippet, and X-Robots-Tag specificationsGoogle · developers.google.com
- X-Robots-Tag header - HTTP | MDNMDN · developer.mozilla.org
- Mobile-first Indexing Best PracticesGoogle · developers.google.com