Writing robots rules into a response header
A response can carry several X-Robots-Tag headers, or one header with a comma-separated list of rules. A crawler name can prefix the rules (for example googlebot: nofollow) to target one search engine. Rules without a crawler name apply to all crawlers. Google treats the header name, the crawler name and the rule values as case-insensitive.
To keep a file out of the index, the header carries the value noindex or none. The header is added in web server configuration; on Apache that means .htaccess or httpd.conf.
Google's example is an ordinary response with one header:HTTP/1.1 200 OKX-Robots-Tag: noindex
To combine rules, Google's example sends two separate X-Robots-Tag headers in one response: noimageindex and unavailable_after.
Standard or not
MDN says the header is not defined in any formal specification; it is a de facto standard that search crawlers support.
Header rules for files and whole URL patterns
Many URLs, one rule. Google names site-wide application as a benefit: one server rule with regular expressions can cover many URLs.
Files without HTML. For PDFs, videos and images, which have no HTML head, Google directs site owners to the X-Robots-Tag header instead.
Limits
If robots.txt disallows the URL, Google never fetches the response, so the X-Robots-Tag rules are ignored.
X-Robots-Tag does not save crawl bandwidth. MDN says robots.txt is the better tool for that.
For header and metadata fixes on a Next.js site, see Next.js SEO services; for setting robots rules in a Next.js app, see Next.js SEO.
Sources
- X-Robots-Tag header - HTTP | MDNMDN · developer.mozilla.org
- Robots meta tag, data-nosnippet, and X-Robots-Tag specificationsGoogle · developers.google.com
- Block Search Indexing with noindexGoogle · developers.google.com