File location, syntax and limits
The file must be named /robots.txt in lowercase and sit at the top-level path of the host. It applies only to the exact host, protocol and port it is served from, so each subdomain needs its own file.
How Google reads the file
Each group starts with a
User-agentline naming its crawler. Google's example:User-agent: GooglebotDisallow: /nogooglebot/User-agent: *Allow: /Sitemap: https://www.example.com/sitemap.xmlGoogle supports four fields (
user-agent,allow,disallow,sitemap) and ignores others such ascrawl-delay.In paths,
*matches 0 or more of any valid character and$marks the end of the URL./*.php$matches paths ending in .php, but not/filename.php?parameters.Under RFC 9309, when several rules match a URL, the most specific rule (the one with the longest matching path) applies.
A sitemap line can point crawlers to sitemaps with a fully qualified URL.
If robots.txt returns a 4xx error (other than 429), Google behaves as if there were no crawl restrictions.
Google enforces a 500 KiB size limit on robots.txt and ignores any content after it.
The file cannot enforce anything. Each crawler decides whether to obey it, and the IETF standard states that robots.txt rules are not a form of access authorization.
Disallow limits and AI crawlers
Google can still index a URL disallowed in robots.txt if other sites link to it, even though it never crawls the content. When Google shows a URL that robots.txt blocks, the search result has no description.
If robots.txt blocks a page, the crawler never sees that page's noindex rule, so the page can stay in search results. Google does not support a noindex line inside robots.txt and says not to use robots.txt to remove a page from Search results.
Crawl budget and launches
A disallow rule frees budget for other pages only when Google is already at the site's crawl capacity limit.
If Google is crawling problem URLs, Google suggests blocking dynamic URLs such as search results or infinite spaces with robots.txt.
Some site owners block all crawling of a new site while in development. Google tells them to prepare the launch robots.txt in advance.
AI crawlers
Google-Extendedcontrols whether crawled content may train future Gemini models or ground Gemini Apps. It does not affect Google Search inclusion or ranking.Per OpenAI, disallowing
GPTBotindicates content should not train its foundation models.OAI-SearchBotis a separate setting.
robots.txt in Pagepro's Nexity starter
Nexity, Pagepro's Next.js and Sanity starter, generates robots.txt and the sitemap automatically. For the wider setup, see Next.js SEO and headless CMS development.
Sources
- Robots.txt Introduction and GuideGoogle · developers.google.com
- RFC 9309: Robots Exclusion ProtocolIETF · rfc-editor.org
- How Google interprets the robots.txt specificationGoogle · developers.google.com
- Block Search Indexing with noindexGoogle · developers.google.com
- Remove Your Site Info from GoogleGoogle · developers.google.com
- Optimize your crawl budgetGoogle · developers.google.com
- URL structure best practices for Google SearchGoogle · developers.google.com
- Site Moves and MigrationsGoogle · developers.google.com
- Create and submit a robots.txt fileGoogle · developers.google.com
- Google's common crawlersGoogle · developers.google.com
- Overview of OpenAI CrawlersOpenAI · developers.openai.com