What one access-log line tells you
The Common Log Format is a standard format that many web servers produce and many log analysis programs read. The Combined Log Format adds a User-Agent field, the identifying string that the client reports about itself. nginx writes access logs in its predefined combined format when no other format is specified.
Key fields:
Request line. Apache's
%rfield logs the request line in double quotes: method, path with query string, and protocol.Time. The
%tfield logs when the server received the request.Status code. The status code in each log line shows whether a request succeeded (2xx), redirected (3xx), or failed on the client (4xx) or server (5xx) side.
Client IP. The client IP in a log line can be a proxy's address rather than the original requester's.
User agent. The
User-Agentrequest header shows whether a request came from Googlebot Smartphone or Googlebot Desktop. Googlebot Desktop can still appear in server logs, for example for product listings and Google for Jobs crawling.
Verifying Googlebot, reading crawl patterns and comparing with Crawl Stats
Google documents how to verify that a request claiming to be Googlebot really comes from Google.
In a reverse DNS lookup of a log IP, a genuine Google request resolves to a googlebot.com, google.com or googleusercontent.com hostname.
Then a forward DNS lookup of that hostname must return the original IP address from the logs.
For large-scale verification, Google publishes its crawlers' IP ranges as JSON files to match log IPs against automatically.
Google's user-triggered fetchers (for example Google Site Verifier) ignore robots.txt, so not every Google request in a log is a robots.txt-respecting crawl.
What the logs show
Crawl spikes. For a sharp increase in crawling, Google strongly recommends checking recent access logs to find the traffic source. Common causes are faceted navigation, sorting and filtering, and calendars with many date URLs.
Overcrawling. When heavy crawling causes availability problems, Google says to find the overcrawling crawler in your logs or Crawl Stats.
Wasted crawling. Google says many known duplicate or unwanted URLs waste a lot of crawling time.
Logs vs Crawl Stats
Google warns that Crawl Stats may miss some requests, so its totals can differ slightly from your logs.
The Crawl Stats report shows only requests to the selected domain.
Crawl Stats counts some failed fetches ('Page could not be reached') that never reached the server, so your logs do not show them.
If Crawl Stats totals are much higher than the Google requests in your logs, Google says this can happen when robots.txt is unavailable for too long.
Crawl Stats counts each hop of a server-side redirect chain as its own request.
For Next.js sites, see Next.js SEO optimization or a technical discovery phase.
Sources
- Log Files - Apache HTTP Server Version 2.4Apache Software Foundation · httpd.apache.org
- Module ngx_http_log_modulenginx · nginx.org
- What Is GooglebotGoogle · developers.google.com
- Mobile-indexing-vLast-final-final.doc (2024)Google · developers.google.com
- Verifying Google's crawlers and fetchersGoogle · developers.google.com
- Crawl Stats report - Search Console HelpGoogle · support.google.com
- Reduce the Google crawl rateGoogle · developers.google.com
- Optimize your crawl budgetGoogle · developers.google.com
- Apache HTTP Server: Log FilesApache Software Foundation · httpd.apache.org