Screaming Frog and Google Search Console tell you what could be crawled and what got indexed. Only server logs tell you what actually happened. This is the difference between guessing and knowing.

1. Why Log Files Outrank Every Other Crawl Data Source

Every SEO tool you already use is inferring bot behavior. Screaming Frog simulates a crawl using its own crawler, not Googlebot. Google Search Console shows you a sampled, delayed, aggregated summary of what Google has decided to tell you. Neither one shows you the actual HTTP requests hitting your server.

A log file is a first-party record. Every request — every bot, every user, every status code, every byte served — is written to your server in real time. When you analyze log files, you stop guessing about crawl behavior and start measuring it directly.

Why this matters more in 2026 Google is no longer the only crawler that determines whether your content gets seen. GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are now crawling e-commerce sites specifically to source answers for AI search and chat interfaces. Log files are the only place you can see how these newer bots behave, because none of them report into GSC-style dashboards.

2. What’s Actually Inside a Log File

A raw server log (Apache/Nginx “combined” format is the most common) looks like this for every single request your server receives:

66.249.66.1 – – [15/Jul/2026:03:12:41 +0000] “GET /products/running-shoes HTTP/1.1” 200 48213 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

Break that line down and you get everything you need for analysis:

• IP address — who made the request (used to verify the bot is genuine, not spoofed)
• Timestamp — when the request happened, down to the second
• Request line — the method (GET/POST), the URL path, and the protocol
• Status code — 200, 301, 404, 500, etc. — what your server actually returned
• Bytes transferred — page weight as delivered, not as measured in a lab tool
• User-agent string — who the requester claims to be (Googlebot, GPTBot, ClaudeBot, a real browser, or a scraper)

3. Getting Access to Your Logs

Where logs typically live

• Traditional hosting (cPanel, VPS, dedicated) — raw access logs, usually under /var/log/ or accessible via the hosting control panel, often rotated daily and kept 7–30 days
• Shopify — no native raw log access; you’ll rely on Shopify’s own crawl stats or a CDN-level log (Cloudflare, Fastly) sitting in front of the store
• WordPress/Kadence stack (like alneeko.com) — depends on host; some (Kinsta, WP Engine) offer downloadable logs, others require asking support directly
• Sites behind a CDN (Cloudflare, Akamai, Fastly) — the CDN log is often more complete than the origin server log, since cached requests never reach the origin

 VERIFIED   Google explicitly recommends log file analysis as a supplementary diagnostic method alongside Search Console, and documents it in their own crawling and indexing guidance for large sites.

4. Tooling: From Spreadsheet to Enterprise

For smaller sites — command line is enough

You don’t need enterprise software to start. grep and awk on a raw access log will get you real answers in minutes:

# Isolate every Googlebot request
grep “Googlebot” access.log > googlebot_requests.log

# Count requests by status code
awk ‘{print $9}’ googlebot_requests.log | sort | uniq -c | sort -rn

# Find the most-crawled URLs
awk ‘{print $7}’ googlebot_requests.log | sort | uniq -c | sort -rn | head -20

For mid-size and agency work

• Screaming Frog Log File Analyser — purpose-built, pairs directly with a Screaming Frog crawl so you can compare “crawled by Google” vs “crawlable at all”
• JetOctopus — cloud-based, handles very large log volumes, strong bot-verification and crawl budget reporting
• Botify — enterprise-grade, log data blended with crawl and ranking data in one platform

For custom / AI-crawler-specific tracking

This is where your systems background is an advantage. A lightweight pipeline — logs into BigQuery or a Postgres table, queried with SQL, visualized in Looker Studio or a simple dashboard — gives you full control over how AI bot traffic is segmented and trended over time, which off-the-shelf tools don’t always do well yet since GPTBot/ClaudeBot/PerplexityBot tracking is still a new category for most vendors.

5. The Step Every Beginner Skips: Verifying the Bot Is Real

A user-agent string is just a claim. Anyone can send a request with “Googlebot” in the header — scrapers and bad actors do this constantly to bypass basic bot-blocking rules. You must verify the IP address actually belongs to the crawler it claims to be.

How to verify

• Run a reverse DNS lookup on the requesting IP
• Confirm the hostname resolves to the expected domain (e.g., googlebot.com or google.com for Googlebot)
• Run a forward DNS lookup on that hostname and confirm it resolves back to the original IP

# Reverse DNS lookup
host 66.249.66.1
# -> 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

# Forward-confirm
host crawl-66-249-66-1.googlebot.com
# -> crawl-66-249-66-1.googlebot.com has address 66.249.66.1

 VERIFIED   Google publishes this exact verification method officially, and also publishes downloadable IP ranges for Googlebot and other Google crawlers so you can automate verification at scale.

 VENDOR CLAIM   OpenAI (GPTBot), Anthropic (ClaudeBot), and Perplexity each publish their own IP ranges or verification guidance for their respective crawlers, but coverage and update frequency vary by vendor and are self-reported rather than independently audited — treat these lists as the vendor’s current claim, and re-check them periodically since ranges do change.

6. Core Metrics to Pull From Any Log Analysis

6.1 Crawl frequency by bot

How often does each bot visit, and which sections of the site get the most attention? A sudden drop in Googlebot visits to your product pages is a leading indicator of a problem — often weeks before it shows up as a ranking or traffic drop in GSC.

6.2 Status code distribution

Segment every bot’s requests by status code. A healthy pattern is dominated by 200s. Watch specifically for:

• High volumes of 301/302 — bots repeatedly hitting redirect chains instead of final URLs, wasting crawl budget
• 404s on URLs that shouldn’t exist — orphaned internal links, old sitemap entries, or scraped/guessed URLs
• 5xx errors — server-side failures bots are hitting that never surface in a manual browser check

6.3 Crawl budget waste

Cross-reference crawled URLs against your canonical URL list. Faceted navigation, session IDs, and parameter-based URLs are the most common culprits — bots spending disproportionate time on filter/sort combinations instead of your actual product and category pages.

6.4 Crawled vs. indexed gap

Compare the list of URLs Googlebot actually requested against what’s indexed in GSC. A large gap (crawled heavily, never indexed) usually points to a content quality or duplication signal, not a crawlability problem — a distinction log data makes obvious that GSC alone doesn’t.

6.5 Orphaned but crawled pages

Pages bots keep visiting that no longer appear in your current site architecture or sitemap. These often indicate old external backlinks pointing at deprecated URLs — worth a redirect, not a removal.

7. Reading AI Crawler Behavior Specifically

This is the extension of your llms.txt and agentic browsing work: log files are where you find out whether GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are actually visiting your site, and which content they prioritize.

What to isolate and compare

• Which AI bots visit at all — many sites get zero ClaudeBot or GPTBot traffic simply because those crawlers haven’t discovered the site yet
• Which pages AI crawlers request most — often your comparison pages, FAQ content, and structured/schema-rich pages over generic category pages
• Crawl frequency trend over time — a rising trend line for AI bots after publishing llms.txt or improving schema is a measurable signal that your GEO work is having an effect
• Whether AI bots respect robots.txt disallow rules — most legitimate AI crawlers do, but this is worth confirming per bot rather than assuming

 DISPUTED   Whether AI crawler visit frequency correlates with actual citation or inclusion in AI-generated answers is not yet independently proven — being crawled is necessary but not sufficient for being cited, and no vendor currently publishes a reliable causal link between the two.

# Isolate each AI crawler for a trend comparison
grep “GPTBot” access.log | wc -l
grep “ClaudeBot” access.log | wc -l
grep “PerplexityBot” access.log | wc -l
grep “Google-Extended” access.log | wc -l

8. A Worked Example: 15-Minute First Pass

• 1. Pull the last 7–30 days of raw access logs from your host or CDN
• 2. grep for each bot user-agent you care about into separate files
• 3. Spot-check 10–20 IPs per bot with reverse DNS to confirm they’re genuine, not spoofed
• 4. Run the status-code breakdown per bot — flag anything outside the 90%+ “200 OK” range
• 5. Extract the top 20 most-requested URLs per bot and sanity-check them against your actual priority pages
• 6. Cross-reference against GSC’s Crawled – Currently Not Indexed report to find the crawled-but-ignored gap
• 7. Document findings against your 8-phase audit framework as a new “Crawl Behavior” phase entry

9. Key Takeaways

• Log files show you what actually happened, not what a tool infers happened — treat them as the source of truth above Screaming Frog and GSC, not a replacement for either
• Always verify bot identity via reverse/forward DNS before trusting a user-agent string — spoofing is common and cheap
• Status code and crawl frequency patterns are leading indicators — they show up in logs before they show up in rankings
• AI crawler analysis (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) is the direct measurement layer for your GEO and llms.txt work — use it to prove impact, not just assume it
• Start with grep/awk on a small site before reaching for enterprise tooling — the workflow scales up, not down

References

1. Google Search Central — “Verifying Googlebot and other Google crawlers” (official IP verification method via reverse/forward DNS). developers.google.com/search/docs/crawling-indexing/verifying-googlebot

2. Google Search Central — “Large site owner’s guide to managing your crawl budget” (official guidance referencing log file analysis as a diagnostic method). developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget

3. OpenAI — “GPTBot” documentation and published IP ranges (vendor-published, self-reported). platform.openai.com/docs/gptbot

4. Anthropic — “ClaudeBot” crawler documentation (vendor-published, self-reported). support.claude.com (search “ClaudeBot”)

5. Perplexity — “PerplexityBot” crawler documentation (vendor-published, self-reported). docs.perplexity.ai

6. Screaming Frog — “Log File Analyser User Guide” (tooling reference, vendor documentation). screamingfrog.co.uk/log-file-analyser