Log File Analysis SEO: Prove What Googlebot Actually Crawled
Crawl tools tell you what a bot could fetch. Your access log tells you what it did fetch, how often, and what you served back. We turn that raw ledger into a ranked list of URLs that waste crawl budget and pages that deserve the next visit.
- Verified bot traffic separated from spoofed user agents
- Crawl frequency by template, not by page
- Status-code mix per section, per bot, per day
- Redirect chains triggered by bots, with hop counts
- Priority backlog: what to fix, block, or consolidate first
30 to 90 days of access logs is enough to start. No traffic, no dashboard access required.
What a log line knows that a crawler never will
A crawl shows a hypothetical path through your site. A log shows the real one, including every wrong turn. It answers questions your SEO platform cannot: did Googlebot ever fetch the new collection page, or did it spend the week on filter parameters? Did the retired URL keep getting hits after the redirect went live? Did the bot hit a 503 while your CDN was revalidating?
That matters because crawl budget is finite. On a 3,000-URL catalog, Googlebot might request a few thousand URLs a day and decide that is enough. If 40% of those requests land on parameter noise, your new buying guide waits behind a queue of pages that will never convert.
Logs also expose the difference between "indexed" and "useful". A page can return 200, be crawled twice a day, and still earn zero impressions. That pattern is your strongest argument for noindex, consolidation or removal — much stronger than an opinion about thin content.
The fields we actually parse
Standard combined format gives us enough: client IP, timestamp with timezone offset, method, full request path including query string, protocol, status code, response size, referrer and user agent. Where available we also take response time and edge cache status.
Two fields do most of the work. The request path reveals the template and the parameter pattern. The status code reveals whether the bot got the page or a detour. Everything else supports attribution: which bot, from where, at what hour, and whether the response was served from cache.
CDN logs are welcome but must include origin fetches. A log that only records edge hits will under-report the pages that actually cost you rendering time.
Reading the ledger: three lines that change a roadmap
Below is a trimmed sample from a real catalog audit. Nothing here is unusual — that is the point. The waste is normal until someone counts it.
| Time | Status | Request path | Bot | Action |
|---|---|---|---|---|
| 09:14:03 | 200 | /search?q=flange&page=3 | Googlebot | Block internal search, keep results out of crawl |
| 09:14:07 | 301 | /old-flange → /flange → /flange-150 | Googlebot | Collapse to one hop, update internal links |
| 09:15:40 | 404 | /products/flange?ref=newsletter | Googlebot | Fix broken parameter route, add 410 where permanent |
| 09:16:02 | 503 | /catalog/hose?p=4 | Googlebot | Raise render timeout, cache the paginated view |
| 09:18:44 | 200 | /filter?color=red&size=3&stock=1 | Bingbot | Noindex faceted sets, keep one canonical combination |
Crawl frequency bars: 28-day window, desktop Googlebot only, excluding asset requests.
Bot identification is a verification problem, not a regex
Any script can send a Googlebot user agent. Before we count a single hit, we confirm the IP belongs to the published range with a reverse-then-forward DNS check. Everything else is labelled "claimed" and kept out of the crawl budget math.
Googlebot
66.249.64.0/19 and related ranges. Mostly desktop and mobile smartphone agents on the same infrastructure.
We split desktop from smartphone-first crawling, because a mobile-first journey starts and ends in different folders.
Bingbot
40.77.167.0/24 and 157.55.39.0/24 in most datasets. Often crawls different templates than Google, which is a useful second opinion.
Bing traffic on faceted URLs is a common early warning for duplicate discovery.
Everything else
AI crawlers, uptime monitors, scrapers, security scanners, your own CDN health checks. Some are worth allowing; most are worth rate-limiting.
We list them separately so you can decide with evidence instead of blanket rules.
Turning rows into a ranked backlog
A thousand URLs sorted by hits is a spreadsheet, not a plan. We rank by cost: requests multiplied by median response time, weighted by whether the page can ever convert. A filter URL that burns 900 requests a day and returns 200 is more urgent than a single missing page.
Each finding becomes one of five actions. Robots and noindex for pages that should never appear. Canonical consolidation for pages that duplicate a real one. Redirect repair where chains add hops. Internal link removal where the only path to a junk URL is your own navigation. Cache and timeout fixes where bots hit slow server responses.
Once the backlog is ordered, the log analysis becomes part of a wider technical program — the same workstream we run inside Technical SEO Services, where crawl budget, rendering and index hygiene are tracked together instead of as isolated tickets.
What the report contains
One file, four tabs. Raw extraction with bot verification. Template-level crawl distribution. Status and response-time summary. A prioritised action list with owner and expected effect.
You also get the queries we would run next month to check ourselves: hits per template, share of 4xx and 5xx, redirect hops, and the count of URLs fetched more than 30 times without earning a single impression.
If the site is large, we add a per-section budget table so your team can see which folder consumes the crawl share it deserves.
From request log to crawl backlog in four passes
Raw files in, nothing filtered. We take 30–90 days, normalise timestamps to one timezone, strip asset requests, and verify every claimed bot IP before counting.
Group URLs by pattern, not by string. Parameter order, trailing slashes and case variants collapse into one template, which is the only level where crawl budget numbers mean anything.
Compare crawl share to value. A section that receives a third of bot hits but a tenth of revenue is the first conversation. We also flag discovery gaps: pages linked internally that the bot never visited.
Fix, deploy, then re-measure. Rules go live, sitemaps update, redirects collapse. We re-run the same window after 28 days and publish the before-and-after hit counts per template.
Results from the last three log audits
An industrial parts catalog was serving 41% of Googlebot hits to faceted filter URLs. After noindex rules and canonical consolidation, junk-crawl share fell 62% and product pages were fetched 3.1 times more often.
Industrial supply / 3 monthsA B2B furniture exporter had 900 orphaned URLs discovered only through an old sitemap. Pruning the sitemap and repairing internal links moved organic inquiries from 5 to 23 per month.
Furniture export / 5 monthsA wholesale electronics distributor fixed 300 redirect chains and a 6-second render path. Googlebot's median fetch time dropped, and index coverage for the catalog rose 1.9 times within a quarter.
Wholesale electronics / 4 monthsScope, access, and a straight quote
Log analysis is priced by data volume and template complexity, not by the number of pages you own. A 400-page brochure site with clean URLs is a small job. A marketplace with 60 filter combinations per category is not.
We quote after a 20-minute look at a sample file: one day of logs, the user-agent mix, and the top 50 request paths tells us almost everything about effort.
If the audit uncovers architecture problems that need engineering work, we scope that separately rather than burying it in an hourly estimate. The analysis itself stays fixed-price.
When logs are not available
Some hosted platforms expose nothing at origin level. In that case we fall back on crawl stats, server response headers, and a controlled test crawl from verified IPs — still directional, but honest about the gap.
A full crawl comparison belongs to SEO Audit Services, which pairs our crawl with your analytics and backlink data when logs are thin.
Have a log file? Send one day.
We will come back with the three templates eating your crawl budget and what to do about them — no obligation, no deck.
Frequently asked
How much log data do you need?
Thirty days minimum, ninety preferred. Shorter windows are noisy because crawl scheduling varies by day of week. If you only keep seven days, we can still work, but we will validate findings against crawl stats before recommending structural changes.
Can we just use Search Console crawl stats?
They help, but they are aggregated by response and purpose, not by URL. You cannot see that a specific filter pattern absorbs 40% of requests. Logs give page-level truth; crawl stats give a sanity check.
Do you need server credentials?
No. A compressed export over SFTP, S3, or a shared link is enough. We ask that the file includes the full request path with query strings, the status code and the user agent — nothing else is mandatory.
What about CDN logs?
Useful when they include origin requests and cache status. Edge-only logs overstate hits on cached pages and hide the slow origin responses that actually delay crawling. Send both if you have them.
How quickly do crawl patterns change?
Usually two to six weeks after a fix, sometimes longer on large catalogs. Crawl budget reallocates gradually, so we measure at 28 and 56 days rather than the morning after deployment. On a monthly retainer we re-run the window every month and report the delta.
Is blocking bots risky?
Blocking the right URLs is low risk when done at template level and with a sitemap that stays clean. We never block a section that still earns organic traffic, and every rule is staged and reversible before it reaches production — no cloaking, no fake status codes.
Will this improve rankings directly?
Not by itself. It improves discovery and freshness for the pages that matter, which is usually a precondition for ranking. We do not promise positions from a log cleanup.