Server Log Analysis Workbook Template

Log Analysis Workbook (fill-in)

When to use: One row per URL class: verified bot hits, status mix, crawl frequency, wasted crawl and the action with an owner.

Server log analysis workbook

Log source: [server, CDN or load balancer] · Date range: [at least 30 days] · Total verified Googlebot requests: [number]

Sheet 1: verification

Sheet 2: URL classes

Sheet 3: key pages check

Wasted crawl = verified hits on URL classes you don't want in search, plus hits that end in redirects, soft 404s or errors. Write the share as a fraction of all verified hits and keep the formula next to it.

Completion check: spoofed bots excluded, every class has a wanted yes or no, every wasted class has an action and owner, and a re-pull date is set.

Do You Even Need This

When to use: Decide whether crawl budget is actually a problem worth your time before diving into logs.

Does crawl budget matter for you?

Crawl budget is the number of URLs a search engine is willing and able to crawl on your site over a given period. For most small and medium sites, this is not a problem worth solving.

Google can usually crawl a few thousand URLs without breaking a sweat.

Google's crawl budget guide gives rough thresholds. It is written for:

• sites with about 1 million or more unique pages that change weekly;
• sites with 10,000 or more pages that change daily;
• sites with a large share of URLs stuck as "Discovered - currently not indexed".

Logs are still useful on smaller sites for spotting errors and redirect loops Googlebot keeps hitting.

Red flags that justify this work

• Faceted navigation or filters creating endless URL combinations
• Important pages not indexed weeks after publishing
• Search Console showing many "Discovered – currently not indexed" URLs
• Server logs full of bot hits to parameters, sorting, or pagination

[Site URL]: note your approximate total URL count here: [Total URLs]. If it is small and pages index quickly, skip the rest and focus elsewhere.

Get & Prepare Log Files

When to use: Collect raw server access logs and clean them into something you can actually analyze.

Where to find your logs

Server access logs are the most reliable record of how bots and users actually hit your site.

Ask your hosting provider, devops team, or CDN for raw access logs covering at least a few weeks, ideally a full month so you capture normal crawl cycles.

Common sources include your web server (Apache, Nginx), your CDN (such as Cloudflare or Fastly), and any load balancer in front of them. Pull from [Log Source] for the period [Date Range].

Prepare the data

1. Combine logs from all servers so you do not miss requests served by different machines.
2. Confirm each line includes timestamp, requested URL, status code, user agent, and IP address.
3. Verify Googlebot: run a reverse DNS lookup on the IP and check the host ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP. For large files, match IPs against the IP range lists Google publishes for its crawlers instead. Never trust the user agent string alone.
4. Filter to verified search engine bots before drawing conclusions about crawl behavior.

Store the cleaned dataset somewhere repeatable so you can re-run the same analysis next quarter.

Analyze Crawl Patterns

When to use: Understand what Googlebot actually crawls, how often, and which status codes it hits.

What to look at first

With verified Googlebot requests isolated, build a picture of real crawl behavior rather than guessing. Focus on volume, frequency, and response codes: these reveal where crawl effort is going.

Key questions to answer

• Which URLs get crawled most? Compare top-crawled URLs against your most important pages. Mismatches are a signal.
• How often are key pages crawled? Money pages should be visited regularly; rarely-crawled important URLs get stale.
• What status codes does Googlebot see? A healthy site is mostly 200s. Watch for spikes in 404s, 301/302 redirects, and 5xx server errors.
• Which sections dominate? Group hits by directory to see whether a low-value section is eating crawl activity.

Record your findings for [Top Crawled Section] and note any section that is over-crawled relative to its value: [Over-crawled Path].

Heavy crawling of redirects or errors is wasted effort that should be cleaned up in later steps.

Find Crawl Waste

When to use: Pinpoint the low-value, duplicate, and broken URLs that drain crawl resources.

Where crawl budget leaks

Crawl waste happens when bots spend time on URLs that should not be crawled at all. On large sites this can be the difference between important pages getting refreshed daily or monthly.

Common sources of waste

• Faceted & parameter URLs: filters, sorting, and tracking parameters that multiply a single page into thousands of variants.
• Duplicate content: the same page reachable via multiple URLs, trailing slashes, or upper/lowercase paths.
• Soft 404s: empty results, expired listings, or thin pages that return 200 but offer no value.
• Redirect chains: URLs that bounce through several hops before resolving.
• Infinite spaces: calendars, search results, and session IDs that generate near-endless URLs.
• Removed pages that still return content: Google's guide says permanently removed pages should return 404 or 410, not a 200.

From your log analysis, list the worst offenders here: [Waste Pattern 1], [Waste Pattern 2]. Estimate the share of Googlebot hits going to these low-value URLs: that number is your opportunity.

Prioritize fixes by how much crawl activity each pattern consumes.

Improve Crawl Efficiency

When to use: Take concrete actions so search engines spend their crawl effort on pages that matter.

Direct crawlers to what matters

Once you know where waste lives, the goal is to consolidate signals and steer bots toward valuable, canonical URLs.

1. Strengthen internal links to important pages so they are easy to discover and signal their priority.
2. Keep XML sitemaps clean: include only canonical, indexable URLs and remove redirects, errors, and noindex pages.
3. Use robots.txt for URLs you never want crawled, such as internal search and infinite parameter spaces. Google advises against using it to shift budget around temporarily, and blocking crawling is not the same as removing from the index.
4. Don't rely on noindex to save crawl. Google still has to request the page to see the tag.
5. Handle parameters with canonical tags pointing to the clean version, and avoid linking to parameterized URLs internally.
6. Fix redirect chains by pointing the first hop directly to the final destination with a single 301.
7. Resolve soft 404s by returning a real 404 or 410, or improving the page so it earns its crawl.

Log your top three actions: [Action 1], [Action 2], [Action 3]. Roll out changes gradually so you can measure their effect.

Monitor with Crawl Stats

When to use: Track crawl behavior over time using Search Console and ongoing log checks.

Close the loop

Crawl efficiency is not a one-time project. After making changes, monitor whether search engines respond the way you expect.

What to watch

• Search Console Crawl Stats (Settings, root-level properties only): review total crawl requests, average response time, and the breakdown by response, file type, purpose (discovery or refresh) and Googlebot type.
• Host status: covers robots.txt fetching, DNS and server connectivity over the past 90 days. A slow or erroring server makes Google crawl less.
• Status code trends: confirm that 404s, redirects, and 5xx errors are trending down after your fixes.
• Indexing reports: check whether previously stuck pages move into the indexed state.

Build a routine

Re-pull server logs on a regular cadence (monthly or quarterly) and compare against your baseline. Record the review date here: [Next Review Date] and the owner responsible: [Owner].

If waste creeps back in, you will catch it early instead of after important pages fall out of the index.

Worked Example (fictional marketplace)

When to use: A filled-in workbook for an invented secondhand camera marketplace with filter URLs eating crawl.

Worked example: LensLoop

Example only. The company, people, URLs and figures are invented to show what a usable, filled-in version looks like.

30 days of CDN logs. 412,000 requests claimed to be Googlebot; 398,000 passed verification.

What the numbers said

About 62% of verified hits went to URLs the team didn't want in search: filters, sold listings and redirects. New listings were being found, but category pages were refreshed slowly.

The first fix was the filter rules, because that class was the largest.

Next step

Trace the chains you found with the Redirect Checker, rebuild clean sitemaps with the XML Sitemap Generator, and draft parameter rules with the Robots.txt Generator before testing them.

Run a website audit to catch status and redirect problems your logs point to. For a done-for-you log audit, our partner agency SEOquick handles technical SEO projects.