Google Not Indexing My Website? Fix Crawl Issues Fast

Crawl control room · coverage triage

Google Not Indexing My Website? Read the Coverage Report Like a Server Room

Google not indexing my website is almost never one fault. In the crawl logs we pull, a URL gets discovered, fetched, sometimes rendered, and then quietly dropped at a gate nobody checked. This page is the diagnostic order we use in the console: seven gates, each with a pass rule, so you stop resubmitting URLs and start clearing the blockage.

Crawl access repair: robots.txt, WAF rules, rate limits, 5xx storms
Indexability cleanup: meta robots, X-Robots-Tag, canonical chains, pagination
Sitemap architecture: canonical-only XML, split by template, monitored weekly
Render-path fixes for JavaScript templates and lazy-loaded copy

Log-level diagnosis · no lock-in retainer · reply within one business day

Signal, not superstition

Why a page can sit in “Discovered” for a month

Discovery is cheap. Indexing is a judgment call, and three things push a URL to the back of the queue: it duplicates something already stored, its template is expensive to render, or the host answers slowly.

A crawler stores a URL the moment it sees a link, a sitemap entry or a redirect. Spending a fetch on that URL is a separate decision, made against a budget and a quality bar. That is why the Search Console report can show a page as “Discovered — currently not indexed” for weeks without ever flagging an error. Nothing is broken in the eyes of the crawler; the page simply never earned a fetch.

This matters because most remediation effort goes into the wrong place. Teams resubmit URLs, rewrite meta descriptions, and add schema while robots.txt still blocks the template, or while every product variant canonicalises to a page that itself canonicalises somewhere else. Each of those actions feels like progress and moves nothing.

Work order matters. The seven gates below are sequential. A gate can only be judged once the gate above it passes. Fixing canonical tags on a URL that robots.txt hides is wasted effort, and fixing internal links on a page that returns a soft 404 is worse than wasted effort — it hides the real fault.

Read the board from top to bottom, mark each gate pass or fail, and only then decide what to change.

Check order · 01–07

Seven checks, in the order that saves the most time

Each entry lists the symptom you will see in the console, the test that settles it, and the pass rule that tells you to move on.

robots.txt or a firewall rule is blocking the crawl Excluded

Open /robots.txt as an anonymous visitor, not as a logged-in admin. Look for a leftover blanket Disallow from staging, then check whether your WAF returns 403 to Googlebot while returning 200 to your browser. Hosting panels, password gates and country blocks all produce the same “Excluded” badge, and none of them look like a CMS problem.

Pass rule A fetch of a sample URL as Googlebot returns 200, and the response body contains the page copy — not a challenge page or a login form.

noindex survives template inheritance Excluded

Check the rendered head for a robots meta tag, then check response headers for an X-Robots-Tag set at the CDN or by a staging module. Both can be applied globally by a plugin switch nobody remembers. Pagination and faceted URLs are the usual casualties: a developer added noindex “just to be safe” two years ago and the rule still ships on every listing page.

Pass rule Target URL carries no noindex directive in head or headers. Directives that only limit snippet length are acceptable; directives that block indexing are not.

Canonical tags point somewhere they should not Discovered

Three canonical faults account for most of the backlog we see. The tag points at a redirect. The tag points at a parameter variant. The tag points at a page that canonicalises to a third URL, so the crawler follows a chain and gives up. Multi-currency and multi-warehouse catalogues hit all three at once: every variant claims the parent is canonical while the parent claims the default variant.

Pass rule Every indexable URL is self-canonical with an absolute URL, no chains, and no cross-domain canonical unless it is genuine syndication you can document.

The sitemap lists the wrong URLs Discovered

A sitemap is a hint, not an instruction, but a missing entry costs weeks for new URLs that have no internal links yet. Worse is the sitemap stuffed with 301s, noindex pages and parameter duplicates: that file teaches the crawler to distrust everything you submit. Split the XML by template so a broken product feed cannot poison the page feed.

Pass rule Every URL in the sitemap returns 200, is self-canonical and is indexable; the file is referenced in robots.txt and submitted per property, with a lastmod you can defend.

Internal links leave the page orphaned Discovered

If the only path to a page is the XML sitemap, it is orphaned in practice. Crawlers allocate attention by link equity and click depth. A page four or more clicks from the homepage, reachable only through a filter or a JavaScript menu, gets fetched last — or never. Audit with a crawl of the live site, not the CMS tree, because the CMS tree is a fiction the crawler cannot see.

Pass rule Every revenue URL is reachable within three clicks of the homepage through crawlable anchor tags, with at least two internal links from contextually related pages.

JavaScript hides the content and the tags Excluded

Disable JavaScript and load the page. If the copy, internal links, canonical tag or robots meta only appear after a client-side fetch, the renderer has to succeed before indexing is even possible. Renderers time out, and they are blocked by the same WAF rules that block other data-centre traffic. An API that refuses the renderer’s IP range looks identical to a page with no content.

Pass rule Primary copy, internal links, canonical and meta robots exist in the initial HTML, or server-side rendering produces them within the render budget.

Duplicates, crawl budget and server faults Discovered

Bursts of 5xx responses, slow time-to-first-byte, soft 404s that return 200, and parameter explosions all burn crawl budget. A site that answers in 2.8 seconds with 8,000 near-duplicate filter URLs will index slowly no matter how clean the sitemap looks. Treat this gate as capacity planning: the crawler spends what you give it, and every avoidable URL taxes the pages that matter.

Pass rule Error rate under 1% for 14 consecutive days, p75 time-to-first-byte under 800ms, faceted URLs handled deliberately, and unique copy on every indexable template.

Gate sequence

The four gates every URL has to pass

A URL is not indexed or excluded. It is at a specific stage, and each stage fails for different reasons.

GATE 1
Discovered
A link, sitemap entry or redirect put the URL in the queue. Failures here are low link equity, sitemap omission and orphan pages.
GATE 2
Crawled
A fetch was attempted and returned HTML. Failures here are robots blocks, 403s, 5xx storms, timeouts and redirect chains.
GATE 3
Rendered
The renderer executed the page and captured the copy and links. Failures here are client-only content and blocked renderer requests.
GATE 4
Indexed
The page survived deduplication and quality thresholds. Failures here are duplicates, thin copy, noindex inheritance and canonical conflicts.

Diagram

How robots.txt and sitemap.xml actually talk to the crawler

They are not two checklists. One controls permission, the other controls suggestion, and they meet at the fetch queue.

Fig. 1 — Permission, suggestion, queue, judgment. Fixing a lower gate never repairs a higher one.

Engagement

What changes when we take over the console

We work from server logs, rendered HTML and the coverage export — not from a template audit that tells you meta descriptions are short.

Log-file forensics

We map every Googlebot request to a template, separate fetch from render, and quantify how much crawl budget goes to URLs that can never rank. Our Technical SEO Services team runs this as a fixed-scope diagnostic with a written gate report, not a dashboard you have to interpret alone.

Indexability map

One sheet: every template, its canonical rule, its robots directive, its sitemap section and its owner. Conflicts become visible in minutes instead of surfacing one page at a time in Search Console.

Render parity checks

We compare raw HTML against rendered HTML for copy, links, canonical and schema, then hand developers a diff they can act on instead of a vague instruction to “make it SSR”.

Crawl-budget steering

Parameter handling, pagination rules, internal search blocking and sitemap segmentation. When we need to test template behaviour at scale without touching production, we use a Spider pool for rent to simulate fetch patterns safely.

Internal-link release

Module-level links from high-equity templates to commercial pages, with anchor text that matches the query rather than the internal project code name.

Index monitoring

Weekly coverage diffing with alerts when newly published URLs fail gate two or gate three, so a launch never again waits a month for its first fetch.

Remediation sequence

Five steps from “not indexed” to a stable queue

The order is deliberate. Each step removes a class of false signals before the next measurement.

Baseline the coverage report

Export every URL with its status, last crawl, canonical and sitemap membership. Freeze the list. Without a baseline you cannot prove that a change helped, and you will re-litigate the same pages next month.
Fix permission before quality

Clear robots blocks, WAF rules, noindex inheritance and 5xx bursts. Nothing downstream can be measured while the crawler is being refused at the door.
Resolve duplication and canonicals

One canonical per URL, no chains, filter URLs handled by rule rather than by accident. Merge or rewrite templates that differ only by a city name or a colour swatch.
Rebuild the path to the page

Add contextual internal links, submit a clean template-level sitemap, and request indexing only for pages that already pass gates one to three. Requests are a nudge, not a lever.
Verify and monitor

Re-crawl after seven days, compare against the baseline, then set weekly alerts. The end state is not “indexed today” but “new URLs reliably indexed within days”.

Field results

What the board looked like after remediation

Representative engagements, described by sector. Metrics are from the client’s own Search Console and analytics exports.

62 → 940

Industrial fasteners exporter

Indexed URLs in nine weeks after removing a global noindex and consolidating 300 canonical chains. Organic inquiries moved from 3 to 14 per month by month five.

4,100 → 180

Laboratory equipment manufacturer

Discovered backlog cut after faceted URLs were canonicalised and the sitemap was split into four template feeds. Organic inquiries 5 to 23 per month over six months.

−38% crawl

B2B packaging supplier

Crawl requests to filter pages fell 38%, while product page impressions rose 61% in the same quarter. Fewer fetches, more of the right ones.

Scope

Scope, engagement and what we need from you

Every engagement starts with a paid diagnostic because the fix list is different for a 200-page catalogue and a 40,000-URL marketplace. The diagnostic produces the gate report, the priority order, and a scope-based quote for implementation. No price appears on this page because a number without your log data would be a guess.

You provide read access to Search Console, one server or CDN log export covering 14 days, and a developer who can deploy header and template changes. We provide the diagnosis, the exact change list, the verification crawl, and a weekly coverage diff until the queue is stable.

“We had resubmitted the same forty product pages four times. The log file showed Googlebot never reached them: a staging password rule on the subdomain was still live, nine months after launch. Two days to fix, eleven weeks of revenue lost.”

— Technical lead, industrial components group

Questions

Answers before you open a ticket

How long until a fixed page is indexed?

Permission fixes often show a fetch within 24 to 72 hours because the URL is already in the queue. Canonical and duplication fixes take longer, typically two to six weeks, because the crawler has to re-evaluate what it already stored. Sitemap-only URLs with no internal links can wait months regardless of what you do.

Does resubmitting a URL in Search Console speed things up?

It can trigger a fetch, but a fetch is not an index. If the page fails the render or duplication gate, the request only adds a fresh “crawled, not indexed” timestamp. We use requests after the gates pass, never as a substitute for fixing them.

Why are only some of my pages indexed?

Usually because templates are treated differently by inheritance. A product template may be clean while its variant template carries a noindex from a theme update. Compare the indexed and excluded lists by template, not by individual URL, and the pattern appears quickly.

Is a sitemap enough to get pages indexed?

No. A sitemap gets a URL discovered; it does not earn a fetch or clear a quality threshold. Pages that exist only in XML tend to sit in “Discovered” indefinitely. Internal links from pages that already rank are the stronger signal.

Can you fix this without full developer access?

Sometimes. robots.txt, sitemap segmentation, internal linking and content consolidation sit with marketing. Header-level noindex, render-path fixes and WAF rules need engineering. When access is blocked, we write the change ticket with the exact diff so your team can deploy it.

How do you measure success?

Three numbers: valid indexed URLs per template, median time from publish to first crawl, and the share of crawl budget spent on URLs that can rank. Organic sessions follow, but coverage is the leading indicator and the one we report against.

Next step

Send us the coverage export. We will tell you which gate is failing.

One log file, one export, one written diagnosis. If the fault is a five-minute robots rule, we will say so and you can close the ticket yourself.