Technical explainer · machine cutaway
Search Engine Optimization: How Stuff Works Inside the Machine
A search engine works in five stages: it discovers a URL, crawls it, indexes the content, ranks stored pages for a query, and serves a result page. Search engine optimization is the work of keeping each stage unblocked and passing the right material to the next one. Most ranking complaints are actually failures one or two stages earlier.
Five stages. Five failure modes. One diagnosis order.
How does a search engine actually work?
A search engine discovers URLs, fetches them with a crawler, parses and stores the content in an index, scores that index against a query, and serves a ranked result page. Each stage passes a smaller set of pages to the next, which is why crawled, indexed, and ranked are three different outcomes rather than one.
01Why the sequence matters more than any single fix
A page that is blocked from crawling cannot be indexed, and a page that is not indexed cannot rank. When a site loses traffic, the temptation is to rewrite titles and add keywords. But if the URL dropped out of the index three weeks earlier, the title was never the problem.
The cutaway gives you a diagnosis order. Start at discovery and move right until you find the first chamber that is failing. Everything downstream of that chamber is a symptom, and fixing the symptom first usually wastes the quarter.
What each chamber receives and passes on
Discovery passes a list of URLs. Crawling passes HTML and headers. Indexing passes stored documents with a canonical and a language. Ranking passes a scored subset for one query. Serving passes a layout.
When someone says "the page is not working", ask which pass they mean. The answer is usually visible in a coverage report within a minute.
02Inside the five chambers
Discovery
learns that a URL existsThe engine learns about a URL from a sitemap, an internal link, an external link, or a manual submission. Discovery only fills a queue. It does not fetch anything and it does not index anything.
What you control: a sitemap that lists canonical URLs only, internal links from pages that are already crawled, and a clean status code at the URL itself. A sitemap full of orphan pages trains the engine to ignore your sitemap.
Crawling
fetches and rendersThe crawler requests the URL, follows redirects, reads robots.txt, and renders JavaScript when it has budget to do so. The status code decides what happens next: 200 passes content forward, 301 passes the target, 404 and 410 retire the URL, and a 5xx sends it back to the queue.
What you control: response time, redirect depth, robots rules, and how many near-duplicate URLs you expose through filters and parameters. On a large catalogue, thousands of low-value URLs can consume the attention that product pages needed.
Indexing
stores and canonicalizesThe engine parses the response, extracts text and links, chooses a canonical, and stores the document in a structure it can query. Duplicates collapse here, noindex is honoured here, and empty renders are discarded here.
What you control: one indexable version of each page, a self-referencing canonical, server-rendered content for anything that must be read, and no accidental noindex left from staging. A page can be crawled daily for a month and still never enter the index.
Ranking
scores pages against a queryFor a query, the engine retrieves candidate documents from the index and scores them on relevance, links, freshness where it applies, locale, and page experience. It ranks a page for a query in a context, not a site in general.
What you control: whether the page clearly answers the query, whether other pages support it with internal links, whether the format matches what already ranks, and whether the content is specific enough to be chosen over a generic competitor.
Serving
assembles the result pageServing builds what the searcher sees: ten links, a map pack, a shopping strip, a video row, a featured snippet, or an AI summary. The same index can produce very different layouts for two similar queries.
What you control: title and snippet inputs, structured data that qualifies the page for a feature, and the page experience behind the click. A ranking you cannot see is often a feature that pushed the blue links down.
Send a domain and one query. We trace it stage by stage and name the block.
03What failure looks like at each stage
The symptom is rarely in the same chamber as the cause. Use the third column as a starting point, then walk left until the report changes.
| Stage | Typical failure | Symptom you observe |
|---|---|---|
| Discovery | Orphan pages, a stale sitemap, no internal links into a new section | New URLs sit in Discovered - currently not indexed for weeks |
| Crawling | robots.txt blocks, redirect chains, repeated 5xx responses | Blocked by robots.txt, or crawl stats falling while the site grows |
| Indexing | noindex left on, canonical pointed at another URL, thin duplicate pages | Excluded by noindex tag or Alternate page with proper canonical tag |
| Ranking | Wrong intent, thin answer, no supporting internal links | Indexed, but no impressions for the query you built the page around |
| Serving | Weak title and snippet, missing structured data, slow page | Impressions with a click rate far below the results around you |
04Indexed is not ranked
An index is a library. Ranking is a librarian answering one question at a time. A book can sit in the library for years and never be handed to anyone, and that is not a shelving error.
Index membership means the engine has stored a document and considers it eligible. Ranking means the engine retrieved that document for a specific query and placed it above others. The two are separated by relevance, competition, and context, and none of them can be fixed by resubmitting a URL.
This distinction explains the most common false diagnosis in search work: a page is indexed, the coverage report is clean, and the team keeps requesting indexing. Nothing changes, because the failure is in the query-to-page match, not in storage.
Fig. 2 — inputs, machine, outputs.
05What this means for a real site
If you take one habit from the cutaway, take the left-to-right check. Before writing a new title, confirm the URL is discoverable, returns 200, is indexable, and has at least one internal link from a page that is already crawled. That check takes minutes and eliminates a large share of "mystery" ranking problems.
The underlying concepts are covered in plain language in Whats Is SEO?, which is a better starting point than a tool dashboard if the team is new to this. When the diagnosis needs log files, render checks, and a crawl budget plan, Technical SEO Services covers the machine room rather than the copy.
How we read a site in one pass
- Coverage first
Which URLs are discovered but not indexed, and when did that start? The date usually points at a deploy.
- Crawl second
Response codes, redirect depth, robots rules, and where the crawl attention is going instead of product pages.
- Index third
Canonicals, duplicates, rendered content, and whether the page the engine stores is the page you wrote.
- Rank and serve last
Only now compare titles, snippets, and intent. Fixing these first is the expensive way to learn the same lesson.
06Questions about how search works
What is the difference between crawling and indexing?
Crawling is fetching a URL and reading the response. Indexing is deciding whether the content is worth storing and retrieving later. A page can be crawled many times and never indexed, which is why the two appear as separate items in a coverage report.
How long does indexing take?
From hours to several weeks. It depends on how often the site changes, how many internal links point at the page, and how much crawl attention the site has earned. No submission tool changes that relationship, though it can shorten the queue.
Why is my page indexed but not ranking?
Indexing only means the page is stored. Ranking compares stored pages against a specific query, so a page can be indexed and never retrieved for the phrase you care about. Look at whether the page answers that query better than the pages already shown.
Does a sitemap guarantee indexing?
No. A sitemap helps discovery and can carry hints, but the engine still decides whether to crawl and whether to index. A sitemap full of orphan pages is discovered and then largely ignored.
What is crawl budget?
The amount of crawling a site can expect in a period. Large sites waste it on filtered URLs, redirects, and error pages, which leaves fewer fetches for the pages that should be indexed. The fix is usually fewer URLs, not more server capacity.
Can a page rank without being indexed?
No. Serving draws from the index, so a page outside it cannot appear. What can happen is that a different URL from your site is indexed and served in place of the one you intended.
Find the chamber that is failing before you rewrite the copy
Send your domain and the query you expected to rank for. We will trace it through discovery, crawl, index, rank, and serve, then tell you which stage is actually blocking it.