Stop one
Crawling: can a bot reach the page?
Crawling is discovery plus fetch. A crawler finds URLs from links, sitemaps, and previously known pages, then requests them. What it gets back decides whether the rail continues. This stop is entirely mechanical, which makes it the easiest to verify and the most embarrassing to skip.
- A page is live but never appears in coverage reports.
- Server logs show bots hitting category pages but never product pages.
- Crawl requests return 5xx under load and 200 when tested alone.
- Robots rules block CSS or JS that the page depends on.
- Unblock the path in robots, then resubmit the sitemap.
- Link new URLs from pages that are already crawled.
- Fix rate limits and timeouts on the origin server.
- Keep important URLs out of infinite parameter loops.
Most crawl problems are link problems wearing a technical costume. If nothing links to a page, the crawler has no reason to visit it. The mechanics of making a site fetchable are covered in our Technical SEO Services page, which lists the checks in the order they fail.