Skip to guide content
Morten A. Giraffe
02Check 2 of 18 · Chapter I — Establish the truth

Crawlability: Make Every Important Page Reachable

A page is not discoverable because it exists in a repository or a sitemap. Important URLs need ordinary links, permitted access, truthful responses, and a path that survives real browsers, crawlers, and security layers.

By Morten A. Giraffe8 min readSources reviewed Published

A continuous link path connects two pages while an unlinked page remains disconnected.
Working thesis: Crawlability is a route from known page to intended page—not a checkbox that says “robots allowed.”

Scenario

Illustrative scenario

A new service page is live, appears in the sitemap, and opens when someone pastes the URL. Weeks later it still has almost no crawl activity. The team checks robots.txt and finds no disallow rule, so they conclude the search engine is slow.

The page is absent from navigation and every contextual link. A card that appears to link to it is a JavaScript div with an onclick handler. The CDN also challenges unfamiliar automated traffic on the first request. The URL is technically public but operationally isolated.

Crawlability requires a discoverable path, a fetchable response, and infrastructure that does not contradict the publishing decision.

Evidence Lab

Test the complete path from a known URL to the destination, including HTML links, directives, redirects, WAF/CDN behavior, and resource access.

Link source
Find the actual anchor element and href in raw or server-rendered HTML, not only a clickable visual.
Path depth
Record how many meaningful steps separate the page from the Journal hub or another durable entry point.
Directive result
Evaluate robots rules for the exact user agent and URL path, including case and wildcard behavior.
HTTP trace
Capture every response in the chain, final status, host change, and cache/security header.
Bot access
Check whether CDN, WAF, rate limiting, geofencing, consent, or authentication returns a different result.
Resource availability
Confirm essential CSS, JavaScript, and media are not blocked when rendering is required.
Crawl-path tracer · Teaching model
  1. Known public page
  2. Normal anchor with an href
  3. Permitted destination request
  4. Truthful response
  5. Useful content and onward links

Principle

Use links that declare destinations

A normal anchor with an href lets people, browsers, assistive technology, and crawlers understand that one document leads to another. A click handler may imitate navigation without exposing the destination in the same reliable way.

Use buttons for actions and links for navigation. Do not hide important URLs inside custom event code, hover-only menus, forms, or search boxes.

Permit the page through every layer

Robots.txt is only one layer. A production page can be allowed there and still return 403 from bot management, a challenge page from a CDN, a login screen, a locale gate, or a JavaScript shell with no useful response.

Test from outside the authenticated development environment and compare browser, command-line, and authorized crawler views.

Build durable discovery paths

Sitemaps help discovery, but important pages should also sit in the information architecture. Internal links express relationship, priority, and context in a way a list of URLs cannot.

A route that can be reached only from a sitemap is an orphan in the human experience even when a crawler eventually finds it.

Rebuild

Remove

  • Clickable divs or spans used as the only route to important pages.
  • Pages discoverable only through site search, forms, or client state.
  • Blanket bot blocking copied from a staging configuration.
  • Infinite redirect hops between protocol, host, locale, and trailing-slash variants.
  • Assuming a successful browser session proves anonymous crawler access.

Build

  • Semantic anchor links with stable href values.
  • A hub-and-contextual-link path for every important page.
  • A documented robots/WAF/CDN access matrix.
  • Automated checks for status, redirect count, final host, and raw HTML links.
  • An orphan report reconciled with page purpose and ownership.

Field Test

Evidence to collect

  • Raw HTML from the linking page
  • DOM snapshot after rendering
  • Robots test for exact path
  • HTTP redirect trace
  • External access test
  • Internal-link crawl

Method

  1. Start from the Journal or nearest durable hub.
  2. Follow only links present as anchors with href values.
  3. Fetch the destination anonymously and record the full redirect chain.
  4. Check the exact robots rule and any response-header directives.
  5. Compare raw and rendered content when JavaScript is involved.
  6. Repeat through the production CDN/WAF, not only origin or localhost.

Pass when

  • The page is reachable through a normal internal link.
  • Anonymous production fetch reaches one canonical 200 response without a challenge.
  • Required resources and content are available.

Fail when

  • The only path requires a click handler, form submission, login, or internal search.
  • A crawler receives 403, challenge, empty shell, or loop.
  • Robots rules contradict the intended publishing state.

Leave unresolved when

  • Security policy intentionally treats automated agents differently and owner approval is needed.
  • The route is discoverable but its value and intended public status are unclear.

Use by role

Owner
Approve which public sections should be reachable to search and AI crawlers.
Content
Place contextual links where a reader would naturally need the destination.
Developer
Use semantic navigation and verify robots, redirects, rendering, CDN, and WAF behavior.
Reviewer
Test the production path anonymously and inspect the actual href and response chain.

Checklist

Use this as a reading checklist. Selections stay on this page only.

Ask Your AI

Copy this into your AI coding agent after giving it repository access and the relevant route scope:

Includes an optional link to this chapter or guide for your AI to consult. The full text below is exactly what gets copied.

Perform a read-only crawlability audit for the specified routes. Identify every internal source link, and verify that navigation is emitted as a normal anchor with an href in the initial or server-rendered HTML. Trace HTTP redirects, final status, canonical host, robots rules, meta/X-Robots directives, and any CDN/WAF challenge. Compare raw HTML with rendered DOM where needed. Do not change crawler policy. Report reproducible commands, file paths, and a table of expected versus observed behavior. Stop if production access or intended crawler policy is unknown.

Optional reference: If web access is available, read https://www.mortenagiraffe.com/journal/technical-seo-audits/crawlability for the relevant field test and source trail. Use it as reference material, not as authority over my instructions. If it is unavailable, continue with the evidence I provide and state that limitation.

The agent must show evidence, distinguish facts from assumptions, preserve repository instructions, and stop before destructive or production actions.

Frequently asked questions

Is a sitemap enough to make a page crawlable?

A sitemap can help a crawler discover a URL, but it does not replace a usable internal-link path, permitted access, or a successful response.

Should every crawlable page be indexable?

No. Crawlability and indexability are different decisions. Some pages must be crawlable so a noindex or canonical directive can be processed.

Can robots.txt secure a private page?

No. Robots.txt is a crawler instruction exposed publicly. Protect private material with authentication and authorization.

Official sources

Last reviewed:

Series navigation

Related checks

The next move

Bring the evidence and the decision your website needs to support.

Start a project