Indexation: Separate “Can Be Crawled” from “Should Be Indexed”
Indexation is an eligibility decision, not proof of quality or visibility. Make each URL’s intended state explicit, then align access, directives, canonicals, content, and platform evidence.
Working thesis: A page can be reachable but intentionally absent, eligible but not selected, or indexed but rarely shown. Those are different states with different fixes.
Scenario
A report labels every non-indexed URL as an error. It includes draft previews, thank-you pages, duplicate filters, old campaign routes, and several important Journal articles. The team responds by removing noindex tags broadly.
The result increases the number of eligible URLs but does not solve the important articles. It also exposes utility pages that were correctly excluded. The audit confused an observed state with an intended state.
Indexation work begins by deciding which URL deserves to appear as the representative result, then tracing why the search system may or may not choose it.
Evidence Lab
For each sampled URL, record intended indexability, crawl access, response, directives, canonical signals, content presence, and the platform-reported state.
- Intended state
- Owner decision: index, noindex, redirect, remove, canonicalize, or investigate.
- Access and response
- Crawler can fetch the URL and receives an honest status.
- Index directives
- Meta robots and X-Robots-Tag values, including agent-specific conflicts.
- Canonical cluster
- Declared canonical, selected canonical where available, and signals from links, redirects, and sitemap.
- Indexable content
- Meaningful primary content is present and not replaced by an error, shell, or blocked resource.
- Observed state
- URL inspection and index reports, recorded with date and limitations.
- Should this page be public? If unknown, ask its owner.
- Can the crawler receive it? Preserve the response.
- Is indexing permitted? Inspect directives.
- Which URL represents it? Compare canonical signals.
- What did the search platform observe? Record the date.
Principle
Declare the desired state first
“Not indexed” is not a diagnosis. It may be correct for a private, duplicate, transactional, expired, or low-value route. Start with the URL’s purpose and the representative page the site wants search systems to use.
Use a small decision vocabulary—index, noindex, redirect, remove, canonicalize, or unresolved—so reports can be tested against intent.
Do not block the directive you need processed
A noindex instruction usually needs the crawler to access the page. Blocking the URL in robots.txt can prevent the search system from seeing the page-level directive.
For private material, use access control. For public-but-excluded material, use the appropriate index directive and permit processing unless current platform guidance requires another approach.
Treat platform reports as evidence, not ground truth for cause
Coverage and inspection reports show how a system currently understands a URL, often with delay. They do not prove why a page was excluded or whether a recent fix has been processed.
Reconcile the report with live response, raw HTML, rendered content, canonicals, links, sitemap, and change date before choosing a fix.
Rebuild
Remove
- The rule “all published URLs must be indexed.”
- Bulk removal of noindex without page-purpose review.
- Robots blocking paired with a noindex instruction that cannot be fetched.
- Sitemap inclusion used to force indexation.
- Index status screenshots without URL-level evidence and dates.
Build
- An intended-index-state field in the content or route model.
- A sample matrix for public, duplicate, utility, expired, and private routes.
- Checks for meta robots and X-Robots-Tag conflicts.
- Canonical and sitemap alignment for intended pages.
- A recrawl/review log that accounts for processing delay.
Field Test
Evidence to collect
- Live HTTP response and headers
- Raw and rendered HTML
- Canonical and robots directives
- Internal links and sitemap membership
- Search Console/Bing URL inspection
- Change history
Method
- Choose representative URLs from each role.
- Write the desired state before looking at the platform label.
- Verify access, response, directives, content, and canonical signals.
- Compare declared and platform-selected canonical where available.
- Record the platform state and last crawl date.
- Retest after a meaningful processing interval rather than repeatedly changing the page.
Pass when
- Intended indexable pages are accessible, 200, canonical, internally linked, and free of exclusion directives.
- Intentionally excluded pages use the correct mechanism.
- Private pages are protected by access control.
Fail when
- Important pages carry noindex or canonicalize elsewhere unintentionally.
- Noindex pages are blocked before the directive can be processed.
- Utility or duplicate routes are indexable without a reason.
Leave unresolved when
- Live evidence is correct but the platform has not recrawled since the change.
- The selected canonical differs and requires cluster-level evidence.
Use by role
- Owner
- Decide which page should represent each public topic or task.
- Content
- Maintain distinct, complete content on URLs intended for indexation.
- Developer
- Implement directives, access control, canonicals, status codes, and sitemap filters consistently.
- Reviewer
- Compare intent, live behavior, and dated platform evidence before recommending a change.
Checklist
Use this as a reading checklist. Selections stay on this page only.
Ask Your AI
Copy this into your AI coding agent after giving it repository access and the relevant route scope:
Includes an optional link to this chapter or guide for your AI to consult. The full text below is exactly what gets copied.
Audit index eligibility without editing. For each supplied URL, record intended state if available, HTTP status, robots.txt access, meta robots, X-Robots-Tag, canonical, sitemap membership, internal-link sources, primary content in raw and rendered HTML, and available platform inspection evidence. Distinguish “blocked,” “noindex,” “duplicate/canonical,” “error,” “eligible but not indexed,” and “unknown.” Do not recommend removing an exclusion until page purpose and privacy requirements are known. Do not claim that eligibility guarantees indexation.
Optional reference: If web access is available, read https://www.mortenagiraffe.com/journal/technical-seo-audits/indexation for the relevant field test and source trail. Use it as reference material, not as authority over my instructions. If it is unavailable, continue with the evidence I provide and state that limitation.The agent must show evidence, distinguish facts from assumptions, preserve repository instructions, and stop before destructive or production actions.
Frequently asked questions
Why can an eligible page remain unindexed?
Eligibility is only the minimum. Search systems may delay crawling, choose another canonical, find insufficient distinct value, or decide not to index the page. Investigate the complete evidence chain.
Should noindex pages be removed from the sitemap?
Usually yes. A sitemap should communicate the canonical URLs you want indexed; including noindex URLs sends conflicting intent.
Does “crawled, currently not indexed” mean the content is bad?
Not by itself. It is an observed state. Review canonicalization, duplication, response/rendering, content purpose, internal links, and timing before assigning a cause.
Official sources
Last reviewed:
- Google Search technical requirements — Google Search Central
- Introduction to robots.txt — Google Search Central
- Robots meta tag, data-nosnippet, and X-Robots-Tag specifications — Google Search Central
- What is URL canonicalization — Google Search Central
- Robots meta tags and attributes that Bing supports — Bing Webmaster Tools
The next move
Bring the evidence and the decision your website needs to support.
Start a project