URL Parameters and Facets: Stop Infinite Crawl Spaces
Filters, sorts, tracking values, and state parameters can create more URLs than the site has meaningful pages. Decide which combinations deserve stable public URLs and prevent everything else from becoming a crawlable maze.
Working thesis: A parameter is not automatically a problem. An uncontrolled parameter space is.
Scenario
A collection has five filters, four sort orders, pagination, tracking parameters, and a view toggle. The interface exposes combinations as crawlable links. A simple crawl expands from dozens of products to tens of thousands of URL states.
Most combinations contain the same items in a different order or leave only one result. Canonicals point back to the collection, but the crawler must still discover and request the variants before it can interpret the hint.
The durable fix is to define the small set of combinations that serve real demand, then stop generating or linking the rest.
Evidence Lab
Measure how parameters are created, linked, combined, and requested before selecting robots, noindex, canonical, or routing controls.
- Parameter registry
- Name, source, allowed values, order sensitivity, persistence, and whether it changes primary content.
- Generation path
- Navigation, forms, JavaScript state, tracking links, feeds, pagination, and external campaigns.
- Combination growth
- Theoretical and observed URL counts, including repeated values and alternate order.
- Demand/value
- Search demand, user task, inventory depth, uniqueness, conversion role, and maintenance owner.
- Crawler behavior
- Log requests, crawl reports, response codes, canonicals, and query-state rendering.
- Control interaction
- Whether blocking prevents processing of canonical or noindex signals, and whether controls affect users.
- 3 hypothetical colors × 4 sizes × 2 sort orders = 24 combinations
- Meaningful demand and distinct content determine which combinations deserve pages.
- This illustrates multiplication; it is not a measured crawl budget.
Principle
Classify parameters by what they change
Tracking parameters usually do not change primary content. Sorting changes order. Filtering changes the set. Pagination changes the slice. Session and view state may change presentation only. These roles need different treatment.
A registry prevents one blanket rule from breaking useful pages or leaving infinite variants open.
Create indexable combinations deliberately
An indexable facet should have a stable URL, meaningful demand, enough distinct inventory or content, unique metadata, internal-link placement, and an owner. It should not exist because every possible combination can be serialized.
Prefer a curated landing-page model over automated indexation of every filter state.
Reduce generation before adding crawler rules
The strongest control is often not emitting unnecessary URLs as links. Normalize parameter order, remove tracking values from internal navigation, constrain combinations, and use state that does not create a new crawlable URL when no durable page is intended.
Robots, noindex, and canonical each solve different problems and can conflict when combined without a plan.
Rebuild
Remove
- Tracking parameters in internal links.
- Every filter and sort rendered as a crawlable URL.
- Multiple parameter orders producing the same state.
- Blanket robots blocking applied before understanding noindex/canonical processing.
- Canonical tags used as permission to generate unlimited variants.
Build
- A parameter registry and indexability decision table.
- Curated, stable landing URLs for valuable combinations.
- Normalized parameter order and value validation.
- Non-link UI controls for states that are not intended as documents, while preserving accessibility.
- Log and crawl monitoring for new parameter families.
Field Test
Evidence to collect
- Route/query parsing code
- UI link output
- Parameter crawl sample
- Server logs
- Canonical/index directives
- Search landing/query data
Method
- List every parameter and the primary content effect.
- Generate combinations for representative categories and count growth.
- Identify which combinations have a distinct task, demand, and maintainable result set.
- Inspect whether unwanted states are emitted as anchors.
- Test response, canonical, noindex, and robots interaction for each parameter class.
- Monitor logs after changes for new crawl paths.
Pass when
- Only approved combinations become durable crawlable pages.
- Tracking and display-state parameters are absent from internal links.
- Equivalent orders normalize to one URL.
- Controls align with desired index and crawl behavior.
Fail when
- Crawl count grows combinatorially with each filter.
- Empty or trivial combinations remain indexable.
- The crawler is blocked from seeing the directive intended to exclude a URL.
- Internal links continuously regenerate variants.
Leave unresolved when
- A facet has demand but insufficient stable inventory or content.
- A product/collection decision is needed before technical controls can be chosen.
Use by role
- Owner
- Approve which filter combinations deserve a maintained public landing page.
- Content
- Provide distinct context and metadata for curated landing pages.
- Developer
- Control parameter generation, normalization, routing, linking, and directives.
- Reviewer
- Test combinatorial growth and control interactions with logs and crawls.
Checklist
Use this as a reading checklist. Selections stay on this page only.
Ask Your AI
Copy this into your AI coding agent after giving it repository access and the relevant route scope:
Includes an optional link to this chapter or guide for your AI to consult. The full text below is exactly what gets copied.
Audit URL parameters and faceted navigation without changing code. Extract parameter names and allowed values from routes, components, forms, and link builders. Classify each as tracking, sort, filter, pagination, session, view state, search, or unknown. Estimate combination growth, identify which states are emitted as anchors, and sample their status, canonical, robots/noindex behavior, content difference, and sitemap presence. Use logs or crawl evidence when available. Recommend generation controls before crawler directives. Do not approve indexable facets without evidence of distinct user value and ownership.
Optional reference: If web access is available, read https://www.mortenagiraffe.com/journal/technical-seo-audits/url-parameters-and-facets for the relevant field test and source trail. Use it as reference material, not as authority over my instructions. If it is unavailable, continue with the evidence I provide and state that limitation.The agent must show evidence, distinguish facts from assumptions, preserve repository instructions, and stop before destructive or production actions.
Frequently asked questions
Should all parameter URLs be blocked in robots.txt?
No. Some parameters represent useful pages, and blocking can prevent processing of page-level signals. Choose controls by parameter role and intended state.
Can a canonical solve facet crawl waste?
It can help consolidate duplicates after they are fetched, but it does not stop the site from generating or crawlers from discovering an enormous space. Reduce URL generation first.
When should a filtered page be indexable?
When it serves a distinct, durable user need, has enough stable content or inventory, can be linked and maintained, and is not merely a temporary interface state.
Official sources
Last reviewed:
- Managing crawling of faceted navigation URLs — Google Search Central
- URL structure best practices — Google Search Central
- What is URL canonicalization — Google Search Central
- Introduction to robots.txt — Google Search Central
- Robots meta tag, data-nosnippet, and X-Robots-Tag specifications — Google Search Central
The next move
Bring the evidence and the decision your website needs to support.
Start a project