Follow the response.
Check a small batch of URLs. See the status, the redirect path and where each request actually ends.
How this works
The checker verifies robots.txt, makes an identified GET request and closes the response without downloading the page body. It follows at most three validated redirects and checks each new origin’s crawler rules.
What to keep in mind
Servers may respond differently to different clients. A 200 response does not establish indexing or content quality. Blocked requests, unsupported targets and exhausted quotas are reported as unavailable, not as a status-code guess.