Workflow guide · 4 minute read
Check page indexability
A page can load perfectly and still tell search engines not to index it. Find the affected URLs, read the reason, and decide whether the exclusion belongs there before changing a template or opening a ticket.
Best for launch checks, migrations, and unexpected visibility changes
16-second workflow
A 200 response is only part of the story.
Recorded in the installed SEO Crawler 0.1.8 app using deliberate public fixtures. The demo starts after the crawl and ends on the indexing reason; it does not change the site.
What happens in the demo
Start with the completed crawl
All 14 fixture URLs are visible. The relevant columns are already enabled and no cell is selected.
Choose Non-indexable in Issues
The table narrows to 12 URLs. The panel stays open so the selected issue and its result can be read together.
Identify the page, then check its response
The fixture index URL is selected first. The next selection highlights its 200 OK response: the page loads, but that does not tell the whole indexing story.
Read the indexing reason
The horizontal scrollbar brings Indexing Reason into view. The final selection highlights Meta robots noindex for the same first row.
Catch the exclusion before chasing the wrong problem.
A successful response code can make a launch look healthy while a staging directive is still present. Checking page source one URL at a time is slow; this view keeps the URL, response, and indexing finding in the same review.
Use it after moving staging to production, changing SEO settings in a CMS, rebuilding a shared template, or investigating a group of pages that should be discoverable. Start with important public pages, not a target of making every URL indexable.
Review a page from URL to reason.
- 1.
Crawl the site
Enter the site URL, run the crawl, and wait for it to finish. The demo starts with completed results from our public fixture site.
- 2.
Show the evidence columns
In Pages, open Filter and enable Status Code, Indexability, and Indexing Reason. URL stays visible by default. Hide unrelated columns to keep the review focused.
- 3.
Filter through Issues
Open Indexability in the Issues panel and choose Non-indexable. Keep the panel open. For a broader audit, review Blocked separately too; it is a different issue group in this crawl.
- 4.
Follow one row across the table
Select the page URL, check its Status Code, then use the bottom horizontal scrollbar to reach Indexing Reason. Read the reason before deciding which setting needs attention.
Practice with the public crawler fixtures. Their exclusions are intentional test data, not settings to remove from the demo site.
Read the finding before choosing a fix.
- 200 OK + Meta robots noindex
- This is the first row in the demo. The request succeeds, but the page declares noindex. For an intended public search landing page, check the CMS field or template that produces the directive. For a deliberately excluded page, keep it.
- More than one reason
- The migration checklist also shows Canonicalized. Review each signal; removing a noindex directive does not resolve an unrelated canonical target. Follow the canonical review workflow for that comparison.
- Redirects and blocked results
- The same filtered table contains 302 responses with blank reason cells. Read Status Code as well. The fixture 404 appears under Blocked instead; that label alone is not proof of a robots.txt restriction.
Verify the correction and keep the evidence.
If the exclusion is unintended, correct the setting at its source, deploy it, and crawl the affected URL again. Check the rendered directive or response header where relevant, and make sure another template rule is not adding the exclusion back.
A useful handoff includes the URL, status code, exact indexing reason, intended visibility, and the template or configuration to inspect. Recheck related pages when the cause is shared. An eligible page still is not guaranteed to be indexed.
Turn the findings into a fix list