Feature guide · 5 minute read
Find CSS and JavaScript blocked by robots.txt
A page can return 200 OK while the files that shape or power it are unavailable to crawlers. Bring those resources into the Pages table, isolate the blocked paths, and decide whether the robots.txt rule or the page dependency is the real problem.
16-second workflow
Confirm a known robots.txt block without guessing.
Recorded in the installed SEO Crawler 0.1.8 app using a deliberate public JavaScript fixture. This direct check is repeatable in Free mode. Automatic sitewide CSS and JavaScript discovery requires Crawl full resources. The matching light or dark recording loads with the viewer's theme; the other file is not fetched.
What happens in the demo
Start with a known JavaScript resource
The public fixture makes the check safe and repeatable without relying on private client data.
Crawl the resource directly
The file returns 200 OK, but the crawler still classifies it as Blocked because access policy is a separate signal from response status.
Choose Blocked in the Issues panel
The issue selection stays visible beside the filtered row, connecting the summary count to its evidence.
Read the blocking reason and content type
Move across the table to confirm Disallowed in robots.txt beside application/javascript before deciding what should change.
The quiet failure
The HTML loaded. The useful page may not have.
Blocked CSS can hide layout and visibility problems from a crawler. Blocked JavaScript can prevent rendered navigation, copy, and internal links from being understood. Looking only at the document status makes these failures easy to miss.
Collect the evidence this check needs.
Standard crawls focus on HTML pages. Enable full resource crawling before the crawl so supporting files can appear as their own rows and feed the Resources issue groups.
- 1.
Clear the current crawl
Crawl settings are locked while results are open. Clear the current crawl before changing resource collection.
- 2.
Open Crawl Configuration
Open Settings and enable Crawl full resources. This adds selected supporting files, including JavaScript and CSS, as rows in Pages.
- 3.
Keep robots.txt enforcement selected
Leave Respect crawl directives enabled and keep robots.txt selected so the crawl records the same restriction a compliant crawler encounters.
- 4.
Run a fresh crawl
Start from the site or section you need to audit and wait for resource processing to finish before reviewing issue counts.
Crawl full resources is a paid feature. A free crawl can still audit the HTML pages, links, metadata, headings, and indexability included in the free limits.
Use Resources to reduce the crawl to one cause.
Expand Resources in the Issues panel. Choose the JavaScript or CSS robots.txt group first, then read the filtered rows before changing any directive.
| Issue group | What it means | Check first |
|---|---|---|
| JS files (blocked by robots.txt) | JavaScript resources discovered by a page match a disallow rule. | Confirm whether the script affects navigation, rendering, or visible content. |
| CSS files (blocked by robots.txt) | Stylesheets discovered by a page match a disallow rule. | Check whether the blocked stylesheet controls layout or content visibility. |
| JS or CSS files (4xx/5xx) | The resource was requested but returned an error response. | Repair the URL, deployment, or server response instead of editing robots.txt. |
Read the resource row beside the rule.
Keep URL, Content Type, Indexability, Indexing Reason, and Discovery Source visible. Together they show what file was discovered, what kind of resource it is, and why it was classified as blocked.
Example directive
User-agent: *
Disallow: /assets/private/
Disallow: /scripts/blocked-render.jsCompare each filtered resource URL with the disallow patterns. Broad directory rules, stale asset paths, and copied staging directives are common causes. Confirm the dependency on the affected page before editing a rule that may protect other paths.
Fix the narrowest responsible layer.
A public page depends on the blocked file
Narrow or remove the disallow rule, then verify the resource is crawlable.
The path is private, administrative, or intentionally restricted
Keep the rule and remove any public-page dependency on that resource.
The asset moved or no longer exists
Update the page to request the correct file and review the separate 4xx/5xx group.
A broad pattern catches unrelated assets
Replace it with the narrowest rule that protects only the intended paths.
Do not open every blocked directory simply to clear the report. The goal is to make resources required by public pages crawlable while keeping intentional restrictions precise.
Deploy, recrawl, and inspect the page again.
After changing robots.txt or the page dependency, run a fresh full-resource crawl. Confirm the resource has left the robots-blocked group, returns the intended response, and is still requested from the correct public page.
A useful handoff names the affected resource URL, the matching directive, the page that depends on it, and the expected post-fix behavior. That gives developers enough context to repair the cause without turning robots.txt into a guessing exercise.
Export the affected resources for handoff