SEO Crawler & Technical SEO Audit Tool
Find the issue. Trace the source. Hand off the fix.
Crawl websites locally on your Mac. Find broken links, redirect chains, duplicate content, metadata, and indexability issues, then export a developer-ready CSV fix list.
Free: 100 pages · 100 links · 100 resources · No card required
From crawl to correction
Run a technical SEO audit from crawl to fix list.
For migration checks, launch-day QA, and the next client handoff. Keep the evidence and the work in one place.
Find what needs attention.
Open Issues to see problems grouped by category, from response codes and indexability to duplicate content and redirects.
Get to the affected URLs.
Select an issue, narrow the table with filters, and inspect the pages or links behind it. Move from the count to the evidence.
Leave with a fix list.
Export a CSV of issues and suggested fixes, or export Pages and Links for a deeper review with your team.
Your site. Your crawl.
A website crawler built for technical SEO investigations.
Choose what to crawl, control the pace, and examine the details that explain why a page needs work.
Explore the documentation- Precise scope
- Crawl a domain, subdomain, subfolder, or a list of URLs. Include URLs discovered through XML sitemaps.
- A pace you control
- Pause and resume a crawl. Configure processing concurrency and crawl delay before you start.
- Indexability & metadata
- Inspect robots directives, canonical tags, page titles, descriptions, and heading issues together.
- Duplicate content
- Find duplicate body content alongside duplicate titles and descriptions, so repeated pages are easier to investigate.
- Redirect chains
- Follow discovered redirect chains and identify links that should point directly to the final destination.
- Selected resource crawling
- Audit up to 100 selected resources per crawl in Pages. Choose images, CSS, JavaScript, media, or other assets.
- JavaScript rendering · Pro & Agency
- Render JavaScript to discover client-side content, links, and resources.
- Advanced Link Analysis · Pro & Agency
- Analyze unlimited hyperlinks and see every page-to-resource relationship for the resource types you choose.
Link diagnostics
Find broken links across your website and trace every source page.
Review internal and external links with source and destination evidence, link type, and crawl status in one place. Broken paths and redirecting links are easier to trace back to the page that needs the fix.
- Find broken internal links before users or search engines hit them.
- Spot redirecting internal and external links that need cleanup.
- See source and destination links in context.
Public demo crawl · Link detail
404 · Not Found- Filter the Destination URL column
- Paste a destination URL to isolate every link pointing to it. To match multiple destinations, separate them with the regex pipe (
|). - Destination
- /seo-crawler/fixtures/broken-target
Example fix list
The handoff should already make sense.
Crawl results are meant to become developer-ready fixes, not another raw export you have to rewrite before anyone can use it.
Download example fix list CSVSelected rows from a crawl scoped to /seo-crawler/fixtures/.
| Issue | Severity | Where | Fix |
|---|---|---|---|
| 4xx Client Errors | High | /seo-crawler/fixtures/broken-target | Restore the page, redirect it to a relevant live URL, or remove internal references. |
| Canonical points elsewhere | Medium | /seo-crawler/fixtures/blog/migration-checklist | Confirm the canonical target is intentional; otherwise point it at this page. |
| Missing title | Medium | /seo-crawler/fixtures/missing-metadata | Add a descriptive page title. |
| Multiple H1s | Low | /seo-crawler/fixtures/services/technical-audit | Keep one primary H1 and use lower-level headings for supporting sections. |
| Redirect chains | Medium | /seo-crawler/fixtures/redirect-a | Update links so they point directly to the final destination. |
Need Windows?
Join the Windows release list.
Available now for macOS. Join the list and we will email when the Windows version is ready to try.
Why we built this
“I've been doing SEO and auditing client sites for over 20 years. I built this crawler for the way I work: local, fast, and no cloud project required.”
Start with a real audit
Start with a free website crawler for Mac.
Audit larger sites in smaller batches. All four crawl scopes and resource-type filters are included in Free mode. Manually target sections or supply URL lists, with separate allowances for 100 page URLs, 100 selected resource URLs, and 100 analyzed hyperlinks per crawl.
| Feature | Free mode | Pro & Agency |
|---|---|---|
| Page URLs | 100 pagesIncluded per crawl | UnlimitedIncluded |
| Resource URLsSelected images, CSS, JavaScript, media, and other assets | 100 resourcesIncluded per crawl | UnlimitedIncluded |
| Links analyzed | 100 hyperlinksIncluded per crawl | UnlimitedIncluded |
| All crawl scopesDomain, Subdomain, Subfolder, URL list | Included | Included |
| CSV fix listWith suggested fixes | Included | Included |
| JavaScript rendering | Not included | Included |
| Advanced Link AnalysisEvery page-to-resource relationship for the resource types you choose appears in Links | Not included | Included |
Free mode limits are separate: up to 100 page URLs, 100 configured hyperlink records, and 100 unique selected resource URLs per crawl. Selected resources appear in Pages. Advanced Link Analysis, available with Pro and Agency, removes hyperlink limits and shows every page-to-resource relationship in Links for the resource types you choose.
Free mode needs no license key or payment card.
Try Free modePlans and checkout
Choose a plan and get your license key.
Start free for focused crawls. Upgrade for unlimited pages and resources, JavaScript rendering, and Advanced Link Analysis.
Questions
SEO crawler and technical audit questions
How the crawler works, what it checks, and the essentials on Free mode, device allowances, license transfers, and refunds.
What is an SEO crawler?
An SEO crawler follows links across a website and collects technical signals such as response codes, indexability, canonical tags, metadata, duplicate content, and redirects. Lil Robots SEO Crawler runs as a native macOS desktop app and processes crawl data locally.
What does Lil Robots SEO Crawler check during a technical SEO audit?
It checks pages, links, and the selected resource types you choose for broken URLs, redirect chains, indexability problems, canonical and metadata issues, duplicate content, and other crawl findings. Results can be filtered and exported as a CSV fix list.
Can SEO Crawler find broken links and their source pages?
Yes. The Links table shows the source URL, destination URL, link type, and crawl status so you can identify each page that links to a broken or redirecting destination.
Is there a free version?
Yes. Free mode includes up to 100 page URLs, 100 hyperlink records, and 100 unique selected resource URLs per crawl, with domain, subdomain, subfolder, and URL list scopes included. These three limits are separate. Selected resources appear in Pages, while Links remains limited to configured hyperlinks. You can choose which resource types to audit and target larger sites in sections or URL lists. Upgrade for unlimited page and resource crawling, JavaScript rendering, and Advanced Link Analysis, which includes unlimited hyperlinks and shows every page-to-resource relationship in Links for the resource types you choose.
Does the app need an internet connection?
Crawl data is processed locally on your Mac. You need network access to reach the sites you crawl, and paid features require an internet connection for license activation and periodic validation. Local processing does not mean offline licensing.
How do I use my license?
After purchase, your license key is saved in your SEO Crawler account. Copy it into the macOS desktop app to unlock paid access on the number of active devices included with your plan.
Can I transfer my license to another device?
Yes. Deactivate the license on one machine and activate it on another. Your plan controls how many devices can stay active at the same time.
What is the difference between Pro Monthly, Pro Yearly, and Agency Yearly?
Pro Monthly is $19/mo for 2 active devices. Pro Yearly is $179/yr for 3 active devices. Agency Yearly is $399/yr for 15 active devices and is the only plan that supports additional device packs.
Can I buy more devices?
Yes, on Agency Yearly. Agencies can add additional device packs for $99/yr per 5 active devices during checkout or later from the license dashboard.
What is the refund policy?
Pro Monthly subscriptions can be canceled anytime and stay active until the paid period ends. Pro Yearly and Agency Yearly subscriptions include a 7-day refund window from the purchase date, then the purchase is final.