Crawler transparency

Lil Robots SEO Crawler user agent

If you see Lil Robots SEO Crawler in your logs, this page explains what it is, why it visits sites, and how site owners can control access.

User agent

Lil-Robots-Crawler/1.0 (+https://lilrobots.com/seo-crawler/bot)

This URL is included in the user agent so server owners can identify the crawler and find contact details quickly.

Identifies itself

Requests use a clear user agent that points back to this page.

Respects robots.txt by default

Site owners can allow, disallow, or scope crawler access with standard robots.txt rules.

Built for SEO checks

The crawler is meant for technical SEO audits, resource checks, metadata review, and client QA.

Purpose

A local SEO audit tool, not a search engine bot.

Lil Robots SEO Crawler is a desktop app used by SEOs, developers, and agencies to audit websites they are responsible for. It helps users find technical SEO issues, blocked resources, metadata gaps, redirects, indexability problems, and handoff items before they affect rankings, launches, or client results.

What the crawler may inspect

Depending on the user's crawl settings and the public pages being audited, the App may inspect:

  • • HTTP status codes, redirects, canonical tags, robots directives, and indexability signals.
  • • Page titles, meta descriptions, headings, links, resources, and other SEO-relevant page structure.
  • • Resource availability signals such as blocked CSS, JavaScript, images, or other linked assets.

What it is not designed to do

  • • Bypassing authentication, paywalls, security controls, or private areas.
  • • Collecting passwords, payment details, cookies, authorization headers, or form submissions.
  • • Ignoring robots.txt rules when the user has not explicitly configured otherwise.