Crawler utility

Robots.txt Tester

Check whether a crawler can access specific URLs before launch, migration, or indexability QA. Paste up to 1,000 full URLs and test them against each site's robots.txt file.

Checked

0

Allowed

0

Blocked

0

Errors

0

0 URLs in input0 skipped or invalidTesting as Googlebot

URLs to test

Use one full URL per line. The tool fetches robots.txt once per domain, then checks each URL against the selected crawler.

What this checks

  • Fetches each domain's robots.txt file.
  • Tests the path and query string for each URL.
  • Reports blocked URLs that need rule review.

Results

Sort the diagnostic table or copy only the allowed, blocked, or error URLs for handoff.

No robots results yet

Add one or more URLs, choose the crawler user agent, and run the test to see what robots.txt allows or blocks.