Scan a website

CrawlMirror

Check AI Bot Rules in robots.txt

Resolve per-bot robots.txt rules for AI search and training crawlers, then choose a desired policy in the report.

Measure access first. Choose your desired AI crawler policy in the report — no need to decide before the scan.

  • No account required
  • Public pages only
  • Evidence-based results
Advanced scan options
Scan scope

Related checks