AI Crawlability Check
Compare robots.txt access rules and page responses for AI crawlers.
About this check
Check the specific crawler and URL before changing a site-wide rule.
How to use it
- Enter the exact URL to test.
- Read the rule and response for the crawler being investigated.
Frequently asked questions
Does an allowed rule prove a crawler visited?
Check access logs to confirm a visit.
What if the server denies a permitted path?
Inspect authentication, firewall and rate-limit rules as well as robots.txt.
