← All tools

AI Crawlability Check

Compare robots.txt access rules and page responses for AI crawlers.

About this check

Check the specific crawler and URL before changing a site-wide rule.

How to use it

  1. Enter the exact URL to test.
  2. Read the rule and response for the crawler being investigated.

Frequently asked questions

Does an allowed rule prove a crawler visited?

Check access logs to confirm a visit.

What if the server denies a permitted path?

Inspect authentication, firewall and rate-limit rules as well as robots.txt.