Browse Code
AI Crawler Access Checker is a free browser extension from USA Marketing Pros that shows which AI and search engine crawlers a website's robots.txt file allows or blocks.
Open any website and click the icon. The extension reads that site's robots.txt and checks 17 crawlers, grouped by what they affect: search and AI answer crawlers like Googlebot, Bingbot, OAI-SearchBot and PerplexityBot; user-requested agents like ChatGPT-User; and AI training crawlers like GPTBot, ClaudeBot and Google-Extended. Each result shows allowed, blocked or partly blocked, with a plain-language note on what it means.
It only runs when you click it, reads only the site's public robots.txt file, and never stores or sends anything.
Available for Chrome, Microsoft Edge and other browsers, plus a free web version at usamarketingpros.com. Made by USA Marketing Pros, a web design and SEO agency in Arlington, Virginia.
Features
- Checks 17 AI and search engine crawlers in one click
- Covers GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Googlebot
- Groups results by search, user-requested and AI training crawlers
- Shows allowed, blocked or partly blocked for each crawler
- Plain-language notes on what each rule means
- Runs only when clicked, no data collected
Project Samples
