Search crawlers
Protect Googlebot and Bingbot access when organic visibility matters.
AI ROBOTS.TXT CHECKER
Know which bots
can fetch your site.
Interpret wildcard and explicit user-agent groups for major search, user-triggered, and AI training crawlers. Keep organic discovery separate from training policy.
AI search
Review search and user-triggered agents separately from model-training bots.
Training crawlers
Use explicit groups when you need access control, not only a preference declaration.