Start with wildcard access
Allow public pages for conventional crawlers if organic indexing is a goal.
BLOCK AI TRAINING CRAWLERS
Keep search open.
Limit training access.
Generate explicit GPTBot, CCBot, and Google-Extended groups while preserving the wildcard access needed by conventional search crawlers.
Add explicit bot groups
Place targeted user-agent rules after the general group and test the final policy.
Declare use preference
Add ai-train=no to communicate intent, while recognizing that support is voluntary.