CrawlerPolicy

CONTENT-SIGNAL GENERATOR

State how content
may be used.

Build a copy-ready Content-Signal line for search indexing, real-time AI input, and model training—then check it against the rest of your robots.txt.

Generate a policy →

SEARCH

Links and snippets

Express whether content may be used to build an index and return search results.

AI-INPUT

Answers and grounding

Express whether content may be supplied to models for real-time responses.

AI-TRAIN

Model development

Express whether content may be used to train or fine-tune models.