Create crawler directives for Googlebot, Bingbot, and all bots. Then test if a URL is blocked or allowed under your rules.

Rule Builder

User Agents (one per line) Allow Paths (one per line) Disallow Paths (one per line)

Crawl-delay

Sitemap URL

Generated robots.txt

Ready.

URL Path or Full URL

Bot

No test yet.

Note: robots.txt controls crawling, not indexing. Use `noindex` in page-level meta tags when needed.