Build a valid robots.txt file with preset configurations for common scenarios: allow all, block all, or block AI crawlers. Supports multiple user-agents, crawl delay, and sitemap declaration.
Start from a preset rather than a blank slate — "allow all" is the right default for almost every public website, since the more common failure mode is accidentally blocking pages you wanted indexed, not under-blocking. Only reach for "block all" on a staging or internal subdomain that should never appear in search results at all.
If you add custom rules, double-check the path patterns before publishing. Disallow: /admin blocks /admin and everything under it, including /administrator if you weren't careful — robots.txt matches by prefix, not by exact folder name, so a too-broad rule can silently take far more pages out of the index than intended.
The "block AI crawlers" preset adds disallow rules for GPTBot, CCBot, Google-Extended, ClaudeBot, and the other major training crawlers in one click. Worth reading first: blocking training crawlers and blocking retrieval crawlers (like PerplexityBot or ChatGPT-User) have very different consequences — one opts you out of model training, the other makes you uncitable in AI-generated answers. Our full breakdown of AI crawler user-agents covers which is which before you apply a blanket block.
Always end the file with a Sitemap: directive pointing at your XML sitemap — it's a one-line addition that helps crawlers that don't rely on Search Console submission find your full URL list directly.
Robots.txt is a text file placed at the root of your website (e.g., example.com/robots.txt) that tells crawlers which pages or sections they are allowed or disallowed from visiting.
Use the 'Block AI crawlers' preset. It adds Disallow: / rules for GPTBot, CCBot, Google-Extended, and ClaudeBot. Note: only compliant bots respect robots.txt; it is not a technical enforcement mechanism.
Yes. Disallowing your important pages stops Google from indexing them. A misconfigured robots.txt is one of the most common causes of pages dropping out of search results entirely.
Rules
robots.txt
User-agent: * Allow: / Disallow: /admin/ Disallow: /cart/ Disallow: /*?* Sitemap: https://sluggenius.com/sitemap.xml
An empty Disallow means "allow everything." Place the file at your site root: yoursite.com/robots.txt
Generate a valid robots.txt file in seconds — control which crawlers access which pages of your site.
Start with Allow all, Block all, or Block AI crawlers — or configure from scratch.
Set user-agents, allow and disallow paths, crawl delay, and sitemap URL.
See the exact robots.txt content in real time before downloading.
Place the file at yoursite.com/robots.txt to activate crawl rules immediately.
The robots.txt file is deceptively simple — a plain text file placed at your domain root that tells web crawlers which pages they may or may not visit. Despite its simplicity, a misconfigured robots.txt is one of the most common causes of entire website sections disappearing from Google's index. Accidentally writingDisallow: /under User-agent: *blocks every crawler from every page — a mistake that can take weeks to recover from after you push it live.
Crawl budget management is where robots.txt delivers real SEO value for large sites. Search engines allocate a finite crawl budget to each domain — they can only visit so many pages per day. By disallowing low-value pages (admin panels, faceted navigation URLs, session-based query strings like?sort=price&page=3), you direct that crawl budget toward your most important content instead. Sites with hundreds of thousands of pages benefit enormously from this optimization.
Blocking AI training crawlers has become a serious concern for content publishers. Bots like GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended, and ClaudeBot respect robots.txt directives and will honor requests to stop crawling when you specifyDisallow: /for those user-agents. While robots.txt is not a technical enforcement mechanism — non-compliant bots can ignore it — it is the universally recognized standard for expressing crawling preferences, and all major AI companies officially claim to respect it.
Disallow: / for all agents (blocks everything)/admin/)/*?*)