Build a valid robots.txt file with preset configurations for common scenarios: allow all, block all, or block AI crawlers. Supports multiple user-agents, crawl delay, and sitemap declaration.
Robots.txt is a text file placed at the root of your website (e.g., example.com/robots.txt) that tells crawlers which pages or sections they are allowed or disallowed from visiting.
Use the 'Block AI crawlers' preset. It adds Disallow: / rules for GPTBot, CCBot, Google-Extended, and ClaudeBot. Note: only compliant bots respect robots.txt; it is not a technical enforcement mechanism.
Yes. Disallowing your important pages stops Google from indexing them. A misconfigured robots.txt is one of the most common causes of pages dropping out of search results entirely.
Rules
robots.txt
User-agent: * Allow: / Disallow: /admin/ Disallow: /cart/ Disallow: /*?* Sitemap: https://sluggenius.com/sitemap.xml
An empty Disallow means "allow everything." Place the file at your site root: yoursite.com/robots.txt
Generate a valid robots.txt file in seconds — control which crawlers access which pages of your site.
Start with Allow all, Block all, or Block AI crawlers — or configure from scratch.
Set user-agents, allow and disallow paths, crawl delay, and sitemap URL.
See the exact robots.txt content in real time before downloading.
Place the file at yoursite.com/robots.txt to activate crawl rules immediately.
The robots.txt file is deceptively simple — a plain text file placed at your domain root that tells web crawlers which pages they may or may not visit. Despite its simplicity, a misconfigured robots.txt is one of the most common causes of entire website sections disappearing from Google's index. Accidentally writingDisallow: /under User-agent: *blocks every crawler from every page — a mistake that can take weeks to recover from after you push it live.
Crawl budget management is where robots.txt delivers real SEO value for large sites. Search engines allocate a finite crawl budget to each domain — they can only visit so many pages per day. By disallowing low-value pages (admin panels, faceted navigation URLs, session-based query strings like?sort=price&page=3), you direct that crawl budget toward your most important content instead. Sites with hundreds of thousands of pages benefit enormously from this optimization.
Blocking AI training crawlers has become a serious concern for content publishers. Bots like GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended, and ClaudeBot respect robots.txt directives and will honor requests to stop crawling when you specifyDisallow: /for those user-agents. While robots.txt is not a technical enforcement mechanism — non-compliant bots can ignore it — it is the universally recognized standard for expressing crawling preferences, and all major AI companies officially claim to respect it.
Disallow: / for all agents (blocks everything)/admin/)/*?*)