Robots.txt Generator
SEO Tools
Build a valid robots.txt visually — user-agent groups, allow/disallow paths, crawl-delay, sitemap URLs. Output ready for your site root.
Runs entirely in your browserAbout Robots.txt Generator
robots.txt is the file at the root of every website that tells search-engine crawlers (and any other automated visitor) which paths they're allowed to fetch. It's not a security boundary — anyone can ignore it — but it's the standard polite signal that Google, Bing, and every legitimate bot honor. Mistakes here can hide your site from search entirely (disallowing `/` by accident) or leak private paths into Google's index (forgetting to disallow them).
This generator gives you a form for the four directives that actually matter: User-agent groups, Allow/Disallow paths, optional Crawl-delay (which Google ignores but Bing respects), and Sitemap URLs. You can have multiple groups targeting different bots — e.g., a permissive group for Googlebot and a restrictive one for all other bots. The output is a properly-formatted robots.txt ready to drop at the root of your site (`/robots.txt`). For testing what your rules will actually match, pair this with the Robots.txt Tester.
How to use
- 1
Set up the default group
The default group targets `User-agent: *` (all bots). Add Allow or Disallow rules with `+ Allow` / `+ Disallow` buttons; each rule takes a path or pattern.
- 2
(Optional) Add bot-specific groups
Click 'Add group' to target a specific bot (`Googlebot`, `Bingbot`, etc.). Rules in a specific group override the `*` group for that bot.
- 3
(Optional) Set Crawl-delay
Seconds to wait between requests. Google ignores this; Bing, Yandex, and others respect it. Use sparingly — most sites don't need it.
- 4
Add Sitemap URLs
Absolute URLs to your sitemap files. These are global (not per-group). One per line.
- 5
Place at your site root
Copy the output and save it as `/robots.txt` at the root of your domain (e.g., `https://example.com/robots.txt`). Verify it's reachable in a browser.
Examples
Basic 'allow everything, hide admin'
Output
User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /
Sitemap: https://example.com/sitemap.xmlDifferent rules for Googlebot
Output
User-agent: *
Disallow: /private/
User-agent: Googlebot
Allow: /
Sitemap: https://example.com/sitemap.xmlFrequently asked questions
Is robots.txt a security measure?+
No. Anyone — humans, scrapers, malicious bots — can read your robots.txt and ignore it. It's a polite signal honored by Google, Bing, and other reputable crawlers. For actual access control, use HTTP authentication or server-side rules.
What's the difference between Allow and Disallow?+
Disallow tells the bot not to crawl matching paths. Allow explicitly permits paths (useful for unblocking a sub-path of an otherwise-disallowed directory: `Disallow: /admin/`, `Allow: /admin/public/`).
Does Google respect Crawl-delay?+
No. Google's official position is to ignore Crawl-delay. To slow Googlebot, use Search Console → Settings → Crawl rate. Crawl-delay is respected by Bing, Yandex, and some smaller crawlers.
Can I have multiple sitemap URLs?+
Yes. List each on its own `Sitemap:` line. Google reads all of them. Useful when you split a large sitemap into multiple files or have separate sitemaps per content type (blog, products, etc.).
What's the wildcard syntax?+
`*` matches any character sequence. `$` matches end-of-URL. So `Disallow: /*.pdf$` blocks all URLs ending in `.pdf`. These wildcards are unofficial but supported by all major search engines.
What if I block a page that's already indexed?+
Counter-intuitively, blocking via robots.txt does NOT remove the page from Google's index — it just stops re-crawling. To remove a page, use `<meta name="robots" content="noindex">` on the page itself (and don't disallow it, so Google can re-crawl and see the noindex). Then once removed, you can disallow.
Related tools
Meta Tag Generator
Build a complete set of HTML meta tags — basic SEO, Open Graph, Twitter Cards, viewport — with character-count warnings.
SERP Snippet Preview
Preview how your title, URL, and meta description render in Google search results — desktop and mobile.
hreflang Generator
Build the hreflang link tag set for multilingual or multi-regional sites with BCP 47 validation.
Readability Score
Score your content with six readability formulas — Flesch, Gunning Fog, SMOG, Coleman-Liau, ARI.
Keyword Density Analyzer
Analyze 1-, 2-, and 3-word phrase frequency and density in your content with stopword filtering.
Robots.txt Tester
Test which URLs your robots.txt allows or blocks. Multi-path testing with rule-match explanation.