Robots.txt Generator
What Is a robots.txt File?
A robots.txt file, placed at your site's root (yoursite.com/robots.txt), tells search engine crawlers which parts of your site they're allowed to crawl and where to find your sitemap. It's the first thing a well-behaved crawler checks before indexing anything — get it wrong and you can accidentally block Google from your entire site.
Enter your sitemap URL and any paths you want to disallow, and this tool generates a correctly-formatted robots.txt file.
Why Use This robots.txt Generator
✅ Correct Syntax, Guaranteed
Avoid a hand-typo that accidentally blocks your whole site from search engines.
🗺️ Sitemap Reference Included
Automatically adds a Sitemap: directive pointing crawlers to your XML sitemap.
⚡ Instant, Copy-Ready Output
Generate and download the file in seconds.
How to Create a robots.txt File
- Enter your Sitemap URL — e.g.
https://yoursite.com/sitemap.xml. - List Disallowed Paths, one per line — folders or pages you don't want crawlers indexing, like
/admin/or/cart/. - Copy the generated file and upload it as
robots.txtto your site's root directory (not a subfolder).
robots.txt Mistakes That Actually Hurt SEO
1. Accidentally disallowing the entire site
A stray Disallow: / blocks every crawler from every page — this is one of the most common and most damaging robots.txt mistakes, often introduced during a staging-to-production migration.
2. robots.txt doesn't remove pages already indexed
Disallowing a page in robots.txt stops future crawling, but a page Google already indexed can still appear in search results (as a URL-only listing) unless you also use a noindex tag or removal request.
3. It must be at the domain root
A robots.txt at /blog/robots.txt is ignored — it only works at yoursite.com/robots.txt, the exact root of the domain.
Frequently Asked Questions (FAQ)
Q: Does robots.txt guarantee a page won't appear in Google?
A: No — it only tells crawlers not to fetch the page's content. If other sites link to it, Google can still index the URL itself. Use a noindex meta tag for a hard guarantee.
Q: Where do I upload the robots.txt file?
A: To your website's root directory, so it's accessible at exactly yoursite.com/robots.txt — not inside any subfolder.
Q: Do I need a robots.txt file at all?
A: Not strictly — without one, crawlers assume they can access everything. But having one that at least declares your sitemap location is standard SEO practice and gives you explicit control.