Robots.txt Generator
Choose whether to allow or block crawling, list any paths to disallow, and add your sitemap URL.
How it works
This generates a single User-agent: * block applying the same rules to every crawler. Each path you list becomes a Disallow: line; a leading slash is added automatically if you leave it off. The file must be saved as robots.txt in your site's root directory (e.g. https://example.com/robots.txt) to take effect.
robots.txt is a request, not a lock. Well-behaved crawlers (Google, Bing) respect it, but it doesn't prevent access — don't rely on it to keep anything truly private.
Frequently asked
Does robots.txt stop a page from being indexed?
Not reliably — a blocked page can still appear in search results (without a description) if other sites link to it. To reliably keep a page out of search results, use a noindex meta tag on the page itself instead.
Where do I put the robots.txt file?
In your site's root directory, so it's reachable at yourdomain.com/robots.txt exactly — not in a subfolder. On cPanel, that's usually the same folder as your homepage's index.html.