Robots.txt Generator
Build a correct robots.txt file for your site.
Loading tool…
About Robots.txt Generator
robots.txt is a small file that tells search engine bots which parts of your site they may crawl. Choose what to allow or block, add your sitemap link, and download the file. A WordPress preset is included.
How to use Robots.txt Generator
- 1Pick a preset or set rules for all bots.
- 2Add folders to block and your sitemap URL.
- 3Download robots.txt and upload it to your site root.
Why use this tool?
- Private. What you type stays in your browser.
- Free. No sign up, no limits, no watermark.
- Fast. Results in seconds, no waiting in a queue.
- Any device. Works on computers, tablets and phones.
Frequently asked questions
Where does robots.txt go?
In the main folder of your site, so it opens at yourdomain.com/robots.txt.
Does blocking a page remove it from Google?
Not always. To keep a page out of results, use a noindex meta tag instead.
Should I block AI crawlers?
That is your choice. The tool has an option to block common AI training bots like GPTBot and CCBot.