Home / Free Tools / Robots.txt Generator
Robots.txt Generator
Fill in your site's crawl rules, get a ready-to-upload robots.txt.
FAQ
Do I need a robots.txt if I want everything crawled?
Not strictly, but having one that explicitly allows everything and references your sitemap is still good practice.
Not strictly, but having one that explicitly allows everything and references your sitemap is still good practice.
Where do I put the file?
At the root of your domain:
At the root of your domain:
https://yourdomain.com/robots.txt — it won't work in a subfolder.Does crawl-delay affect Google?
Googlebot ignores crawl-delay; it's respected by some other crawlers (Bing, Yandex). Use Search Console's crawl-rate settings for Google instead.
Googlebot ignores crawl-delay; it's respected by some other crawlers (Bing, Yandex). Use Search Console's crawl-rate settings for Google instead.