Robots.txt Generator
Robots.txt GeneratorBuild a robots.txt file easily. Fast, free and secure — works in your browser. By AK Computer, Dwarka.
Processing...
How to use
- 1 Choose which user agents the rules should apply to.
- 2 Specify the paths you want to allow or disallow.
- 3 Add your sitemap URL so crawlers can find it.
- 4 Copy the generated robots.txt and place it in your site root.
Frequently Asked Questions
What is a robots.txt file for?
It tells search engine crawlers which parts of your site they may or may not access.
Does disallowing a page hide it from Google?
Not necessarily; robots.txt blocks crawling but a page can still be indexed if linked elsewhere, so use a noindex tag to keep it out of results.
Where should the robots.txt file go?
It must be placed in the root directory of your domain so it is reachable at yourdomain.com/robots.txt.