Generate Robots.txt files quickly and easily with this robots.txt generator. Help the search engines properly index your page by using a robots.txt file.

By default this tool will allow the major search engines to crawl every part of your webpage - if there are areas you wish to exclude, simply add them to this file and upload it to your root directory.

Some areas you may wish to exclude from the bots include:

• Login Pages
• Your contact page
• Privacy Policy
• Internal areas
• Media files you do not wish to see in search results
• Image folders if you do not want to share your images

Instructions

Select folders and/or files

• If there are areas of your page you do not wish made public, simply indicate the directories using the tool above. If you choose to hide particular pages, then you must do them individually for each subdirectory - otherwise they will be parsed.

Saving your robots.txt file

• After you generate your robots.txt code, simply copy the text that appears in the code-box and save it as a file named: robots.txt
  

Uploading your robots.txt file

• Then you must upload it to your ROOT directory. Search engines will automatically look for a file named robots.txt stored in your site's root folder, if they do not find one, then they look at individual pages. You may also add meta robots tags to your individual pages for additional tweaking. Visit a meta tag generator to create meta robots tags.

NOTE: A robots.txt file does not make areas on your site secure! It will not block bad bots nor protect sensitive areas of your site. There are other techniques you can also use to block bad bots, such as your .htaccess file.