Robots.txt Generator

Search Engine Optimization

Robots.txt Generator

Default - All Robots are:  
Sitemap: (leave blank if you don't have) 
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo MM
  Yahoo Blogs
  DMOZ Checker
  MSN PicSearch
Restricted Directories: The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

The robots.txt file is a type of guide that lets you tell search engines where websites/blogs can and cannot be accessed. This means that if you do not want to inform the search engines about all the content of your website, you can use robots.txt to keep them separate from the search engines. As a result, search engines do not have access to these files. Search engines cannot crawl the robots.txt file.

Each search engine has its own robot or bot which also has its own name like Googlebot in case of Google, Yahoo bot in case of Yahoo etc.

Our Populer Free Tools

How does robots.txt work?

  • Web crawling to find content.
  • Indexing the content so that those who are searching on Google for various topics can find it very easily.