How To Add Custom Robots.txt File In Google's Blogger? 6 Facts You Must Know

                

custom-robots-txt-file

Robot.txt is  a  simple  code  of  text saved in  the  website or Blogger Server  which instructs  the 'Web Crawlers' on how to index your entire blog  and show them in the search results. With the help of this  'text file' you can restrict the 'Web Crawlers' not to index some of  your unimportant pages such as label/search/privacy policy/contact/terms & conditions or any unimportant articles. The robots.txt file includes instruction to the 'Web Crawlers' which pages of your blog are to be indexed and which pages are not.

All blogs have a default robots.txt file as shown below :-

User-agent:  Mediapartners-Google

Disallow:

User-agent:*

Disallow: /search

Allow: /

Sitemap: http://example.blogspot.com/feeds/posts/default?orderby:=UPDATED

When we include this code in blog (Sitemap: http://example.blogspot.com/feeds/posts/default?orderby:=UPDATED) it helps 'Web Crawler' to index  our blog  faster and  easily. When you attempt to enter this code by going to the 'Settings' page you will find a warning there. It says :-

"Warning! Use with caution. Incorrect use of these features can result your blog being ignored by search engines". 

So, one has to be very careful while adding this code to the blog. Follow the following steps :-

1) Log in to your Dashboard

2) Click : Settings

3) Click : Search preferences

4) Now under 'Crawlers and indexing'

5) Against 'Custom robots.txt' click 'Edit'

6) Now click 'Yes' and a small box opens

7) Now copy the code given above in bold & italic and paste it in the small box

8) Now replace 'example.blogspot.com' with the url of your blog. And you have done it. 

Explanation of the robots.txt file :

User-agent: Mediapartners-Google

     This code is meant for 'Google Adsense robots' which helps them to serve appropriate ads related to the niche of your blog. This code should not be disturbed.

User-agent:*

    

     This code is meant for all robots marked with asterisk(*) an instruction to all 'Web Crawlers' that the blog or website can be crawled by all search engines and the blog is visible to all search engines. In default settings in blogger, the label pages are restricted to be indexed by search engine crawlers with the help of the following code:

Disallow:/search

This code instructs the crawlers not to index the search page and therefore, will be ignored by all the crawlers. In the absence of the code all the crawlers will index the contents of all the pages.

Allow:/

This line is an instruction to all the search engine crawlers to index the blog's entire home page.

Sitemap:http://example.blogspot.com/feeds/posts/default?orderby:=UPDATED

This is the sitemap of a blog which includes all our published posts/articles. By adding this code in the blog one can easily improve the crawling rate. Whenever the 'Web crawler' starts indexing the pages of a blog it will be easier for the crawler to index all the published posts of the blog.

The above sitemap is meant for indexing of the last 25 posts. To increase the number of posts of the blog to be indexed, just replace the above code with the following one:-

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

If your blog contains more than 500 posts, then two codes as shown below may be included:

____________________________________________

Sitemap: http://example.blogspot.com/atom.sml?redirect=false&start-index=1&max-results=500

____________________________________________

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000 

____________________________________________ 

Now, to test if the robot.txt is entered in the right manner, just open a window and type http://your-blog-name.blogspot.com/robots.txt 

you will find your robots.txt.

Read :-

How to add a xml sitemap of your blog to 'Webmaster Tools'

How to add a sitemap to your blog