How To Add A Robots.txt File To Blogger?

Home » Unlabelled » How To Add A Robots.txt File To Blogger?

How To Add A Robots.txt File To Blogger?

Posted by Unknown

Robots.txt ?

What does really robots.txt file do to your site/blog? They give guidelines for search engine bots (tells robotic crawlers how to crawl on your site and what to index). We normally have a blogger page without a robots tag assigned. If you bought a new domain and want to move your blog to that domain. Here is where you need the custom robot file assigned. Else you will be visited by lots and lots of crawlers, resulting in un-natural views. Also some search engines cant crawl unless they found a valid robots.txt file. All these can result in poor SEO (Search engine optimization)

Where To Add Robots.txt ?

The robots.txt must be added to the root of your blogger. For this, navigate to

Blogger button (left side)---From the drop down, select settings

Select search preferences--There you can find custom robots.txt with an edit on the side

What to enter in space provided ?

There are many robots.txt file generators available, but the best practice is to try them out yourself manually. Only problem is that a misconfigured robot directory causes search engines to avoid your site.

There are 3 types of robot files added

1. Allow all bots and allow any action (not preffered by me)

Copy and Enter the following below. (in red colour)
(copy to a notepad before pasting. Else it may return an error)

User-agent: *
Disallow:
Sitemap: http://www.YOUR-SITE.com/atom.xml?redirect=false&start-index=1&max-results=500

(please be sure to edit YOUR-SITE to your site address.)

Explanation

The command allows all robots to crawl through your site without limitations and acess all labels via search.

2. Allow all bots with restricted acess

Copy and Enter the following below. (in red colour)

(copy to a notepad before pasting. Else it may return an error)

User-agent: *
Disallow:/search

Allow:/
Sitemap: http://www.YOUR-SITE.com/atom.xml?redirect=false&start-index=1&max-results=500

(please be sure to edit YOUR-SITE to your site address.)

Explanation

The command allows all bots to crawl through with limitations to search indicated by disallow:/search and allow homepage indexing indicated by allow:/

3. Allow only specific bots

Copy and Enter the following below. (in red colour)

(copy to a notepad before pasting. Else it may return an error)

User-agent:BOT-NAME
Disallow:

Sitemap: http://www.YOUR-SITE.com/atom.xml?redirect=false&start-index=1&max-results=500

(please be sure to edit YOUR-SITE to your site address,and BOT-NAME to bot you wish to allow)

Explanation

The command only allow specific bots and may result in low traffic. But it is necessary for specific customers having trouble with crawling bots.The disallow being left blank indicates,no restriction is present there.

Digitally Blind