Set Custom robots txt

SamplePost

Set Custom robots.txt

You can use the robots.txt file to give instructions about your site to web robots. By default, your blog uses it immediately after you created the blog. To check this file, go to http://example.blogspot.com/robots.txt.

You might see the file like below if your site is not visible to Search engines —

01 02 03 04 05 06 07 08 09 10

User-agent: Mediapartners-Google Disallow:   User-agent: * Disallow: /search Disallow: /   User-Agent: googlebot Disallow: /search Allow: /

• User-agent: * — It means this section applies to all robots.
• Disallow: / — It tells the robot that it should not visit any pages on the site.
• Disallow: /search — It means the links having keyword search just after the blog address will be ignored. Therefore, a page url like http://example.blogspot.com/search/label/labelname will never be crawled or indexed.
• Allow: / — It means that the robot may crawl your homepage.

You can generate a custom robots.txt file for your blog from here. After generating the file —

1. Click “Yes” to enable custom robots.txt content.
2. Copy and paste the generated code in this field.
3. Click Save changes.

After a few moments, the file will be updated to this new code.

Set Custom Robots Header tags

You can customize your blog’s robots header tags for individual pages using this feature. To enable custom robots header tags —

1. Click Edit button
2. Select “Yes”
3. Click Save changes

Once you’ve enabled it, you can see the custom robots tags is available for every post.

Warning: Before using this feature you should have knowledge about Robots Header Tags. If you don’t know anything about it then I recommend you not to enable it.

Robots Header Tags with Meaning

Directive	Meaning
all	There are no restrictions for indexing or serving. Note: this directive is the default value and has no effect if explicitly listed.
noindex	Do not show this page in search results and do not show a “Cached” link in search results.
nofollow	Do not follow the links have been used on this page
none	Equivalent to noindex, nofollow
noarchive	Do not show a “Cached” link in search results.
nosnippet	Do not show a snippet (meta or search description) in the search results for this page
noodp	Do not use metadata from the Open Directory project for titles or snippets shown for this page.
notranslate	Do not offer translation of this page in search results.
noimageindex	Do not index images on this page
unavailable_after: [RFC-850 date/time]	Do not show this page in search results after the specified date/time. The date/time must be specified in the RFC 850 format.

Robots Header Tags Live Experiment

I am using Chrome browser to see the robot header tags.

1. Open Chrome and go to your site.
2. Right-click on the page, then click Inspect or press Ctrl+Shift+I.
3. Select the Network tab.
4. You may need to reload the page once again. Press F5 to reload.
5. Scroll up and click your site name (example: www.egtutorial.com).
6. If Custom robots header tags are enabled then you will see X-Robots-Tag like the image below.