Robots meta tags and the robots.txt file

Unlike human editors, search engines use digital robots, which act like mini vacum cleaners, sucking up information about your web site that will allow it to add each web page it visits to it's search index accurately.

In the meta tags <robots> you should instruct the visiting robot on whether you want it to see what you have on a web page, or decide not to allow it to see your private content. To allow this to happen, you have 2 choices. You can either add the <robots> meta tag or add a robots.txt file. Both are explained here.

The robots meta tag is a tag to tell a robot if it is ok to index this page or not. It also is used to invite a spider to walk down through all your pages. It is growing in importance.

Some search engines, such as Inktomi now fully obey the Robots Meta Tag. Inktomi will crawl down through a site if the Index,Follow syntax is used.

Robots meta tag format.

The Robots META tag is placed in the HEAD section of your HTML document:
The format is quite simple: (case is not significant)

<HTML>
<HEAD>
<TITLE>Your web page title </TITLE>
<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">
</HEAD> <BODY>

Robot meta tag options

There are four directives that can be placed in a robots meta tag. The CONTENT section of the meta tag can contain: index,noindex,follow,nofollow and are separated by commas.

At this point, only the following combinations make sense:
The INDEX directive tells the robot it is ok to index the page.

The FOLLOW directive tells the robot it is ok to follow the links found on this page. Some search engine articles on Robots Meta tag say the predefined defaults are INDEX and FOLLOW, not true with Inktomi. The default with Inktomi is index,nofollow .

There are also, two global directives that can specify both actions: ALL=INDEX,FOLLOW, and NONE=NOINDEX,NOFOLLOW .

Robots Meta Tag Examples :
<META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">
<META NAME="ROBOTS" CONTENT="NOINDEX,FOLLOW">
<META NAME="ROBOTS" CONTENT="INDEX,NOFOLLOW">
<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">

To make your own robot.txt file online - Robot.txt Generator

Allowing the robots to visit is becoming search engine friendly!

Meta tags » Title meta tag » description meta tag » keyword meta tag » robots meta tag

Homepage | Blog | Internet Speed Test | Meta Tag Generator | Google Indexing Tool

Robots meta tags | robots.txt improve search engine ranking.

Your search engine ranking can improve with a robots meta tags.

Robots meta tag format.

Robot meta tag options

Allowing the robots to visit is becoming search engine friendly!