Only search engine crawlers will interpret the values within the meta-element “robots”. In most cases, the values “INDEX” and “FOLLOW” are used to instruct the crawler to include the present page in their index and follow all links on the page.
Your page may be added to the index and all links may be crawled, if you add the following markup to the source code of your page:
<meta name="robots" content="index, follow">Advantage/purpose of the NOINDEX value
If you do not want to have a specific URL in the Google index (or in the index of any other search engine), you should add the value NOINDEX to the meta-element “robots” in the part of the pages source code.
By adding the following markup in the source code of a page, you will make sure that this page will not be included in the index:
<meta name="robots" content="noindex">Does it make sense to use the values NOINDEX and FOLLOW together?
This depends on the purpose you are trying to achieve. What could make you want to keep a page from being indexed by search engines while still wanting all the links on the page to be crawled?
Matt Cutts gave us a great example in one of his Google webmasters videos:
Let us assume you have a HTML sitemap with a few hundred internal links and, for some reason, this sitemap is not pretty and therefore not very useful to a visitor. Therefore, a user would probably not be very pleased if he was send to this page from the search results. In order to avoid the former, while still making sure that the search engines can follow all the links on the page, you would use the NOINDEX and FOLLOW values together.
By adding this markup to the source code of a page you will keep the page from being indexed, while all links on the page can still be crawled:
<meta name="robots" content="noindex, follow">Another possible scenario could be tag pages on blogs:
There are cases where webmasters are saying that the tag pages of their blogs got better rankings than corresponding articles. The tag pages ranked about 2 to 3 positions above the corresponding articles. After the tag pages were changed to “noindex” and “follow”, the article rankings improved by these 2 to 3 positions.
