tag for Google

Question

I would like to tell Google not to index certain parts of the page. In Yandex (russian SE) there's a very useful tag called <noindex>. How can it be done with Google?

Answer 1

Create a robots.txt file at your root level and insert something like the following:

Block Google:

User-agent: Googlebot
Disallow: /myDisallowedDir1/
Disallow: /myDisallowedPage.html
Disallow: /myDisallowedDir2/

Block all bots:

User-agent: *
Disallow: /myDisallowedDir1/
Disallow: /myDisallowedPage.html
Disallow: /myDisallowedDir2/

A handy robots.txt generator:

http://www.mcanerin.com/EN/search-engine/robots-txt.asp

Answer 2

No, Google does not support the <noindex> tag. Virtually no one does.

Answer 3

You can prevent Google from seeing portions of the page by putting those portions in iframes that are blocked by robots.txt.

robots.txt

Disallow: /iframes/

index.html

This text is crawlable, but now you'll see 
text that search engines can't see:
<iframe src="/iframes/hidden.html" width="100%" height=300 scrolling=no>

/iframes/hidden.html

Search engines cannot see this text.

Instead of using using iframes, you could load the contents of the hidden file using AJAX. Here is an example that uses jquery ajax to do so:

his text is crawlable, but now you'll see 
text that search engines can't see:
<div id="hidden"></div>
<script>
    $.get(
        "/iframes/hidden.html",
        function(data){$('#hidden').html(data)},
    );
</script>

Answer 4

According to Wikipedia¹, there are some rules some spiders follow:

<!--googleoff: all-->
This should not be indexed by Google. Though its main spider, Googlebot,
might ignore that hint.
<!--googleon: all-->

<div class="robots-nocontent">Yahoo bots won't index this.</div>

<noindex>Yandex bots ignore this text.</noindex>
<!--noindex-->They will ignore this, too.<!--/noindex-->

Unfortunately, they could not agree on a single standard it seems – and to my knowledge, there's nothing to keep all spiders off...

The googleoff: comment seems to support different options, though I'm not sure where there's a complete list. There's at least:

• all: completely ignore the block
• index: content doesn't go into Google's index
• anchor: anchor text for links will not be associated with the target page
• snippet: text will not be used to create snippets for search results

Note as well that (at least for Google) this will only affect the search index, not the page ranking etc. Furthermore, as Stephen Ostermiller correctly pointed out in his comment below, googleon and googleoff only work with the Google search appliance and have no effect on normal Googlebot, unfortunately.

There's also an article on the Yahoo part² (and an article describing that Yandex also honors <noindex>⁶). On the googleoff: part, also see this answer, and the article I took most of the related information from.³

---

Additionally, Google Webmaster Tools recommend using the rel=nofollow attribute⁴ for specific links (e.g. ads or links to pages not accessible/useful to the bots, such as login/signup). That means, the HTML a rel Attribute should be honored by the Google bots – though that's mainly related to page rank, not to the search index itself. Unfortunately, it seems there's no rel=noindex^5,7. I'm also not sure if this attribute could be used for other elements as well (e.g. <DIV REL="noindex">); but unless crawlers honor "noindex", that wouldn't make sense either.

---

Further references:

• How to Noindex parts of a web page?
• Excluding crawler from sections of pages (Spiderline crawler; you see, other crawlers might use other proprietary markers (see also AddSearch crawler). I wish they simply made REL="noindex" a standard instead to be used with any HTML tag such as DIV/SPAN/P/A!)
• Preventing Google from indexing the contents of a div by reversing the string
• Methods for preventing search engines from indexing irrelevant content on a page

---

¹ Wikipedia: Noindex
² Which Sections of Your Web Pages Might Search Engines Ignore?
³ Tell Google to Not Index Certain Parts of Your Page
⁴ Use rel="nofollow" for specific links
⁵ Is it a good idea to use <a href=“http://name.com” rel=“noindex, nofollow”>name</a>?
⁶ Using HTML tags — Yandex.Help. Webmaster
⁷ existing REL values