How robots.txt file should be properly written for

问题:

Can someone explain me how should i write a robots.txt file if i want that all crawlers index root and some specific subdomains

User-agent: *
Allow: /
Allow: /subdomain1/
Allow: /subdomain2/

Is this right? And where should i put it? In the root (public_html) folder or in each subdomain folder?

回答1:

There is no way to specify rules for different subdomains within a single robots.txt file. A given robots.txt file will only control crawling of the subdomain it was requested from. If you want to block some subdomains and allow others, then you need to serve different robots.txt files from the different subdomains.

For example, if you want to allow crawling of http://crawlme.example.com/, but you want to block crawling of http://nocrawl.example.com/ then:

http://crawlme.example.com/robots.txt should contain:

# Allow everything:
User-agent: *
Disallow:

and http://nocrawl.example.com/robots.txt should contain:

# Block everything:
User-agent: *
Disallow: /