How do I stop google to crawl to certain urls in my application?
For example: I want google to stop crawling all the URLs that starts with http://www.myhost-test.com/
What should I add in my robot.txt?
How do I stop google to crawl to certain urls in my application?
For example: I want google to stop crawling all the URLs that starts with http://www.myhost-test.com/
What should I add in my robot.txt?
The answer can be found directly here:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
But it looks like you add "disallow" and your url.