How to hide website directory from search engines

问题:

We know we can stop search engines from indexing directories on our site using robots.txt.

But this of course has the disadvantage of actually publicising directories we don't want found to possible attackers.

Password protecting the directory using .htaccess or other means is obviously the best way to keep the directory private.

But what if, for reasons of convenience, we didn't want to add another layer of security to the directory and just wanted to add another level of obfuscation? To hide, for example, an admin login page.

Is there another way to "hide" the directory without broadcasting its location in a robots.txt file?

回答1:

Here is what to do, please note as you haven't mentioned any particular technology I haven't included how to do it.

If you configure your web server to output the following meta tag in the directory listing HTML page, it will prevent your page from being indexed by compliant search engines.

<meta name="robots" content="noindex">

Adding this would probably require implementing a custom module within your web server that will override the default directory listing output page.

回答2:

Try using a random string. Something like http://website.com/some-random-string-here/file.html

The remember not use some-random-string-here in your robots.txt file or on any links.