Extract URL's from a website?

问题:

I would like to migrate a site, but the new URL's are completely different from the old URL's. I do not want to break my old sites links that are out there on the web. Is there a way to extract all the links from a site so I can generate a redirect table (I will probably put the redirects on the web server level)? The old is on ASP.NET with SQL server 2008 by the way. Am open to crawling the site or mining the SQL database, but need some help or advice.

回答1:

You could use IIS Seo Optimalization Toolkit for that. http://www.iis.net/download/seotoolkit

With that tool you can spider the entire website. Then when the report finishes, go to Content -> Host summary and double click the host of the website. There you have a complete list of all spedered URL's.

And ofcourse, when you have these url's, you can make URL rewrite of routing mechanisms available to 301 to the right content on the new website.

回答2:

The best way is to actually put a 301 redirect to notify all engines that your URL's have permanently moved Search for 301 in this http response code documentation for more info on 301. That is if I am understanding your problem (that you want links from other sites and searches and whatnot to realize that your site moved?).

回答3:

It doesn't sound like you're running Apache. However, Apache has an available module called mod_rewrite. There may be a similar module for other web servers. mod_rewrite allows you to do exactly what you are describing.

回答4:

Perhaps with litle js magic if works for you