crawl links of sitemap.xml through wget command

问题:

I try to crawl all links of a sitemap.xml to re-cache a website. But the recursive option of wget does not work, I only get as respond:

Remote file exists but does not contain any link -- not retrieving.

But for sure the sitemap.xml is full of "http://..." links.

I tried almost every option of wget but nothing worked for me:

wget -r --mirror http://mysite.com/sitemap.xml

Does anyone knows how to open all links inside of a website sitemap.xml?

Thanks, Dominic

回答1:

It seems that wget can't parse XML. So, you'll have to extract the links manually. You could do something like this:

wget --quiet http://www.mysite.com/sitemap.xml --output-document - | egrep -o "https?://[^<]+" | wget -i -

I learned this trick here.

回答2:

You can use one of the sitemapping tools. Try Slickplan. It has the site crawler option and by using it you can import a structure of existing website and create a visual sitemap from it. Then you can export it to Slickplan XML format, which contains* not only links, but also SEO metadata, page titles (product names), and a bunch of other helpful data.

回答3:

While this question is older, google send me here.

I finally used xsltproc to parse the sitemap.xml:

sitemap-txt.xsl:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
            xmlns:sitemap="http://www.sitemaps.org/schemas/sitemap/0.9"
            xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output method="text" version="1.0" encoding="UTF-8" indent="no"/>
    <xsl:template match="/">
        <xsl:for-each select="sitemap:urlset/sitemap:url">
            <xsl:value-of select="sitemap:loc"/><xsl:text>&#xa;</xsl:text>
        </xsl:for-each>
    </xsl:template>
</xsl:stylesheet>

Using it (in this case it is from a cache-prewarming-script, so the retrieved pages are not kept ("-o /dev/null"), only some statistics are printed ("-w ....")):

curl -sS http://example.com/sitemap.xml | xsltproc sitemap-txt.xsl - | xargs -n1 -r -P4 curl -sS -o /dev/null -w "%{http_code}\t%{time_total}\t%{url_effective}\n"

(Rewriting this to use wget instead of curl is left as an exercise for the reader ;-) ) What this does is:

1. Retrieve sitemap.xml
2. Parse sitemap, output url-list as texts (one url per line)
3. use xargs to call "curl" on each url, using 4 requests in parallel)