Get first lines of Wikipedia Article

Question

I got a Wikipedia-Article and I want to fetch the first z lines (or the first x chars, or the first y words, doesn't matter) from the article.

The problem: I can get either the source Wiki-Text (via API) or the parsed HTML (via direct HTTP-Request, eventually on the print-version) but how can I find the first lines displayed? Normaly the source (both html and wikitext) starts with the info-boxes and images and the first real text to display is somewhere down in the code.

For example: Albert Einstein on Wikipedia (print Version). Look in the code, the first real-text-line "Albert Einstein (pronounced /ˈælbərt ˈaɪnstaɪn/; German: [ˈalbɐt ˈaɪ̯nʃtaɪ̯n]; 14 March 1879–18 April 1955) was a theoretical physicist." is not on the start. The same applies to the Wiki-Source, it starts with the same info-box and so on.

So how would you accomplish this task? Programming language is java, but this shouldn't matter.

A solution which came to my mind was to use an xpath query but this query would be rather complicated to handle all the border-cases. [update]It wasn't that complicated, see my solution below![/update]

Thanks!

Answer 1

You need a parser that can read Wikipedia markup. Try WikiText or the parsers that come with XWiki.

That will allow you to ignore anything you don't want (headlines, tables).

Answer 2

I opened the Albert Einstein article in Firefox and I clicked on View source. It's pretty easy to parse using an HTML parser. You should focus on the <p> and strip the other html from within it.

Answer 3

I was also in the same need and wrote some Python code to do that.

The script downloads the wikipedia article with given name, parses it using BeautifulSoup and returns first few paragraphs.

Code is at http://github.com/anandology/sandbox/blob/master/wikisnip/wikisnip.py.