lexers / parsers for (un) structured text document

问题:

There are lots of parsers and lexers for scripts (i.e. structured computer languages). But I'm looking for one which can break a (almost) non-structured text document into larger sections e.g. chapters, paragraphs, etc.

It's relatively easy for a person to identify them: where the Table of Contents, acknowledgements, or where the main body starts and it is possible to build rule based systems to identify some of these (such as paragraphs).

I don't expect it to be perfect, but does any one know of such a broad 'block based' lexer / parser? Or could you point me in the direction of literature which may help?

回答1:

Many lightweight markup languages like markdown (which incidentally SO uses), reStructured text and (arguably) POD are similar to what you're talking about. They have minimal syntax and break input down into parseable syntactic pieces. You might be able to get some information by reading about their implementations.

回答2:

1. Define the annotation standard, which indicates how you would like to break things up.
2. Go on to Amazon Mechanical Turk and ask people to label 10K documents using your annotation standard.
3. Train a CRF (which is like an HMM, but better) on this training data.

If you actually want to go this route, I can elaborate on the details. But this will be a lot of work.

回答3:

Most of the lex/yacc kind of programs work with a well defined grammar. if you can define your grammar in terms of a BNF like format (which most of the parsers accept similar syntax) then you can use any of them. That may be stating the obvious. However you can still be a little fuzzy around the 'blocks' (tokens) of text which would be part of your grammar. After all you define the rules for your tokens.

I have used Parse-RecDescent Perl module in the past with varying levels of success for similar projects.

Sorry, it may not be a good answer but more sharing my experiences on similar projects.

回答4:

try: pygments, geshi, or prettify

They can handle just about anything you throw at them and are very forgiving of errors in your grammar as well as your documents.

References:
gitorius uses prettify,
github uses pygments,
rosettacode uses geshi,