Stanford NLP Tagger via NLTK - tag_sents splits ev

Question

I'm hoping someone has experience with this as I'm unable to find any comments online besides a bug report from 2015 regarding the NERtagger which is probably the same.

Anyway, I'm trying to batch process text to get around the poor performing base tagger. From what I understand, tag_sents should help.

from nltk.tag.stanford import StanfordPOSTagger
from nltk import word_tokenize
import nltk

stanford_model = 'stanford-postagger/models/english-bidirectional-distsim.tagger'
stanford_jar = 'stanford-postagger/stanford-postagger.jar'
tagger = StanfordPOSTagger(stanford_model, stanford_jar)
tagger.java_options = '-mx4096m'
text = "The quick brown fox jumps over the lazy dog."
print tagger.tag_sents(text)

Except no matter what I pass to the tag_sents method, the text gets split up into chars instead of words. Anyone know why it doesn't work properly? This works as expected...

tag(text)

I tried splitting the sentence into tokens as well to see if that helped but same treatment

Answer 1

The tag_sents function takes a list of list of strings.

tagger.tag_sents(word_tokenize("The quick brown fox jumps over the lazy dog."))

Here's a useful idiom:

 tagger.tag_sents(word_tokenize(sent) for sent in sent_tokenize(text))

where text is a string.

Answer 2

Another variation of what alvas said, which worked for me: tagger.tag_sents([[text]]).