I would like to vectorize with scikit learn a list who has lists. I go to the path where I have the training texts I read them and then I obtain something like this:
corpus = [["this is spam, 'SPAM'"],["this is ham, 'HAM'"],["this is nothing, 'NOTHING'"]]
from sklearn.feature_extraction.text import CountVectorizer
vect = CountVectorizer(analyzer='word')
vect_representation= vect.fit_transform(corpus)
print vect_representation.toarray()
And I get the following:
return lambda x: strip_accents(x.lower())
AttributeError: 'list' object has no attribute 'lower'
Also the problem with this are the labels at the end of each document, how should I treat them in order to do a correct classification?.
For everybody in the future this solve my problem:
corpus = [["this is spam, 'SPAM'"],["this is ham, 'HAM'"],["this is nothing, 'NOTHING'"]]
from sklearn.feature_extraction.text import CountVectorizer
bag_of_words = CountVectorizer(tokenizer=lambda doc: doc, lowercase=False).fit_transform(splited_labels_from_corpus)
And this is the output, when I use the .toarray() function:
[[0 0 1]
[1 0 0]
[0 1 0]]
Thanks guys
First you should separate labels from texts. If you want to use CountVectorizer you have to transform your texts one by one:
corpus = [["this is spam, 'SPAM'"],["this is ham, 'HAM'"],["this is nothing, 'NOTHING'"]]
from sklearn.feature_extraction.text import CountVectorizer
... split labels from texts
vect = CountVectorizer(analyzer='word')
vect_representation= map(vect.fit_transform,corpus)
...
As another option, you can use TfidfVectorizer with list of lists directly.
