How to use vector representation of words (as obta

问题:

I am familiar with using BOW features for text classification, wherein we first find the size of the vocabulary for the corpus which becomes the size of our feature vector. For each sentence/document, and for all its constituent words, we then put 0/1 depending on the absence/presence of that word in that sentence/document.

However, now that I am trying to use vector representation of each word, is creating a global vocabulary essential?

回答1:

Suppose the size of the vectors is N (usually between 50 or 500). The naive way of generalizing the traditional of generalizing BOW is just replacing 0 bit (in BOW) with N zeros, and replacing 1 bit (in BOW) with the the real vector (say from Word2Vec). Then the size of the features would be N * |V| (Compared to |V| feature vectors in the BOW, where |V| is the size of the vocabs). This simple generalization should work fine for decent number of training instances.

To make the feature vectors smaller, people use various techniques like using recursive combination of vectors with various operations. (See Recursive/Recurrent Neural Network and similar tricks, for example: http://web.engr.illinois.edu/~khashab2/files/2013_RNN.pdf or http://papers.nips.cc/paper/4204-dynamic-pooling-and-unfolding-recursive-autoencoders-for-paraphrase-detection.pdf )

回答2:

To get a fixed length feature vector for each sentence, although the number of words in each sentence is different, do as follows:

1. tokenize each sentence into constituent words
2. for each word get word vector (if it is not there ignore the word)
3. average all the word vectors you got
4. this will always give you a d-dim vector (d is word vector dim)

below is the code snipet

def getWordVecs(words, w2v_dict):
    vecs = []
    for word in words:
        word = word.replace('\n', '')
        try:
            vecs.append(w2v_model[word].reshape((1,300)))
        except KeyError:
            continue
    vecs = np.concatenate(vecs)
    vecs = np.array(vecs, dtype='float')
    final_vec = np.sum(vecs, axis=0)
return final_vec

words is list of tokens obtained after tokenizing a sentence.