Simplest feature selection algorithm

Question

I am trying to create my own and simple feature selection algorithm. The data set that I am going to work with is here (very famous data set). Can someone give me a pointer on how to do so?

I am planning to write a feature rank algorithm for a text classification. This is for a sentiment analysis of movie reviews, classifying them as either positive or negative.

So my question is on how to write a simple feature selection for a text data set.

Answer 1

I currently use this approach:

calculate mean value and variance of data for each class. A good feature candidate should have small variance and the mean value should be different from mean values of other classes.

Currently having only < 50 features I select them manually. For automation of this process one could calculate variances of average values among all classes and give the higher prioritization to those, having bigger variance. Then, select first those, having smaller variance within one class.

Of cause this doesn't removes redundant features.

Answer 2

Feature selection methods are a big topic. You can start with following:

1. Chi square

2. Mutual information

3. Term frequency

etc. Read this paper if you have time: Comparative study on feature selection in text categorization this will help you lot.

The actual implementation depends on how you pre-process the data. Basically its keeping the counts, be it hash table or a database.

Answer 3

Random features work well, when you are then building ensembles. It's known as feature bagging.

Answer 4

Here's one option: Use pointwise mutual information. Your features will be tokens, and the information should be measured against the sentiment label. Be careful with frequent words (stop words), because in this type of task they may actually be useful.