NLP: any easy and good methods to find semantic si

Question

I don't know whether StackOverflow covers NLP, so I am gonna give this a shot. I am interested to find the semantic relatedness of two words from a specific domain, i.e. "image quality" and "noise". I am doing some research to determine if reviews of cameras are positive or negative for a particular attribute of the camera. (like image quality in each one of the reviews).

However, not everybody uses the exact same wording "image quality" in the posts, so I am out to see if there is a way for me to build something like that:

"image quality" which includes ("noise", "color", "sharpness", etc etc) so I can wrap all everything within one big umbrella.

I am doing this for another language, so Wordnet is not necessarily helpful. And no, I do not work for Google or Microsoft so I do not have data from people's clicking behaviour as input data either.

However, I do have a lot of text, pos-tagged, segmented etc.

Answer 1

Word-Space is definitely the way to go here.If LSA is to slow for your application and if the semantics in random-indexing is too shallow the you should consider api.cortical.io . This REST API can give you the semantic fingerprint representation of any word. This semantic fingerprint contains all the different contexts to which the words belong. You can disambiguate any word wit one call like "organ" returns (muscle, piano, church, membership...) And for each of the contexts you can get contextual terms: "piano" will give (organ, clarinet, violin, flute, cello, compositions, harpsichord, orchestral) Concerning your last aspect, these semantic fingerprints are fully language independent. Currently cortical.io API covers: English, Spanish, French, German, Danish, Arabic, Russian, Chinese. More languages are being published until the end of 2014.

Answer 2

I saw word2vec on HackerNews a couple of weeks ago, looks pretty close to what you want.

Answer 3

Re your comment:

1. Classifiation through machine learning is being used for NLP all the time.
2. Regarding semantic similarity between concepts, see Dekang Lin's information theoretic definition of similarity.

Please also see these questions: finding related words,semantic similarity of two phrases.

Answer 4

Check out google similarity distance - http://arxiv.org/abs/cs.CL/0412098 eg. if lots of webpages include them both, theyre probably related.

demo program at http://mechanicalcinderella.com

Other than that, you could try to translate a project like wordnet ((google translate could help), or start a collaborative ontology.

Answer 5

In order to find semantic similarity between words, a word space model should do the trick. Such a model can be implemented very easily and fairly efficiently. Most likely, you will want to implement some sort of dimensionality reduction. The easiest one I can think of is Random Indexing, which has been used extensively in NLP.

Once you have your word space model, you can calculate distances (e.g. cosine distance) between words. In such a model, you should get the results you mentioned earlier (distance between "focus" and "Details" should be higher than "camera weight" vs "flash").

Hope this helps!

Answer 6

Take a look at Latent Semantic Indexing http://en.wikipedia.org/wiki/Latent_semantic_indexing it specifically addresses your problem. However you need to come up with some way to correlate these meta concepts with either positive or negative sentiments. Sentiment analysis http://en.wikipedia.org/wiki/Sentiment_analysis should help you.