I'm looking for a table or heuristic library that can convert extended characters like the o with the hat above it to a regular ascii o.
I'm looking to do this for search indexing purposes since most people are not going to type the o with the hat.
For example I type "Cote" into the search but I want my search to include things like "Côte".
It appears Solr does not convert these.
I tried java.text.Normalizer and friends but that did not work. did work see solution below.
You want to use the ASCIIFoldingFilterFactory when performing your indexing and query analysis in Solr. If you want this standalone you could just grab the source code and does whatever it does.
I did some quick googling and really didn't come up with any available libraries that will do character translation. I could be missing something though.
I did find the solr javadoc for all their factory classes:
http://lucene.apache.org/solr/api/org/apache/solr/analysis/package-summary.html
I'm sure you've seen the above already, but perhaps there is something there that you can use.
My personal suggestion is that you will have to write your filter. Which I don't believe will be very easy. If you accept only unicode, things will be easier, but if you're using a webapp, then you will have to be concerned about characterset coming in from your user's browser and then convert that characterset to unicode.
Best of luck.
This does appear to work:
import java.text.Normalizer;
Normalizer.normalize("ô", Normalizer.Form.NFD).replaceAll("[^\\p{ASCII}]","");
You've got the right general idea but the wrong specific approach.
Note, that in many European languages, it's a very bad idea to strip accents. People type them for a reason. Very different words differ only by an accent. 'papa' is not the same as 'papá'. Not even close. Users of these languages expect to type them in and expect to get search hits that respect the distinctions.
Second, there is plenty of existing work in this area. See lucene.apache.org, which has a set of token filters that do a variety of things you might or might not want.
There are also commercial offerings with other potentially useful behaviors, such as mapping 'running' to 'run' and all that.
If you are really sure you want to do this, first normalize to 'decomposed' (NFKD), and then just remove the accents. Viol(å).
