Don't split on underscore with solr.StandardTo

Question

I'm using solr, I'm using StandardTokenizerFactory in the text field but I don't want to split on the underscore. Do I have to use another toknizer like PatternTokenizerFactory or I can do this with StandardTokenizerFactory ? as I need the same functionality of StandardTokenizerFactory but without split on underscore.

Answer 1

I don't think you can do it in StandardTokenizerFactory. One solution is to first replace underscores with something the StandardTokenizerFactory won't process and something your documents won't otherwise contain. For example, you can first replace _ with QQ everywhere with PatternReplaceCharFilterFactory and pass through StandardTokenizerFactory and then replace QQ with _ using PatternReplaceFilterFactory. Here is the fieldType definition to do it:

<fieldType name="text_std_prot" class="solr.TextField" positionIncrementGap="100">
    <analyzer>
        <charFilter class="solr.PatternReplaceCharFilterFactory" 
                    pattern="_" 
                    replacement="QQ"/>
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.PatternReplaceFilterFactory" 
                pattern="QQ" 
                replacement="_"/>
        ...
    </analyzer>
</fieldType>

And here is a screen shot of what happens:

Answer 2

Adding just following seems to do trick for StandardTokenizerFactory as StandardTokenizerFactory splits at hyphen "-".

<charFilter class="solr.PatternReplaceCharFilterFactory" 
                    pattern="_" 
                    replacement="-"/>
      <tokenizer class="solr.StandardTokenizerFactory"/>