Removing Solr duplicate values into multivalued fi

问题:

My Solr index contains a multivalued field with duplicate values. How can I remove the duplicates ?

Is it possible to overwrite duplicate values into the multivalued field when indexing ?

Thanks

回答1:

I was struggling to accomplish the same. This worked for me. Add the below processor to your solrconfig.xml

<updateRequestProcessorChain name="deduplicateMultiValued" default="true">
        <processor class="org.apache.solr.update.processor.UniqFieldsUpdateProcessorFactory">
            <lst name="fields">
                <str>multivaluedFieldXYZ</str>
            </lst>
        </processor>
        <processor class="solr.RunUpdateProcessorFactory" />
 </updateRequestProcessorChain>

回答2:

Really late to the party, but the top answer did not work for me in Solr 6.0 for attempting to add a duplicate entry on a multivalued field. it was missing a processor right before UniqFieldsUpdateProcessorFactory. So adding something like this to my solrconfig.xml worked:

<updateRequestProcessorChain name="uniq-fields">
<processor class="org.apache.solr.update.processor.DistributedUpdateProcessorFactory"/>
<processor class="org.apache.solr.update.processor.UniqFieldsUpdateProcessorFactory">
  <str name="fieldName">YourFieldA</str>
  <str name="fieldName">yourFieldB</str>
</processor>
<processor class="solr.RunUpdateProcessorFactory" />

Where YourFieldA and YourFieldB are defined fields in your schema.xml. Note that you must also add this to the proper requestHandler ie:

  <requestHandler name="/update" class="solr.UpdateRequestHandler" >
<lst name="defaults">
  <str name="update.chain">uniq-fields</str>
</lst>

This will not only prevent duplicates from being added, but also remove all duplicates from your index upon update for the specified fields.

回答3:

You would need to handle it on the Client side to remove the duplicate values.

You can customize the implementation like RemoveDuplicatesTokenFilterFactory (works for same text at same position) to filter out the tokens. Write an extension basically. OR

Also, If using the multivalued field for just faceting ,the value in faceted field is counted just once. So even if you add multiple same values, that would be reflected as a single value in the facet count entry. Have tested this. you too can confirm.

However, the duplicate values would cause the change in the lengthNorm and hence can have an effect on the scoring.

回答4:

I am using solrJ to bind documents, and to avoid duplicated values I defined my multivalued field as a HashSet.

@Field("description")
public Collection<String> description = new HashSet<>();

回答5:

Or you could handle it in Solr, but in an UpdateRequestProcessor so that it happens before indexing and you don't need to learn about analysis chain.

You can use java or a number of scripting languages with the ScriptUpdateProcessor

回答6:

This configuration works for Solr 5.3.1

<updateRequestProcessorChain name="distinct-values" default="true">
    <processor class="solr.DistributedUpdateProcessorFactory"/>
    <processor class="solr.UniqFieldsUpdateProcessorFactory">
        <str name="fieldName">field1</str>
        <str name="fieldName">field2</str>
    </processor>
    <processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>