Solr DataImportHandler is not indexing all data de

Question

I am using solr5.3.

I am trying to upload wikipedia page article dump to solr using "DataImportHandler" but I am getting only id and title files when i am querying.

Below is my data-config.xml

<dataConfig>
        <dataSource type="FileDataSource" encoding="UTF-8" />
        <document>
        <entity name="page"
                processor="XPathEntityProcessor"
                stream="true"
                forEach="/mediawiki/page/"
                url="/mnt/TEST/enwiki-20150602-pages-articles1.xml"
                transformer="RegexTransformer,DateFormatTransformer"
                >
            <field column="id"        xpath="/mediawiki/page/id" />
            <field column="title"     xpath="/mediawiki/page/title" />
            <field column="revision"  xpath="/mediawiki/page/revision/id" />
            <field column="user"      xpath="/mediawiki/page/revision/contributor/username" />
            <field column="userId"    xpath="/mediawiki/page/revision/contributor/id" />
            <field column="text"      xpath="/mediawiki/page/revision/text" />
            <field column="timestamp" xpath="/mediawiki/page/revision/timestamp" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss'Z'" />
            <field column="$skipDoc"  regex="^#REDIRECT .*" replaceWith="true" sourceColName="text"/>
       </entity>
        </document>
</dataConfig>

Also I have added below entires to schema.xml.

 <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
    <field name="title"     type="string"  indexed="true" stored="false"/>
    <field name="revision"  type="int"    indexed="true" stored="true"/>
    <field name="user"      type="string"  indexed="true" stored="true"/>
    <field name="userId"    type="int"     indexed="true" stored="true"/>
    <field name="text"      type="text_en"    indexed="true" stored="false"/>
    <field name="timestamp" type="date"    indexed="true" stored="true"/>
    <field name="titleText" type="text_en"    indexed="true" stored="true"/>

I have copied schema.xml from "example/example-DIH/solr/solr/conf/schema.xml" and removed all field entries with few exceptions as mentioned in comments.

After importing data I am just trying to fetch all fields but I am getting only "Id" and "Title".

Also I tried to run documentImport using debug mode so that I can get some information regarding indexing, but at whenever i am selecting debug mode it is only importing 2 documents. I am not sure why? Due to this reason I am not able to debug the indexing process.

Please guide me further.

EDIT-I am now sure that other fields are not getting indexed because when I am specifying df=user or text, I am getting below message.

"msg": "undefined field user",

I am querying like below: http://localhost:8983/solr/wiki/select?q=%3A&fl=id%2Ctitle%2Ctext%2Crevision&wt=json&indent=true&debugQuery=true

Answer 1

My dear friend you have simply mis-typed one of the fields. Try this link and you'd want to laugh and cry at the same time.

http://localhost:8983/solr/wiki/select?q=*%3A*&fl=id+titleText+user+revision&wt=json&indent=true

The title you mentioned in the schema is "titleText" and your limit mentioned "title" and "text" seperately. So God speed and you can stay in touch with me via hangouts: [e-mail]

Answer 2

The provided setting will work fine with classic schema only. But at solrconfig by default managed schema was enabled. Due to which I was not getting text. For managed schema I need not to define "schema.xml" and I should define fields in data-config.xml like below.

 <field column="id"        xpath="/mediawiki/page/id" />
            <field column="title_s"     xpath="/mediawiki/page/title" />
            <field column="revision"  xpath="/mediawiki/page/revision/id" />
            <field column="user_s"      xpath="/mediawiki/page/revision/contributor/username" />
            <field column="userId"    xpath="/mediawiki/page/revision/contributor/id" />
            <field column="text_s"      xpath="/mediawiki/page/revision/text" />
            <field column="timestamp" xpath="/mediawiki/page/revision/timestamp" dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss'Z'" />
            <field column="$skipDoc"  regex="^#REDIRECT .*" replaceWith="true" sourceColName="text"/>

Answer 3

I was recently trying the same wikipedia import with Solr 7. The reason text isn't returned is because that field in the managed_schema is set to stored="false":

<field name="text" type="text_en" indexed="true" stored="false"/>

Changing it to stored="true" will return the text.

The current accepted answer suggests using the text_s field, which was probably stored in the managed_schema of the Solr version OP was using. Note that searching for terms included in any field that is not stored will still return the relevant document, only the text itself is not returned. See the answer here for more info: Solr index vs stored