parse Solr xml files to SolrInputDocument

Question

If I have individual files in the expected Solr format (having just ONE doc per file):

<add>
  <doc>
    <field name="id">GB18030TEST</field>
    <field name="name">Test with some GB18030 encoded characters</field>
    <field name="features">No accents here</field>
    <field name="features">ÕâÊÇÒ»¸ö¹¦ÄÜ</field>
    <field name="price">0</field>
  </doc>
</add>

Is not there a way to easily marshal that file into a SolrInputDocument? Do I have to do the parsing myself?

EDIT: I need it in java pojo cause I want to modify some fields before indexing it with SolrJ...

Answer 1

This is best done programmatically. I know you're looking for a Java solution, but I'd personally recommend groovy.

The following script processes XML files found in the current directory.

//
// Dependencies
// ============
import org.apache.solr.client.solrj.SolrServer
import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer
import org.apache.solr.common.SolrInputDocument

@Grapes([
    @Grab(group='org.apache.solr', module='solr-solrj', version='3.5.0'),
])

//
// Main
// =====
SolrServer server = new CommonsHttpSolrServer("http://localhost:8983/solr/");

new File(".").eachFileMatch(~/.*\.xml/) { 

    it.withReader { reader ->
        def xml = new XmlSlurper().parse(reader)

        xml.doc.each { 
            SolrInputDocument doc = new SolrInputDocument();

            it.field.each {
                doc.addField([e-mail](), it.text())
            }

            server.add(doc)
        }
    }

}

server.commit()

Answer 2

EDIT: In order to convert XML to POJO, please reference this previous SO Question - Is there a library to convert Java POJOs to and from JSON and XML?

Since you already have your documents in the expected format, you can just use the post.jar or post.sh script file as shown in the Solr Tutorial - Indexing Data that both accept xml files as input.

Also, there is a toSolrInputDocument() method in the SolrJ ClientUtils library that may be useful for you. Granted you would need to marshal the files into the SolrDocument class in order to use the toSolrInputDocument() method.

Answer 3

In Java you can do this.

private void populateIndexFromXmlFile(String fileName) throws Exception {

    UpdateRequest update = new UpdateRequest();

    update.add(getSolrInputDocumentListFromXmlFile(fileName));

    update.process(server);

    server.commit();
}

private List<SolrInputDocument> getSolrInputDocumentListFromXmlFile(
        String fileName) throws Exception {

    ArrayList<SolrInputDocument> solrDocList = new ArrayList<SolrInputDocument>();

    File fXmlFile = new File(fileName);

    DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
    DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
    Document doc = dBuilder.parse(fXmlFile);

    NodeList docList = doc.getElementsByTagName("doc");

    for (int docIdx = 0; docIdx < docList.getLength(); docIdx++) {

        Node docNode = docList.item(docIdx);

        if (docNode.getNodeType() == Node.ELEMENT_NODE) {

            SolrInputDocument solrInputDoc = new SolrInputDocument();

            Element docElement = (Element) docNode;

            NodeList fieldsList = docElement.getChildNodes();

            for (int fieldIdx = 0; fieldIdx < fieldsList.getLength(); fieldIdx++) {

                Node fieldNode = fieldsList.item(fieldIdx);

                if (fieldNode.getNodeType() == Node.ELEMENT_NODE) {

                    Element fieldElement = (Element) fieldNode;

                    String fieldName = fieldElement.getAttribute("name");
                    String fieldValue = fieldElement.getTextContent();

                    solrInputDoc.addField(fieldName, fieldValue);
                }

            }

            solrDocList.add(solrInputDoc);
        }
    }

    return solrDocList;

}