Solr delta import does not work

2019-02-28 04:03发布

问题:

I am using solr 4.2. Note that full import works but somehow delta import doesn't. Delta import does not give any error but never fetches any changes. Here's the data config file.

<dataConfig> 
<dataSource type="JdbcDataSource" 
                   driver="com.microsoft.sqlserver.jdbc.SQLServerDriver" 
                   url="jdbc:sqlserver:testsql;databaseName=test" 
                   user="dba" 
                   password="dba"/> 


  <script>
    <![CDATA[
    function metadataTransformer (row) {
        var attributeName = row.get("attribute_name");
        var attributeValue = row.get("attribute_value");
        row.put(attributeName, attributeValue);

        return row;
    }
    ]]>
  </script>

<document name="PRODUCTS">

<entity name="product" query="select distinct  p.product_id as id from products p
                              inner join products_meta pm on p.product_id = pm.product_id
                              inner join meta m on pm.meta_id = m.meta_id
                              where m.meta_type_id = 11 order by id desc" 

deltaImportQuery="select distinct  p.product_id as id from products p
                              inner join products_meta pm on p.product_id = pm.product_id
                              inner join meta m on pm.meta_id = m.meta_id
                              where m.meta_type_id = 11 and p.product_id='${dih.delta.product_id}'"

deltaQuery= "select distinct  product_id as id from products 
                              where updtime > '${dih.last_index_time}'">

<field column="id" name="id"/> 

<entity name="attribute" query="select attribute_name,attribute_value from solr_import
                                where id =${product.id}" transformer= "script:metadataTransformer">

</entity>
</entity>

</document> 
</dataConfig>

Here's what I have tried without any luck.

changing p.product_id='${dih.delta.product_id} to p.product_id='${dih.delta.id} and other way also. changing where updtime > '${dih.last_index_time}'" to where updtime > '${dih.last_index_time}'"

Please help.

回答1:

This is fixed. The issue was solr box was on UTC timezone. Once I converted that to ET timezone, delta import started working fine. Hope this helps someone else.



回答2:

If you don't want to use ${dih.last_index_time} (server's time config problem), you can try mysql query like:

deltaImportQuery="SELECT * FROM table_name where id='${dataimporter.delta.id}'"
deltaQuery="select id from table_name where update_time  > SUBTIME( NOW( ) , '0:15:0' )

I used with crontab and worked fine!



标签: sql solr