Cassandra and Solr integration Using JDBC: org.apa

2019-08-05 09:01发布

问题:

I'm trying to integrate cassandra and solr. I'm using solr -6.6.0 version, cassandra 3.10 version and java 8.

After adding all the necessary config files, I could not get the integration going. I'm I missing any jars or conflicting jars? I tried with solr 7.1.0 as well and stuck with same issue. Any pointers would help.

Added to solarconfig.xml

  <lib dir="/Office/solr-6.6.0/dist" regex="solr-dataimporthandler-6.6.0.jar" />
  <lib dir="/Office/solr-6.6.0/dist" regex="solr-dataimporthandler-extras-6.6.0.jar" />      
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="asm-5.0.3.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="asm-analysis-5.0.3.jar" />        
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="asm-commons-5.0.3.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="asm-tree-5.0.3.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="asm-util-5.0.3.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="cassandra-driver-core-3.3.0.jar" />        
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="cassandra-driver-extras-3.3.0.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="cassandra-jdbc-1.2.5.jar" /> 
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="cassandra-jdbc-driver-0.6.4.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="cassandrajdbc1.1.jar" />        
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="guava-19.0.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="jffi-1.2.10-native.jar" /> 
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="jffi-1.2.10.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="jnr-constants-0.9.0.jar" />        
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="jnr-ffi-2.0.7.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="jnr-posix-3.0.27.jar" />   
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="jnr-x86asm-1.0.2.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="joda-time-2.9.9.jar" />        
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="jsqlparser-0.9.6.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="lz4-1.3.0.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="metrics-core-3.2.2.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="netty-buffer-4.0.47.Final.jar" />        
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="netty-codec-4.0.47.Final.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="netty-common-4.0.47.Final.jar" /> 
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="netty-handler-4.0.47.Final.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="netty-transport-4.0.47.Final.jar" />        
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="slf4j-api-1.7.25.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="snakeyaml-1.18.jar" /> 
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="snappy-java-1.1.4.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc/Workingcombo2" regex="tinylog-1.2.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc" regex="cassandra-all-3.11.1.jar" />  
  <lib dir="/Office/solr-6.6.0/cassandrajdbc" regex="cassandra-thrift-3.11.1.jar" /> 
  <lib dir="/Office/solr-6.6.0/cassandrajdbc" regex="libthrift-0.10.0.jar" />                

dataconfig.xml

<dataConfig>
<dataSource type="JdbcDataSource"
driver="org.apache.cassandra.cql.jdbc.CassandraDriver"
url="jdbc:cassandra://127.0.0.1:9042/test"
autoCommit="true"/>
<document name="content">
<entity name="test"
query="SELECT a,b from test" autoCommit="true">
<!--
<field column="is_first" name="is_first" />
<field column="is_last" name="is_last" />
-->
<field column="id" name="id" />
<field column="a" name="a_s" />
<field column="b" name="b_s" />
</entity>
</document>
</dataConfig>

schema.xml

<schema name="cassandra" version="1.1">
<uniqueKey>id</uniqueKey>
    <types>
            <fieldType name="string" class="solr.StrField" omitNorms="true"/>
            <fieldType name="double" class="solr.DoublePointField" omitNorms="true"/>
            <fieldType name="tdoubles" class="solr.DoublePointField" omitNorms="true"/>
            <fieldType name="int" class="solr.IntPointField" omitNorms="true"/>
            <fieldType name="long" class="solr.LongPointField" omitNorms="true"/>
            <fieldType name="tlongs" class="solr.LongPointField" omitNorms="true"/>
            <fieldType name="uuid" class="solr.UUIDField" indexed="true" />
            <fieldType name="text_general" class="solr.TextField" indexed="true" />
            <fieldType name="booleans" class="solr.BoolField" indexed="true" />
            <fieldType name="tdates" class="solr.DatePointField" docValues="true" multiValued="true"/>
    </types>

    <fields>
        <field name="id" type="string" indexed="true" stored="true" required="true"/>
        <field name="a" type="string" indexed="true" stored="true" required="true"/>
        <field name="b" type="string" indexed="true" stored="true"/>
        <field name="_version_" type="long" indexed="true"  stored="true"/>
    </fields>
</schema>

Error:

2017-12-19 00:29:08.675 INFO  (Thread-14) [   x:cassandraconnect] o.a.s.u.p.LogUpdateProcessorFactory [cassandraconnect]  webapp=/solr path=/dataimport params={core=cassandraconnect&optimize=false&indent=on&commit=true&name=dataimport&clean=true&wt=json&command=full-import&entity=test&_=1513643343716&verbose=false} status=0 QTime=25{deleteByQuery=*:* (-1587170087790444544)} 0 275
2017-12-19 00:29:08.677 ERROR (Thread-14) [   x:cassandraconnect] o.a.s.h.d.DataImporter Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: SELECT a,b from test Processing Document # 1
    at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:270)
    at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:415)
    at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:474)
    at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:457)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: SELECT a,b from test Processing Document # 1
    at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:416)
    at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
    at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
    ... 4 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: SELECT a,b from test Processing Document # 1
    at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)
    at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:327)
    at org.apache.solr.handler.dataimport.JdbcDataSource.createResultSetIterator(JdbcDataSource.java:288)
    at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:283)
    at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:52)
    at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)
    at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
    at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267)
    at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
    at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
    ... 6 more
Caused by: java.sql.SQLNonTransientConnectionException: org.apache.thrift.transport.TTransportException: Read a negative frame size (-2062548992)!
    at org.apache.cassandra.cql.jdbc.CassandraConnection.<init>(CassandraConnection.java:159)
    at org.apache.cassandra.cql.jdbc.CassandraDriver.connect(CassandraDriver.java:92)
    at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:192)
    at org.apache.solr.handler.dataimport.JdbcDataSource$1.call(JdbcDataSource.java:172)
    at org.apache.solr.handler.dataimport.JdbcDataSource.getConnection(JdbcDataSource.java:528)
    at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:317)
    ... 14 more
Caused by: org.apache.thrift.transport.TTransportException: Read a negative frame size (-2062548992)!
    at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:134)
    at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:424)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
    at org.apache.cassandra.thrift.Cassandra$Client.recv_describe_cluster_name(Cassandra.java:1247)
    at org.apache.cassandra.thrift.Cassandra$Client.describe_cluster_name(Cassandra.java:1235)
    at org.apache.cassandra.cql.jdbc.CassandraConnection.<init>(CassandraConnection.java:130)
    ... 19 more

Please help!

回答1:

You're trying to connect to Cassandra using JDBC protocol on CQL port 9042. You need to enable Thrift support by calling nodetool enablethrift and after this you should be able to connect by using port 9160 (or whatever you have in your Cassandra config)