Oracle replication data using Apache kafka

Question

I would like to expose the data table from my oracle database and expose into apache kafka. is it technicaly possible? As well i need to stream data change from my oracle table and notify it to Kafka. do you know good documentation of this use case? thanks

Answer 1

1. You need Kafka Connect JDBC source connector to load data from your Oracle database. There is an open source bundled connector from Confluent. It has been packaged and tested with the rest of the Confluent Platform, including the schema registry. Using this connector is as easy as writing a simple connector configuration and starting a standalone Kafka Connect process or making a REST request to a Kafka Connect cluster. Documentation for this connector can be found here

2. To move change data in real-time from Oracle transactional databases to Kafka you need to first use a Change Data Capture (CDC) proprietary tool which requires purchasing a commercial license such as Oracle’s Golden Gate, Attunity Replicate, Dbvisit Replicate or Striim. Then, you can leverage the Kafka Connect connectors that they all provide. They are all listed here

3. Debezium, an open source CDC tool from Redhat, is planning to work on a connector that is not relying on Oracle Golden Gate license. The related JIRA is here.

Answer 2

At the surface this is technically feasible. However, understand that the question has implications on downstream applications. So to comprehensively address the original question regarding technical feasibility, bear in mind the following:

1. Are ordering/commit semantics important? Particularly across tables.
2. Are continued table changes across instance crashes (Kafka/CDC components) important?
3. When the table definition changes - do you expect the application to continue working, or will resort to planned change control?
4. Will you want to move partial subsets of data?
5. What datatypes need to be supported? e.g. Nested table support etc.
6. Will you need to handle compressed logical changes - e.g. on update/delete operations? How will you address this on the consumer side?

Answer 3

You can use Kafka Connect for data import/export to Kafka. Using Kafka Connect is quite simple, because there is no need to write code. You just need to configure your connector.

You would only need to write code, if no connector is available and you want to provide your own connector. There are already 50+ connectors available.

There is a connector ("Golden Gate") for Oracle from Confluent Inc: https://www.confluent.io/product/connectors/