Debezium Extension (Early Access)

The Debezium extension allows reading from a Debezium source connector and optionally writing from that input out to a JDBC interface.

Setup

In order to be able to use the Debezium extension, first include it as a dependency in your application's pom.xml file.

...
<dependency>
  <groupId>ext.grainite</groupId>
  <artifactId>grainite-debezium</artifactId>
  <version>{GRAINITE-VERSION}</version>
</dependency>
...

Replace {GRAINITE-VERSION} with the version of Grainite you are also using for libgrainite (the Grainite Client library for Java).

Contents

The Debezium Extension includes:

• DebeziumTask: Task to set up and read from a debezium source connector in a parallel manner.

• DebeziumCDCProcessor: Handler that can receive output from a DebeziumTask and convert it into a format that’s suitable for sending to a JDBCWriterHandler (adds the changeType and other fields as needed).

DebeziumTask has been implemented in a generic fashion that should allow it to work with any Debezium connector but has not been tested with every connector available. If you find a bug when using a particular connector, please share the name of the connector and a description of the issue encountered via email to [e-mail]

DebeziumTask

The DebeziumTask has some Grainite-only configuration, and additionally supports all of the configuration parameters that Debezium or the source connector may support. For example, here are the Oracle connector properties and the Mongo connector properties.

Usage

To include this task in your application, you must specify the taskClass ext.grainite.tasks.debezium.DebeziumTask and taskInstanceClass ext.grainite.tasks.debezium.DebeziumInstance in your application's configuration YAML file.

app.yaml

...
tasks:
  - task_name: my_debezium_task
    taskClass: ext.grainite.tasks.debezium.DebeziumTask
    taskInstanceClass: ext.grainite.tasks.debezium.DebeziumInstance
    config:
...

Below are the configuration options that can be passed in under config:

Property	Required?	Value	Description
connector.class	REQUIRED	Example: io.debezium.connector.mongodb.MongoDbConnector	Class name of the Debezium Source Connector
debug	Optional	true or false Default: false	Prints some additional debugging information when true.
max_poll_interval	Optional	Example: 30 Default: 60	max delay in seconds between successive polls. The task exponentially backs off until this delay value.
combine_txn	Optional	true or false Default: false	Will emit a list of all the updates (across tables specified) within the same message, if the updates were part of the same txn
tables.property.name	Optional	Example: my_table_action	Property name, specific to the connector, that indicates the tables being queried for CDC. When combine_txn is false, this will result in parallel reads from each table.
useObjectId	Optional	true or false Default: false	NOTE: This is only for MongoDB. Emits events with the Document ID in the key.
...			Additional properties for the specific connector.

Additionally, all properties provided by Debezium connectors are applicable here.

https://debezium.io/documentation/reference/stable/connectors/index.html

DebeziumCDCProcessor

Processor to take a DebeziumTask input and generate a message compatible with the JDBCWriterHandler

Usage

To include this handler in your application, you must specify the class_name ext.grainite.handlers.debezium.DebeziumCDCProcessor in your application's configuration YAML file.

app.yaml

...
tables:
  - table_name: my_debezium_cdc_processor
    key_type: string
    action_handlers:
      - name: my_debezium_cdc_processor
        type: java
        class_name: ext.grainite.handlers.debezium.DebeziumCDCProcessor
        config:
...

Below are the configuration options that can be passed in under config:

Property	Required?	Value	Description
output.topic	Optional	Example: my_events	Grainite topic to emit output of this task to
output.table	Optional	Example: my_table	Grainite table to emit output of this task to via Grain message (output.table_action must also be provided)
output.table_action	Optional	Example: my_table_action	Given action to invoke via Grain Message to the table specified in output.table

Debezium is a trademark of Red Hat, Inc.