Extensions

Grainite is often deployed in environments where other services or products are already running, and in many cases, there is need to either consume data from or push data to those other products. As opposed to building integrations for those products into the core Grainite product, we've introduced Extensions. Extensions are individual packages that contain Tasks, Handlers and other common code needed to integrate with those products.

Benefits of Grainite Extensions:

Grainite Extensions enjoy the same platform guarantees as any Grainite application. They are

• No additional infrastructure requirement. Extensions run directly on Grainite platform eliminating additional resource and operational costs.

• No data loss, duplication, or missed records. Once a record is pulled from the source, it is guaranteed to be processed and written to the sink.

• Source or sink data from/to external components at scale with sub-second latencies.

• High availability with built-in fault tolerance.

• Built-in monitoring and observability

Grainite offers two types of Extensions:

• Standard Extensions - Included with the product at no additional cost

• Paid Extensions - 3rd party connectors that are separately licensed

Note that Extensions are not the same as Tasks. Extensions are packages that may contain one or more tasks, one or more handlers, and other common library code that we want to share with users.

Versioning and Packaging

Extensions are versioned the same way as the main product, so Grainite version 2315 will also have extensions versioned as 2315. However, Extensions are not automatically installed on the server; they are designed to be integrated and packaged along with client applications.

For Java extensions, we publish Maven packages within our Gitlab Maven repository for customer applications to declare dependencies on and package extensions along with their application packages.

Example

An example of how an extension may be used within a client application is below. In this case, the application wants to read data from the CDC tables for SQLServer and pull them into a Grainite topic. From there, further processing can be done by Grainite handlers. Instead of writing custom code to do so, using the Tasks framework, the pre-built grainite-sqlserver extension can be used.

app.yaml

app_name: payments_app
package_id: org.samples
jars:
  - target/sqlserver-jar-with-dependencies.jar

tasks:
  - task_name: sqlserver_cdc
    taskClass: ext.grainite.tasks.sqlserver.SQLServerCDCReaderTask
    taskInstanceClass: ext.grainite.tasks.sqlserver.SQLServerCDCReaderInstance
    config:
      connectionUrl: $secret:sqlserver_connectionUrl
      database: TestDB
      tables: cards;transactions
      userid: $secret:sqlserver_userid
      password: $secret:sqlserver_password
      combine_txn: true
    output:
      topic: payments_topic
      
topics:
  - topic_name: payments_topic
    key_name: id
    key_type: string
    value_type: json

pom.xml

    <dependencies>
      ...
      <dependency>
        <groupId>com.grainite</groupId>
        <artifactId>libgrainite</artifactId>
        <version>2315</version>
      </dependency>
      <dependency>
        <groupId>ext.grainite</groupId>
        <artifactId>grainite-sqlserver</artifactId>
        <version>2315</version>
      </dependency>
      ...
    </dependencies>

The pages within this section provide usage details for each individual extension.