App Configuration Options

YAML file configuration options

The tables and topics for Grainite applications are defined in a YAML configuration file:

Java config

app.yaml

app_name: my_app
package_id: org.sample
env_name: test
# Location of jar file relative to this config file
jars:
    - target/my_app-jar-with-dependencies.jar

topics:
-   topic_name: my_topic
    key_name: key_id
    key_type: string
    value_type: json

exception_handlers:
  DefaultExceptionHandler:
    class_name: org.sample.my_app.MyExceptionHandler
    method_name: handleRuntimeException
    exceptions:
      - java.lang.RuntimeException
  StatusCodeExceptionHandler:
    status_code: 13
    reason: User not found.
    exceptions:
      - org.sample.my_app.UserNotFound

imports:
-   name: DatabaseCDC
    config:
      connection: https://database:1234
      table: my_table

tables:
-   table_name: my_table
    key_type: string
    value_type: json
    auto_index_json: true
    maps:
    -   id: 1
        name: stats
        key_type: string
        value_type: double
    action_handlers:
    -   name: MyActionHandler
        type: java
        class_name: org.sample.my_app.MyActionHandler
        timeout: 30
        config:
            param_one: foo
        actions:
        -   action_name: handleOrders
            # `method_name` is optional. Default method_name is the `action_name`
            method_name: handleOrders
            # `handles_multiple_actions` is optional. Default is `false`.
            handles_multiple_actions: false
            error_sink_topic: handleOrderErrorSink
            exception_handlers: 
                - DefaultExceptionHandler
                - StatusCodeExceptionHandler
            subscriptions:
                - subscription_name: mySubscription
                  topic_name: my_topic
                  topic_key: key_id

Python config

app.py.yaml

app_name: my_app
python_dir: src
env_name: test

topics:
-   topic_name: my_topic
    key_name: key_id
    key_type: string
    value_type: json

tables:
-   table_name: my_table
    key_type: string
    value_type: json
    auto_index_json: true
    maps:
    -   id: 1
        name: stats
        key_type: string
        value_type: double
    action_handlers:
    -   name: MyActionHandler
        type: python
        script_path: handlers/MyActionHandler.py
        timeout: 30
        config:
            param_one: foo
        actions:
        -   action_name: handle_orders
            # `method_name` is optional. Default method_name is the `action_name`
            method_name: handle_orders
            # `handles_multiple_actions` is optional. Default is `false`.
            handles_multiple_actions: false
            subscriptions:
                - subscription_name: mySubscription
                  topic_name: my_topic
                  topic_key: key_id

App Name

app_name: my_app

Name of the app to load onto the server. With this, all topics and tables within this app will have their names prepended with the app name. This allows for multiple apps to have the same table and topic names, as well as integrate with each other.

Package ID (Java/Kotlin only)

package_id: org.sample

Project-related package ID. This field is used by gx gen all to generate classes and files. For example, for the YAML above, gx gen all -M would generate the files with the following structure:

src
└── main
    └── java
        └── org
            └── sample
                └── my_app
                    ├── Client.java
                    ├── Constants.java
                    └── MyHandler.java

pagegx CLI

Python Directory (Python only)

python_dir: src

This is where you specify the location of the Python source code files, if you application uses any Python code. This is used as the path prefix for the script_path of your Python action handlers (see the Python example in Action Handlers below). For example, you would put src here if your Python client and action handler scripts were stored like the following:

src
├── Client.py
└── handlers
    └── ActionHandler.py

Then the script_path of your action handlers defined in ActionHandler.py would be handlers/ActionHandler.py.

Environment Name

env_name: test

The name of the environment for this application. This can be overridden for various gx commands by using --env_name.

Applications are scoped by environments, allowing for instances of the same application to run on a single server in complete isolation.

Environment names must not start with a double leading underscore (__) and must not contain a colon (:).

JARs (Java only)

jars: 
    - target/my_app-jar-with-dependencies.jar

The path, relative to the config file, of the JAR files that contain the action handlers. Generally, all action handlers are packaged within a single JAR, but if that is not the case, it is possible to provide multiple JAR files:

jars: 
    - target/jar1.jar
    - target/jar2.jar
    - target/jar3.jar

The JARs provided to Grainite need to be fat/uber JARs, which means that they need to contain all their dependencies. In order to build the fat JAR, you will need to provide additional arguments or options, depending on your build system.

Head over to the Maven/Gradle Setup page for information on how to package your application.

Topics

topics:
-   topic_name: my_topic
    key_name: key_id
    key_type: string
    value_type: json

A topic has a topic_name, key_name ,key_type and a value_type.

The topic_name uniquely identifies a topic within the app, the key_name provides a way for grains to subscribe to topics, while key_type and value_typedefine the type of the key and value.

Exception Handlers (Java only)

There are two kinds of exception handlers:

1. Standard Exception Handler

exception_handlers:
  DefaultExceptionHandler:
    class_name: org.sample.my_app.MyExceptionHandler
    method_name: handleRuntimeException
    exceptions:
      - java.lang.RuntimeException

An exception handler must have a method_name and a list of exceptions. Optionally, a class_name can be specified as well. If the class_name is not specified, it defaults to the action handler's parent class.

The exception handler name (DefaultExceptionHandler in the example above) serves as an identifier to allow re-using exception handlers with multiple actions.

When an action handler throws a specified exception, the specified "exception handler" is invoked and its result is used.

2. Status Code Exception Handler

exception_handlers:
  StatusCodeExceptionHandler:
    status_code: 13
    reason: User not found.
    exceptions:
      - org.sample.my_app.UserNotFound

A status code exception handler must have a status_code and a list of exceptions. Optionally, a reason can be specified as well.

When an action handler throws a specified exception, the result for the action is automatically converted into an ActionResult.failure object with the provided status_code and reason.

To attach an exception handler to an action, please refer to the Actions section.

Please refer to the GrainContext API page for instructions on writing an exception handler.

Tables

tables:
-   table_name: my_table
    key_type: string
    value_type: long
    maps:
    -   id: 1
        name: stats
        key_type: string
        value_type: double
    action_handlers:
     ...

A table has a table_name, key_type , value_type maps and action_handlers.

The table_name uniquely identifies a table, the key_type and value_type define the type of key and value for each grain in the table, and maps define the structure for an entry in a map. maps can be identified by either id and/or name . action_handlers contain definitions for action handlers in each grain in the table.

pageData Types and Schemas

Automatic Secondary Indexes

This feature was added in 2315.

auto_index_json: true

auto_index_json can be used to have Grainite automatically create secondary indices for all top level properties for a json stored in a Grain’s value.

For example, consider this:

Java

String jsonString = """
{ 
  "name": "John",
  "location": "Los Angeles",
   courses: [
     "Computer Science",
     "Physics"
   ]
}
""";

context.setValue(Value.of(jsonString));

Python

json_string = '''
{ 
  "name": "John",
  "location": "Los Angeles",
   courses: [
     "Computer Science",
     "Physics"
   ]
}
'''
context.set_value(Value.of(json_string))

If auto_index_json: true is set for this Table, Grainite will automatically create indices for name == "John" and location == "Los Angeles" for this Grain.

These indices can then be queried using the table.find() API:

Java

Iterator<Grain> iter = table.find("name == 'John' && location == 'Los Angeles'");

// This will print the json from above.
System.out.println(iter.next().getValue().asString());

Python

iter = table.find("name == 'John' && location == 'Los Angeles'")

# This will print the json from above.
print(next(iter).get_value().as_string())

table.find() was added for Python in 2321.

Action Handlers

Java config

action_handlers:
    -   name: MyActionHandler
        type: java
        class_name: org.sample.my_app.MyActionHandler
        timeout: 30
        actions:
        ...

Python config

action_handlers:
    -   name: MyActionHandler
        type: python
        script_path: handlers/MyActionHandler.py
        timeout: 30
        actions:
        ...

An Action Handler has a name, type, actions and some additional properties based on the provided type.

The name serves as the identifier for the action handler. This field is required if there are more than one action handlers for a table.

The type defines the language that the action handler is written in. Currently, there are three types that are supported:

1. java

2. python

action_handlers used to be called action_classes in previous versions. Functionally, both are the same and either one of them can be used.

However, it is highly recommended to use action_handlers instead of action_classes as it more accurately defines their roles.

Java config

-   type: java
    class_name: org.sample.my_app.MyActionHandler

Java action handlers require a class_name which is the full class name of the defined Java class. These classes should be part of the project package (defined by package_id).

Python config

-   type: python
    script_path: handlers/MyActionHandler.py

Python action handlers require a script_path which is the path to the Python file where this action handler function is defined. The prefix of this path is the application's python_dir , see Python Directory above).

timeout was added in 2229 and enables users to configure timeouts (in seconds) for all actions in their action handlers. If an action takes longer than the provided timeout, it is automatically retried. By default, this timeout is set to 60 seconds.

Actions

actions:
-   action_name: myAction
    method_name: handleMyAction
    error_sink_topic: handleOrderErrorSink
    handles_multiple_actions: false
    exception_handlers: 
        - DefaultExceptionHandler
        - StatusCodeExceptionHandler

actions define the types of actions that are expected to be handled in the provided action handler. Note that method_name and handles_multiple_actions are both optional.

action_name defines the name of the action, while method_name defines the name of the method (in the action handler), that is expected to handle the action. With this, it is possible for a single method to handle multiple actions.

If method_name is not provided for an action, the method_name is the same as the action_name.

error_sink_topic defines a topic in which all failed events for this action are sent.

exception_handlers define the types of exception handlers associated with this action. When an action throws an exception, the exception handler corrospending to the exception is called and its result is used instead.

At the moment,error_sink_topicand exception_handlers are not available in Python. Only action handlers implemented in Java can make use of these features.

There are two types of methods that can be defined within action handlers:

Single-Action Variant

This method takes a single Action and returns a single ActionResult for a per-action action handler.

Multi-Action Variant

This method takes a list of Action and returns a list of ActionResult for a batch-action action handler.

handles_multiple_actions can be set to true to use the Multi Action Variant, or false to use the Single Action Variant.

By default, handles_multiple_actions is set to false.

Subscriptions

subscriptions:
    - subscription_name: orderUpdates
      topic_name: orders_topic
      topic_key: order_id

A method can have multiple subscriptions to topics. A subscription is defined by a subscription_name, topic_name and topic_key. When a new event is appended to a topic, all subscriptions for that topic are triggered and the method corresponding to the subscription is invoked.

The subscription_name can be used to control the subscription (defer, pause, etc.), while the topic_name and topic_key identify the topic and key (for that topic) for the subscription.

To subscribe to a topic in another Application, simply specify the app and topic name, separated by a colon, in the YAML - <app_name>:<topic_name>.

subscriptions:
    - subscription_name: ordersSubscription
      topic_name: food_app:orders_topic
      topic_key: order_id

Configuration

config:
    param_one: foo

config allows users to pass data from the configuration file to action handlers or tasks and access it via GrainContext:

Java

String paramOne = context.getConfig().get("param_one");

Python

param_one = context.get_config()["param_one"]

Secrets

A secret can be passed to the configuration using the prefix $secret. So for a secret named my_secret which is stored in Grainite, it can be passed in through the configuration file as below:

config:
    some_token: $secret:my_secret

This can be used by calling the resolveProperty GrainiteContext API:

String token = context.resolveProperty(context.getConfig().get("my_secret"));

Starting in release version 2327, string interpolation for secrets is now supported.

config:
    some_token: $secret:{my_secret}-abc

Assuming the value for my_secret is 123, the above will resolve to 123-abc when context.resolveProperty("some_token") is called in the Action Handler.

🧪 Imports (Early Access)

imports:
-   name: DatabaseCDC
    config:
      connection: https://database:1234
      table: my_table

imports provide a convenient way to incorporate complex configurations into your application, enabling Java programs to generate configurations programmatically, based on user-supplied configuration parameters. imports can even be used to share complex application configuration and logic with others through simple configurations.

An import is a way for applications to programmatically provide their required resources (tables, topics, etc.) to gx.

Imports are resolved at the time an app is loaded (via gx load). To view the resolved app config without running gx load, you can use gx beta app-resolve.

Each import config must contain a name which will be used by the gx CLI to search for a properties file - <name>GrainiteImport.properties, in the provided JAR file(s). Consider the example from above; since the name of the import is DatabaseCDC, gx will look for a DatabaseCDCGrainiteImport.properties file in the root of the provided JAR file.

The GrainiteImport.properties must contain a config_provider_class property to point to the class which will generate an application configuration. For instance, the DatabaseCDC properties file could look like this:

# DatabaseCDCGrainiteImport.properties
config_provider_class=my.pkg.database_cdc.ConfigProvider

By default, gx will invoke the generateGrainiteConfig method but this can be changed by providing a config_provider_method property.

The method header for the "config generator" method should be the following:

public Map<String, Object> generateGrainiteConfig(Map<String, Object> userConfig)

The method takes in a map representing the config options defined by the user in their app.yaml. For the DatabaseCDC example, the map will consist of:

connection: https://database:1234
table: my_table

The method must return a Map<String, Object>, representing the resources required by this import. Currently, the following resources can be generated by an import:

• Topics

• Tables

• Tasks

Here is an example of what my.pkg.database_cdc.ConfigProvider might generate:

public Map<String, Object> generateGrainiteConfig(Map<String, Object> userConfig) {
  Map<String, Object> resources = new HashMap<>();
  
  // Declare topics needed by this application.
  List<Map<String, Object>> topics = new ArrayList<>();
  Map<String, Object> inputTopic = Map.of("topic_name", "cdc_topic", "key_name", "table_id", "key_type", "string");
  topics.add(intputTopic);
  resources.put("topics", topics);
  
  // Declare tables needed by this application.
  List<Map<String, Object>> tables = new ArrayList<>();
  Map<String, Object> processorTable = Map.of("table_name", userConfig.get("table"), "key_type", "string", "action_handlers", List.of(...));
  tables.add(processorTable);
  resources.put("tables", tables);
  
  return resources;
}

The configuration returned will be merged with the existing app.yaml configuration, and will be utilized by gx load.

In the event of conflicting imports, the one declared first in the app.yaml file will take precedence. If an import's configuration clashes with the user-defined configuration specified in the app.yaml, the user-defined configuration will take precedence.