HOCON Configuration Merging

Engine Configuration Merging

Prior to StreamBase 10.5.0, a given engine was allowed one engine configuration; StreamBase now supports merging configurations of certain properties within respective engine configurations (namely, the StreamBaseEngine, the LDMEngine, and the JavaEngine configuration root objects). Multiple configurations can now have affinity with an engine.

Upon configuration activation (which can occur during deploy and runtime), a root object's contents are merged with those of other root objects of the same class. Configuration merging should not be planned to occur during "design-time settings"; the process of activation configuration requires load configuration before activation can occur.

On configuration replacement, the new root object contents replace the previous one's and are re-merged. Merge semantics are specific to each root object. Note that the engine(s) that the configuration belongs to must be stopped before you can activate an updated configuration. The rest of the node can still be running.

These engine configurations contain settings that (1) can overlap between design time and deploy time, and (2) can apply to multiple fragments (top-level and dependent) that comprise the application running on an engine. Consider how you want your application to handle these settings when designing your application.

Essentially merging can occur at deploy time, where an administrator can decide to add additional configuration to the configuration, which is embedded in the fragment and application archive using the epadmin install node --nodedeploy command.

Configuration Merge Strategies

Configuring the following merge-supported properties (where it makes sense to do so) is recommended in the top-level fragment rather than a dependent fragment:

Last-one-wins Strategy

Successive configurations' values overwrite earlier ones (and therefore are not intended for merging) for these supported properties:

jvmTuning

Note that this property value is itself an object, so the most recent object replaces the current settings in their entirety. All the nested values are engine-singleton in nature, so this is the most intuitive behavior — for example dependent fragment engine configurations should never set these values.

This property is supported in the StreamBaseEngine, the LDMEngine, and JavaEngine configuration root objects.

streamBase

This is a deeply nested object and replaces the current object in its entirety with no merging. All its nested properties are engine-singleton in nature.

This object is supported in the StreamBaseEngine and LDMEngine configuration root objects.

ldm

This is a deeply nested object and replaces the current object in its entirety with no merging. All its nested properties are engine-singleton in nature.

This object is supported in the LDMEngine configuration root object.

List-appended Strategy

Successive configurations' values are APPENDED to a combined list. The list elements are processed last-one-wins for the following:

jvmArgs

Successive values are appended. Using the same arguments in multiple configurations is not recommended.

This property is supported in the StreamBaseEngine, the LDMEngine, and JavaEngine configuration root objects.

Note

As of 10.5.0, a known limitation prevents jvmArgs property values from being successfully appended.

List-prepended Strategy

Successive configurations' values are PREPENDED to a combined list. The list elements are processed first-one-wins for these supported properties:

externalClassPath

This property is supported in the StreamBaseEngine, the LDMEngine, and JavaEngine configuration root objects.

externalNativeLibraryPath

This property is supported in the JavaEngine configuration root object.

Configuration Merge With Overwrite Strategy

All configurations' name-value pairs are put into a combined location, so subsequent values with the same name replace earlier values for the systemProperties property:

systemProperties

Maintains JVM last-one-wins argument semantics. Using the same system properties in multiple configurations is not recommended.

This property is supported in the StreamBaseEngine, the LDMEngine, and JavaEngine configuration root objects.

To summarize: you can merge JVM arguments, system properties, and external paths within an engine configuration singleton.

Singleton Configuration Overview

StreamBase defines a small number of cardinality behaviors, with respect to configuration. Some of these behaviors are singletons in nature (node, engine), others are named entities. To maintain consistency and minimize complexity, all root objects fall into one of the following singleton categories.

Node Singleton

A given node can have exactly one instance of the root object. Any attempt to activate a configuration containing a second instance will fail. New versions of the configuration containing the node singleton root object are permitted, with the new root object replacing the existing one. Node singleton objects do not have merge semantics.

Engine Singleton

A given node can have any number of instances of the root object, but a given engine can have affinity with exactly one instance of the root object. Any attempt to activate a configuration containing a second instance having affinity with the same engine will fail. New versions of the configuration containing the engine singleton root object are permitted, with the new root object replacing the existing one.

Again, multiple versions of the same engine singleton are allowed — the most recent version "wins". Engine configurations support merging as explained above.

Named Entities

A given node can have any number of instances of the root object. The root object has a name that uniquely identifies it (or contains sub-objects that are uniquely named, typically in an associative array). Any attempt to activate a configuration containing another root object or sub-object with the same name will fail. New versions of the configuration containing the named entity are permitted, with the new root object replacing the existing one.

Named entities contain settings that are shared by other configuration objects. Imagine an authentication realm shared by multiple listeners, or a JDBC data source shared by multiple engines, or parameter settings shared by multiple EventFlow deployments.

Named entities do not have merge semantics. All you can do is replace a named entity with a newer version of the entity; you cannot take two named entities defined in different configurations and combine them.

Custom Merge Semantics

A given node can have any number of instances of the root object. The root object is identified by class and owning configuration name+type+version (recall a configuration file of a given type can contain zero or one of each of the root objects of that type). Upon configuration activation, its contents are merged with those of other root objects of the same class. Upon configuration replacement, the new root object contents replace the previous one's and are re-merged. Merge semantics are specific to each root object.

List of Root Objects and Their Cardinalities

Application Definition

Node singleton.

Fragment Definition

Named entities: routers, data mappers, quorum notifiers.

Node Deploy

Node singleton.

Kerberos Authentication Realm

Node singleton (Kerberos limitation). Also named entity, with the namespace including ALL authentication realm types. That is, you cannot have a Kerberos realm foo and an LDAP realm foo.

LDAP Authentication Realm

Named entity, with the namespace including ALL authentication realm types. That is, you cannot have an LDAP realm foo and a Kerberos realm foo.

Local Authentication Realm

Named entity, with the namespace including ALL authentication realm types. That is, you cannot have a local realm foo and a Kerberos realm foo.

OIDC Authentication Realm

Named entity, with the namespace including ALL authentication realm types. That is, you cannot have an OIDC realm foo and a Kerberos realm foo.

Role to Privilege Mappings

Custom merge semantics. Entities (roles) are named, but you can have multiple roles with the same name in different role-to-privilege mapping root objects, and the result is a merge of all the privileges in those roles.

Secure Communication Client Profile

Named entity, with the namespace across client AND server profiles. That is, you cannot have a client profile foo and a server profile foo.

Secure Communication Server Profile

Named entity, with the namespace across client AND server profiles. That is, you cannot have a client profile foo and a server profile foo.

Trusted Hosts Set

A trusted host set consists of trusted host identifiers, which are names that have no associated content. A global set of identifiers (names, addresses, wildcard host name expressions, Classless Inter-Domain Routing (CIDR) expressions) is maintained and updated as individual trusted host objects are added, updated, and removed. It is not considered an error for multiple root objects to contribute the same identifier because there is no content associated with the identifier that could be different.

LiveView Client API Listener Configuration

Engine singleton.

LiveView Internal Credentials Configuration

Engine singleton.

Artifact Distribution Service Configuration

ADS replication configuration is in the context of having a single configuration that gets packaged in the application archive, since the ADS replication applies to the whole node. You are not prohibited from having multiple configurations, but only the first one read is actually used so it is not technically merging. First one wins.

Artifact configuration specifies the artifacts that are packaged with the fragments, and these fragments are automatically loaded into ADS when the fragment starts. Since each fragment might have its own set of artifacts, each is allowed to have its own artifact configuration(s).

As EventFlow fragment developer, you would add an artifact's configuration to your fragment defining the artifacts packaged in the fragment. Configuration merging applies here because all of the artifacts' configuration are "merged" together to create the set of artifacts that are loaded into ADS. That means you can develop and configure each fragment independently.

JDBC Data Source Group

Named entity (data source in group's associative array).

Adapter Group

Named entity (adapter in group's associative array).

JMS Adapter

Engine singleton.

Parameter Settings Group

Named entity (settings object in group's associative array).

Service Discovery Adapter Configuration

Engine singleton.

EventFlow Client API Listener Configuration

Engine singleton.

Cluster Monitor Service Configuration

Engine singleton.