Runtime Application Configuration

Overview

This article provides a reference for writing a StreamBase Runtime Application configuration file where the HOCON type is com.tibco.ep.dtm.configuration.application.

Application definition configuration is used to define application and fragment configuration values that are known at design-time. There are two top-level configuration objects used to configure applications and fragments:

  • ApplicationDefinition — use to configure an application.

  • FragmentDefinition — use to configure a fragment.

The ApplicationDefinition configuration is packaged in an application archive, while the FragmentDefinition configuration is generally packaged in a fragment archive.

A single application definition configuration file may contain both an ApplicationDefinition configuration object and zero or more FragmentDefinition configuration objects. All of the FragmentDefinition configuration objects are processed first. If there are multiple FragmentDefinition objects in a configuration file, the order in which they are processed is undefined.

Audits

The application definition configuration is audited whenever it changes state (for example, load-to-active state). There are detailed audits for each of the configuration objects in the application definition configuration file. Also, the following audits are enforced on application definition configuration:

  • An application definition configuration is optional — an application archive does not have to contain one.

  • If an application definition is contained in an application archive, there can only be one ApplicationDefinition configuration object.

  • The ApplicationDefinition configuration can only be contained in an application archive. It cannot be contained in a fragment archive. An ApplicationDefinition configuration found in a fragment archive will fail an audit during application installation.

  • Each fragment archive may optionally contain a FragmentDefinition configuration object.

  • One or more FragmentDefinition configuration objects can be contained in an application archive.

Required Header Lines

Each configuration file must contain the following header lines, typically found at the beginning of each file:

name

Specifies an arbitrary, case-sensitive string to name this configuration, which must be unique among other files with the same type, if any. Configuration files can refer to each other by this name. Select a name that reminds you of this configuration's type and purpose. For example:

name = "FullAppConfig"
version

Specifies an arbitrary version number that you can use to keep track of file versions for this configuration type in your development project. The maintenance of version numbers is under user control; StreamBase does not compare versions when loading configuration files during the fragment launch process. The version number is a string value, and can contain any combination of characters and numbers. For example:

version = "1.0.0"
type

This essential setting specifies the unique HOCON configuration type described on this page.

type = "com.tibco.ep.dtm.configuration.application"

The three header lines taken together constitute a unique signature for each HOCON file in a project's configurations folder. Each project's configurations folder can contain only one file with the same signature.

The top-level configuration object defines the configuration envelope the same way for all HOCON file types.

configuration

On a line below the header lines, enter the word configuration followed by an open brace. The configuration object is a sibling of the name, version, and type identifiers, and serves to define the configuration envelope around this type's objects as described on this page. The file must end with the matching close brace.

configuration = {
...
...
}

Root Objects

The application configuration type contains two root objects:

This section provides detailed descriptions for all properties in each application configuration object.

ApplicationDefinition Properties

The ApplicationDefinition object is the top-level container for all of the application definition configuration including fragment-specific data mappings, fragment specific notifiers, application execution resources, and data distribution policies. The following shows the relationships to other configuration objects.

ApplicationDefinition

Application definition configuration root object.

description

String. Human-readable description. Optional.

execution

The execution resources associated with an instantiated application, which includes data transport, transaction and node types. A single set of execution resources applies to the whole deployed cluster. Optional. If unset, the execution object applies a set of default values to its child objects.

buildType

Build to use when executing your application; this property is used only during the boot process. Do not change the default value PRODUCTION unless instructed by TIBCO Support to diagnose problems. DEVELOPMENT builds require additional libraries (available from Support) that contain diagnostics that will affect performance. Optional.

buildType = PRODUCTION
transaction

Transaction configuration object. Optional.

timeoutSeconds

Long. The number of seconds to wait for a distributed lock. If this time value expires, a deadlock is thrown and the transaction is retried. Must be a positive number. Optional. Default value is 60 seconds.

For example:

timeoutSeconds = 60
numberCompletedTransactions

Long. The maximum number of committed or aborted transactions to retain for each remote node. This property is used for recovery processing to determine the outcome of global transactions if a remote node crashes and is restarted with a loss of shared memory. Optional. Default value is 1000.

For example:

numberCompletedTransactions = 1000
maximumBackoffMilliseconds

Long. The maximum amount of time, in milliseconds, to back off during deadlock resolution. Deadlock backoff times will never exceed this value. Optional. The default value is 10000 milliseconds (10 seconds).

For example:

maximumBackoffMilliseconds = 10000
dataTransport

DataTransport configuration object. Optional

keepAliveSendIntervalSeconds

Long. Keep-alive send interval. Must be a positive number. Optional. Default value is 1 second.

For example: keepAliveSendIntervalSeconds = 1
nonResponseTimeoutSeconds

Long. Keep-alive non-response timeout interval following a send failure on a network interface to a remote node. When the non-response timeout expires on all configured interfaces, the remote node is marked down. Must be a positive number. Optional. Default value is 2 seconds.

For example:

nonResponseTimeoutSeconds = 2
deferredWritesEnabled

Bool. Control deferred writes distribution protocol. A value of true causes writes to be deferred until a transaction commits. A value of false causes writes to immediately follow a field modification. Optional. Default value is true.

For example:

deferredWritesEnabled = true
nodeActiveTimeoutSeconds

Long. The amount of time in seconds to wait for a remote node to move into the Active state before a resource unavailable exception is raised. The wait is done for a partition's active node when defining a partition, or for each remote node when a discover cluster administration command is executed. This value must be greater than 0. Optional. Default value is 60 seconds.

For example:

nodeActiveTimeoutSeconds = 60
tcpNoDelayEnabled

Bool. Control enabling of TCP_NODELAY socket option on connection establishment. Optional. Default value is true.

For example:

tcpNoDelayEnabled = true
maximumPDUSizeBytes

Long. Control maximum protocol data unit (PDU) size used for distributed communications. This value must be greater than 4000. Optional. Default value is 1000000 bytes.

For example:

maximumPDUSizeBytes = 1000000
nodeTypes

Associative array of NodeType configuration objects keyed by node-type-name. Optional, with no default.

For example, a nodeType named nodetype1 containing the following objects.

nodetype1

String. NodeType name example.

description

String. Human-readable description. For example:

description = "node type 1"
fragments

An array of fully qualified fragment names that specified the fragments to be run on this node type. If set, only these fragments may run on nodes of this type, and only these fragments' data distribution policy bindings are associated with this node type. If unset, nodes of this type can run all application fragments, and all data distribution policy bindings are associated with this node type.

If a fragment specified here has dependent fragments, you only need to specify the top-level fragment. Dependent fragments are determined and included automatically.

For example:

ApplicationDefinition = {
   execution = {
     nodeTypes = {
      "DataService" = {
       description = "Way fast data collector"
       fragments = [
         "com.example.dept.WayFastCollector"
         "com.example.dept.SlowItDownNow"
       ]
     }
     ...
sharedMemory

Shared memory setup for this node type. Optional. If unset, a set of default values are applied to sharedMemory's child objects.

memoryAllocators

Long. Number of concurrent shared memory allocators. This property is used only during the boot process. This property is optional and its default value is based on the number of cores on the machine and the size of shared memory.

For example:

memoryAllocators = 8
memoryType

Type of memory to use for system shared memory. The value FILE indicates a memory mapped file. The value SYSTEM_V_SHARED_MEMORY is System V Shared memory. This property is used only during the boot process. This optional property's default value is FILE.

For example:

memoryType = "FILE"
memorySizeBytes

Long. Shared memory size in bytes. This property is used only during the boot process. This optional property's default and minimum value is 512 MBytes.

For example:

memorySizeBytes = 2g
memoryThrottling

Memory throttling. This is optional. If unset, a set of default values are applied to the memoryThrottling's child objects.

thresholdPercentage

Long. A number treated as a percentage at which to start shared memory throttling. This property is optional and its default value is 50.

For example:

thresholdPercentage = 50
checksPerSecond

Long. The number of shared memory utilization checks per second. This property is optional and its default value is 10.

For example:

checksPerSecond = 2
cacheFlush

Memory cache flush settings. This is optional. If unset, a set of default values are applied to cacheFlush's child objects.

flushIntervalSeconds

Long. The number of seconds to sleep between runs of the flusher. A value of 0 disables the flusher; its default value is 1 second.

For example:

flushIntervalSeconds = 5
maximumObjectsPerType

Long. Control the number of objects per type that will be flushed per flusher run. A value of 0 indicates no limit; its default value is 0.

For example:

maximumObjectsPerType = 0
ipc

Shared memory IPC configuration parameters. Optional. If unset, a set of default values are applied to ipc's child objects.

noDestinationTimeoutSeconds

Long. The number of seconds to block waiting for a target engine for a method invocation. The current transaction is aborted when the timeout value is exceeded. This property is optional and its default value is 10 seconds.

For example:

noDestinationTimeoutSeconds = 10
dataDistributionPolicies

An associative map of data distribution policies keyed by policy name. This object is optional and has no default value. If not present, the application's managed objects have no high availability.

type

Enumeration. The data distribution policy type. Valid values are DYNAMIC or STATIC. Optional. Default value is DYNAMIC.

For example:

type = DYNAMIC
customQuorumNotifier

String. Custom quorum notifier names associated with the data distribution policy. The custom quorum notifier name must be defined in a FragmentDefinition configuration in the application. Optional. No default.

For example:

customQuorumNotifiers = ["quorumNotifier", "anotherQuorumNotifier"]
dataMappers

String. Data mapper names associated with the data distribution policy. Data mapper name must be defined in a FragmentDefinition configuration in the application (see FragmentDefinition for more information). Optional. No default.

disableAction

Enumeration. Valid values are LEAVE_CLUSTER or LEAVE_CLUSTER_FORCE.

Actions performed when a node leaves this group. LEAVE_CLUSTER means the node performs any object migrations needed to migrate partitions owned by this node to the first order replica node, and removes this node from all replica lists.

Before the disable is executed, an audit is done to insure that no partitions will be moved to the UNAVAILABLE state. LEAVE_CLUSTER_FORCE works exactly as above, but no audit step is performed. Optional. The default value is LEAVE_CLUSTER_FORCE.

For example:

disableAction = LEAVE_CLUSTER_FORCE
enableAction

Enumeration. Actions performed when a node joins this group. Valid values are JOIN_CLUSTER, JOIN_CLUSTER_PURGE, or JOIN_CLUSTER_RESTORE.

JOIN_CLUSTER means the node performs any object migrations needed to activate the partitions defined for this group. Use this option for newly added nodes.

JOIN_CLUSTER_PURGE means before migrating the partition data to the joining node, the node removes any instances that exist on the local node. Use this option for nodes where the local data does not need to be preserved.

JOIN_CLUSTER_RESTORE means for all partitions where the node joining the cluster is the current active node, the node accesses the remote node where the partition should be restored from, and compares the objects between the local node and the remote node. If an object on the local node was updated, or a new object created, the update is pushed to the remote node. If the remote node was updated, the change is copied to the local node. If a state conflict is detected, or a duplicate object is found, remove the local object and copy the remote node object to the local node. For all partitions where the node joining the cluster is acting as a replica, the replica objects are deleted and instances migrated from the remote node.

Warning

JOIN_CLUSTER_RESTORE is performance hostile, and should only be done when resolving a multi-master scenario in a cluster.

Optional. The default value is JOIN_CLUSTER.

For example:

enableAction = JOIN_CLUSTER
forceReplication

Bool. Determine how objects are replicated during the migration of an object partition. When set to true, a copy of partitioned objects to all preexisting replica nodes is done. By default, objects are only copied to new replicas as they are added to the cluster since the objects should already exist on the existing replica nodes.

The default behavior allows replica nodes to be incrementally added to a cluster without having to copy the objects to the existing replica nodes. However, if one or more replicas nodes were offline, or were not discovered when the high-availability group was first enabled, this property can be set to insure that objects are pushed to all replicas in the cluster.

Warning

Setting this property to true is performance hostile, and should only be done if a replica node cannot be manually taken offline and restored.

Optional. Default value is false.

For example:

forceReplication = false
numberOfThreads

Long. Define the number of threads used when performing a migration. When migration occurs, the work is done in multiple threads in order to scale to the number of CPUs available in the machine. If the number of partitioned objects is less than the value of objectsLockedPerTransaction, only one thread is used. If objectsLockedPerTransaction is zero, or the calling transaction has one or more partitioned objects locked in the transaction, numberOfThreads is ignored, and all work is done in the caller's transaction. Optional. The default value is 1.

For example:

numberOfThreads = 1
objectsLockedPerTransaction

Long. Define the number of objects locked in a transaction when performing a partition migrate or update. When a partition is migrating or being updated the work is done in object chunks defined by this value. This allows applications to concurrently run while the work is in progress, otherwise all partitioned objects would be locked in the transaction performing the work, conflicting with application code also attempting to write lock on the active node or establishing a read lock on replica nodes.

If this value is set to zero, all instances will be processed in the work initiator's transaction. If the calling transaction has one or more partitioned objects locked in the transaction, the objectsLockedPerTransaction value is ignored, and all work is done in the caller's transaction. To ensure that migration or update work minimizes the number of locks taken, it should be run in separate transactions that have no partitioned objects locked. Optional. The default value is 1000.

For example:

objectsLockedPerTransaction = 1000
remoteEnableAction

Enumeration. Valid values are ENABLE_PARTITION or LEAVE_DISABLED.

When a remote node enables a partition, it enables the partition on all nodes in the node list, and updates the partition status and state. This is done even on nodes that have explicitly disabled the partition. This behavior can be changed by setting this value to LEAVE_DISABLED. Note that this only controls the behavior on the local node; it does not affect partitions defined on remote nodes. Optional. The default value is ENABLE_PARTITION.

For example:

remoteEnableAction = ENABLE_PARTITION
staticDataDistributionPolicy

Static data distribution policy configuration. Can only be specified if data distribution policy type is STATIC. Optional.

foreignAudit

A foreign partition is a partition that does not have the local node as the active node, or in the replica node list. When enabled, no data migration takes place; instead the node list is verified against the node list that is present on the active node to verify they match. When restoring multiple nodes, it can be necessary to disable the auditing of node lists, since the active node may have a node list that is the result of a previous failover.

Options are one of IGNORE_NODE_LIST or VERIFY_NODE_LIST. This property is optional and its default value is VERIFY_NODE_LIST

For example:

foreignAudit = IGNORE_NODE_LIST

In StreamBase releases before 10.3.0, the foreignAudit property was known as sparseAudit and foreign partitions were known as sparse partitions. These terms are now deprecated.

replicaAudit

Define audit of replica nodes when enabling a partition. This property can be set to WAIT_ACTIVE or DISCARD_REPLICA. If set to WAIT_ACTIVE, block until each replica node is active before enabling the partition. If set to DISCARD_REPLICA and the replica node state is not active, the replica node is removed from the node list. This property is optional and its default value is WAIT_ACTIVE.

For example:

replicaAudit = WAIT_ACTIVE
dynamicDataDistributionPolicy

Dynamic data distribution policy configuration. Can only be specified if data distribution policy type is DYNAMIC. Optional.

numberOfPartitions

Long. Number of high availability partitions that are used to distribute the data in this dynamic partition group.

For example:

numberOfPartitions = 64
numberOfClones

Long. Number of entries in the consistent hash ring buffer used to distributed keys across partitions. Increasing this number may result in more uniform distribution of keys, but increases the Java heap memory overhead. This property is optional and its default value is 1000.

For example:

numberOfClones = 1000
primaryDataRedundancy

Data redundancy for the primary group. If this optional object is unset, a set of default values are applied to primaryDataRedundancy child objects.

numberOfReplicas

Long. Number of replicas, which controls the contents of the partitions created. This property is optional and its default value is 2.

For example:

numberOfReplicas = 2
replicationType

The type of replication, either SYNCHRONOUS or ASYNCHRONOUS. This property is optional and its default value is SYNCHRONOUS.

For example:

replicationType = SYNCHRONOUS
backupDataRedundancy

Data redundancy for the backup group. If this optional object is unset, a set of default values are applied to backupDataRedundancy's child contained objects.

numberOfReplicas

Long. Number of replicas, which controls the contents of the partitions created. This property is optional and its default value is 2.

For example:

numberOfReplicas = 1
replicationType

The type of replication, either SYNCHRONOUS or ASYNCHRONOUS. This property is optional and its default value is SYNCHRONOUS.

For example:

replicationType = ASYNCHRONOUS

FragmentDefinition Properties

FragmentDefinition

The FragmentDefinition root object defines fragment-specific configuration, including data mappings and notifiers. This configuration is associated with the fragment archive in which it is packaged. On configuration activation, a set of runtime managed objects is created to support activation of a node deployment file.

description

String. Fragment description. Optional. No default.

For example:

description = "my application"
dataMappers

Implements a specific data distribution policy. Data mapper names must be globally unique within an application. Optional, with no default.

The following are optional properties that you can add to the dataMappers object:

deferMappings

Bool. If set to true, the mapper installation should be deferred until the type exists in shared memory. Default is false.

roundRobinDataMappers

This mapper applies to statically or dynamically partitioned classes, and will distribute managed objects across all partitions in the contained availability zone in a round robin fashion.

className

Mapper type identifier. This property is required and has no default value. For example:

className = "com.tibco.ep.dtm.configuration.node.policy11.
  ClassA"
distributedHashDataMappers

Defines partitioned classes that should use a distributed hash partitioning policy, and can be any valid string. You can specify a distributedHashDataMapper for both static and dynamic data distribution policies.

className

String. Mapper type identifier. This property is required and has no default value. Note that one key or field property must be specified. For example:

className = "com.tibco.ep.dtm.configuration.node.policy21.
  ClassA"
key

String. If set, distribute the data by the named key.

field

String. If set, distribute the data by the named field. For example:

field = "fieldA"
customDataMappers

Defines an array of partitioned classes in a custom partition mapper, and can be any valid string. You can specify a custom data mapper only with a static data distribution policy.

className

String. Mapper type identifier. This property is required and has no default value. For example:

className = "com.tibco.ep.dtm.configuration.node.
  policy41.ClassA"
factoryClassName

Name of the factory class that creates mapper objects used to perform distribution.

For example:

factoryClassName = "com.tibco.ep.dtm.configuration.
  node.policy41.ClassAMapperFactory"
value

Data to pass to the instantiated mapper. Its value is a single string interpreted by the mapper factory. This property is required and has no default value.

For example:

value = "10"
routers

Associative router configuration objects keyed by router names. Router names must be globally unique within an application. This object is optional with no default.

className

Router class name. This property is required and has no default value.

For example, for a router named myHashRouter, a className of:

className = "com.tibco.haservice.sample.HashRouter"
customQuorumNotifiers

Associative objects keyed by quorum notifier names. One or more CustomQuorumNotifiers can be specified. Custom quorum notifier names (quorumNotifier and anotherQuorumNotifier in the example below) must be globally unique within an application. Optional. No default.

notifierClassName

Quorum notifier Java class, which is called when the runtime detects the loss of quorum.

For example:

customQuorumNotifiers = {
  quorumNotifier = {
    notifierClassName = "com.tibco.ep.dtm.configuration.node.MyQuorumNotifier"
  }
  anotherQuorumNotifier = {
    notifierClassName = "com.tibco.ep.dtm.configuration.node.AnotherQuorumNotifier"
}

ApplicationDefinition Configuration File Sample

Long lines wrap to the next line in this example for clarity.

name = "appdef"
type = "com.tibco.ep.dtm.configuration.application"
version = "1.0.0"
configuration = {
  ApplicationDefinition = {
    dataDistributionPolicies = {
      sample-data-distribution-policy-0 = {
        dataMappers = [ "sample-data-mapper-name" ]
        disableAction = "LEAVE_CLUSTER_FORCE"
        dynamicDataDistributionPolicy = {
          backupDataRedundancy = {
            numberOfReplicas = 2
            replicationType = "SYNCHRONOUS"
          }
          numberOfClones = 1000
          numberOfPartitions = 64
          primaryDataRedundancy = {
            numberOfReplicas = 2
            replicationType = "SYNCHRONOUS"
          }
        }
        enableAction = "JOIN_CLUSTER"
        forceReplication = false
        numberOfThreads = 1
        objectsLockedPerTransaction = 1000
        remoteEnableAction = "ENABLE_PARTITION"
        staticDataDistributionPolicy = {
          foreignAudit = "VERIFY_NODE_LIST"
          replicaAudit = "WAIT_ACTIVE"
        }
        type = "DYNAMIC"
      }
    }
    description = "sample application description"
    execution = {
      buildType = "PRODUCTION"
      dataTransport = {
        deferredWritesEnabled = true
        keepAliveSendIntervalSeconds = 1
        maximumPDUSizeBytes = 1000000
        nodeActiveTimeoutSeconds = 60
        nonResponseTimeoutSeconds = 2
        tcpNoDelayEnabled = true
      }
      nodeTypes = {
        sample-node-type-0 = {
          description = "sample node type description"
          fragments = [ "sample-fragment-ID" ]
          sharedMemory = {
            cacheFlush = {
              flushIntervalSeconds = 1
              maximumObjectsPerType = 0
            }
            ipc = {
              noDestinationTimeoutSeconds = 10
            }
            memoryAllocators = 0
            memorySizeBytes = "512M"
            memoryThrottling = {
              checksPerSecond = 10
              thresholdPercentage = 50
              }
            memoryType = "FILE"
          }
        }
      }
      transaction = {
        maximumBackoffMilliseconds = 10000
        numberCompletedTransactions = 1000
        timeoutSeconds = 60
      }
    }
  }
}

FragmentDefinition Configuration File Sample

Long lines wrap to the next line in this example for clarity.

name = "FullFragmentConfig"
version = "1.0"
type = "com.tibco.ep.dtm.configuration.application"
configuration = {
  FragmentDefinition = {
    description = "my application"
      dataMappers = {
        fragmentMapperA = {
          roundRobinDataMappers = [
            {
              className = "com.example.ClassName"
            }
          ]
          distributedHashDataMappers = [
            {
              className = "com.example.ClassName"
              field = "sample-field-value"
            }
          ]
          customDataMappers = [
            {
              className = "com.example.ClassName"
              factoryClassName = "com.example.FactoryClassName"
              value = "sample-factory-data-value"
            }
          ]
        }
      }
      routers = {
        myHashRouter = {
          className = "com.tibco.ep.dtm.ha.sample.HashRouter"
        }
          myRoundRobinRouter = {
            className = "com.tibco.ep.dtm.ha.sample.RoundRobinRouter"
        }
          myBroadcastRouter = {
            className = "com.tibco.ep.dtm.ha.sample.BroadcastRouter"
        }
          myPartitionedRouter = {
            className = "com.tibco.ep.dtm.ha.sample.PartitionedRouter"
        }
          myNodeRouter = {
            className = "com.tibco.ep.dtm.ha.sample.NodeRouter"
        }
      }
      customQuorumNotifiers = {
        myNotifier = {
          notifierClassName = "com.tibco.ep.dtm.configuration.node.QuorumNotifier"
        }
      }
  }
}