Reference for DTM Application Configuration

Overview

This article provides a reference for writing a DTM Application HOCON configuration file.

Required Header Lines

Each configuration file must contain the following header lines, typically found at the beginning of each file:

name

Specifies an arbitrary, case-sensitive string to name this configuration, which must be unique among other files with the same type, if any. Configuration files can refer to each other by this name. Select a name that reminds you of this configuration's type and purpose. For example:

name = "FullAppConfig"
version

Specifies an arbitrary version number that you can use to keep track of file versions for this configuration type in your development project. The maintenance of version numbers is under user control; StreamBase does not compare versions when loading configuration files during the fragment launch process. The version number is a string value, and can contain any combination of characters and numbers. For example:

version = "1.0.0"
type

This essential setting specifies the unique HOCON configuration type described on this page.

type = "com.tibco.ep.dtm.configuration.application"

The header lines taken together constitute a unique signature for each HOCON file in a project's configurations folder. Each project's configurations folder can contain only one file with the same signature.

In addition, the configuration top-level element is the same for all HOCON file types.

configuration

On a line below the header element lines, enter the word configuration followed by an open brace. The configuration element is a sibling of the name, version, and type elements, and serves as a wrapper around this type's elements described below. The file must end with the matching close brace.

configuration = {
...
...
}

HOCON Elements Explained

This section provides detailed descriptions for all properties in each application definition configuration object.

ApplicationDefinition Elements

The ApplicationDefinition object is the top-level container for all of the application definition configuration, including the FragmentDefinition object. The following shows the relationships to other configuration objects.

ApplicationDefinition

Application Definition Configuration

description

Human-readable description

execution

The execution resources associated with an instantiated application, which includes data transport, transaction and node types. A single set of execution resources applies to the whole deployed cluster. This is optional; if unset, the execution object applies a set of default values to its child objects.

buildType

For example:

buildType = DEVELOPMENT
transaction

Transaction configuration object. Optional.

timeoutSeconds

Long. The number of seconds to wait for a distributed lock. If this time value expires, a deadlock is thrown and the transaction is retried. Must be a positive number. Optional. Default value is 60 seconds.

For example:

timeoutSeconds = 60
numberCompletedTransactions

Long. The maximum number of committed or aborted transactions to retain for each remote node. This is used for recovery processing to determine the outcome of global transactions if a remote node crashes and is restarted with a loss of shared memory. Optional. Default value is 1000.

For example:

numberCompletedTransactions = 1000
maximumBackoffMilliseconds

Long. The maximum amount of time, in milliseconds, to back off during deadlock resolution. Deadlock backoff times will never exceed this value. Optional. The default value is 10000 milliseconds (10 seconds).

For example:

maximumBackoffMilliseconds = 10000
dataTransport

DataTransport configuration object. Optional

keepAliveSendIntervalSeconds

Long. Keep-alive send interval. Must be a positive number. Optional. Default value is 1 second.

For example: keepAliveSendIntervalSeconds = 1
nonResponseTimeoutSeconds

Long. Keep-alive non-response timeout interval following a send failure on a network interface to a remote node. When the non-response timeout expires on all configured interfaces, the remote node is marked down. Must be a positive number. Optional. Default value is 2 seconds.

For example:

nonResponseTimeoutSeconds = 2
deferredWritesEnabled

Bool. Control deferred writes distribution protocol. A value of true causes writes to be deferred until a transaction commits. A value of false causes writes to be done immediately following a field modification. Optional. Default value is true.

For example:

deferredWritesEnabled = true
nodeActiveTimeoutSeconds

Long. The amount of time in seconds to wait for a remote node to move into the Active state before a resource unavailable exception is raised. The wait is done for a partition's active node when defining a partition, or for each remote node when a discover cluster administration command is executed. This value must be greater than 0. Optional. Default value is 60 seconds.

For example:

nodeActiveTimeoutSeconds = 60
tcpNoDelayEnabled

Bool. Control enabling of TCP_NODELAY socket option on connection establishment. Optional. Default value is true.

For example:

tcpNoDelayEnabled = true
maximumPDUSizeBytes

Long. Control maximum protocol data unit (PDU) size used for distributed communications. This value must be greater than 4000. Optional. Default value is 1000000 bytes.

For example:

maximumPDUSizeBytes = 1000000
nodeTypes

Associative array of NodeType configuration objects keyed by node-type-name. Node types supported by the application. Optional. No default.

For example, a nodeType named nodetype1 containing the following name-value pairs.

nodetype1

String. NodeType name example.

description

String. Human-readable description.

For example:

description = "node type 1"
sharedMemory

Shared memory setup for this node type. This is optional. If unset, a set of default values are applied to sharedMemory's child objects.

memoryAllocators

Long. Number of concurrent shared memory allocators. This name-value pair is used only during the boot process. This name-value pair is optional and its default value is based on the number of cores on the machine and the size of shared memory.

For example:

memoryAllocators = 8
memoryType

Type of memory to use for system shared memory. The value FILE indicates a memory mapped file. The value SYSTEM_V_SHARED_MEMORY is System V Shared memory. This field is used only during the boot process. This name-value pair is optional and its default value is FILE.

For example:

memoryType = "FILE"
memorySizeBytes

Long. Shared memory size in bytes. This field is used only during the boot process. This field is optional and its default value is 512MBytes.

For example:

memorySizeBytes = 2g
memoryThrottling

Memory throttling. This is optional. If unset, a set of default values are applied to the memoryThrottling's child objects.

thresholdPercentage

Long. A number treated as a percentage at which to start shared memory throttling. This field is optional and its default value is 50.

For example:

thresholdPercentage = 50
checksPerSecond

Long. The number of shared memory utilization checks per second. This field is optional and its default value is 10.

For example:

checksPerSecond = 2
cacheFlush

Memory cache flush settings. This is optional. If unset, a set of default values are applied to cacheFlush's child objects.

flushIntervalSeconds

Long. The number of seconds to sleep between runs of the flusher. A value of 0 disables the flusher; its default value is 1 second.

For example:

flushIntervalSeconds = 5
maximumObjectsPerType

Long. Control the number of objects per type that will be flushed per flusher run. A value of 0 indicates no limit; its default value is 0.

For example:

maximumObjectsPerType = 0
ipc

Shared memory IPC configuration parameters. This is optional. If unset, a set of default values are applied to ipc's child objects.

noDestinationTimeoutSeconds

Long. The number of seconds to block waiting for a target engine for a method invocation. The current transaction is aborted when the timeout value is exceeded. This field is optional and its default value is 10 seconds.

For example:

noDestinationTimeoutSeconds = 10
dataDistributionPolicies

An associative map of data distribution policies keyed by policy name. This name-value pair is optional and has no default value. If not present, the application's managed objects have no high availability.

type

Enumeration. The data distribution policy type. Valid values are DYNAMIC or STATIC. Optional. Default value is DYNAMIC.

For example:

type = DYNAMIC
customQuorumNotifiers

String. Custom quorum notifier names associated with the data distribution policy. Custom quorum notifier name must be defined in a FragmentDefinition configuration in the application (see FragmentDefinition for more information). Optional. No default.

dataMappers

String. Data mapper names associated with the data distribution policy. Data mapper name must be defined in a FragmentDefinition configuration in the application (see FragmentDefinition for more information). Optional. No default.

disableAction

Enumeration. Valid values are LEAVE_CLUSTER or LEAVE_CLUSTER_FORCE.

Actions performed when a node leaves this group. LEAVE_CLUSTER means the node performs any object migrations needed to migrate partitions owned by this node to the first order replica node, and removes this node from all replica lists.

Before the disable is executed, an audit is done to insure that no partitions will be moved to the UNAVAILABLE state. LEAVE_CLUSTER_FORCE works exactly as above, but no audit step is performed. Optional. The default value is LEAVE_CLUSTER_FORCE.

For example:

disableAction = LEAVE_CLUSTER_FORCE
enableAction

Enumeration. Actions performed when a node joins this group. Valid values are JOIN_CLUSTER, JOIN_CLUSTER_PURGE, or JOIN_CLUSTER_RESTORE.

JOIN_CLUSTER means the node performs any object migrations needed to activate the partitions defined for this group. This option should be used for newly added nodes.

JOIN_CLUSTER_PURGE means before migrating the partition data to the joining node, the node removes any instances that exist on the local node. This option should be used for nodes where the local data does not need to be preserved.

JOIN_CLUSTER_RESTORE means for all partitions where the node joining the cluster is the current active node, the node accesses the remote node where the partition should be restored from, and compares the objects between the local node and the remote node. If an object on the local node has been updated, or a new object created, the update is pushed to the remote node. If the remote node has been updated, the change is copied to the local node. If a state conflict is detected, or a duplicate name-value pair is found, remove the local object and copy the remote node object to the local node. For all partitions where the node joining the cluster is acting as a replica, the replica objects are deleted and instances migrated from the remote node.

Warning

JOIN_CLUSTER_RESTORE is performance hostile, and should only be done when resolving a multi-master scenario in a cluster.

Optional. The default value is JOIN_CLUSTER.

For example:

enableAction = JOIN_CLUSTER
forceReplication

Bool. Determine how objects are replicated during the migration of an object partition. When set to true, a copy of partitioned objects to all pre-existing replica nodes is done. By default, objects are only copied to new replicas as they are added to the cluster since the objects should already exist on the existing replica nodes.

The default behavior allows replica nodes to be incrementally added to a cluster without having to copy the objects to the existing replica nodes. However, if one or more replicas nodes were offline, or were not discovered when the high-availability group was first enabled, this property can be set to insure that objects are pushed to all replicas in the cluster.

Warning

Setting this property to true is performance hostile, and should only be done if a replica node cannot be manually taken offline and restored.

Optional. Default value is false.

For example:

forceReplication = false
numberOfThreads

Long. Define the number of threads used when performing a migration. When migration occurs, the work is done in multiple threads in order to scale to the number of CPUs available in the machine. If the number of partitioned objects is less than the value of objectsLockedPerTransaction, only one thread will be used. If objectsLockedPerTransaction is zero, or the calling transaction has one or more partitioned objects locked in the transaction, numberOfThreads is ignored, and all work is done in the caller's transaction. Optional. The default value is 1.

For example:

numberOfThreads = 1
objectsLockedPerTransaction

Long. Define the number of objects locked in a transaction when performing a partition migrate or update. When a partition is migrating or being updated the work is done in object chunks defined by this value. This allows applications to concurrently run while the work is in progress, otherwise all partitioned objects would be locked in the transaction performing the work, conflicting with application code also attempting to write lock on the active node or establishing a read lock on replica nodes.

If this value is set to zero, all instances will be processed in the work initiator's transaction. If the calling transaction has one or more partitioned objects locked in the transaction, the objectsLockedPerTransaction value is ignored, and all work is done in the caller's transaction. To insure that migration or update work minimizes the number of locks taken, it should be run in separate transactions that have no partitioned objects locked. Optional. The default value is 1000.

For example:

objectsLockedPerTransaction = 1000
remoteEnableAction

Enumeration. Valid values are ENABLE_PARTITION or LEAVE_DISABLED.

When a remote node enables a partition, it will enable the partition on all nodes in the node list, and update the partition status and state. This is done even on nodes that have explicitly disabled the partition. This behavior can be changed by setting this value to LEAVE_DISABLED. Note that this only controls the behavior on the local node; it does not affect partitions defined on remote nodes. Optional. The default value is ENABLE_PARTITION.

For example:

remoteEnableAction = ENABLE_PARTITION
staticDataDistributionPolicy

Static data distribution policy configuration. Can only be specified if data distribution policy type is STATIC. Optional.

sparseAudit

A sparse Partition is a partition that doesn't have the local node as the active node, or in the replica node list When enabled, no data migration takes place; instead the node list is verified against the node list that is present on the active node to verify they match. When restoring multiple nodes, it may be necessary to disable the auditing of node lists, since the active node may have a node list that is the result of a previous failover.

Field is one of IGNORE_NODE_LIST or VERIFY_NODE_LIST. This field is optional and its default value is VERIFY_NODE_LIST

For example:

sparseAudit = IGNORE_NODE_LIST
replicaAudit

Define audit of replica nodes when enabling a partition. Field is one of WAIT_ACTIVE or DISCARD_REPLICA. If set to WAIT_ACTIVE, block until each replica node is active before enabling the partition. If set to DISCARD_REPLICA and the replica node state is not active, the replica node is removed from the node list. This field is optional and its default value is WAIT_ACTIVE.

For example:

replicaAudit = WAIT_ACTIVE
dynamicDataDistributionPolicy

Dynamic data distribution policy configuration. Can only be specified if data distribution policy type is DYNAMIC. Optional.

numberOfPartitions

Long. Number of high availability partitions that are used to distribute the data in this dynamic partition group.

For example:

numberOfPartitions = 64
numberOfClones

Long. Number of entries in the consistent hash ring buffer used to distributed keys across partitions. Increasing this number may result in more uniform distribution of keys, but increases the Java heap memory overhead. This field is optional and its default value is 1000.

For example:

numberOfClones = 1000
primaryDataRedundancy

Data redundancy for the primary group. This is optional. If unset, a set of default values are applied to primaryDataRedundancy's child objects.

numberOfReplicas

Long. Number of replicas, which controls the contents of the partitions created. This field is optional and its default value is 2.

For example:

numberOfReplicas = 2
replicationType

The type of replication, either SYNCHRONOUS or ASYNCHRONOUS. This field is optional and its default value is SYNCHRONOUS.

For example:

replicationType = SYNCHRONOUS
backupDataRedundancy

Data redundancy for the backup group. This is optional. If unset, a set of default values are applied to backupDataRedundancy's child contained objects.

numberOfReplicas

Long. Number of replicas, which controls the contents of the partitions created. This field is optional and its default value is 2.

For example:

numberOfReplicas = 1
replicationType

The type of replication, either SYNCHRONOUS or ASYNCHRONOUS. This field is optional and its default value is SYNCHRONOUS.

For example:

replicationType = ASYNCHRONOUS

FragmentDefinition Elements

FragmentDefinition

The FragmentDefinition object defines fragment specific configuration. This configuration is associated with the fragment archive in which it is packaged.

description

String. Fragment description. Optional. No default.

For example:

description = "my application"
dataMappers

Associative array of DataMapper objects keyed by data mapper names. Data mapper names must be globally unique within an application. Optional. No default.

The following example shows a dataMapper named fragmentMapperA, which includes the following name-value pairs:

deferMappings

Bool. Defer mappings. If set to true, the installation of the mapper should be deferred until the type exists in shared memory. This field is optional and its default value is false.

For example:

deferMappings = false
roundRobinDataMappers

List of Round-robin data mappers. This field is optional.

className

String. Name of the class being distributed. This field is required.

For example:

className = "com.tibco.ep.dtm.configuration.node.policy11.ClassA"
distributedHashDataMappers

List of distributed hash mappers. This field is optional.

className

String. Name of the class being distributed. This field is required and has no default value.

For example:

className = "com.tibco.ep.dtm.configuration.node.policy21.ClassA"
key

String. If set, distribute the data by the named key. Note that one of key or field is required.

field

String. If set, distribute the data by the named field. Note that one of key or field us required.

For example:

field = "fieldA"
customDataMappers

CustomDataMapper array of configuration objects. Only valid with static data distribution policies. Optional. No default.

className

String. Name of the application class being distributed. This field is required and has no default value.

For example:

className = "com.tibco.ep.dtm.configuration.node.policy41.ClassA"
factoryClassName

String. Name of the factory class that creates mapper objects used to perform distribution. This field is required and has no default value.

For example:

factoryClassName = "com.tibco.ep.dtm.configuration.node.policy41.ClassAMapperFactory"
value

String. Data to pass to the instantiated mapper. Its value is a single string interpreted by the mapper factory. This field is required and has no default value.

For example:

value = "10"
routers

Associative array of Router objects keyed by router names. Router configuration. Router names must be globally unique within an application. Optional. No default.

className

Router class name. This field is required and has no default value.

For example, for a router named myHashRouter, a className of:

className = "com.tibco.haservice.sample.HashRouter"
customQuorumNotifiers

Associative array of CustomQuorumNotifier objects keyed by quorum notifier names. Custom quorum notifier configuration. Only a single CustomQuorumNotifier object may be specified. Custom quorum notifier names must be globally unique within an application. Optional. No default.

HOCON Configuration File Samples

The following are two examples of the com.tibco.ep.dtm.configuration.application type.

ApplicationDefinition HOCON File Sample

name = "FullAppConfig"
version = "1.0"
type = "com.tibco.ep.dtm.configuration.application"

configuration = {
  ApplicationDefinition = {
    description = "my application"
       
        
    execution = {
      buildType = DEVELOPMENT
      transaction = {
          timeoutSeconds = 60
          numberCompletedTransactions = 1000
          maximumBackoffMilliseconds = 10000
      }
      dataTransport = {
          keepAliveSendIntervalSeconds = 1
          nonResponseTimeoutSeconds = 2
          deferredWritesEnabled = true
          nodeActiveTimeoutSeconds = 60
          tcpNoDelayEnabled = true
          maximumPDUSizeBytes = 1000000
      }
      nodeTypes = {
          nodetype1 = {
              description = "node type 1"
              fragments = [
                  "fragment1"
              ]
              sharedMemory = {
                  memoryAllocators = 8
                  memoryType = "FILE"
                  memorySizeBytes = 2g
                  memoryThrottling = {
                      thresholdPercentage = 50
                      checksPerSecond = 2
                  }
                  cacheFlush = {
                      flushIntervalSeconds = 5
                      maximumObjectsPerType = 0
                  }
                  ipc = {
                      noDestinationTimeoutSeconds = 10
                  }
              }
          }
      }
  }
        
  dataDistributionPolicies = {
      my-dynamic-policy = {
          type = DYNAMIC
          dataMappers = [ ]
          customQuorumNotifiers = [ ]          
          forceReplication = false
          objectsLockedPerTransaction = 1000
          enableAction = JOIN_CLUSTER
          remoteEnableAction = ENABLE_PARTITION
          disableAction = LEAVE_CLUSTER_FORCE
          numberOfThreads = 1
          dynamicDataDistributionPolicy = {
              numberOfPartitions = 64
              numberOfClones = 1000
              primaryDataRedundancy = {
                  numberOfReplicas = 2
                  replicationType = SYNCHRONOUS
              }
              backupDataRedundancy = {
                  numberOfReplicas = 1
                  replicationType = ASYNCHRONOUS
              }
          }
      }
      my-static-policy = {
          type = STATIC
          dataMappers = [ "fragmentMapperA" ]
          customQuorumNotifiers = [ "myNotifier" ]
          forceReplication = false
          objectsLockedPerTransaction = 1000
          enableAction = JOIN_CLUSTER
          remoteEnableAction = ENABLE_PARTITION
          disableAction = LEAVE_CLUSTER_FORCE
          numberOfThreads = 1
          staticDataDistributionPolicy = {
              sparseAudit = IGNORE_NODE_LIST
              replicaAudit = WAIT_ACTIVE
          }
      }
   }
        
 }
}

FragmentDefinition HOCON File Sample

Long lines wrap to the next line in this example for clarity.

name = "FullFragmentConfig"
version = "1.0"
type = "com.tibco.ep.dtm.configuration.application"

configuration = {
  FragmentDefinition = {
      description = "my application"
        
      dataMappers = {
          fragmentMapperA = {
              deferMappings = false
              roundRobinDataMappers = [
                  {
                      className = "com.tibco.ep.dtm.configuration.node.policy11.ClassA"
                  }
              ]
              distributedHashDataMappers = [
                  {
                      className = "com.tibco.ep.dtm.configuration.node.policy21.ClassA"
                      field = "fieldA"
                  }
              ]
              customDataMappers = [
                  {
                      className = "com.tibco.ep.dtm.configuration.
                       node.policy41.ClassA"
                      factoryClassName = "com.tibco.ep.dtm.configuration.
                       node.policy41.ClassAMapperFactory"
                      value = "10"
                  }
              ]
          }
      }

      routers = {
          myHashRouter = {
              className = "com.tibco.haservice.sample.HashRouter"
          }
          myRoundRobinRouter = {
              className = "com.tibco.haservice.sample.RoundRobinRouter"
          }
          myBroadcastRouter = {
              className = "com.tibco.haservice.sample.BroadcastRouter"
          }
          myPartitionedRouter = {
              className = "com.tibco.haservice.sample.PartitionedRouter"
          }
          myNodeRouter = {
              className = "com.tibco.haservice.sample.NodeRouter"
          }
      }

      customQuorumNotifiers = {
           myNotifier = {
                  notifierClassName = "com.tibco.ep.dtm.configuration.node.
                    QuorumNotifier"
           }
      }
        
  }
}