Nodes

As part of node installation, default application and node configuration information is installed. Both the application and node configuration information can be overridden by specifying a node deploy configuration file when installing a node. The default configuration can also be augmented using the node deploy configuration file. Node deployment configuration files are described in detail in Node Deploy Configuration.

When an application is initialized there is a well defined order in which configuration is processed:

Configuration specified in the node deploy global or per-node configuration sections is processed in the order in which it is specified in the file — the first configuration is processed first, the second next, and so on.

This implies that configuration loaded after other configuration can override previously loaded and activated configuration (see the Configuration chapter in the Architects Guide for details on configuration replacement). A failure to load or activate a configuration file will cause the node startup to fail. See Starting for more details.

Fragment Configuration Load Order

When an application is installed, fragment configuration is loaded using the classpath of the top-level fragment (Fragment-A and Fragment-B in Figure 2, “Application dependencies”) engines. For each engine, fragment configuration is loaded in reverse classpath order. If there are duplicate dependencies in the classpath, the configuration for the duplicate dependency is loaded only once when the dependency is first processed during start-up. If an application archive itself has multiple classpath elements containing configuration, this configuration is also loaded in reverse classpath order. Finally, if there are multiple configuration files in a single classpath element, they are loaded in ASCII sort order.

Configuration load ordering between engines is undefined.

Application dependencies

Using the application dependencies in Figure 2, “Application dependencies”, configuration would be loaded in this order (assuming Fragment-A engine is loaded before Fragment-B engine — this is undefined behavior).

1. Fragment-A engine

   1. c.conf

   2. a.conf

2. Fragment-B engine

   1. d.conf

   2. e.conf

   3. (c.conf skipped, already loaded as part of Fragment-A engine processing)

   4. b.conf

3. Application

   1. x.conf

When all dependencies are not packaged into an application archive, they must be made available to each node running the application using either engine configuration or deploy directories.

The JavaEngine.externalClassPath and JavaEngine.externalNativeLibraryPath configuration values can be used to define locations on the local machine running a node where Java and native library dependencies can be located. See JavaEngine for more details on these configuration values.

Engine Deploy Directories

Engine deploy directories apply to the engine whose name matches a directory name. See Node Deploy Configuration for details on how to set an engine name in the node deploy configuration.

Engine deploy directories are optional and they are not automatically created; they must be manually created.

Engine deploy directories are located within existing node deploy directories.

Figure 3, “Deploy directories” shows an example where the node deploy directories were set to /a/jars and /b/wars at node installation time, and two engine deploy directories named EngineOne and EngineTwo were created under them with the JAR and WAR files installed as shown in Figure 3, “Deploy directories”.

Deploy directories

When EngineOne and EngineTwo are started they will be have these JAR and WAR files and directories added to their class paths:

• EngineOne — /a/jars/EngineOne/configuration.jar:/a/jars/EngineOne/process_v2.jar:/a/jars/EngineOne:/a/jars:/b/wars/EngineOne/admin-rest.war:/b/wars/EngineOne:/b/wars

• EngineTwo — /a/jars/EngineTwo/administration.jar:/a/jars/EngineTwo/process_v1.jar:/a/jars/EngineTwo:/a/jars:/b/wars/EngineTwo/healthcheck.war:/b/wars/EngineTwo:/b/wars

A node inherits all environment variables set in the environment in which the node was installed. This provides a mechanism to make environment variables available to deployed applications. The inherited environment is printed in the node coordinator's log file. Any changes to environmental variables after a node is installed do not affect the node's inherited environment.

There is special handling of the executable search path, such as PATH, and the shared library search path, such as DYLD_LIBRARY_PATH. Both the executable and library search paths inherited from the environment are appended to the paths defined by the node. This ensures that a setting in the inherited environment cannot cause incompatibilities with a node's required environment.

When a node is installed, it has the default security policies described on Default Security Configuration

A user is defined in the default local authentication realm when a node is installed using the user name determined from the operating system context in which the node installation was done. This is the name of the user logged into the operating system when performing a node installation. By default this user has a random password assigned and they are granted administrator role privileges, which give that user full administrative access to the node. This user can only access the node from a trusted host since the password is unknown. The password should be changed to a known value (see ????) to allow this user to access the node from a non-trusted host.

A known password can also be set for the operating system user on the command line using the password parameter to install node. For example,

//
//    The operating system user will have a password of admin 
//    instead of a random password
//
epadmin --password=admin install node --nodename=A.X

A user definition is also automatically defined in the local authentication realm for the optional username and password parameters specified when installing a node. This user definition uses the value of the username parameter as the principal name and the value of the password parameter as the text credential. This principal definition is granted administrator role privileges. For example, this command will define a user named guest with a password of guest for the node being installed.

epadmin --username=guest --password=guest install node --nodename=A.X

The operating system user name, and the user name specified when installing a node, may be the same or different. Both users are granted the same privileges and access to the node. The only difference is that the operating system user can perform administrative commands on the local node without specifying a user name or password because of the default trusted host configuration described next.

Additional local authentication realm users can be defined during node installation by including a LocalAuthenticationRealm (see ????) configuration as described in Overriding Default Configuration.

The default trusted host configuration defines localhost, and the actual host name where the node is being installed, as trusted hosts. This allows the operating system user to administer the node without specifying a user name or password when running administrative commands from the same machine on which the node was installed. This is accomplished by epadmin using the operating system identity when running administrative commands if no user name and password is specified. Since the operating system user was defined as a principal with administrative privileges on the node during installation, they are allowed to run any administrative command.

See ???? for additional details on the security architecture and configuration.

The current status of a node can be displayed using this command:

epadmin --servicename=A.X display node

This information is displayed for the target node:

Example output from the display node command:

epadmin --serviceprefix=false --servicename=A.sbuser display node
Node Name = A.sbuser
Node Description = No description
Node State = Started
Host Name = bwrought.local
Administration Port = 54220
Discovery Service Implementation = UDP
Discovery Service Port = 54321
Discovery Service State = Running
Node Directory = /Users/bwright/Documents/ws/Studio_1060G/.nodes/A.sbuser
Deployment Directories = /Users/bwright/Documents/ws/Studio_1060G/.nodes/A.sbuser/deploy
Install Time = 2020-05-01 13:21:41 -0400 UTC
Last Start Time = 2020-05-01 13:21:48 -0400 UTC
Build Type = PRODUCTION
Product Version = TIBCO StreamBase Runtime 10.6.0-SNAPSHOT (build 2004291638)
Product Installation Directory = /Users/bwright/Applications/TIBCO Streaming 10.6.0
Sensitive Configuration Data Encryption = Disabled
Secure Communication Profile Name = None

Once a node has been installed, it must be started.

When a node is started the following occurs:

Any failures detected while loading and activating configuration causes any previously processed configuration files to be deactivated and removed, and node start-up fails.

This command can be used to start a node:

epadmin --servicename=A.X start node

Optionally, this command can be used to start all nodes in cluster X:

epadmin --servicename=X start node

When a node is stopped the following occurs:

Any configuration that was loaded and activated after a node was started are not deactivated and removed when the node is stopped.

Any failures detected while deactivating and removing configuration are logged, but stopping the node continues.

This command can be used to stop a node:

epadmin --servicename=A.X stop node

Optionally, this command can be used to stop all nodes in cluster X:

epadmin --servicename=X stop node

A stopped node can be restarted to resume application execution.

A node must be stopped before it can be removed. When a node is removed the following occurs:

This command can be used to remove a node:

epadmin --servicename=A.X remove node

Optionally, this command can be used to remove all nodes in cluster X:

epadmin --servicename=X remove node

A node must be reinstalled after it has been removed.

When a machine on which a node is running is shutdown, the node must be quiesced before shutting down the machine.

When a node is quiesced the following occurs:

These commands are used to quiesce a node:

epadmin --servicename=A.X stop node
epadmin --servicename=A.X terminate node

When the machine is rebooted this command must be used to restart the node:

epadmin start node --installpath=<node-directory>/<node-name>

Please see Operating System Service Management for details on integration with operating system specific service management subsystems to perform these actions automatically.

A complete discussion of configuration life cycle can be found in the Spotfire Streaming Architecture Guide. In summary configuration follows this life-cycle:

The target node must be in the Started state to perform these configuration actions.

Configuration is loaded into a node using this command:

epadmin --servicename=A.X load configuration --source=myConfiguration.conf

The Ignore memory threshold checkbox is used to allow configuration data to be loaded in shared memory even if the configured shared memory throttle utilization is exceeded. Without this override, attempting to load configuration data into congested shared memory will fail.

All loaded configuration data can be displayed from the Configuration tab for a node, using this command:

epadmin --servicename=A.X display configuration

This information is displayed for each loaded configuration file:

Configuration activation, deactivation, and removal is performed using these commands:

//
//  Activate a deactive configuration
//
epadmin --servicename=A.X activate configuration --type=com.tibco.ep.dtm.configuration.node --name=payments --version=1.0

//
//  Deactivate an active configuration
//
epadmin --servicename=A.X deactivate configuration --type=com.tibco.ep.dtm.configuration.node --name=payments --version=1.0

//
//  Remove configuration from shared memory
//
epadmin --servicename=A.X remove configuration --type=com.tibco.ep.dtm.configuration.node --name=payments --version=1.0

Configuration can be exported using this command:

epadmin --servicename=A.X export configuration --type=com.tibco.ep.dtm.configuration.node --name=payments --version=1.0

Exported configuration can be used to determine exactly how a node is configured and also to easily make modifications to a node's configuration.

To make changes to an existing configuration, export the configuration, modify the configuration values as required, update the version number, and reload and activate the configuration on the node.

When a configuration is exported it contains any values initially set in the configuration along with all default values currently being used because the value was not explicitly set in the configuration when it was loaded.

Metrics are managed using the metric administration target (see epadmin-metric(1) for command details). Support is provided for these commands:

For example, the display command displays all metrics supported by a node.

epadmin --serviceprefix false --servicename A.X display metric
Name = builtin.cpu.idle.utilization.percentage
Type = HISTOGRAM
Description = Percent idle CPU utilization for machine hosting node A.X

Name = builtin.cpu.system.utilization.percentage
Type = HISTOGRAM
Description = Percent system CPU utilization for machine hosting node A.X

Name = builtin.cpu.user.utilization.percentage
Type = HISTOGRAM
Description = Percent user CPU utilization for machine hosting node A.X

Name = builtin.engine.my-engine.heap.memory.utilization.bytes
Type = HISTOGRAM
Description = Heap memory used (bytes) for engine my-engine

Name = builtin.engine.my-engine.heap.memory.utilization.percentage
Type = HISTOGRAM
Description = Percent heap memory used for engine my-engine

Name = builtin.engine.my-engine.queue.default:ParallelSequence.depth.second
Type = METER
Description = Queue default:ParallelSequence depth per second for engine my-engine

Name = builtin.engine.my-engine.tuples.rate
Type = METER
Description = Tuple rate for engine my-engine

Name = builtin.node.shared.memory.kilobytes
Type = HISTOGRAM
Description = Shared memory used (kilobytes) for node A.X

Name = builtin.node.shared.memory.percentage
Type = HISTOGRAM
Description = Percent shared memory used for node A.X

Name = builtin.node.transactions.deadlocks.rate
Type = METER
Description = Transaction deadlock rate for node A.X

Name = builtin.node.transactions.latency.average.microseconds
Type = METER
Description = Average transaction latency (microseconds) for node A.X

Name = builtin.node.transactions.total.rate
Type = METER
Description = Transaction rate for node A.X

The current value of a specific metric can be read using the read command.

epadmin --serviceprefix false --servicename A.X read metric \
    --name builtin.node.transactions.latency.average.microseconds
Last Reset = 2020-04-15 07:41:35.317-0700
Last Updated = 2020-04-16 15:32:51.090-0700
Count = 5972713
Mean One Minute Rate = 99.39
One Minute Rate = 85.98
Five Minute Rate = 93.78
Fifteen Minute Rate = 98.22

A metric can be reset using the reset command.

epadmin --serviceprefix false --servicename A.X reset metric \
    --name builtin.node.transactions.latency.average.microseconds
epadmin --serviceprefix false --servicename A.X read metric \
    --name builtin.node.transactions.latency.average.microseconds
Last Reset = 2020-04-16 15:33:19.700-0700
Last Updated = 2020-04-16 15:33:27.204-0700
Count = 757
Mean One Minute Rate = 100.76
One Minute Rate = 90.0
Five Minute Rate = 90.0
Fifteen Minute Rate = 90.0

Metrics are also available via a web service. See Metrics.

Docker container logging is enabled by configuring a console appender in an application. For example:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} %10.10thread %-5level %20.20logger{5} : %msg%n</pattern>
        </encoder>
    </appender>
    <root level="INFO">
        <appender-ref ref="Console" />
    </root>
</configuration>

When the container containing the application is started it must have a pseudo-tty allocated for the console appender to work correctly. This is done using the tty parameter to the run command:

docker run --tty --detach --name A.sb10 --network demo-network sb10-demo-application A.sb10

When the container is started all engine logging will be sent to the container logs as shown in Figure 4, “Docker container logging”.

Docker container logging

Container logging is also available from the docker command line using the logs command:

docker logs --follow A.sb10

Log configuration uses standard Logback XML or Groovy configuration files with a few extensions. See the Logback configuration documentation for details on the standard configuration values.

<dependency>
    <groupId>org.codehaus.groovy</groupId>
    <artifactId>groovy</artifactId>
    <version>3.0.1</version>
</dependency>

The supported extensions are:

While it is possible to move the location of the engine log files outside of the <node-directory>/logs directory using configuration, if this is done, engine log messages are sent to both the configured location of the engine log file and also the default engine log in the logs directory. This is done to ensure that snapshot files include all logging from the node for diagnostic purposes. If an engine log file is configured to not use the default engine log file name, but the log file is left in the logs directory, only the configured log file name is used.

This example shows the use of these extensions in a Logback XML configuration file.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appender name="FILE" class="ch.qos.logback.core.FileAppender">
    <file>${com.tibco.ep.dtm.logging.logDirectory}/${com.tibco.ep.dtm.logging.logFileNamePrefix}.log</file>
    <encoder>
      <pattern>d{HH:mm:ss.SSS} %usec %process %thread %transaction %-5level %logger: %msg%n</pattern>
    </encoder>
  </appender>
  <root level="INFO">
    <appender-ref ref="FILE"/>
  </root>
</configuration>  

And here is the same configuration file in Groovy.

import ch.qos.logback.classic.encoder.PatternLayoutEncoder
import ch.qos.logback.core.FileAppender

import static ch.qos.logback.classic.Level.INFO

appender("FILE", FileAppender) {
  file = "${com.tibco.ep.dtm.logging.logDirectory}/${com.tibco.ep.dtm.logging.logFileNamePrefix}.log"
  encoder(PatternLayoutEncoder) {
    pattern = "d{HH:mm:ss.SSS} %usec %process %thread %transaction %-5level %logger: %msg%n"
  }
}
root(INFO, ["FILE"])

Logging configuration is processed in this order; last configuration found is used:

The Logback configuration file resolution rules use the standard LogBack conventions described here. The logging configuration can also be changed at runtime using the logging administration target. See Administration.

Details on packaging logging configuration into fragment and application archives is in the Transactional Memory Developers Guide.

The runtime logging level can be changed using the logging administration target. There is support for both getting and setting the current log level for all loggers running in each engine in a node.

To set the log level for a logger, the logger name must be known. The loggers that are displayed using the display logging command were:

For example:

//
//    Get loggers and logging level for the Pattern_sbapp0 engine
//
epadmin --servicename=A.X display logging --enginename=Pattern_sbapp0
Engine Name = Pattern_sbapp0
Logger Name = ROOT
Log Level = INFO

Engine Name = Pattern_sbapp0
Logger Name = com.sun.jersey
Log Level = WARN

Engine Name = Pattern_sbapp0
Logger Name = com.tibco.ep.dtm.audit
Log Level = INFO

Engine Name = Pattern_sbapp0
Logger Name = org.apache.shiro
Log Level = DEBUG

//
//    Set the ROOT logging level to DEBUG for all engines
//
epadmin --servicename=A.X set logging --loggername=ROOT --level=DEBUG
Engine Name = Pattern_sbapp0
Logger Name = ROOT
Previous Log Level = INFO
New Log Level = DEBUG

Engine Name = System::administration
Logger Name = ROOT
Previous Log Level = INFO
New Log Level = DEBUG

//
//    Load a new log configuration file on all engines
//
epadmin --servicename=A.X --set logging --configuration=new-logging-config.xml

Complete details on the logging target can be found in epadmin-logging(1).