Using StreamBase Logging

For all supported operating systems, StreamBase uses the industry-standard SLF4J logging system with the Logback framework. See Windows Event Logging for additional considerations for Windows.

All subsystems in EventFlow and LiveView fragments use Logback logging. You can configure the level of logging information emitted, and can direct logs to different appenders using standard Logback configuration files in XML format.

To configure a custom Logback configuration files see Custom Configuration and StreamBase Logback Extensions below. The default logs subdirectory location can be overridden as described in StreamBase Logback Extensions.

The StreamBase Runtime processes log configuration in the following order:

Logging configurations do not merge. That is, a later configuration in the processing order sequence overrides an earlier one. For this reason, any custom logging configuration file should specify settings for both RootFileAppender and RootConsoleAppender to avoid error messages.

Logging settings are ultimately specified per engine. If your fragment has more than one engine, each engine can inherit the settings made for the fragment and application as a whole.

The logging level and logging configuration can be changed at runtime per engine using the logging target of the epadmin command, as described in Managing Logging at Runtime.

In addition to the logging messages emitted by EventFlow and LiveView as a whole, you can add logging instructions to any EventFlow module with the Log Output adapter. The adapter has a sample described in Log Output Adapter Sample that illustrates formatting options you can perform with the Log Output adapter.

Do not confuse the Java logging facility discussed here with the StreamBase error stream feature. Logging is for writing your own logging messages and exception stack traces to one or more log files. By contrast, the error stream feature is like a module-level throw that ascends up the call stack to the containing module, and from there to the top-level EventFlow module instance running the fragment. StreamBase sends an error tuple to the error stream when it encounters an exception during a tuple processing method. See Using Error Ports and Error Streams for further information.

If you write custom operators or adapters, you can use an import org.slf4j.Logger statement in your code to define a logger for your operator or adapter. The base class, Operator, provides the convenience method getLogger(), which returns an org.slf4j.Logger object that should thereafter be used for logging messages and exceptions in your code. For the full list of methods available on Logger objects, refer to the Logger class in the SLF4J Javadoc.

The StreamBase installation includes a logging sample that illustrates the difference between using a Log Output adapter and using getLogger() methods in your custom code. This sample compares both methods with a third module that does not log.

StreamBase uses the standard log level meanings provided by the Logback framework, as defined in the Logger API.

The following table maps the log level integer settings used in StreamBase releases before 10.2.0 to Logback message levels.

Most StreamBase adapters allow you to set a log level independently for the adapter than for the hosting EventFlow module:

For example, an adapter might set the log level to WARN or ERROR for messages emitted from the adapter while the hosting fragment is still set to its default INFO level. This allows you to troubleshoot adapter connectivity issues without having to wade through hundreds of verbose node messages from the fragment.

Log files are generated at the node and engine levels. Log files are written by default into a subdirectory named logs in a fragment's node directory:

You can change the location and name of the node-directory/logs directory using StreamBase Logback extensions.

The log files generated in the node-directory/logs directory are:

Another set of log files is generated in the node-directory/logs/bootstrap directory. These files are populated during the early node startup timeframe before the Logback system has initialized. All files in logs/bootstrap are of interest only to StreamBase® engineers as part of deep-level debugging of a snapshot file.

StreamBase's default settings for Logback logging are as follows:

The components of the default log message pattern have the meanings shown in the following table.

Consult the Logback documentation for PatternLayout for the available log message pattern components.

You can override any of the default settings with a custom Logback configuration.

The following example shows Default Streaming Logback Configuration in XML format:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<turboFilter class="com.tibco.ep.dtm.logging.SBDuplicateMessageFilter">
<AllowedRepetitions>9</AllowedRepetitions>
</turboFilter>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${com.tibco.ep.dtm.logging.logDirectory}/${com.tibco.ep.dtm.logging.logFileNamePrefix}.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>
${com.tibco.ep.dtm.logging.logDirectory}/${com.tibco.ep.dtm.logging.logFileNamePrefix}-%i.log
</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>10</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>10485760</maxFileSize>
</triggeringPolicy>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger'{'35'}' - %msg%n</pattern>
</encoder>
</appender> 
  <root level="INFO">
    <appender-ref ref="FILE" />
  </root>
 <logger name="org.apache.shiro" level="WARN"/>
  <logger name="org.atmosphere" level="WARN"/>
  <logger name="org.eclipse.jetty" level="WARN"/>
  <logger name="org.hibernate" level="WARN"/>
  <logger name="org.springframework" level="WARN"/>
  <logger name="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer" level="ERROR"/>
</configuration>

To configure Logback logging, specify a Logback configuration file. StreamBase accepts files in either the traditional XML format, often named logback.xml. In either case, place your configuration files in the src/main/resources folder of your Studio project.

You can also use the src/test/resources folder for a logging configuration you only need when running Maven tests. To have one configuration used both for standard fragment launches and tests, use src/test/resources.

The StreamBase logging sample includes an example logback.xml configuration file that illustrates a setup for both a console appender and a rolling file appender.

In general, StreamBase Logback configuration files use standard Logback patterns as described in the Logback documentation for PatternLayout. StreamBase provides the following extensions to the standard Logback patterns:

The following example shows the use of these extensions in a Logback XML configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<variable name="com.tibco.ep.dtm.logging.logDirectory" value="/Users/sbuser/logs"/>
<variable name="com.tibco.ep.dtm.logging.logFileNamePrefix" value="testrun"/>
  <appender name="FILE" class="ch.qos.logback.core.FileAppender">
    <file>${com.tibco.ep.dtm.logging.logDirectory}/${com.tibco.ep.dtm.logging.logFileNamePrefix}.log</file>
    <encoder>
      <pattern>d{HH:mm:ss.SSS} %usec %process %thread %transaction %-5level %logger: %msg%n</pattern>
    </encoder>
  </appender>
  <root level="INFO">
    <appender-ref ref="FILE"/>
  </root>
</configuration>

Search for "Variable substitution" on this Logback documentation page to learn more about Logback variables.

StreamBase creates entries in an audit log for every node. The audit log contains audit messages for:

Example:

//
// Audit message for display web command
//
2019-08-12 07:16:46.000-0700: Audit: "epadmin --servicename A.X --username sbuser --password
*** display web" executed by Steve succeeded.
//
// Audit message for Health Check endpoint with trusted user
//
2019-08-12 07:17:18.303-0700: 10.0.0.164 "GET /healthcheck/v1/status HTTP/1.1" 200 by -

The audit log:

A deadlock log is configured for every node, and cannot be disabled.

The deadlock log:

See Analyzing Deadlocks in the Tuning Guide for details on the deadlock log message format.

You can enable Docker container logging by configuring a console appender in a Docker-hosted StreamBase application. For example:

<xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <pattern>%d{HH:mm:ss.SSS} %10.10thread %-5level %20.20logger{5} :
%msg%n</pattern>
    </encoder>
  </appender>
  <root level="INFO">
    <appender-ref ref="Console"/>
  </root>
</configuration>

When the container containing the application is started it must have a pseudo-tty allocated for the console appender to work correctly. To do so, use the tty parameter with the run command:

docker run --tty --detach --name A.sb10 --network example.com firstapp
A.sb10

When the container is started all engine logging is sent to the container logs.

docker logs --follow A.sb10