The TIBCO StreamBase® Log File Output Adapter is an embedded adapter that sends its input tuples to both its output port (if configured to do so) and to the currently configured StreamBase Java logging system, which is described in Using StreamBase Logging. Each input tuple that enters the Log adapter is sent to the logging system, and then passes unchanged to the adapter's output port.
The Log adapter participates automatically in the logging system currently in use by its containing StreamBase Server. This means that by default, the Log adapter uses:
-
Logback logging when run in Studio or when run from the command line with sbd and no
-bswitch. -
Log4j logging when run as a Windows service or when run from the command line with sbd -b.
If you have configured StreamBase Server to use an alternative logging system (such as using Log4j under all circumstances), the Log adapter still participates in the configured logging system. You can configure logging for the Log adapter as appropriate for the logging system in use. See Using StreamBase Logging for more on the StreamBase logging system.
The Log Output adapter itself adds negligible overhead to each tuple when the configured log level is such that no logging actually occurs. When the log level is such that tuples are actually sent into the logging system, the overhead can become significant, depending on the type of appenders configured. Even logging appenders that are said to be asynchronous can introduce surprising delays.
To avoid all overhead delays introduced by logging system appenders, be sure that the Log Asynchronously check box is set (which is its default state). With this option enabled, each incoming tuple is sent immediately to the adapter's output port (if the adapter is configured to emit tuples), and a copy of the tuple is sent to a queue. The queue is managed by a separate thread that immediately retrieves the tuple from the queue and sends it to the currently configured logging system. Because each tuple's retrieval from the queue is immediate, this option's actions do not constitute buffering, just separately-threaded logging. When the Log Asynchronously option is enabled, the profiling Size attribute for the Log adapter reports the size of this queue.
There can be unintended consequences of enabling the Log Asynchronously option:
-
In a large application with multiple instances of the Log Output adapter, if all Log adapter instances are configured to be synchronous, or all Log adapter instances are configured to be asynchronous (the default state), then logged events are written to log appender locations in the downstream order of the various Log adapter instances, as expected.
-
However, if you mix synchronous and asynchronous Log adapters in the same execution region, you have no guarantee that all Log adapters write their log entries in sequence.
To set the synchronous state of all Log adapters in an application at the same time, you can use the Java system property,
streambase.log-adapter-async. Setting this property avoids having to edit the Properties view for all Log adapters in the application. Set streambase.log-adapter-async=true to force all Log adapters to enable the Log Asynchronously option. Set streambase.log-adapter-async=false to restore the default asynchronous state of all Log adapters.
The Log adapter provides two format types for the message portion of the log emitted from the adapter, CSV and JSON.
Note
The overall format of emitted log messages is still controlled by the logging system in use and by its configuration file.
Logback logging is configured with a file usually named logback.xml, while Log4j logging is configured with log4j.configuration or log4j.xml. In both cases, a PatternLayout string defines the format and structure of all log messages emitted from StreamBase.
The Log adapter provides ways to format only the %m portion of the log message.
The Log Format property determines which whether the Log adapter should use CSV or JSON formats for logging the messages. Each format option has a corresponding tab that allows for further message customization.
The Logger Name or Logger Name Expression property must be specified, and corresponds to the %c portion of the log message pattern string. Use different Logger names for Log adapters in different portions of your application.
By default, the Log adapter emits the fields of its input tuple preceded by the name of the Logger that you assign. For example,
a Log adapter instance might be configured with the Logger1 Logger Name but no other formatting options. With the input tuple {IBM, 164.35, 1000}, the default-configured adapter emits the following message:
Logger1 - IBM,164.35,1000.0
If using the CSV format, you can optionally precede the emitted tuple with a string prefix that helps further identify log messages emitted from this instance of the adapter. For example:
Logger1 - Incoming tuple: IBM,164.35,1000.0
The CSV format also allows you to optionally precede each field value with its corresponding field name (with or without the
message prefix). In this case, the listing of input fields is preceded by an internal counting field, tupleid. For example:
Logger1 - Incoming tuple: (tupleid=0,symbol="IBM",price=164.35,numshares=1000.0)
With the CSV format, instead of emitting the input tuple's fields in order, you can provide a string pattern or an expression.
-
The string pattern in the format defined by java.text.MessageFormat, with zero-based positional parameters in the form
{0},{1}, and so on, filled in with input field values in input schema order. The message pattern option allows you to construct log message text in phrase or sentence form, with some or all of the log message text provided by the value of fields in the input tuple. For the same input tuple as above, with the message format string{2, number, integer} shares of {0} traded at {1, number, currency}, the adapter emits the following:Logger1 - 1,000 shares of IBM traded at $164.35.
Note
Do not confuse the
java.text.MessageFormatstring you provide to the Log adapter with thePatternLayoutstring you provide in Logback and Log4J configuration files. The two message pattern standards are not the same and are not interchangeable. -
The expression can be any arbitrary StreamBase expression using any of the incoming fields and will be evaluated for each tuple sent to the adapter.
If using the JSON format, you can further specify if the logged message should be displayed in a Map or List format.
An instance of the Log adapter has a single input port and a single output port. This is a pass-through adapter, so the schema of the output port is always the same as the schema of the input port.
Name: Use this required field to specify or change the name of this instance of this component, which must be unique in the current EventFlow module. The name must contain only alphabetic characters, numbers, and underscores, and no hyphens or other special characters. The first character must be alphabetic or an underscore.
Adapter: A read-only field that shows the formal name of the adapter.
Class name: Shows the fully qualified class name that implements the functionality of this adapter. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.
Start with application: If this field is set to Yes (default) or to a module parameter that evaluates to true, this instance of this adapter starts as part of the JVM engine that runs this EventFlow module. If this field is set to
No or to a module parameter that evaluates to false, the adapter instance is loaded with the engine, but does not start until you send an sbadmin resume command, or until you start the component with StreamBase Manager.
Enable Error Output Port: Select this check box to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.
Description: Optionally enter text to briefly describe the component's purpose and function. In the EventFlow canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.
The Log adapter has the following properties on the Adapter Settings tab in its Properties View in StreamBase Studio:
| Property | Data Type | Description |
|---|---|---|
| Log Level | drop-down list | Specify the logging message type you want to send to the logging system from this adapter. The choices are to send Error, Warn, Info, Debug, or Trace log messages.
|
| Log Level Override Field | string | Specify the name of an input tuple field of type string used to override the adapter's configured log level. If unspecified,
the adapter logs at the level configured in its Log Level property. If a log level override field is specified, its value overrides the adapter's configured log level on a per-tuple
basis. The field can contain any of the following values: Error, Warn, Info, Debug, or Trace. Any other value, including null, causes the adapter use the configured log level for that tuple.
|
| Use Logger Name Format | radio button | Determine which property to use when setting the loggers name. |
| Logger Name | string | Specify the name to use in creating the logger object for this adapter instance. The logger name appears in each logged message
if the %c pattern is specified in the message pattern string in the <layout> element of the Logback or Log4j configuration file. (StreamBase's default Logback message configuration includes %c.)
|
| Logger Name Expression | string | Specify the expression for the name to use in creating the logger object for this adapter instance. If the expression evaluates
to null the default of 'Logger' will be used. The logger name appears in each logged message if the %c pattern is specified in the message pattern string in the <layout> element of the Logback or Log4j configuration file. (StreamBase's default Logback message configuration includes %c.)
|
| Log Asynchronously | check box | Selected be default. Each incoming tuple is sent immediately to the output port, if the Emit Logged Tuples property is enabled, and a copy of the tuple is sent to a queue managed by a separate thread that immediately retrieves the
tuple from the queue and sends it to the currently configured logging system. See The Log Asynchronously Option above for cautions and considerations when using this option.
When cleared, each incoming tuple passes through each configured appender for the current logging system, and then passes to the adapter's output port. |
| Emit Logged Tuples | check box | Selected be default, causing logged tuples to be emitted on the log adapter's output port. If this property is disabled, no output port is present, and logged tuples are not emitted. |
| Capture Transform Strategy | radio button | The strategy to use when transforming capture fields for this operator: FLATTEN or NEST. |
| Compress Frequent Messages | check box |
Compress similar frequent messages into a single message with a count of repetition over time. Note that this may cause data loss on crash. When enabled the system will track each tuple and only output a log messages as follows: The first tuple is received and stored in memory and is not logged yet. The second tuple is received and compared to the first, if different the first tuple is logged and the second is now in memory for comparing. If the second tuple matches the previous and the time difference between the first and second tuple is within the max window size the tick count is incremented and nothing is logged. When either the a tuple is received and the max window is reached or the tuples are different the first tuple stored in memory will be emitted to the log with a suffix of 'Occured x times in y seconds'.Example: 'Symbol IBM Price 165.34 Occured 50 times in 1 seconds' Note: When the log entry is created for multiple tuples that are matched, the first tuple is used to create the log entry. This means if there is a Timestamp field output it will be from the time the first tuple was received for this window of tuples. |
| Max Window Size | double | The maximum number of seconds to allow the logger to wait for similar messages when compress few messages is enabled. Note that larger window sizes means more data can be lost if there is a crash. Decimal values are allowed for sub-second logging. |
The Log adapter has the following properties on the CSV tab in its Properties View in StreamBase Studio:
| Property | Data Type | Description |
|---|---|---|
| Message Prefix | string | The specified prefix string is prepended to the log message text for each tuple emitted from this Log adapter instance. You can use this feature with several instances of the Log adapter to prefix different strings to the log messages emitted from different portions of your application to help identify the source of messages. |
| Display Tuple Field Names | check box | Enable to display tuple field names in logged messages. |
| Quote Null Strings | check box | When the Display Tuple Field Names property is set, this option is enabled. Select it to place quotes around string fields containing null (the default); clear it to not display quotes.
|
| Hex Encode Blobs | check box | Hex encode blob contents, otherwise display as String. |
| Use Custom Format | radio button |
Determines which custom message format should be used. Custom MessageFormat (Basic), or Custom Expression MessageFormat (Expression) |
| Custom MessageFormat | string |
Use Custom MessageFormat to specify a string that defines a pattern in the format defined in java.text.MessageFormat, where the For example, if the adapter's incoming tuple consists of fields {2, number, integer} shares of {0} traded at {1, number, currency}.You can specify field names directly instead of by their indexes by providing each field name as an exact literal. Leading
and trailing whitespace is ignored. Normal format type and format style arguments may follow as normal. Field names with escaped
identifiers (e.g.,
{numshares, number, integer} shares of {symbol} traded at {price, number, currency}. Note: This field cannot be used if you enabled the Display Tuple Field Names option |
| Custom Expression MessageFormat | string |
Use Custom Expression MessageFormat to specify an arbitrary StreamBase expression that will evaluate to your log message. This option is only available when the Use Custom Format is set to Expression Note: This field cannot be used if you enabled the Display Tuple Field Names option |
The Log adapter has the following properties on the JSON tab in its Properties View in StreamBase Studio:
| Property | Data Type | Description |
|---|---|---|
| Include Null Fields | check box | Enable to include null fields in your logger message. |
| Timestamps as long | check box | If enabled, this will convert timestamp fields to a long value. |
| Timestamp Format | string | The format to use when converting timestamp values to a string. |
| Encode Sub Type | radio button |
Determines which transformation should be used when the tuple contains subtuples, Map or List. |
Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.
Caution
Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.
Typechecking detects a value for the output that is null or unspecified.