The Spotfire Streaming Log File Output Adapter is an embedded adapter that sends its input tuples to both its output port (if configured to do so) and to the currently configured StreamBase Logback logging system, which is described in Using StreamBase Logging in the Administration Guide. Each input tuple that enters the Log adapter is sent to the logging system, and then passes unchanged to the adapter's output port.
You can configure logging for the Log adapter as appropriate for the logging system in use, as described in Using StreamBase Logging.
The Log Output adapter itself adds negligible overhead to each tuple when the configured log level is such that no logging actually occurs. When the log level is such that tuples are actually sent into the logging system, the overhead can become significant, depending on the type of appenders configured. Even logging appenders that are said to be asynchronous can introduce surprising delays.
To avoid all overhead delays introduced by logging system appenders, be sure that the Log Asynchronously checkbox is selected (which is its default state). With this option enabled, each incoming tuple is sent immediately to the adapter's output port (if the adapter is configured to emit tuples), and a copy of the tuple is sent to a queue. The queue is managed by a separate thread that immediately retrieves the tuple from the queue and sends it to the currently configured logging system. Because each tuple's retrieval from the queue is immediate, this option's actions do not constitute buffering, just separately-threaded logging. When the Log Asynchronously option is enabled, the profiling Size attribute for the Log adapter reports the size of this queue.
There can be unintended consequences of enabling the Log Asynchronously option:
-
In a large application with multiple instances of the Log Output adapter, if all Log adapter instances are configured to be synchronous, or all Log adapter instances are configured to be asynchronous (the default state), then logged events are written to log appender locations in the downstream order of the various Log adapter instances, as expected.
-
However, if you mix synchronous and asynchronous Log adapters in the same execution region, you have no guarantee that all Log adapters write their log entries in sequence.
To set the synchronous state of all Log adapters in an application at the same time, you can use the Java system property,
streambase.log-adapter-async. Setting this property avoids having to edit the Properties view for all Log adapters in the application. Set streambase.log-adapter-async=true to force all Log adapters to enable the Log Asynchronously option. Set streambase.log-adapter-async=false to restore the default asynchronous state of all Log adapters.
The Log adapter provides two format types for the message portion of the log emitted from the adapter, CSV and JSON.
Note
The overall format of emitted log messages is still controlled by the logging system in use and by its configuration file.
Logback logging is configured with a file usually named logback.xml, where a PatternLayout string defines the format and structure of all log messages emitted from StreamBase. The default log message is described
in Logback Default Configuration.
The Log adapter provides ways to format only the %m portion of the log message.
The Log Format property determines whether the Log adapter is to use CSV or JSON formats for logging messages. Each format option has a corresponding tab that allows for further message customization.
The Logger Name or Logger Name Expression property must be specified, and corresponds to the %c portion of the log message pattern string. Use different Logger names for Log adapters in different portions of your application.
By default, the Log adapter emits the fields of its input tuple preceded by the name of the Logger that you assign. For example,
a Log adapter instance might be configured with the Logger1 Logger Name but no other formatting options. With the input tuple {IBM, 164.35, 1000}, the default-configured adapter emits the following message:
Logger1 - IBM,164.35,1000.0
If using the CSV format, you can optionally precede the emitted tuple with a string prefix that helps further identify log messages emitted from this instance of the adapter. For example:
Logger1 - Incoming tuple: IBM,164.35,1000.0
The CSV format also allows you to optionally precede each field value with its corresponding field name (with or without the
message prefix). In this case, the listing of input fields is preceded by an internal counting field, tupleid. For example:
Logger1 - Incoming tuple: (tupleid=0,symbol="IBM",price=164.35,numshares=1000.0)
With the CSV format, instead of emitting the input tuple's fields in order, you can provide a string pattern or an expression.
-
The string pattern in the format defined by java.text.MessageFormat, with zero-based positional parameters in the form
{0},{1}, and so on, filled in with input field values in input schema order. The message pattern option allows you to construct log message text in phrase or sentence form, with some or all of the log message text provided by the value of fields in the input tuple. For the same input tuple as above, with the message format string{2, number, integer} shares of {0} traded at {1, number, currency}, the adapter emits the following:Logger1 - 1,000 shares of IBM traded at $164.35.
Note
Do not confuse the
java.text.MessageFormatstring you provide to the Log adapter with thePatternLayoutstring you provide in Logback configuration files. The two message pattern standards are not the same and are not interchangeable. -
The expression can be any arbitrary StreamBase expression using any of the incoming fields and will be evaluated for each tuple sent to the adapter.
If using the JSON format, you can further specify whether the logged message should be displayed in a Map or List format.
An instance of the Log adapter has a single input port and a single output port. This is a pass-through adapter, so the schema of the output port is always the same as the schema of the input port.
Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.
Adapter: A read-only field that shows the formal name of the adapter.
Class name: Shows the fully qualified class name that implements the functionality of this adapter. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.
Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this adapter starts.
Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.
Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.
The Log adapter has the following properties on the Adapter Settings tab in its Properties View in StreamBase Studio:
| Property | Data Type | Description |
|---|---|---|
| Log Level | drop-down list | Specify the logging message type you want to send to the logging system from this adapter. The choices are to send Error, Warn, Info, Debug, or Trace log messages.
|
| Log Level Override Field | string | Specify the name of an input tuple field of type string used to override the adapter's configured log level. If unspecified,
the adapter logs at the level configured in its Log Level property. If a log level override field is specified, its value overrides the adapter's configured log level on a per-tuple
basis. The field can contain any of the following values: Error, Warn, Info, Debug, or Trace. Any other value, including null, causes the adapter use the configured log level for that tuple.
|
| Use Logger Name Format | radio button | Determine which property to use when setting the loggers name. |
| Logger Name | string | Specify the name to use in creating the logger object for this adapter instance. The logger name appears in each logged message
if the %c pattern is specified in the message pattern string in the <layout> element of the Logback or Log4j configuration file. (StreamBase's default Logback message configuration includes %c.)
|
| Logger Name Expression | string | Specify the expression for the name to use in creating the logger object for this adapter instance. If the expression evaluates
to null the default of 'Logger' will be used. The logger name appears in each logged message if the %c pattern is specified in the message pattern string in the <layout> element of the Logback or Log4j configuration file. (StreamBase's default Logback message configuration includes %c.)
|
| Log Asynchronously | check box | Selected be default. Each incoming tuple is sent immediately to the output port, if the Emit Logged Tuples property is enabled, and a copy of the tuple is sent to a queue managed by a separate thread that immediately retrieves the
tuple from the queue and sends it to the currently configured logging system. See The Log Asynchronously Option above for cautions and considerations when using this option.
When cleared, each incoming tuple passes through each configured appender for the current logging system, and then passes to the adapter's output port. |
| Emit Logged Tuples | check box | Selected be default, causing logged tuples to be emitted on the log adapter's output port. If this property is disabled, no output port is present, and logged tuples are not emitted. |
| Capture Transform Strategy | radio button | The strategy to use when transforming capture fields for this operator: FLATTEN or NEST. |
| Compress Frequent Messages | check box |
Compress similar frequent messages into a single message with a count of repetition over time. Note that this may cause data loss on crash. When enabled the system will track each tuple and only output a log messages as follows: The first tuple is received and stored in memory and is not logged yet. The second tuple is received and compared to the first, if different the first tuple is logged and the second is now in memory for comparing. If the second tuple matches the previous and the time difference between the first and second tuple is within the max window size the tick count is incremented and nothing is logged. When either the a tuple is received and the max window is reached or the tuples are different the first tuple stored in memory will be emitted to the log with a suffix of 'Occured x times in y seconds'.Example: 'Symbol IBM Price 165.34 Occured 50 times in 1 seconds' Note: When the log entry is created for multiple tuples that are matched, the first tuple is used to create the log entry. This means if there is a Timestamp field output it will be from the time the first tuple was received for this window of tuples. |
| Max Window Size | double | The maximum number of seconds to allow the logger to wait for similar messages when compress few messages is enabled. Note that larger window sizes means more data can be lost if there is a crash. Decimal values are allowed for sub-second logging. |
The Log adapter has the following properties on the CSV tab in its Properties View in StreamBase Studio:
| Property | Data Type | Description |
|---|---|---|
| Message Prefix | string | The specified prefix string is prepended to the log message text for each tuple emitted from this Log adapter instance. You can use this feature with several instances of the Log adapter to prefix different strings to the log messages emitted from different portions of your application to help identify the source of messages. |
| Display Tuple Field Names | check box | Enable to display tuple field names in logged messages. |
| Quote Null Strings | check box | When the Display Tuple Field Names property is set, this option is enabled. Select it to place quotes around string fields containing null (the default); clear it to not display quotes.
|
| Hex Encode Blobs | check box | Hex encode blob contents, otherwise display as String. |
| Use Custom Format | radio button |
Determines which custom message format should be used. Custom MessageFormat (Basic), or Custom Expression MessageFormat (Expression) |
| Custom MessageFormat | string |
Use Custom MessageFormat to specify a string that defines a pattern in the format defined in java.text.MessageFormat, where the For example, if the adapter's incoming tuple consists of fields {2, number, integer} shares of {0} traded at {1, number, currency}.You can specify field names directly instead of by their indexes by providing each field name as an exact literal. Leading
and trailing whitespace is ignored. Normal format type and format style arguments may follow as normal. Field names with escaped
identifiers (e.g.,
{numshares, number, integer} shares of {symbol} traded at {price, number, currency}. Note: This field cannot be used if you enabled the Display Tuple Field Names option |
| Custom Expression MessageFormat | string |
Use Custom Expression MessageFormat to specify an arbitrary StreamBase expression that will evaluate to your log message. This option is only available when the Use Custom Format is set to Expression Note: This field cannot be used if you enabled the Display Tuple Field Names option |
The Log adapter has the following properties on the JSON tab in its Properties View in StreamBase Studio:
| Property | Data Type | Description |
|---|---|---|
| Include Null Fields | check box | Enable to include null fields in your logger message. |
| Timestamps as long | check box | If enabled, this will convert timestamp fields to a long value. |
| Timestamp Format | string | The format to use when converting timestamp values to a string. |
| Encode Sub Type | radio button |
Determines which transformation should be used when the tuple contains subtuples, Map or List. |
Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.
Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.
Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.
Caution
Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.
Typechecking detects a value for the output that is null or unspecified.
-
Using Embedded AdaptersUsing Embedded Adapters