Contents
The Spotfire Streaming Adapter for Kinesis Consumer allows the system to consume data from connected kinesis data stream.
We use the default AWSCredentialProvider chain that the KCL library provides out of the box. Before using the adapters, we need to set 2 environment variables: export AWS_ACCESS_KEY_ID="*******" export AWS_SECRET_ACCESS_KEY="******"
This section describes the properties you can set for this adapter, using the various tabs of the Properties view in StreamBase Studio.
Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.
Adapter: A read-only field that shows the formal name of the adapter.
Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this adapter starts.
Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.
Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.
| Property | Type | Description |
|---|---|---|
| Kinesis Stream | string | The Kinesis Stream where the data is published. |
| Region | string | The AWS region where the stream is present. |
| Consumer Name | string | The name of the enhanced fan out consumer to use when fetching records from Kinesis. |
| Data Fetch Policy | drop list | The policy to use when fetching the records from AWS Kinesis. |
| Timestamp | string | Used only when the 'Data fetch policy' is 'AT_TIMESTAMP' in the above property. |
| Log Level | INFO | Controls the level of verbosity the adapter uses to issue informational traces to the console. This setting is independent of the containing application's overall log level. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE. |
Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.
Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.
Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.
Caution
Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.
The data port is the default output port for the Kinesis Consumer adapter, and is always enabled. Use the data port to receive message data from the connected Kinesis Data Stream.
The default schema for the data output port is:
-
sequenceNumber, string. The sequence number for this message.
-
subSequenceNumber, string. The subsequence number this message.
-
data, string. The content of this message.
-
partitionKey, string. A partition key is used to group data by shard within a stream.
-
explictHashKey, string. The hash value used to explicitly determine the shard the data record is assigned to by overriding the partition key hash.
You can optionally enable the status output port for this adapter instance by means of the Enable Status Port property in the Adapter Properties page of the Properties view. Use the status port to retrieve status output messages from the adapter.
-
status, string. The type of status information emitted. Status types are:
-
Error—This message relates to an error that occurred.
-
Warn—This message relates to a warning that the user should be aware of.
-
Info—This message relates to extra status information.
-
-
message, string. This is a formatted human readable message that explains the status message.