Contents
The TIBCO StreamBase® Adapter for Kinesis Publisher allows the system to publish data to Kinesis Data Streams.
We use the default AWSCredentialProvider chain that the KCL library provides out of the box. Before using the adapters, we need to set 2 environment variables: export AWS_ACCESS_KEY_ID="*******" export AWS_SECRET_ACCESS_KEY="******"
This section describes the properties you can set for this adapter, using the various tabs of the Properties view in StreamBase Studio.
Name: Use this required field to specify or change the name of this instance of this component, which must be unique in the current EventFlow module. The name must contain only alphabetic characters, numbers, and underscores, and no hyphens or other special characters. The first character must be alphabetic or an underscore.
Adapter: A read-only field that shows the formal name of the adapter.
Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this adapter starts.
Enable Error Output Port: Select this check box to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.
Description: Optionally enter text to briefly describe the component's purpose and function. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.
| Property | Type | Description |
|---|---|---|
| Kinesis Stream | string | The Kinesis Stream where the data is published. |
| Region | string | The AWS region where the stream is present. |
| Partition Key | string | A partition key is used to group data by shard within a stream. |
| Explict Hash Key | string | The hash value used to explicitly determine the shard the data record is assigned to by overriding the partition key hash. |
| Publisher Name | string | The name of the enhanced fan out Publisher to use when fetching records from Kinesis. |
| Enable Buffer Publish | check box | Enable buffered publish for the Kinesis data stream. |
| Producer Buffer Count | int | The number of tuples to hold before doing a network write. |
| Producer Flush Interval (ms) | long | The number of millisecnds to wait before doing a network write. |
| Threading Model | string | the way threads are used in the library, default to thread per request. |
| Thread Pool Size | int | sets the maximum number of threads in the thread pool. |
| Log Level | INFO | Controls the level of verbosity the adapter uses to issue informational traces to the console. This setting is independent of the containing application's overall log level. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE. |
Use the settings in this tab to allow this operator or adapter to start and stop based on conditions that occur at runtime in a cluster with more than one node. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with TIBCO Streaming releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.
Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.
Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.
Caution
Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.
Use the data port to publish data to the kinesis data stream.
The data port's schema is below:
| Field | Data Type | Description |
|---|---|---|
| data | string | The data content tp publish. |
| sequenceNumberForOrdering(optional) | string | If you do not provide 'sequenceNumberForOrdering', we do not send anything for sequencing to AWS Kinesis. |
| partitionKey(optional) | string | A partition key is used to group data by shard within a stream. Kinesis Data Streams segregates the data records belonging to a stream into multiple shards. It uses the partition key that is associated with each data record to determine which shard a given data record belongs to. Partition keys are Unicode strings, with a maximum length limit of 256 characters for each key. An MD5 hash function is used to map partition keys to 128-bit integer values and to map associated data records to shards using the hash key ranges of the shards. When an application puts data into a stream, it must specify a partition key. |
| explicitHashKey(optional) | string | The hash value used to explicitly determine the shard the data record is assigned to by overriding the partition key hash. |
You can optionally enable the status output port for this adapter instance by means of the Enable Status Port property in the Adapter Properties page of the Properties view. Use the status port to retrieve status output messages from the adapter.
-
status, string. The type of status information emitted. Status types are:
-
Error — This message relates to an error that occurred.
-
Warn — This message relates to a warning that the user should be aware of.
-
Info — This message relates to extra status information.
-
-
message, string. This is a formatted human readable message that explains the status message.
-
inputTuple, tuple. Pass through input tuple.