Kinesis Consumer Adapter

Introduction

The Spotfire Streaming Adapter for Kinesis Consumer allows the system to consume data from connected kinesis data stream.

Kinesis AWS Set Up

We use the default AWSCredentialProvider chain that the KCL library provides out of the box. Before using the adapters, we need to set 2 environment variables: export AWS_ACCESS_KEY_ID="*******" export AWS_SECRET_ACCESS_KEY="******"

Adapter Properties

This section describes the properties you can set for this adapter, using the various tabs of the Properties view in StreamBase Studio.

General Tab

Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.

Adapter: A read-only field that shows the formal name of the adapter.

Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this adapter starts.

Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.

Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.

Adapter Properties Tab

Property Type Description
Kinesis Stream string The Kinesis Stream where the data is published.
Region string The AWS region where the stream is present.
Consumer Name string The name of the enhanced fan out consumer to use when fetching records from Kinesis.
Data Fetch Policy drop list The policy to use when fetching the records from AWS Kinesis.
Timestamp string Used only when the 'Data fetch policy' is 'AT_TIMESTAMP' in the above property.
Log Level INFO Controls the level of verbosity the adapter uses to issue informational traces to the console. This setting is independent of the containing application's overall log level. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE.

Cluster Aware Tab

Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.

Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.

Concurrency Tab

Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.

Caution

Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.

Data Output Port

The data port is the default output port for the Kinesis Consumer adapter, and is always enabled. Use the data port to receive message data from the connected Kinesis Data Stream.

The default schema for the data output port is:

  • sequenceNumber, string. The sequence number for this message.

  • subSequenceNumber, string. The subsequence number this message.

  • data, string. The content of this message.

  • partitionKey, string. A partition key is used to group data by shard within a stream.

  • explictHashKey, string. The hash value used to explicitly determine the shard the data record is assigned to by overriding the partition key hash.

Status Output Port

Description

You can optionally enable the status output port for this adapter instance by means of the Enable Status Port property in the Adapter Properties page of the Properties view. Use the status port to retrieve status output messages from the adapter.

Status Port Schema

  • status, string. The type of status information emitted. Status types are:

    • Error—This message relates to an error that occurred.

    • Warn—This message relates to a warning that the user should be aware of.

    • Info—This message relates to extra status information.

  • message, string. This is a formatted human readable message that explains the status message.