Contents
The Spotfire LiveView Publish output adapter allows a StreamBase application to publish data to LiveView tables.
This adapter is best used to publish real-time changes or updates to existing LiveView tables. For best performance of bulk
populating data into a LiveView table, set up a <data-source> or <publisher> configuration element.
The adapter may be used to publish to tables for which the table name or table schema is not known at author time by using the Ad-Hoc mode. When in this mode, the table name is sent in the first element of the input tuple while the data to be published is sent as a JSON string in the second element. The JSON string does not need to contain all the fields of the table, any fields not present are considered to be null while any fields not in the table are ignored.
This section describes the properties you can set for this adapter, using the various tabs of the Properties view for the adapter's icon in StreamBase Studio.
Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.
Adapter: A read-only field that shows the formal name of the adapter.
Class name: Shows the fully qualified class name that implements the functionality of this adapter. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.
Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this adapter starts.
Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.
Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.
This section describes the properties on the Adapter Properties tab in the Properties view for the LiveView Publish adapter.
| Property | Description |
|---|---|
| Set Server URI to encapsulating LiveView | Select this checkbox when the adapter is part of a LiveView project. In this case, the adapter connects to the port of the
running LiveView server. When authentication is enabled, the credentials used to connect to the LiveView server are retrieved
from the liveview.internal.username and liveview.internal.password system properties. The corresponding LiveView user must be configured with the appropriate permissions to allow this adapter
to perform its operations.
|
| LiveView Server URI | A LiveView server URI, or an expression or variable that resolves to a LiveView server URI. For example, the URI for the local
machine and default port is lv://localhost:11080.
|
| Use Runtime URIs | Enables an additional input port to this adapter. The input port expects tuples of the schema
(ControlAction: string, URI: string). The author time URI options are greyed out and
the adapter will wait to connect until it receives a Connect tuple. If the connection fails, the adapters will continue
to
try and connect. If the adapter receives a Disconnect tuple, it will disconnect from its current connection, if it has
one,
or stop trying to connect if it’s currently in failure.
|
| Connection Key | Only enabled if Use Runtime URIs and Share LiveView Connection are enabled. If both are enabled, this is a required
property which defines groups of adapters that will share connections. An author-defined string
of alphanumeric characters is expected here like Pool1 or Pool2. All adapters set to share
connections and use runtime URIs with the Connection Key Pool1 will connect and disconnect
using a single shared connection as Connect/Disconnect tuples are received by any one of the
adapters in the group.
|
| Ad-Hoc mode |
When selected, the Target Table Name field is ignored. Instead, the name of the table to publish data into is taken from the first field of a tuple on the input port. This field must be a string and must contain a valid table name for the specifiedLiveView server (or must contain an expression that resolves to a valid table name). In addition, the second field contains the data to be published. This field takes the form of a JSON string. Any of the table's fields may be present in the JSON string. If an element is not present, it is assumed to be NULL. A The Enforce Strict Schema is disabled and assumed to be false. |
| Get table name from input | When selected, the Target Table Name field is ignored. Instead, the name of the table to publish data into is taken from the first field of a tuple on the input port. This field must be a string and must contain a valid table name for the specified LiveView server (or must contain an expression that resolves to a valid table name). |
| Target Table Name | The name of the LiveView table to publish data to. This control is ignored when the Get table name from input or the Ad-Hoc mode checkbox is selected. |
| Delete Field Name |
If not empty, this field contains the name of the boolean field in the input schema to be used to indicate that the current
tuple should be deleted or published. It is an error to have something in this field and an input field named If the value of the named field is |
| Enforce Strict Schema |
This control is ignored when the Get table name from input or Ad-Hoc mode checkbox is selected. When selected, the schema of the input port must exactly match the schema of the target table specified in the row above,
except that the strict schema can contain an additional field of the name When cleared, the input port accepts tuples with any schema. When used with the Get table name from input check box, the input port's schema's first field must be a string containing or resolving to the name of a table on the specified LiveView server. |
| Copy input to output | When this checkbox is selected, the adapter creates an output port that gets a copy of the tuple sent to the input port. The output tuples are sent regardless of the LiveView connection state. Default state is cleared. |
| Share LiveView Connection | When selected (the default), this instance of the adapter shares a single LiveView connection to the configured LiveView server with all other LiveView adapters in the same container that also have the Share LiveView Connection property selected. |
| Enable Compressed Connection | If checked, compress this adapter's LiveView connection. Not enabled if Use Runtime URI is also selected. To compress a connection defined at runtime, see the "ConnectCompress" control action defined in the Runtime URI Schema |
| Connect Inline |
When this checkbox is cleared (the default state), tuples are sent to the target table after every Publisher Flush Interval. A background reconnection thread in the adapter continuously tries to reconnect to the specified server so that a connection is available when tuples need to be sent. However, a window can occur during which the server is actually up, but the adapter is disconnected, and the reconnection thread is waiting for its next connection attempt. This situation can cause a delay in having the freshest tuple information written to the LiveView table. To avoid this situation, select this checkbox. In this case, if the adapter is currently disconnected from the server, it attempts to reconnect inline each time a tuple or batch of tuples is ready to be sent to the target table. Use this feature carefully, because, while it can speed up the freshness of live data on the server, it can also cause long application delays while each connection is attempted. |
| Publisher Buffer Size | The number of tuples to buffer before forcing a network write to the target table. |
| Publisher Flush Interval (ms) | The maximum number of milliseconds that can elapse before each tuple or batch of tuples is written to the target table. |
| Enable Reliable Publish | If enabled, publisher will use the reliable protocol. See LiveView Data Recovery for a general discussion of reliable publishing. Also see the Javadoc for details about publisher names and sequence numbers. |
| Sequence Number Field Name | The long data type field name in the input schema that will supply the reliable publish sequence number. This number must increment by 1 for each tuple published. |
| Acknowledgment period | The number of milliseconds between acknowledgment status messages. Zero means no acknowledgment messages. |
| Log Level | Controls the level of verbosity the adapter uses to send notifications to the console. This setting can be higher than the containing application's log level. If set lower, the system log level is used. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE. |
Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.
Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.
Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.
Caution
Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.
The LiveView Publish adapter has an input port that accepts tuples to be published to the target table, and a status port to provide information about connection status.
As with other Spotfire Streaming adapters and operators, you can optionally enable an Error Output Port, as described in Using Error Ports and Error Streams.
An optional Use Runtime URI Input Port is also available for Connecting/Disconnecting to LiveView servers at runtime instead of at author time.
If Enforce Strict Schema is set, the schema of the PublishIn input port must exactly match the schema of the LiveView table specified in the Target Table Name control. In this case, fields must match by exact, case-sensitive field name and data type, and in exact field order. However,
even with Enforce Strict Schema set, the input port's schema can also contain an extra boolean field named CQSDelete. If the value of this field is true, then the primary key fields set in the tuple are used to look up and delete the specified row in the table.
With or without Enforce Strict Schema, tuples sent to the PublishIn input port do not need to specify data for every field. You can leave any field null if you only need to update a few fields in the LiveView table.
If Enforce Strict Schema is cleared, then the schema of the PublishIn input port can contain any field names and data types, within the following guidelines:
-
The schema of the PublishIn port can contain any subset of the fields of the LiveView table or tables to which this adapter instance is to publish or delete rows. If a table has many fields, but you only expect to publish updated data to a few of those fields, then only the few target fields need to be in the schema.
-
If Get table name from input is also selected, then the first field of the input schema must be a string that contains or resolves to the valid name of a LiveView table on the specified LiveView server.
-
To successfully use the Get table name from input feature, the second and subsequent fields of the PublishIn port's schema must be a union of the names and data types of all the fields you expect to update in all the LiveView tables that you expect to name at runtime in the first field. For maximum flexibility, you can specify the full schema of each LiveView table you might name in first field. Or, if you know in advance the fields you expect to update in each table, then you can specify only those fields in the schema.
-
After the first field, the order of fields in the PublishIn port's schema does not matter. Fields are matched by name and data type, not field order.
-
If any field has the same name and data type in two or more LiveView tables, you only need to specify it once in the PublishIn port's schema; since the table's name is specified in the first field, the input tuple is matched only against that table's instance of the like-named field.
Deletions are triggered by the use of either a CQSDelete field or a field named by the Delete Field Name property. These fields are optional, but if a CQSDelete field is present, the value of Delete Field Name must either be empty or have the value CQSDelete. The field must be of type boolean and if true, the row determined by the publish data is deleted. Any other value results in a publish action. Except in the
case of Ad-Hoc mode or if Get table name from input being checked, these fields may exist anywhere on the top level of the input tuple, for those, they must come after the required
fields.
When in Ad-Hoc mode, the input must have at least two fields. The first field must be a string and contains the table name or an expression which resolves to a valid table name. The second field is a JSON representation
of the data to be published. Any field in the JSON which exists in the table will be sent, any field which does not exist
in the table will be ignored and any table field which is not present will be set to null. Additional fields may be present in the input and will be ignored except in the case of CQSDelete or a valid value in Delete Field Name.
When configured for reliable publishing, you must supply a sequence number. The sequence number must be incremented by 1 for each tuple sent. If you configure a non-zero Acknowledgment period, the sequence number is the way you can track what tuples the adapter has sent that are committed to the destination LiveView table. See LiveView Data Recovery for a general discussion of reliable publishing, and in particular the difference between last published, and last persisted sequence number.
You must configure a unique, long-lived, publisher name for purposes of reliable publishing. It is this name that ties together publishes from previous instances
of the Publish adapter. You could use a fixed GUID (globally unique identifier), which would ensure there would be no collisions
with other publishes to the LiveView table. It is the responsibility of the author to ensure no collisions are possible. Long-lived means the publisher name must remain unique for at least for the life of the data published to the LiveView table. Some use
cases trim LiveView tables aggressively—like every hour—and some use cases hold data for very extended periods. When a Publish
adapter configured for reliable publishing connects to the server, you must wait for the ACKNOWLEDGMENT message to determine
what the highest sequence number the LiveView table has. The EventFlow author is responsible for ensuring all tuples starting
immediately after the reported highest sequence number, if any, are resent. Publishing then proceeds from the next sequence
number.
Managing the publisher name (and its uniqueness), the publishing sequence number, and retaining unacknowledged tuples associated with published sequence numbers are all the responsibility of the author. The techniques for doing this are as varied as the potential data sources. A Kafka data source provides a simple, concrete example of mapping a data source to the reliable publisher requirements. The publisher name could be a fixed GUID, or even just the Kafka Topic. The publisher sequence number could be the Kafka message bus offset into the topic. When the Publish adapter connects, the latest sequence number returned will be the Topic offset that should be subscribed to from the Kafka message bus.
The adapter can import a proposed schema for the input port that you can use as a starting point.
To import an input schema, click the Import proposed schemas hyperlink in the top right corner above the Adapter Properties tab. From the dialog, click Browse to select an existing destination interface file, or click New File to save a Named schema file to a project folder.
When Use Runtime URIs is enabled on this adapter, so is this port.
This port accepts tuples of the below schema.
| Field | Data Type | Description | |||
|---|---|---|---|---|---|
| ControlAction | string | A value of either
|
|||
| URI | string | The URI of a LiveView server such as lv://localhost:11080. The adapter will attempt to connect
to the LiveView server at this address if Connect or ConnectCompress is supplied in this tuple's ControlAction.
If Disconnect is supplied as the ControlAction,
a value for URI need not be supplied and will be ignored.
|
The LiveView Publish adapter has one output status port. For each publish request sent as an input tuple on the adapter's input port, a connection status report tuple is emitted on the status port. For each publish request that fails, an error tuple is emitted:
-
A CONNECTION message shows the success or failure of the connection to the LiveView server.
-
For failing publish requests, a PUBLISH message shows FAILED or ERROR it its Action field, with an explanatory note in the Message field.
-
An ACKNOWLEDGE action will occur when reliable publish is configured with a non-zero Acknowledgment period. The Message field will contain the last published sequence number and the last persisted sequence number separated by a semicolon.
The status port has the fields shown in the following table:
| Field | Data Type | Description | ||||
|---|---|---|---|---|---|---|
| Type | string | The basic state reported, always one of the following:
|
||||
| Object | string | The name on the EventFlow canvas of the adapter reporting the status. | ||||
| Action | string | The action that occurred, one of the following:
|
||||
| Message | string | A human-readable string that provides additional context for the status message. | ||||
| Time | timestamp | The time the publish request or error occurred. |
Add an instance of the adapter to a new EventFlow application with the following steps:
-
In StreamBase Studio, create a project, including an empty StreamBase EventFlow application file to host the adapter.
-
From the Operators and Adapters drawer of the Palette view, drag the Adapters, Java Operators icon to the canvas. This opens the Insert an Operator or Adapter dialog.
-
In the search field, type a string such as
liveviewto narrow the list of adapters. Select the icon for the LiveView Publish adapter, then click . -
Double-click the adapter's icon and select the Adapter Properties tab in the Properties view.
-
Next, either:
-
If this adapter is part of an EventFlow module that is itself part of a LiveView project, select the Set Server URI to encapsulating LiveView check box. This check box tells the adapter to connect to the host and port for the current project's LiveView server.
-
Or specify the URI for a local or remote LiveView server to connect to (or accept the default URI,
lv:/localhost:11080).
-
-
Add an Input Stream and connect an arc from it to the LiveView Publish adapter's input port.
-
Specify a schema for the Input Stream that matches its intended use, as described in LiveView Publish Adapter Input Port.
-
Add an Output Stream and connect an arc to it from the LiveView Publish adapter's output port.
-
At runtime, send a tuple to this module's input port, specifying the a tuple to publish to the specified LiveView table.
The LiveView Publish adapter uses typecheck messages to help you configure the adapter.
The adapter generates warning messages at runtime under various conditions, including:
-
The configured schema of this adapter does not match the LiveView table, if the Enforce Strict Schema option is set.
-
The configured LiveView server is not available.