The TIBCO StreamBase® Input Adapter for Velocity Analytics Broadcast Server for StreamBase receives real-time market data from the Velocity Analytics Broadcast Server via the real-time multicast interface. It converts each tick into a StreamBase tuple and sends it to StreamBase Server for processing. Additionally, it maintains a set of counters to provide processing statistics and logs events and error conditions via the StreamBase logging facility.
Familiarity with the Thomson Reuters Enterprise Platform for Velocity Analytics is required. In addition, a Velocity Analytics development server environment is required with the following installed:
The following Velocity Analytics documentation, available on the Thomson Reuters Customer Zone, provides additional information about Velocity Analytics, its components, data formats, and the SQL interface to use to query the time series database and User Time Series Store (UTSS).
Installation Guide — provides instructions on installing Velocity Analytics components.
System Admin and Config Guide — provides additional information about data record formats.
Hardware and Software Guide — provides information about current hardware and operating system requirements.
Using SQL with Velocity Analytics — provides information about querying the Velocity Analytics time series bases and UTSS.
This section describes the properties you can set for this adapter, using the various tabs of the Properties view in StreamBase Studio.
Name: Use this required field to specify or change the name of this instance of this component, which must be unique in the current EventFlow module. The name must contain only alphabetic characters, numbers, and underscores, and no hyphens or other special characters. The first character must be alphabetic or an underscore.
Adapter: A read-only field that shows the formal name of the adapter.
Class name: Shows the fully qualified class name that implements the functionality of this adapter. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.
Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this adapter starts.
Enable Error Output Port: Select this check box to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.
Description: Optionally enter text to briefly describe the component's purpose and function. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.
|Connect On Startup||When selected (default), the adapter automatically connects to the multicast when the StreamBase application is started. It
is recommended to enable Connect on Startup. Otherwise, make sure the control port is enabled so that a
|Enable Control Port||Enables the control port and status ports. See ControlStream for further information. The Status port is the last output port and is enabled automatically whenever the control port is enabled.|
|Enable Statistic Port||Enables the statistic port. See Statistics for further information.|
|Number of Output Ports||Set this to the number of data output ports configured via the Port Management tab; those are the ports that are mapped to flex record definitions. This number does not include the control, status, or statistics port.|
|Local IP||The DNS name or IP address of the local machine hosting this adapter.|
|Multicast IP||Specifies the DNS name or IP address or host name of the network card receiving the feed multicast. This address must be on the same subnet as the multicast group IP address for the Velocity Analytics Broadcast Server. See the Velocity Analytics System Admin and Config Guide for more information about group ID and Velocity Analytics Broadcast Server configuration.|
|Multicast Port||Multicast port for the incoming feed broadcast. This must match the setting of the
|Flex Record XML File||The name of the flex record definition file imported into the application. This file must contain definitions corresponding
to the record formats listened to by the adapter, and must match the definitions used by the Velocity Analytics Broadcast
Server. This file is read whenever StreamBase typechecking is performed. An example flex record definition file is provided
|Receive Time Format
Exchange Time Format
|The output time format for the ReceiveTime and ExchangeTime fields. Supported formats include:
See the Velocity Analytics System Admin and Config Guide for more information about time and date formats.
|Log Level||Controls the level of verbosity the adapter uses to send notifications to the console. This setting can be higher than the containing application's log level. If set lower, the system log level is used. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE.|
This section describes the properties on the Advanced Properties tab in the Properties view for the Velocity Analytics Broadcast Server Input adapter.
|Multicast Socket Buffer Size||Buffer size, in bytes, of the multicast socket. The default value is
|Back Channel Retry Time-out||Connection retry timeout, in milliseconds, for back channel TCP request. The default value is
|Feed Handlers||Number of threads used for feed processing. The default value is
|Max Packet Buffer Size||Maximum size of the internal packet hashmap. The default value is
|Max Tuple Queue Size||Size of the queue used to store the tuples. The default value is
|Statistics Interval||Time interval, in milliseconds, between two consecutive statistics outputs. The default value is
This section describes the Port Management tab in the Properties view for the Velocity Analytics Broadcast Server Input adapter.
Use this tab to specify the mapping of output ports to flex record definitions. Flex records are defined in the flex record
definition file corresponding to the flex records multicast from the Velocity Analytics Broadcast Server you are using. You
must import the XML file that defines these record definitions into the Studio project that includes this adapter, and specify
that file's name in the Flex Record XML File property. An example flex record definition file is provided as
VhayuRecordDefinitions.xml in the Velocity Adapters Sample.
The Port Management tab takes the form of a Port to Flex Record Mapping grid where you enter one
port#=FRDef entry per line, with each line corresponding to an output port for this adapter instance. The
FRDef value must correspond to the
name attribute of an
<FRDef> entry in the flex record definition file; examples are
Depth, and so on.
port# value is zero-based. Thus:
port2, and so on.
Each record mapping line in the grid can be in one of two forms:
In this case, all fields defined by the specified flex record are emitted to the specified output port. Examples:
- Port#=flexRecordDefName (
Field1, Field2, ... FieldN)
In this case, only the specified field names are emitted on that port. The
symbolfield is mandatory and the order of fields in the output tuple for this port follows their order in the list of field names. The same flex record name cannot be assigned to two or more ports. All user-defined fields in flexRecordDefName, as well as the following fixed fields are available:
recvTime— The receive time in VHTime format.
recordType— Record type.
symbol— The symbol name.
defName— Flex record definition name.
seqNum— Sequence number of the record provided by the feed.
exchTime— The exchange time, as reported by the exchange, in VHTime format.
subType— Subtype of the record if so provided by the feed.
Time conversion functions are available for converting
recvTimeto other formats. See Time Conversion Functions for further information.
Use the settings in this tab to allow this operator or adapter to start and stop based on conditions that occur at runtime in a cluster with more than one node. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with TIBCO Streaming releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.
Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.
For this adapter, it is recommended to keep Multiplicity set at single.
Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.
Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.
This section demonstrates how to use the Velocity Analytics Broadcast Server Input adapter in a StreamBase application, and describes the use of the adapter's ports. By default, the adapter provides one output port. As shown in the EventFlow diagram below (from one of this adapter's sample applications), the adapter can communicate with the surrounding application by means of one optional input port, a configurable number of output ports, plus a optional status output port. As with other StreamBase adapters and operators, you can optionally enable an Error Output Port.
The Velocity Analytics Broadcast Server Input adapter's ports are used as follows:
ControlStream (input): The optional input stream used to control the adapter during runtime. The adapter can be configured to have a control input port and corresponding status output port through the Enable Control Port check box in the Adapter Properties section of the Properties view. When enabled, they are represented by the last input and output port nodes on the adapter icon. The control port must have a schema of a single field named
string. The following commands can be sent to the Control port to manage the adapter during runtime or to generate a status report. Commands are case insensitive.
Join the multicast group of the Velocity Analytics Broadcast Server and start processing data if not already connected.
Leave the multicast group of the Velocity Analytics Broadcast Server and shut down all processing threads. In order to complete processing for all buffered data, it is recommended to issue a
disconnectcommand before stopping the adapter.
Rejoin the multicast group of the Velocity Analytics Broadcast Server and restart processing data.
portNumto emit data when available. When the adapter is started, all the ports are enabled by default. The changed setting persists after a disconnect.
flexDefNameto emit data when available. When the adapter is started, all the ports are enabled by default. The changed setting persists after a disconnect. Flex definition names are specified in the Port Management tab.
portNumfrom emitting data when available. When the adapter is started, all the ports are enabled by default. The changed setting persists after a disconnect.
flexDefNamefrom emitting data when available. When the adapter is started, all the ports are enabled by default. The changed setting persists after a disconnect.
Emit status of the ports and the connection to the status port. The status port must be connected to an output stream to see its output.
Trade and Quote (output): The output streams corresponding to the output data ports of the adapter. Two such ports are shown in the image above as examples, mapped specifically to the Trade and Quote FR definitions. You must configure at least one output data port, and can configure any number of output data ports. See Port Management Tab for information on configuring data output ports.
Statistics (output): The optional output stream carrying various adapter statistic counters. The counters in the table below are available through the statistics output port of the adapter. The statistics port can be enabled or disabled via the Enable Statistics Port check box in the Adapter Properties tab of the Properties view. The schema of the Statistics port is shown in the following table.
Total bytes, including header, received by the Input Adapter.
Total invalid feed lines received by the Input Adapter.
Total packets received by the Input Adapter.
Total valid feed lines received by the Input Adapter.
Total GetPacketFromCache backchannel data bad.
Total GetPacketFromCache backchannel cache miss.
Total times GetPacketFromCache backchannel closes.
Total times GetPacketFromCache backchannel opens.
Total times GetPacketFromCache backchannel open failures.
Total GetPacketFromCache backchannel receives.
Total GetPacketFromCache backchannel receive failures.
Total GetPacketFromCache backchannel receive too short.
Total GetPacketFromCache backchannel sends.
Total GetPacketFromCache backchannel send failures.
Count of retry timeouts backchannel.
Total duplicate or discarded packets found.
Total multicast reads.
Total multicast reads with inconsistent size.
Total multicast reads with error.
Total bytes received successfully through Datagram.
Total multicast reads too short.
Total multicast reads with zero length.
Total Velocity Analytics Broadcast Server cache read attempts.
Total Velocity Analytics Broadcast Server cache read fails.
Total packets received from Velocity Analytics Broadcast Server cache.
Total packet missed due to full hash map.
Total packet missed due to full feed processor queues.
Total number of tuples sent to the output ports.
Total tuples missed due to an overflow in the tuple queue.
Status (output): The optional output stream carrying adapter status.
Add an instance of the adapter to a new EventFlow application as follows:
In StreamBase Studio, create a project, including an empty StreamBase EventFlow application file to host the adapter.
Insert the adapter into your application. Drag the Input Adapter icon from the Operators and Adapters tray of the Palette view. In the resulting dialog, type
velocityto narrow the list of options, then select Velocity Analytics Broadcast Server from the Global Input Adapters section. (As an alternative, you can invoke > from top-level menu to invoke the same dialog.)
Configure the adapter via its various properties tabs. See Setting Properties for further information.
Import the flex record definitions file corresponding to the flex records multicast to the adapter.
Connect Output Streams to the adapter's output ports corresponding to output port properties.
Enable the Statistics Port, if desired. See Statistics for information.
Enable the Control and Status Ports. See ControlStream for further information. If the Control port is not enabled, make sure Connect On Startup is enabled via the Adapter Properties tab; otherwise, there is no way to inform the adapter to establish connection with the Velocity Analytics Broadcast Server.
Make sure the specified Velocity Analytics Broadcast Server is multicasting packets. For testing, configure the Velocity Analytics Broadcast Server to replay an existing feed file. See the Velocity Analytics System Admin and Config Guide for configuring the Velocity Analytics Broadcast Server for feed replay.
Run the EventFlow application by clicking thebutton in the Studio toolbar.
Observe the output streams.
Before stopping the EventFlow application, issue a
disconnectcommand via the Control port. This insures all buffered data is processed before the application shuts down.
The adapter uses the logging facility provided by StreamBase, which provides built-in support for standard loggers such as Log4J and Logback. See Using StreamBase Logging in the Administration Guide.
Logging is controlled through the XML configuration file,
logback.xml. Using this configuration file, you can customize logging based on your needs. For example, you can change the log message
format, include or remove timestamps, and so on. The logging level for the individual adapter messages can be set through
the Log Level drop-down control in the Adapter Properties section of the Properties view.
The adapter uses typechecking messages to help you configure the adapter within your StreamBase application. In particular, the adapter generates typecheck messages for the following reasons:
VhayuRecordDefinition.xmlfile is not found.
Values in the Properties view tabs are invalid.
The flex record definition used in a port mapping is not found in the
The schema of the Control port is wrong. It must have a single field named
Two or more ports are mapped to the same flex record definition name.
The number of ports specified is more than the number of ports configured in the Port Management tab.
SYMBOLfield is missing from the list of fields for any port definitions that list specific fields.
Empty or duplicate field names exist for a flex record definition.
The syntax of the field list in a port mapping is incorrect. It must have comma separated field names within parentheses.
The adapter generates messages during runtime under various conditions, including:
Multicast IP is invalid.
Local IP is invalid.
Multicast group joining fails.
Out of Memory error, which can happen due to bottlenecks in the network or anything that can lead to queuing data in the adapter's buffers. These messages might also be due to a data rate too high.
When suspended, the adapter stops emitting tuples, and all new incoming records from the Velocity Analytics Broadcast Server are discarded without any processing. The records already processed by the adapter but not sent through its output ports are cached.
The custom functions described in this section are available in the following classes:
These classes are provided as part of the JAR files that implement the Velocity Analytics adapters. See the Velocity Analytics Samples for examples of calling these functions with the StreamBase expression language's
The following sections list the available time conversion functions based on the source time format. In all instances, time parameters are defined as:
||The string representation of a date/time, expressed with the following format:
||StreamBase proprietary time format.|
||UTC time expressed with the ISO C standard time_t type definition. It is the number of seconds since Jan 1, 1970, 00:00:00 (midnight).|
||Velocity Analytics proprietary time format. This is UTC time expressed by the number of tenths of a microsecond (100ns) since
Jan 1, 1601, 00:00:00.000,000,0 (midnight). To convert from serial time to VHTime, use the following formula:
|Destination Time Format||Function|
|Destination Time Format||Function|
|Destination Time Format||Function|