Thomson Reuters Velocity Analytics Samples

Contents

Introduction
Prerequisites
Importing This Sample into StreamBase Studio
Preparing to Run the Sample
Running the Samples in StreamBase Studio
This Sample's Files
Sample Location

Introduction

The applications in this sample group demonstrate the use of the TIBCO StreamBase® Adapter for Thomson Reuters Velocity Analytics Broadcast Server, the TIBCO StreamBase® Adapter for Thomson Reuters Velocity Analytics UTSS, and query operations against the Velocity Analytics time series data store.

The Broadcast Server input adapter receives real-time market data from the Velocity Analytics Broadcast Server via the real-time multicast interface, converts each tick into a StreamBase Tuple, and sends it to the engine for processing. The UTSS adapter inserts data into a user time series store (UTSS) via the Analytics Engine Firebird SQL interface.

Prerequisites

Familiarity with the Thomson Reuters Enterprise Platform for Velocity Analytics is required. In addition, a Velocity Analytics development server environment is required with the following installed:

  • Analytics Engine

  • Broadcast Server

The following Velocity Analytics documentation, available on the Thomson Reuters Customer Zone, provides additional information about Velocity Analytics, its components, data formats, and the SQL interface to use to query the time series database and User Time Series Store (UTSS).

  • Installation Guide — provides instructions on installing Velocity Analytics components.

  • System Admin and Config Guide — provides additional information about data record formats.

  • Hardware and Software Guide — provides information about current hardware and operating system requirements.

  • Using SQL with Velocity Analytics — provides information about querying the Velocity Analytics time series bases and UTSS.

Importing This Sample into StreamBase Studio

In StreamBase Studio, import this sample with the following steps:

  • From the top-level menu, select File>Import Samples and Community Content.

  • Enter velocity to narrow the list of options.

  • Select Thomson Reuters Velocity Analytics adapters from the StreamBase Market Data Adapters category.

  • Click Import Now.

StreamBase Studio creates a single project containing the sample files.

Preparing to Run the Sample

Some of the applications in this sample require multicast Trade and Quote data from the Broadcast Server. This data can be supplied from a market data feed or by configuring the Broadcast Server to replay a sample feed log included in the distribution (recommended). The sections below, which document how to use each sample, assume you are using replay data from the sample feed log.

Configuring Feed Replay

This procedure configures the Broadcast Server to replay market data from the sample feed log provided in the Velocity Analytics distribution at the following location: 

install_dir\Samples\Data\Feed_20070301.log

This file contains trade and quote records for March 1, 2007 within the time range 10:08:47 AM and 10:11:30 AM for the following symbols: 

AAPL.O, AMGN.O, DELL.O, EBAY.O, GOOG.O, INTC.O, MSFT.O, ORCL.O, PFE.N, T.N

  1. Install the Velocity Analytics Broadcast Server, following the instructions in the Velocity Analytics Installation Guide.

  2. Configure playback mode. On the system hosting the Broadcast Server, open the system configuration file, install_dir\Config\TBAdmin.ini, in a text editor.

    1. Locate the playbackmode configuration parameter and set it to 1, as follows: 

      playbackmode=1

    2. Locate the following parameters in the broadcastserver process section and note their settings. They are used when configuring the EventFlow applications. 

      listenip
groupip
portnoofstock

    3. Confirm the following setting in the FeedReplay Configuration section of TBAdmin.ini: 

      FeedName=install_dir\Samples\Data\Feed_20070301.log

    Save and close TBAdmin.ini.

  3. Start the Broadcast Server. From a Windows command prompt, type: 

    net start broadcastserver

    This initiates feed file playback.

  4. Confirm feed replay has successfully started. Look at the Broadcast Server message log, install_dir\Log\BSMessages.log, for a message similar to the following: 

    INFO, 635711,SendP: 1st packet: 200......@@a,FR,INTC.O, Trade,....

Starting the Analytics Engine

  1. From a Windows command prompt, type: 

    net start searchengine

  2. Confirm the Analytics Engine successfully started. Look at the Analytics Engine message log, install_dir\Log\SEMessages.log, for a message similar to the following: 

    INFO,6000018,VA Init: Initialization successfully completed: 
INFO,190142,SETclThreads waiting for accepts with 13 threads

  3. Confirm the Analytics Engine has successfully synchronized with the Broadcast Server. Look at the Analytics Engine message log, install_dir\Log\SEMessages.log, for a message similar to the following: 

    INFO,190931,  ****** Synch record found in file - switching to MC stream 
INFO,190932,  ****** Synch record: 201...,@@a,FR,EBAY.O,Trade,,...,1947080,...

Running the Samples in StreamBase Studio

BSInputAdapter.sbapp

This sample demonstrates ingesting ticks from the Velocity Analytics real-time multicast interface provided by the Broadcast Server and dequeuing them to simple output streams based on record type. It requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.

  1. In the Project Explorer view, double-click to open the BSInputAdapter.sbapp application.

  2. Specify the Broadcast Server:

    • Select the Velocity Analytics Broadcast Server adapter icon to open the Properties view for the adapter.

    • Select the Adapter Properties tab and enter valid values for Local IP, Multicast IP, and Multicast Port. These values correspond as follows to those found in the Broadcast Server configuration.

      Adapter Property Broadcast Server Configuration Parameter
      Local IP listenip
      Multicast IP groupip
      Multicast Port portnoofstock

  3. Click the Run button. This opens the SB Test/Debug perspective and starts the module.

  4. In the Test/Debug Perspective, open the Output Streams view. If connectivity to your Broadcast Server is configured correctly, look for tuples emitted in the Output Streams view indicating the adapter has connected successfully.

  5. When done, press F9 or click the Terminate EventFlow Fragment button.

UTSSOutputAdapter.sbapp

This application receives real-time trade and quote records from the Velocity Analytics Broadcast Server adapter. The components TradeFieldSelection and QuoteFieldSelection filter out the fields that are not acceptable for insertion into a UTSS data store. Trade and Quote records for MSFT.O with BidPrice > 0.0 are filtered and written into an UTSS archive using the adapter. Rejected trades and quotes are sent to RejectedMSFTOTrades and RejectedMSFTOQuotes output streams respectively.

This sample requires the Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.

  1. In the Project Explorer view, double-click to open the UTSSOutputAdapter.sbapp application.

  2. Specify the Broadcast Server:

    • Select the Velocity Analytics Broadcast Server adapter icon to open the Properties view for the adapter.

    • Select the Adapter Properties tab and enter valid values for Local IP, Multicast IP, and Multicast Port. These values correspond as follows to those found in the Broadcast Server configuration.

      Adapter Property Broadcast Server Configuration Parameter
      Local IP listenip
      Multicast IP groupip
      Multicast Port portnoofstock

  3. Specify the UTSS database:

    • Select the Velocity Analytics UTSS adapter icon to open the Properties view for the adapter.

    • Select the Adapter Properties tab and enter valid values for UTSSPath and UTSSName.

  4. Start the Analytics Engine. Refer to Starting the Analytics Engine for details.

  5. Click the Run button. This opens the SB Test/Debug perspective and starts the module.

  6. Select the Manual Input tab and enter the command connect to the ControlStream input stream. Click Send Data to send the command.

  7. In the Test/Debug Perspective, open the Output Streams view. If connectivity to your Broadcast Server and UTSS database is configured correctly, look for tuples emitted on the Statistics stream in the Output Streams view, which indicates that records have been written.

  8. When finished, issue a disconnect command to the Control port of the Broadcast Server adapter. When all buffered data has been processed, press F9 or click the Terminate EventFlow Fragment button.

UTSSSelect.sbapp

This application demonstrates the use of the StreamBase Query operator to retrieve data from a User Time Series Store (UTSS).This sample requires an existing UTSS database, such as created using the UTSSOutputAdapter.sbapp application.

  1. Configure the JDBC data source. In the Project Explorer view, double-click sbd.sbconf. Change the IP address in the Analytics Engine data source definition to that of the Analytics Engine hosting the UTSS. This is the same address as defined with the listenip configuration parameter in the searchengine process section of the Velocity Analytics system configuration file, TBAdmin.ini.

    Save your changes.

  2. Specify the UTSS data base. In the Project Explorer view, double-click UTSSSelect.sbapp. Select the Parameters tab and:

    • Enter values for UTSSPath and UTSSName. These values are those used when the UTSS was created via UTSSOutputAdapter.sbapp.

    • Enter Quote for UTSSTable.

  3. Review the query. In the Project Explorer view, double-click to open the UTSSSelectModule.sbapp application.

    • Select the SelectFromUTSSTable Query operator icon to open its Properties view.

    • Select the Query Settings tab. From this tab you can review the SQL statement to be submitted to the UTSS.

    • Select the Result Settings tab. Make sure Quote_AE is selected as the private schema name.

  4. Run the application. In the Project Explorer view, double-click to open the UTSSSelect.sbapp application. Click the Run button. This opens the SB Test/Debug perspective and starts the module.

  5. In the Test/Debug Perspective, click the Manual Input tab. Enter the desired symbol, plus startTime and endTime values. Times are entered with the format yyyymmddhhmmss.ttt. For this demonstration, enter the following:

    symbol MSFT.O
    startTime A start time well before the time you started the UTSSOutputAdapter.sbapp application.
    endTIme An end time after you stopped the UTSSOutputAdapter.sbapp application.

  6. Click the Send Data button.

  7. In the Test/Debug Perspective, open the Output Streams view. If connectivity to your UTSS is configured correctly, look for tuples emitted in the Output Streams view indicating the adapter has connected successfully.

  8. When finished, press F9 or click the Terminate EventFlow Fragment button.

VWAP.sbapp

This EventFlow application demonstrates the use of the Velocity Analytics UTSS Output Adapter to store a VWAP analytics output in a User Time Series Store (UTSS). It receives real-time records from a Velocity Analytics Broadcast Server Input adapter. In this application, the Broadcast Server Input Adapter is configured to emit only Trade records. The Field Selection component filters out fields that are not needed for VWAP calculations. VWAP is then calculated on the resultant output with a window of 10 seconds. The result is grouped by symbol.

The VWAP output from the aggregate operator contains openTime and closeTime fields in StreamBase timestamp format. Since a UTSS database understands only VHTime, the StreamBase timestamps are converted to VHTime using the time conversion function sbTimestampToVH(). This function is implemented in BSInputAdapter.jar (and also in UTSSOutputAdapter.jar) and can be invoked using StreamBase's calljava() function. After the time is converted, the output is sent to the Velocity Analytics UTSS Output Adapter for subsequent write to the configured UTSS archive.

This sample requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.

  1. In the Project Explorer , double-click VhayuRecordDefinitions.xml. If not already present, add the following flex record definition near the beginning of the file, within the <FRDict> tag. 

    <FRDef name="VWAP" defid="44001">
   <field name="NumTrades"               type="t_u32" />
   <field name="OpenTime"                type="t_VHTime" />
   <field name="CloseTime"               type="t_VHTime" />
   <field name="VWAP"                    type="t_double" />
</FRDef>

    For your convenience, this definition is also included in the file, SampleFRDefinitions.xml, available via the Project Explorer.

  2. In the Project Explorer view, double-click to open the VWAP.sbapp application.

  3. Specify the Broadcast Server:

    • Select the Velocity Analytics Broadcast Server adapter icon to open the Properties view for the adapter.

    • Select the Adapter Properties tab and enter valid values for Local IP, Multicast IP, and Multicast Port. These values correspond as follows to those found in the Broadcast Server configuration.

      Adapter Property Broadcast Server Configuration Parameter
      Local IP listenip
      Multicast IP groupip
      Multicast Port portnoofstock

  4. Specify the UTSS database:

    • Select the Velocity Analytics UTSS adapter icon to open the Properties view for the adapter.

    • Select the Adapter Properties tab and enter valid values for UTSSPath and UTSSName.

  5. If not already running, start the Analytics Engine. Refer to Starting the Analytics Engine for details.

  6. Click the Run button. This opens the SB Test/Debug perspective and starts the module.

  7. Select the Manual Input tab and enter the command connect to the ControlStream input stream. Click Send Data to send the command.

  8. In the Test/Debug Perspective, open the Output Streams view. If connectivity to your Broadcast Server and UTSS data base is configured correctly, look for tuples emitted on the VWAP Output stream in the Output Streams view.

  9. When finished, issue a disconnect command to the Control port of the Broadcast Server adapter. When all buffered data has been processed, press F9 or click the Terminate EventFlow Fragment button.

TimeConversionFunctions.sbapp

This sample demonstrates the use of various time conversion functions to reformat timestamps. It requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.

  1. In the Project Explorer view, double-click to open the TimeConversionFunctions.sbapp application.

  2. Specify the Broadcast Server:

    • Select the Velocity Analytics Broadcast Server adapter icon to open the Properties view for the adapter.

    • Select the Adapter Properties tab and enter valid values for Local IP, Multicast IP, and Multicast Port. These values correspond as follows to those found in the Broadcast Server configuration.

      Adapter Property Broadcast Server Configuration Parameter
      Local IP listenip
      Multicast IP groupip
      Multicast Port portnoofstock

  3. Click the Run button. This opens the SB Test/Debug perspective and starts the module.

  4. In the Test/Debug Perspective, open the Output Streams view. If connectivity to your Broadcast Server is configured correctly, look for tuples emitted on the Output Streams stream for the various types of time conversion formats.

  5. When finished, issue a disconnect command to the Control port of the Broadcast Server Input adapter. When all buffered data has processed, press F9 or click the Terminate EventFlow Fragment button.

QueryAnalyticsEngine.sbapp

This application demonstrates the use of StreamBase JDBC Query operator to retrieve data from the Analytics Engine time series data store using SQL. This sample requires the Analytics Engine to be installed, started, and loaded with data.

If not already running, start the Analytics Engine. Refer to Starting the Analytics Engine for details.

  1. Configure the JDBC data source. In the Project Explorer view, double-click sbd.sbconf. Change the IP address in the Analytics Engine data source definition to that of the Analytics Engine. This is the same address as defined with the listenip configuration parameter in the searchengine process section of the Velocity Analytics system configuration file, TBAdmin.ini.

    Save your changes.

  2. In the Project Explorer view, double-click QueryAnalyticsEngine.sbapp.

  3. Select the Query Settings tab. From this tab you can review the SQL statement to be submitted to the Analytics Engine.

  4. Click the Run button. This opens the SB Test/Debug perspective and starts the module.

  5. In the Test/Debug Perspective, click the Manual Input tab. Enter the desired symbol, plus startTime and endTime values. Times are entered with the format yyyymmddhhmmss.ttt.

  6. Click the Send Data button.

  7. In the Test/Debug Perspective, open the Output Streams view. If connectivity to your Analytics Engine is configured correctly, look for tuples emitted on the output stream indicating the adapter has connected successfully.

  8. When finished, press F9 or click the Terminate EventFlow Fragment button.

This Sample's Files

The set of Velocity Analytics samples includes the following EventFlow applications. These applications can be executed within StreamBase Studio or using the StreamBase Server command line utility from a terminal window on UNIX or StreamBase Command Prompts on Windows.

  • BSInputAdapter.sbapp — demonstrates ingesting ticks from the real-time multicast interface and dequeuing them to simple output streams based on record type.

  • UTSSOutputAdapter.sbapp — demonstrates accepting filtered Trade and Quote records from the Broadcast Server adapter and subsequent writing them to a UTSS.

  • UTSSSelect.sbapp — demonstrates submitting a query to the target UTSS. This application uses UTSSSelectModule.sbapp, itself an EventFlow application, as a module to handle administrative interface details to the UTSS.

  • VWAP.sbapp — demonstrates calculating the VWAP analytic on the event stream. It is an end-to-end application from multicast ingestion to UTSS output.

  • TimeConversionFunctions.sbapp — demonstrates the use of custom time conversion functions.

  • QueryAnalyticsEngine.sbapp — demonstrates the use of the StreamBase Query operator to retrieve data from the Analytics Engine time series data store using standard JDBC access.

Sample Location

When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.

Important

Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.

Using the workspace copy of the sample avoids permission problems. The default workspace location for this sample is: 

studio-workspace/sample_adapter_embedded_thomsonreuters-velocityanalytics

See Default Installation Directories for the default location of studio-workspace on your system.