Thomson Reuters Velocity Analytics Samples

The applications in this sample group demonstrate the use of the TIBCO StreamBase® Adapter for Thomson Reuters Velocity Analytics Broadcast Server, the TIBCO StreamBase® Adapter for Thomson Reuters Velocity Analytics UTSS, and query operations against the Velocity Analytics time series data store.

The Broadcast Server input adapter receives real-time market data from the Velocity Analytics Broadcast Server via the real-time multicast interface, converts each tick into a StreamBase Tuple, and sends it to the engine for processing. The UTSS adapter inserts data into a user time series store (UTSS) via the Analytics Engine Firebird SQL interface.

Prerequisites

Familiarity with the Thomson Reuters Enterprise Platform for Velocity Analytics is required. In addition, a Velocity Analytics development server environment is required with the following installed:

  • Analytics Engine

  • Broadcast Server

The following Velocity Analytics documentation, available on the Thomson Reuters Customer Zone, provides additional information about Velocity Analytics, its components, data formats, and the SQL interface to use to query the time series database and User Time Series Store (UTSS).

  • Installation Guide — provides instructions on installing Velocity Analytics components.

  • System Admin and Config Guide — provides additional information about data record formats.

  • Hardware and Software Guide — provides information about current hardware and operating system requirements.

  • Using SQL with Velocity Analytics — provides information about querying the Velocity Analytics time series bases and UTSS.

This Sample's Files

The set of Velocity Analytics samples includes the following EventFlow applications. These applications can be executed within StreamBase Studio or using the StreamBase Server command line utility from a terminal window on UNIX or StreamBase Command Prompts on Windows.

  • BSInputAdapter.sbapp — demonstrates ingesting ticks from the real-time multicast interface and dequeuing them to simple output streams based on record type.

  • UTSSOutputAdapter.sbapp — demonstrates accepting filtered Trade and Quote records from the Broadcast Server adapter and subsequent writing them to a UTSS.

  • UTSSSelect.sbapp — demonstrates submitting a query to the target UTSS. This application uses UTSSSelectModule.sbapp, itself an EventFlow application, as a module to handle administrative interface details to the UTSS.

  • VWAP.sbapp — demonstrates calculating the VWAP analytic on the event stream. It is an end-to-end application from multicast ingestion to UTSS output.

  • TimeConversionFunctions.sbapp — demonstrates the use of custom time conversion functions.

  • QueryAnalyticsEngine.sbapp — demonstrates the use of the StreamBase Query operator to retrieve data from the Analytics Engine time series data store using standard JDBC access.

Importing This Sample into StreamBase Studio

In StreamBase Studio, import this sample with the following steps:

  1. From the top menu, select FileLoad StreamBase Sample.

  2. Select Thomson Reuters Velocity Analytics from the StreamBase Market Data Adapters category.

  3. Click OK.

StreamBase Studio creates a project for this sample.

Sample Location

When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.

Important

Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.

Using the workspace copy of the sample avoids permission problems. The default workspace location for this sample is:

studio-workspace/sample_adapter_embedded_thomsonreuters-velocityanalytics

See Default Installation Directories for the location of studio-workspace on your system.

In the default TIBCO StreamBase installation, this sample's files are initially installed in:

streambase-install-dir/sample/adapter/embedded/thomsonreuters-velocityanalytics

See Default Installation Directories for the default location of studio-workspace on your system.

Preparing to Run the Sample

Some of the applications in this sample require multicast Trade and Quote data from the Broadcast Server. This data can be supplied from a market data feed or by configuring the Broadcast Server to replay a sample feed log included in the distribution (recommended). The sections below, which document how to use each sample, assume you are using replay data from the sample feed log.

Configuring Feed Replay

This procedure configures the Broadcast Server to replay market data from the sample feed log provided in the Velocity Analytics distribution at the following location:

install_dir\Samples\Data\Feed_20070301.log

This file contains trade and quote records for March 1, 2007 within the time range 10:08:47 AM and 10:11:30 AM for the following symbols:

AAPL.O, AMGN.O, DELL.O, EBAY.O, GOOG.O, INTC.O, MSFT.O, ORCL.O, PFE.N, T.N
  1. Install the Velocity Analytics Broadcast Server, following the instructions in the Velocity Analytics Installation Guide.

  2. Configure playback mode. On the system hosting the Broadcast Server, open the system configuration file, install_dir\Config\TBAdmin.ini, in a text editor.

    1. Locate the playbackmode configuration parameter and set it to 1, as follows:

      playbackmode=1
    2. Locate the following parameters in the broadcastserver process section and note their settings. They are used when configuring the EventFlow applications.

      listenip
      groupip
      portnoofstock
    3. Confirm the following setting in the FeedReplay Configuration section of TBAdmin.ini:

      FeedName=install_dir\Samples\Data\Feed_20070301.log

    Save and close TBAdmin.ini.

  3. Start the Broadcast Server. From a Windows command prompt, type:

    net start broadcastserver

    This initiates feed file playback.

  4. Confirm feed replay has successfully started. Look at the Broadcast Server message log, install_dir\Log\BSMessages.log, for a message similar to the following:

    INFO, 635711,SendP: 1st packet: 200......@@a,FR,INTC.O, Trade,....

Starting the Analytics Engine

  1. From a Windows command prompt, type:

    net start searchengine
  2. Confirm the Analytics Engine successfully started. Look at the Analytics Engine message log, install_dir\Log\SEMessages.log, for a message similar to the following:

    INFO,6000018,VA Init: Initialization successfully completed: 
    INFO,190142,SETclThreads waiting for accepts with 13 threads
  3. Confirm the Analytics Engine has successfully synchronized with the Broadcast Server. Look at the Analytics Engine message log, install_dir\Log\SEMessages.log, for a message similar to the following:

    INFO,190931,  ****** Synch record found in file - switching to MC stream 
    INFO,190932,  ****** Synch record: 201...,@@a,FR,EBAY.O,Trade,,...,1947080,...

Running the Samples in StreamBase Studio

BSInputAdapter.sbapp

This sample demonstrates ingesting ticks from the Velocity Analytics real-time multicast interface provided by the Broadcast Server and dequeuing them to simple output streams based on record type. It requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.

  1. In the Project Explorer, double-click to open the BSInputAdapter.sbapp application.

  2. Specify the Broadcast Server:

    • Select the Velocity Analytics Broadcast Server adapter icon to open the Properties view for the adapter.

    • Select the Adapter Properties tab and enter valid values for Local IP, Multicast IP, and Multicast Port. These values correspond as follows to those found in the Broadcast Server configuration.

      Adapter Property Broadcast Server Configuration Parameter
      Local IP listenip
      Multicast IP groupip
      Multicast Port portnoofstock
  3. Click the Run button. This opens the SB Test/Debug perspective and starts the application.

  4. In the Test/Debug Perspective, open the Application Output view. If connectivity to your Broadcast Server is configured correctly, look for tuples emitted in the Application Output view indicating the adapter has connected successfully.

  5. When done, press F9 or click the Stop Running Application button.

UTSSOutputAdapter.sbapp

This application receives real-time trade and quote records from the Velocity Analytics Broadcast Server adapter. The components TradeFieldSelection and QuoteFieldSelection filter out the fields that are not acceptable for insertion into a UTSS data store. Trade and Quote records for MSFT.O with BidPrice > 0.0 are filtered and written into an UTSS archive using the adapter. Rejected trades and quotes are sent to RejectedMSFTOTrades and RejectedMSFTOQuotes output streams respectively.

This sample requires the Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.

  1. In the Project Explorer, double-click to open the UTSSOutputAdapter.sbapp application.

  2. Specify the Broadcast Server:

    • Select the Velocity Analytics Broadcast Server adapter icon to open the Properties view for the adapter.

    • Select the Adapter Properties tab and enter valid values for Local IP, Multicast IP, and Multicast Port. These values correspond as follows to those found in the Broadcast Server configuration.

      Adapter Property Broadcast Server Configuration Parameter
      Local IP listenip
      Multicast IP groupip
      Multicast Port portnoofstock
  3. Specify the UTSS database:

    • Select the Velocity Analytics UTSS adapter icon to open the Properties view for the adapter.

    • Select the Adapter Properties tab and enter valid values for UTSSPath and UTSSName.

  4. Start the Analytics Engine. Refer to Starting the Analytics Engine for details.

  5. Click the Run button. This opens the SB Test/Debug perspective and starts the application.

  6. Select the Manual Input tab and enter the command connect to the ControlStream input stream. Click Send Data to send the command.

  7. In the Test/Debug Perspective, open the Application Output view. If connectivity to your Broadcast Server and UTSS database is configured correctly, look for tuples emitted on the Statistics stream in the Application Output view, which indicates that records have been written.

  8. When finished, issue a disconnect command to the Control port of the Broadcast Server adapter. When all buffered data has been processed, press F9 or click the Stop Running Application button.

UTSSSelect.sbapp

This application demonstrates the use of the StreamBase Query operator to retrieve data from a User Time Series Store (UTSS).This sample requires an existing UTSS database, such as created using the UTSSOutputAdapter.sbapp application.

  1. Configure the JDBC data source. In the Project Explorer, double-click sbd.sbconf. Change the IP address in the Analytics Engine data source definition to that of the Analytics Engine hosting the UTSS. This is the same address as defined with the listenip configuration parameter in the searchengine process section of the Velocity Analytics system configuration file, TBAdmin.ini.

    Save your changes.

  2. Specify the UTSS data base. In the Project Explorer view, double-click UTSSSelect.sbapp. Select the Parameters tab and:

    • Enter values for UTSSPath and UTSSName. These values are those used when the UTSS was created via UTSSOutputAdapter.sbapp.

    • Enter Quote for UTSSTable.

  3. Review the query. In the Project Explorer, double-click to open the UTSSSelectModule.sbapp application.

    • Select the SelectFromUTSSTable Query operator icon to open its Properties view.

    • Select the Query Settings tab. From this tab you can review the SQL statement to be submitted to the UTSS.

    • Select the Result Settings tab. Make sure Quote_AE is selected as the private schema name.

  4. Run the application. In the Project Explorer view, double-click to open the UTSSSelect.sbapp application. Click the Run button. This opens the SB Test/Debug perspective and starts the application.

  5. In the Test/Debug Perspective, click the Manual Input tab. Enter the desired symbol, plus startTime and endTime values. Times are entered with the format yyyymmddhhmmss.ttt. For this demonstration, enter the following:

    symbol MSFT.O
    startTime A start time well before the time you started the UTSSOutputAdapter.sbapp application.
    endTIme An end time after you stopped the UTSSOutputAdapter.sbapp application.
  6. Click the Send Data button.

  7. In the Test/Debug Perspective, open the Application Output view. If connectivity to your UTSS is configured correctly, look for tuples emitted in the Application Output view indicating the adapter has connected successfully.

  8. When finished, press F9 or click the Stop Running Application button.

VWAP.sbapp

This EventFlow application demonstrates the use of the Velocity Analytics UTSS Output Adapter to store a VWAP analytics output in a User Time Series Store (UTSS). It receives real-time records from a Velocity Analytics Broadcast Server Input adapter. In this application, the Broadcast Server Input Adapter is configured to emit only Trade records. The Field Selection component filters out fields that are not needed for VWAP calculations. VWAP is then calculated on the resultant output with a window of 10 seconds. The result is grouped by symbol.

The VWAP output from the aggregate operator contains openTime and closeTime fields in StreamBase timestamp format. Since a UTSS database understands only VHTime, the StreamBase timestamps are converted to VHTime using the time conversion function sbTimestampToVH(). This function is implemented in BSInputAdapter.jar (and also in UTSSOutputAdapter.jar) and can be invoked using StreamBase's calljava() function. After the time is converted, the output is sent to the Velocity Analytics UTSS Output Adapter for subsequent write to the configured UTSS archive.

This sample requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.

  1. In the Project Explorer , double-click VhayuRecordDefinitions.xml. If not already present, add the following flex record definition near the beginning of the file, within the <FRDict> tag.

    <FRDef name="VWAP" defid="44001">
       <field name="NumTrades"               type="t_u32" />
       <field name="OpenTime"                type="t_VHTime" />
       <field name="CloseTime"               type="t_VHTime" />
       <field name="VWAP"                    type="t_double" />
    </FRDef>

    For your convenience, this definition is also included in the file, SampleFRDefinitions.xml, available via the Project Explorer.

  2. In the Project Explorer, double-click to open the VWAP.sbapp application.

  3. Specify the Broadcast Server:

    • Select the Velocity Analytics Broadcast Server adapter icon to open the Properties view for the adapter.

    • Select the Adapter Properties tab and enter valid values for Local IP, Multicast IP, and Multicast Port. These values correspond as follows to those found in the Broadcast Server configuration.

      Adapter Property Broadcast Server Configuration Parameter
      Local IP listenip
      Multicast IP groupip
      Multicast Port portnoofstock
  4. Specify the UTSS database:

    • Select the Velocity Analytics UTSS adapter icon to open the Properties view for the adapter.

    • Select the Adapter Properties tab and enter valid values for UTSSPath and UTSSName.

  5. If not already running, start the Analytics Engine. Refer to Starting the Analytics Engine for details.

  6. Click the Run button. This opens the SB Test/Debug perspective and starts the application.

  7. Select the Manual Input tab and enter the command connect to the ControlStream input stream. Click Send Data to send the command.

  8. In the Test/Debug Perspective, open the Application Output view. If connectivity to your Broadcast Server and UTSS data base is configured correctly, look for tuples emitted on the VWAP Output stream in the Application Output view.

  9. When finished, issue a disconnect command to the Control port of the Broadcast Server adapter. When all buffered data has been processed, press F9 or click the Stop Running Application button.

TimeConversionFunctions.sbapp

This sample demonstrates the use of various time conversion functions to reformat timestamps. It requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.

  1. In the Project Explorer, double-click to open the TimeConversionFunctions.sbapp application.

  2. Specify the Broadcast Server:

    • Select the Velocity Analytics Broadcast Server adapter icon to open the Properties view for the adapter.

    • Select the Adapter Properties tab and enter valid values for Local IP, Multicast IP, and Multicast Port. These values correspond as follows to those found in the Broadcast Server configuration.

      Adapter Property Broadcast Server Configuration Parameter
      Local IP listenip
      Multicast IP groupip
      Multicast Port portnoofstock
  3. Click the Run button. This opens the SB Test/Debug perspective and starts the application.

  4. In the Test/Debug Perspective, open the Application Output view. If connectivity to your Broadcast Server is configured correctly, look for tuples emitted on the Application Output stream for the various types of time conversion formats.

  5. When finished, issue a disconnect command to the Control port of the Broadcast Server Input adapter. When all buffered data has been processed, press F9 or click the Stop Running Application button.

QueryAnalyticsEngine.sbapp

This application demonstrates the use of StreamBase JDBC Query operator to retrieve data from the Analytics Engine time series data store using SQL. This sample requires the Analytics Engine to be installed, started, and loaded with data.

If not already running, start the Analytics Engine. Refer to Starting the Analytics Engine for details.

  1. Configure the JDBC data source. In the Project Explorer, double-click sbd.sbconf. Change the IP address in the Analytics Engine data source definition to that of the Analytics Engine. This is the same address as defined with the listenip configuration parameter in the searchengine process section of the Velocity Analytics system configuration file, TBAdmin.ini.

    Save your changes.

  2. In the Project Explorer view, double-click QueryAnalyticsEngine.sbapp.

  3. Select the Query Settings tab. From this tab you can review the SQL statement to be submitted to the Analytics Engine.

  4. Click the Run button. This opens the SB Test/Debug perspective and starts the application.

  5. In the Test/Debug Perspective, click the Manual Input tab. Enter the desired symbol, plus startTime and endTime values. Times are entered with the format yyyymmddhhmmss.ttt.

  6. Click the Send Data button.

  7. In the Test/Debug Perspective, open the Application Output view. If connectivity to your Analytics Engine is configured correctly, look for tuples emitted on the output stream indicating the adapter has connected successfully.

  8. When finished, press F9 or click the Stop Running Application button.

Running the Samples in Terminal Windows

This section describes how to run the samples in UNIX terminal windows or Windows command prompt windows. On Windows, be sure to use the StreamBase Command Prompt from the Start menu as described in the Test/Debug Guide, rather than the Windows default command prompt.

BSInputAdapter.sbapp

This sample demonstrates ingesting ticks from the Velocity Analytics real-time multicast interface provided by the Broadcast Server and dequeuing them to simple output streams based on record type. It requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.

  1. Open six terminal windows on UNIX, or six StreamBase Command Prompts on Windows. In each window, navigate to your workspace copy of the sample, as described in Sample Location.

  2. In window 1, type:

    sbd -f sbd.sbconf BSInputAdapter.sbapp
  3. In window 2, type:

    sbc dequeue -v Trade
  4. In window 3, type:

    sbc dequeue -v Quote
  5. In window 4, type:

    sbc dequeue -v Statistics
  6. In window 5, type:

    sbc dequeue -v Status

    This window displays tuples dequeued from adapter's output port for Status information.

  7. Observe that tuples representing Trade data, Quote data, and Statistics information are being emitted from window 2, window 3, and window 4 respectively.

  8. In window 6, type:

    echo status | sbc enqueue ControlStream
  9. In window 6, type the following command to terminate the server and the dequeuers:

    sbadmin shutdown

UTSSOutputAdapter.sbapp

This application receives real-time trade and quote records from Velocity Analytics Broadcast Server adapter. Trade and Quote records for MSFT.O with BidPrice > 0.0 are filtered and written into an UTSS archive using the adapter.

This sample requires the Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.

  1. Open five terminal windows on UNIX, or five StreamBase Command Prompts on Windows. In each window, navigate to your workspace copy of the sample, as described in Sample Location.

  2. In window 1, type:

    sbd -f sbd.sbconf  UTSSOutputAdapter.sbapp
  3. In window 2, type:

    echo connect | sbc enqueue ControlStream

    This establishes a connection between the UTSS Output Adapter and the Analytics Engine.

  4. In window 3, type:

    sbc dequeue -v Status
  5. In window 4, type:

    echo status | sbc enqueue ControlStream
  6. In window 5, type:

    sbc dequeue -v Statistics

    Observe the emitted tuples in the same window. The tuples carry information about SQL inserts being applied to the specified UTSS archive. Use any Firebird SQL client tool to confirm that the data is continuously written to the UTSS archive while the application is running. See the Velocity Analytics guide, Using SQL with Velocity Analytics, available in the Velocity Analytics documentation set for more information about UTSS databases.

  7. In window 4, type the following command to terminate the server and the dequeuers:

    sbadmin shutdown

UTSSSelect.sbapp

This application demonstrates the use of a StreamBase Query operator to retrieve data from a User Time Series Store (UTSS). This sample requires an existing UTSS data base, such as created using the UTSSOutputAdapter.sbapp application. If not already running, start the Analytics Engine. Refer to Starting the Analytics Engine for details.

  1. Open three terminal windows on UNIX, or three StreamBase Command Prompts on Windows. In each window, navigate to your workspace copy of the sample, as described in Sample Location.

  2. In window 1, type:

    sbd -f sbd.sbconf  UTSSSelect.sbapp
  3. In window 2, type:

    sbc dequeue -v SelectOutput

    This window displays the query result set in form of StreamBase tuples.

  4. In window 3, type:

    echo MSFT.O, 20010101, 20020101 | sbc enqueue QueryParameters

    Observe the query result set in window 2.

  5. In window 3, type the following command to terminate the server and the dequeuers:

    sbadmin shutdown

VWAP.sbapp

This EventFlow application demonstrates the usage of the Velocity Analytics UTSS Output Adapter to store an analytics output (VWAP) in a User Time Series Store (UTSS). It receives real-time records from Velocity Analytics Broadcast Server Input Adapter. In this application, the Broadcast Server Input Adapter is configured to emit only Trade records. The VWAP is then calculated on the resultant output with a window of 10 seconds. The result is grouped by symbol.

The VWAP output from the aggregate operator contains openTime and closeTime fields in StreamBase timestamp format. Since a UTSS database understands only VHTime, the StreamBase timestamps are converted to VHTime using the time conversion function sbTimestampToVH(). This function is implemented in BSInputAdapter.jar (and also in UTSSOutputAdapter.jar) and can be invoked using StreamBase's calljava() function. After the time is converted, the output is sent to the Velocity Analytics UTSS Output Adapter for subsequent write to the configured UTSS archive.

This sample requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.

  1. Open four terminal windows on UNIX, or four StreamBase Command Prompts on Windows. In each window, navigate to your workspace copy of the sample, as described in Sample Location.

  2. In window 1, type:

    sbd -f sbd.sbconf  VWAP.sbapp
  3. In window 2, type:

    echo connect | sbc enqueue ControlStream

    This establishes a connection between the UTSS Output Adapter and the Analytics Engine.

  4. In window 3, type:

    sbc dequeue -v VWAPOutput

    This window displays the VWAP calculation output.

  5. Wait until the time window for the VWAP calculation (default 10 seconds) expires and observe the VWAP output in window 3. The same data is also written to the specified UTSS archive in UTSS output adapter properties. Use any Firebird SQL client tool to confirm that the data is continuously written to the UTSS archive while the application is running. See Using SQL with Velocity Analytics, available in the Velocity Analytics documentation set, for more information about UTSS databases.

  6. In window 4, type the following command to terminate the server and the dequeuers:

    sbadmin shutdown

TimeConversionFunctions.sbapp

This sample demonstrates the use of various time conversion functions to reformat timestamps. It requires the Velocity Analytics Broadcast Server to be installed and configured for feed replay. See Configuring Feed Replay for details. Once configured, proceed with the following steps.

  1. Open seven terminal windows on UNIX, or seven StreamBase Command Prompts on Windows. In each window, navigate to your workspace copy of the sample, as described in Sample Location.

  2. In window 1, type:

    sbd -f sbd.sbconf TimeConversionFunctions.sbapp
  3. In window 2, type:

    sbc dequeue -v GMT_ASCII

    This window displays the time field in ASCII format adjusted for GMT time zone.

  4. In window 3, type:

    sbc dequeue -v LOCAL_ASCII

    This window displays the time field in ASCII format adjusted for local time zone.

  5. In window 4, type:

    sbc dequeue -v SERIALTIME

    This window displays the time field in the Serial time format.

  6. In window 5, type:

    sbc dequeue -v SBTIMESTAMP

    This window displays the time field in the Serial time format.

  7. In window 6, type:

    sbc dequeue -v VHTIME

    This window displays the time field in VHTime format.

  8. Observe the time values in GMT_ASCII, LOCAL_ASCII, SERIAL, SBTIMESTAMP, VHTIME format in windows 2,3,4,5, and 6 respectively.

  9. In window 7, type the following command to terminate the server and the dequeuers:

    sbadmin shutdown

QueryAnalyticsEngine.sbapp

This application demonstrates the use of a StreamBase JDBC Query operator to retrieve data from the Analytics Engine time series data store using SQL. This sample requires the Analytics Engine to be installed, started, and loaded with data.

If not already running, start the Analytics Engine. Refer to Starting the Analytics Engine for details.

  1. Open three terminal windows on UNIX, or three StreamBase Command Prompts on Windows. In each window, navigate to your workspace copy of the sample, as described in Sample Location.

  2. In window 1, type:

    sbd -f sbd.sbconf  QueryAnalyticsEngine.sbapp
  3. In window 2, type:

    sbc dequeue -v OutputStream

    This window displays the query result set in the form of StreamBase Tuples.

  4. In window 3, type:

    echo MSFT.O, 20070101, 20080101 | sbc enqueue QueryParameters
  5. Observe the query result set in window 2.

  6. In window 3, type the following command to terminate the server and the dequeuers:

    sbadmin shutdown