Using the TERR Operator

Introduction

The TIBCO StreamBase® TERR operator for TIBCO Enterprise Runtime for R (TERR) allows StreamBase to use TIBCO's implementation of the R language to analyze and manipulate data.

How the Operator Locates TERR

In order to run correctly, the operator assumes that the machine running the StreamBase runtime and your application has a 64-bit version of TERR version 4.0 or later installed locally. The TERR operator has been tested and validated with TERR versions starting with 4.0.

Note

The minimum supported TERR version for macOS is 4.2.0.

The TERR bin directory does not need to be in the system PATH, and no environment variables are required for Linux and Windows. MacOS requires setting or adding to the DYLD_LIBRARY_PATH environment variable:

DYLD_LIBRARY_PATH=/Library/Frameworks/TERR.framework/Versions/version-number/
   Resources/lib/x86_64-apple-darwin

As an alternative, you can configure the <library> element in your sbd.sbconf file as follows:

 <java-vm>
   <library path="/Library/Frameworks/TERR.framework/Versions/
     version-number/Resources/lib/x86_64-apple-darwin"/>
 </java-vm>

(In both examples, the long line is broken into two for clarity. Enter these as one unbroken line.)

The operator recognizes and honors the TERR_HOME environment variable if set, and if it points to a valid local TERR installation directory. However, setting TERR_HOME is not required.

Starting with release 7.6.3, a copy of TERR Developer Edition is installed as part of your StreamBase installation. The Developer Edition edition has restricted license terms, as described on the StreamBase License Considerations page.

TERR Developer Edition is installed in the directory STREAMBASE_HOME/terr and, as a convenience, the environment variable TERR_HOME is set to this directory when using a StreamBase Command Prompt on Windows.

The operator locates the version of TERR to call, using the following formula:

  • If the option Use Embedded TERR is selected on the Operator Properties tab of the operator's Properties view, then the embedded TERR engine is used. If not selected:

    • If you specify a path to a local TERR installation in the TERR Home Path property on the Operator Properties tab of the operator's Properties view, that version of TERR is used first.

    • If that property is left blank, the operator looks for a path specified in the TERR_HOME environment variable.

This sequence lets you override the embedded TERR version with any newer or older version that your application requires.

Installing an Alternate TERR Version

To determine the version of TERR installed with StreamBase 7.6.3 or later:

On Windows, using a StreamBase Command Prompt, run:

%STREAMBASE_HOME%\terr\bin\terr --version

On Linux or macOS in a shell configured with the sbconfig --env command as described in the Installation Guide, run:

$STREAMBASE_HOME/terr/bin/terr --version

TIBCO customers can download TERR from edelivery.tibco.com, or download an evaluation copy of TERR from the TIBCO Access Point.

For Linux

TERR is only provided for 64-bit Linux. Download the tar file provided. Untar the file into a temporary local directory, and run the ./INSTALL file provided. The default installation directory is /opt/tibco/terrver, where ver is the TERR version number.

For Windows

Download the zip file provided; unzip the file to find a single installer executable. Run this installer and accept its suggested default location (C:\Program Files\TIBCO\terrver) or install into the currently recommended location (C:\TIBCO\terrver), where ver is the TERR version number.

On Windows, the TERR installer provides both 32-bit and 64-bit versions of the TERR runtime code. When run on 64-bit Windows, the 64-bit version of TERR is automatically used. Since StreamBase supports only 64-bit Windows, it uses the 64-bit version of TERR.

For macOS

Download the DMG file provided and run the installer.

To connect StreamBase and its TERR operator to your local TERR installation, you must either:

  • Set the TERR Home Path property in the Operator Properties tab of each operator's Properties view, providing the full, absolute path to the TERR installation directory.

  • Set the TERR_HOME environment variable to point to the full, absolute path of your alternate TERR installation directory. Use this method if you anticipate using many operator instances in your StreamBase applications.

How the TERR Operator Works

This operator allows a stream of tuples to be evaluated by an external TERR process, with the results returned as another stream of tuples.

The operator can instantiate multiple TERR instances to improve performance. When more than one instance is required, the tuple execution can no longer be guaranteed to be in order, as the operator now works asynchronously.

The fields in the input tuples terrVars field are converted directly to global TERR variables. The script is then run in that environment and the result variable retrieved and converted to the output tuple. This allows the script to be very short; a simple function call is sufficient as long as the function is defined in the initially loaded model. Having the values directly converted to TERR variables greatly increases both the speed of processing and the size of the input that can be processed for each tuple.

All the tuple entries that are to be read into the TERR process must be in a top level tuple named terrVars. Each element in this tuple is converted into a TERR variable.

A list of integers can be sent using the tuple (1) or (list (1, 2, 3)) or the enhanced form (tuple myInts (names = ["one", "two"], values=[1,2])). All StreamBase data types are supported with the exception of capture fields and functions. See the data conversion section of this document for more information.

Once the variables have been sent to the TERR process, the script is executed and the result is retrieved.

Using the TERR Operator

To use a TERR operator in a StreamBase EventFlow module, drag a token for the operator onto the canvas of your EventFlow Editor. Then select the newly placed operator to rename it and configure its properties.

Placing an Operator on the Canvas

The operator is a member of the Java Operators group in the Palette view in StreamBase Studio. Select the operator from the Insert an Operator or Adapter dialog. Invoke the dialog with one of the following methods:

  • Drag the Adapters, Java Operators token from the Operators and Adapters drawer of the Palette view to the canvas.

  • Click on the canvas where you want to place the operator, then invoke the keyboard shortcut O V.

  • From the top-level menu, invoke InsertOperatorJava.

When the dialog is open, enter terr in the search field to narrow the list of operators.

Properties View Settings

This section describes the properties you can set for the TERR operator, using the various tabs of the Properties view in StreamBase Studio.

In the tables in this section, the Property column shows each property name as found in the one or more adapter properties tabs of the Properties view for this adapter.

General Tab

Name: Use this field to specify or change the component's name, which must be unique in the application. The name must contain only alphabetic characters, numbers, and underscores, and no hyphens or other special characters. The first character must be alphabetic or an underscore.

Operator: A read-only field that shows the formal name of the operator.

Class: A field that shows the fully qualified class name that implements the functionality of this operator. Use this class name when loading the operator in StreamSQL programs with the APPLY JAVA statement. You can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.

Start with application: If this field is set to Yes or to a module parameter that evaluates to true, an instance of this operator starts as part of the containing StreamBase Server. If this field is set to No or to a module parameter that evaluates to false, the adapter is loaded with the server, but does not start until you send an sbadmin resume command, or until you start the component with StreamBase Manager. With this option set to No or false, the operator does not start even if the application as a whole is suspended and later resumed. The recommended setting is selected by default.

Enable Error Output Port: Select this check box to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports and Error Streams to learn about Error Ports.

Description: Optionally enter text to briefly describe the component's purpose and function. In the EventFlow canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.

Operator Properties Tab

Property Data Type Description
Use Embedded TERR Check box When enabled, the operator uses the embedded TERR engine that is bundled with StreamBase (licensed for development use only).
TERR Home Path String When not using the embedded TERR engine, you must supply the home path for the TERR installation to use. You can leave this blank if the TERR_HOME environment variable is set.
Default Script String The default script to execute. If a script is passed into the input port, it overrides this value except when doing init requests.
Enable Status Port Check box When enabled, the adapter reports data on the status port regarding various adapter states.
Log Level Drop-down list Controls the level of verbosity the adapter uses to issue informational traces to the console. This setting is independent of the containing application's overall log level. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE, and ALL.

Advanced Tab

Property Data Type Description
TERR Instances Integer The number of instances of the TERR engine to use with this adapter. NOTE: If greater than 1, the operator becomes asynchronous and tuple order is not guaranteed.
Enable Timing Check box When enabled, the result tuples produced include timing information.
To TERR Date Format String The date format to use when converting tuple data into TERR.
From TERR Date Format String The date format to use when converting TERR data into tuples.
TERR Engine Parameters String The engine parameters to send into the TERR engine.
TERR Java Home Path String The path to the Java Home to use with the TERR instance. If blank, the Java instance embedded with the StreamBase installation is used.
TERR Java Options String The engine parameters to send into the TERR engine.
TERR Instance Process Affinity Map The processor affinity to set for each instance of TERR. Instance values are matched to processors; you can specify an instance number more than once to have multiple processors.
TERR Environment Map The environment to set for each instance of TERR.

Edit Schema Tab

Use the Edit Schema tab to specify the schema of the output tuple for this adapter.

For general instructions on using the Edit Schema tab, see the Properties: Edit Schema Tab section of the Defining Input Streams page.

Use the Import proposed schemas link to import schemas as needed, for the various TERR output types. The list of importable schemas is specified in the Definitions tab of the EventFlow Editor.

Each field specified in the output schema represents an R variable that is retrieved after the R engine execution of an input tuple. The field name is used to match an R variable name to pull the values.

Initialize Tab

Use the Initialize tab to specify one or more R data objects to load at startup into each instance of TERR called by this TERR operator.

Specify one file per row in the File column, providing an optional name for the data object in the Name column. You can specify compressed or uncompressed R data object files, in plain text or serialized format, to be read by one of the read* functions in TERR. The specified files must be at the root of the Studio project folder, or in a directory along the project's resource search path. If specified files cannot be found, the result is a typecheck error.

Concurrency Tab

Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.

Caution

Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.

Use the TERR Instances property on the Advanced tab to enable parallel processing into multiple TERR instances as needed. You can still use the Concurrency tab, but it will have very little impact on performance.

Input Port

The TERR operator has a single input port to handle all interactions. The schema for this can include any field, but the following are used by the operator, with the remaining fields passed through the operator into an inputTuple field on the output stream.

Field Name Field Type Description
script string (Optional) This field contains the script to be executed by the TERR instance. If this script is null or empty and a default script was set, it will be used, unless the init flag is set.
init boolean (Optional) If true the data is executed on all TERR instances. This field is commonly used to load models and scripts with functions to be executed later.
rName string (Optional) The name of the R variable to set when loading R data objects using the rData blob input field.
rData blob (Optional) The R byte data to load. The load is completed differently depending on whether rName is provided. If rName is provided, the rData is loaded via an R command such as 'rName <- unserialize(rData)'. Otherwise the following is performed 'load(rData)'
terrVars tuple (Optional) The tuple data to convert into R variables. This field must be a tuple and each field in the tuple is converted into an R variable based on the field's schema.

Output Ports

The TERR operator has two output ports: a data port and an optional status port.

Data Port

The data port outputs the result of each call into the TERR engine. The result tuple contains two or three fields, depending on whether timing is enabled.

  • terrData — The result data pulled from TERR instance after execution, this field contains all the values specified from the Edit Schema Tab. Each subfield of the terrData field represents a variable from the TERR instance.

  • inputTuple — This tuple contains all the fields from the input tuple.

  • (Optionally) timing — This tuple contains some timing information to help gauge what might be the bottleneck in execution. The timing tuple contains the following fields:

    • eval — The time in nanoseconds it took for the TERR instance to evaluate and execute the R functions.

    • tupleToTerr — The time in nanoseconds it took to convert the input tuple into TERR data objects to send to the TERR instance.

    • terrToTuple — The time in nanoseconds it took to convert the TERR data objects from the TERR instance into the outbound tuple.

    • terrSetVariable — The time in nanoseconds it took to send the TERR data objects into the running TERR instance.

    • terrGetVariable — The time in nanoseconds it took to get the TERR data objects from the running TERR instance.

Status Port

The status port emits tuples that describe the processing status for each input tuple. It is only present when the Enable Status Port property is selected. The schema of the output tuple consists of:

Field Name Field Type Description
type String The type of report, which follows normal log levels: DEBUG, ERROR, INFO, TRACE, and WARN.
action String The action that caused the report. These can be Load R Data Objects, Init, or Execute.
object String An option object that has been affected by this status.
Message String A human-readable status message.
time Tuple The timestamp indicating when the status occurred.
inputTuple Tuple The input tuple that caused this status message. NOTE: This value is null when loading initialization data.

Data Type Conversion

This section describes how data is converted from a tuple into Terr Data objects and back again.

TERR to Tuple

This section describes how data is converted from Terr Data objects into a tuple result. Note that the best data conversion option is highlighted.

Terr Data Type StreamBase Field Types
Terr Byte (vector byte)
  • blob — converts the vector elements into a blob field

  • list(blob) — converts the vector elements into a list with a single blob element

  • list(double) — converts the vector elements to a list of doubles

  • double — converts the first vector element to a double

  • list(boolean) — converts the vector elements to a list of boolean values, with a value of 1 being true and any other value being false

  • boolean — converts the first vector element to a boolean, with a value of 1 being true and any other value being false

  • list(int) — converts the vector elements to a list of ints

  • int — converts the first vector element to a int

  • list(long) — converts the vector elements to a list of longs

  • long — converts the first vector elements to a long

  • string — converts the vector elements to a string

  • timestamp — converts the vector elements to a string and tries to parse as a timestamp value using the given simple date format from the advanced tab

  • tuple(names list(string), values list(blob)) — converts the vector elements to a tuple that contains a list of names and a list of values for each element of the vector

  • * See Terr Generic below for completely generic conversion

Terr Double (vector double)
  • list(double) — converts the vector elements to a list of doubles

  • double — converts the first vector element to a double

  • list(blob) — converts the vector elements to a list of blob each with a single byte

  • blob — converts the first vector element to a blob with a single byte

  • list(boolean) — converts the vector elements to a list of boolean values, with a value of 1 being true and any other value being false

  • boolean — converts the first vector element to a boolean, with a value of 1 being true and any other value being false

  • list(int) — converts the vector elements to a list of ints

  • int — converts the first vector element to a int

  • list(long) — converts the vector elements to a list of longs

  • long — converts the first vector element to a long

  • list(string) — converts the vector elements to a list of strings

  • string — converts the first vector element to a string

  • timestamp — converts the first vector element to a timestamp base on the double being the milliseconds from epoch, January 1, 1970 00:00:00.000 GMT

  • tuple(names list(string), values list(double)) — converts the vector elements to a tuple that contains a list of names and a list of values for each element of the vector

  • * See Terr Generic below for completely generic conversion

Terr Integer (vector integer)
  • list(int) — converts the vector elements to a list of ints

  • int — converts the first vector element to a int

  • list(double) — converts the vector elements to a list of doubles

  • double — converts the first vector element to a double

  • list(blob) — converts the vector elements to a list of blob each with a single byte

  • blob — converts the first vector element to a blob with a single byte

  • list(boolean) — converts the vector elements to a list of boolean values, with a value of 1 being true and any other value being false

  • boolean — converts the first vector element to a boolean, with a value of 1 being true and any other value being false

  • list(long) — converts the vector elements to a list of longs

  • long — converts the first vector element to a long

  • list(string) — converts the vector elements to a list of strings

  • string — converts the first vector element to a string

  • timestamp — converts the first vector element to a timestamp base on the int being the milliseconds from epoch, January 1, 1970 00:00:00.000 GMT

  • tuple(names list(string), values list(int)) — converts the vector elements to a tuple that contains a list of names and a list of values for each element of the vector

  • * See Terr Generic below for completely generic conversion

Terr String (vector string)
  • list(int) — converts the vector elements to a list of ints

  • int — converts the first vector element to a int

  • list(double) — converts the vector elements to a list of doubles

  • double — converts the first vector element to a double

  • list(blob) — converts the vector elements to a list of blob each with a single byte

  • blob — converts the first vector element to a blob with a single byte

  • list(boolean) — converts the vector elements to a list of boolean values, with a value of 1 being true and any other value being false

  • boolean — converts the first vector element to a boolean, with a value of 1 being true and any other value being false

  • list(long) — converts the vector elements to a list of longs

  • long — converts the first vector element to a long

  • list(string) — converts the vector elements to a list of strings

  • string — converts the first vector element to a string

  • timestamp — converts the first vector element to a timestamp parsed using the given simple date format from the advanced tab

  • list(timestamp) — converts the vector elements to a list of timestamps parsed using the given simple date format from the advanced tab

  • tuple(names list(string), values list(string)) — converts the vector elements to a tuple that contains a list of names and a list of values for each element of the vector

  • * See Terr Generic below for completely generic conversion

Terr Logical (vector logical)
  • list(int) — converts the vector elements to a list of ints

  • int — converts the first vector element to a int

  • For all listed below NA is converted to a null StreamBase value.

  • list(double) — converts the vector elements to a list of doubles

  • double — converts the first vector element to a double

  • list(blob) — converts the vector elements to a list of blob each with a single byte

  • blob — converts the first vector element to a blob with a single byte

  • list(boolean) — converts the vector elements to a list of boolean values

  • boolean — converts the first vector element to a boolean

  • list(long) — converts the vector elements to a list of longs

  • long — converts the first vector element to a long

  • list(string) — converts the vector elements to a list of strings

  • string — converts the first vector element to a string

  • timestamp — no conversion available

  • tuple(names list(string), values list(boolean)) — converts the vector elements to a tuple that contains a list of names and a list of values for each element of the vector

  • * See Terr Generic below for completely generic conversion

Terr Factor
  • list(double) — converts the vector elements to a list of doubles

  • double — converts the first vector element to a double

  • list(blob) — converts the vector elements to a list of blob each with a single byte

  • blob — converts the first vector element to a blob with a single byte

  • list(boolean) — converts the vector elements to a list of boolean values, with a value of 1 being true and any other value being false

  • boolean — converts the first vector element to a boolean, with a value of 1 being true and any other value being false

  • list(int) — converts the vector elements to a list of ints

  • int — converts the first vector element to a int

  • list(long) — converts the vector elements to a list of longs

  • long — converts the first vector element to a long

  • list(string) — converts the vector elements to a list of strings

  • string — converts the first vector element to a string

  • timestamp — converts the first vector element to a timestamp base on the int being the milliseconds from epoch, January 1, 1970 00:00:00.000 GMT

  • tuple(names list(string), indexes list(int), levels list(string)) — converts the vector elements to a tuple that contains a list of names, a list of indexes, and a list of levels

  • * See Terr Generic below for completely generic conversion

Terr List
  • list(x) — Terr list types are converted to a StreamBase list type. The elements inside the list determine how the conversion takes place further.

Terr DataFrame
  • tuple(x,y,z) — Terr data frame types will use the names values of the data frame to match sub fields of the tuples and convert each sub field based on the rules already listed.

  • list(x) — Each element in the list of data frames will be converted based on the statement above.

Terr Generic
  • tuple(names list(string), doubles list(tuple(names list(string), values list(double))), integers list(tuple(names list(string), values list(integer))), factors list(tuple(names list(string), indexes list(int), levels list(string))), strings list(tuple(names list(string), values list(string))), logicals list(tuple(names list(string), values list(boolean))), bytes list(tuple(names list(string), values list(blob))) ) — This is a completely generic tuple format that, if specified as the output format, converts the inbound data into the specified data types. Please note that only the names field is required for this kind of generic conversion; you can specify one or all the remaining fields. The Import proposed schemas feature of the adapter does create this full tuple for you as well.

Tuple to TERR

This section describes how data is converted from a tuple into Terr Data objects.

StreamBase Field Type Terr Data Types
boolean TerrLogical — NULL values are converted to NA values.
list(boolean) TerrLogical — NULL values are converted to NA values.
tuple(names list(string), values list(boolean)) TerrLogical — converts the list elements inside the tuple to a logical vector with names supplied.
int TerrInteger
list(int) TerrInteger
tuple(names list(string), values list(int)) TerrInteger — converts the list elements inside the tuple to a int vector with names supplied.
long TerrDouble
list(long) TerrDouble
tuple(names list(string), values list(long)) TerrDouble — converts the list elements inside the tuple to a double vector with names supplied.
double TerrDouble
list(double) TerrDouble
tuple(names list(string), values list(double)) TerrDouble — converts the list elements inside the tuple to a double vector with names supplied.
blob TerrByte
list(blob) TerrByte — All bytes from all the elements in the list are copied into a single Terr Byte
tuple(names list(string), values list(blob)) TerrByte — converts the list elements inside the tuple to a byte vector with names supplied.
string TerrString
list(string) TerrString
tuple(names list(string), values list(string)) TerrString — converts the list elements inside the tuple to a string vector with names supplied.
timestamp TerrString
list(timestamp) TerrString
tuple(names list(string), values list(timestamp)) TerrString — converts the list elements inside the tuple to a string vector with names supplied.
tuple(names list(string), indexes list(int), levels list(string)) TerrFactor — converts the list elements inside the tuple to a factor vector with names supplied.
tuple(x, y, z) TerrData (DataFrame) — Each sub field of the tuple is converted to a field in the data frame with the tuples field name being the names supplied to the TerrData objects. The object types are converted based on the rules supplied in this list.
Function Function fields not supported.
Capture Field Capture Fields are not supported.

Typecheck and Error Handling

Typechecking fails when:

  • Any required fields are not filled in.

  • The Embedded Engine property is disabled and no TERR Home is set or no TERR_HOME environment variable is found.

  • Process Affinity is not an integer greater than 0.

  • Any R data object file specified on the initialize tab cannot be located.

Suspend and Resume Behavior

On suspension, the TERR operator finishes processing the current tuple or tuples (depending on the TERR instance count), outputs the result tuples, then pauses, waiting for input.

On resumption, the TERR operator continues processing with the next input tuple.

The TERR instance or instances remain running during suspension.