Using the Python Operator

Contents

Introduction
Python and Python Instance Operators
Python Compatibility
Data Type Conversion
Configuring a Global Python Instance
Python Operator Properties
Input and Output Port
Python Versions Supported

Disclaimer

Artifact Management Server (AMS) is removed in Spotfire® Streaming 11.1.x.

Introduction

  The Python operator allows you to run any valid Python code within a StreamBase module. The Python operator and its companion Python Instance operator allow Python-centric teams to reuse existing Python code in an event processing context without requiring major rewrites to that code. The Python operators allow the execution of Python-based statistical modeling, data science processing, and machine learning produced with Python packages such as SciPy and TensorFlow.

The Python operators do not turn the StreamBase EventFlow language into a Python interpreter. Instead, Python stateful sessions managed by an external Python interpreter are attached as child processes to EventFlow modules. The Python operators interact with these sessions by setting input variables, running the script, and reading output variables, potentially emitting Python results as fields of StreamBase tuples. The Python operator guarantees that these three Python management operations are executed sequentially, even if there are multiple operator instances touching the same session, or if the operator is running in asynchronous mode.

The Python operators support several Python interpreters compliant with Python 2.7 or 3.4+. The Python script provided to a particular Python operator must be compliant with the Python interpreter configured for the containing EventFlow application to use. This means that the Python script can only use the libraries and language structures available to the configured Python interpreter. The operator treats the script code as opaque and does not attempt to parse or compile it before sending it to the interpreter. Thus, all the power of the configured interpreter's Python libraries is accessible from the script.

StreamBase includes a set of samples of using the Python and Python Instance operators, described in Python Operator Samples.

Python and Python Instance Operators

Use the Python operator to run Python code and to optionally emit the results of that execution.

By contrast, you use the Python Instance operator to specify and configure the name and features of a constrained local instance execution environment for the set of Python operators that you configure to use that name and environment.

By default, Python operators run in the global execution environment of the containing EventFlow module. However, it is possible to use EventFlow concurrency features to run multiple copies of a single module that contains Python operators. In this case, you can use a Python Instance operator to define and name a local execution environment for the Python operators in that concurrent module. In this way, the execution of Python in one concurrent instance of that module does not interfere with other instances.

You can also configure a Python Instance operator with a control port that lets you enable and disable at runtime the execution of Python operators configured to use that Python instance.

Python Compatibility

The Python operator integration layer uses a minimal set of features from Python 2.7 and Python 3.4+, requiring the pickle library and TCP/IP networking. The constructs used are compatible with Python 2.7 and 3.4+.

The following table provides a summary of the Python interpreters supported by the Python operator. See Python Versions Supported for detailed information for each operating system.

Runtime Version Equivalent to ...
Python 2 2.7.x. Requires 2.7.15 and above for Numpy dtype support. —
Python 3 3.4 or newer —
PyPy 5.x Python 2.7
PyPy 7.x Python 2.7, 3.5, or 3.6

Data Type Conversion

The Python operator's input tuple can include a single field of type tuple whose name is inputVars. The type of data passed to the Python script from the inputVars tuple is inferred from the data type of each field.

StreamBase type to Python
boolean truth
int int
double float
string unicode (Python2), str (Python3)
timestamp datetime.datetime (absolute), datetime.timedelta (interval)
blob bytes
list list
tuple dict
capture not supported
function not supported

The operator optionally emits a field of type tuple named outputVars. When you define the data type for the outputVars tuple fields, the operator makes its best effort to cast the Python objects to StreamBase data types. The following table summarizes the data type conversions.

from Python StreamBase type
truth, int, float boolean
truth, int, float int
truth, int, float double
str, bytes, bytearray, Unicode (Python2) string
datetime.datetime, datetime.date, datetime.time (absolute), datetime.timedelta (interval) timestamp
bytes, bytearray blob
list, tuple, array.array, materialized generator (list) list
dict tuple
numpy.dtype.int8, numpy.dtype.int16, numpy.dtype.int32 int
numpy.dtype.int64 long
numpy.dtype.bool boolean
numpy.dtype.str String
numpy.dtype.float16, numpy.dtype.float32, numpy.dtype.float64 double (possible data loss for huge numbers)
numpy.dtype.uint8, numpy.dtype.uint16, numpy.dtype.uint32, numpy.dtype.uint64 long (possible data loss for huge numbers)
unsupported capture
unsupported function

To use any of the dtype data types, you must install the Numpy Python package, and must be using Python 2.7.15 and above, or Python 3.4+.

Configuring a Global Python Instance

The global Python instance is the default, module-scoped execution environment in which all Python operators in a module perform their operations. The global Python instance environment is configured in the module's configuration file, src/main/resources/adapter-configurations.xml.

By contrast, the Python Instance operator is configured using the operator's Properties view on the EventFlow canvas.

Use a configuration file of type com.tibco.ep.streambase.configuration.adapter to define the parameters of the module's global Python environment. The following table describes the supported properties.

Property Type Default Description
instance string   Provides an arbitrary name for the global instance. This name is displayed in the dropdown list for the Global Instance ID control in the Properties view for both Python and Python Instance operators. This name links the two operators together when using a Python Instance operator.
executable string python Specifies the full path to the Python executable. Use forward slash path separators, even on Windows. If not specified, the operator invokes the command python, which is assumed to be on the PATH. This default Python command is not likely to be the same if you develop on Windows or Mac and deploy on Linux, so be sure to specify the exact path to the Python interpreter you intend to use for each platform. See Python Versions Supported below for example paths.
useTempFile boolean false This flag indicates that the operator's integration layer should create a temporary file with Python code wrapping the interactions with StreamBase, instead of pushing it through standard input. Using standard input works for most Python interpreters and is the default.
captureOutput boolean false Modifies the stdout and stderr behavior of the operator. By default, both are chained to the parent process's stdout and stderr. For running tests that include output, you can set this flag to capture the output.
workingDir string . Specifies the working directory for the launched process. When not specified, the process starts in the same directory as the parent StreamBase process.
envVariables section/ setting   Specifies environment variables to be passed before launching the Python interpreter, potentially overriding variables in the platform's environment. You can use more than one <setting> line.
arguments section/ setting   Specifies arguments to be passed to the Python interpreter (not the Python script), and can be defined multiple times. The usual use for this property it to pass -u, which forces Python to use unbuffered stdin, stdout, and stderr streams. Consult the following references for information on Python launch parameters:

The following shows an example Python.conf file for a global instance named pythonic that uses Python 2.7 on Windows. 

name = "Python.conf"
type = "com.tibco.ep.streambase.configuration.adapter"
version = "1.0.0"
configuration = {
        
// An adapter group type defines a collection of EventFlow adapter configurations, indexed by adapter type.
  AdapterGroup = {
        
// A collection of EventFlow adapter configurations, indexed by adapter type. This object is required
// and must contain at least one configuration.
    adapters = {
        
// The root section for an EventFlow adapter configuration.
      python = {
        
// Section list. This array is optional and has no default value.
        sections = [
        
// A configuration for an EventFlow adapter named section.
          {
        
// Section name. The value does not have to be unique; that is, you can have multiple sections with
// the same name in the same array of sections. This property is required.
            name = "python"
        
// Section for setting adapter properties. All values must be strings. This object is optional
// and has no default value.
              settings = {
                executable = "C:/Python27/python.exe"
                instance = "pythonic"
                useTempFile = "false"
                workingDir = "."
                captureOutput = "false"
            }
          }
        ]
      }
    }
  }
}

Python Operator Properties

This section describes the properties you can set for the Python operator, using the various tabs of the Properties view in StreamBase Studio.

General Tab

Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.

Operator: A read-only field that shows the formal name of the operator.

Class name: Shows the fully qualified class name that implements the functionality of this operator. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.

Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this operator starts.

Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.

Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.

Operator Properties Tab

Property Type Description
Python Configuration Edit Button Shortcut to the StreamBase Configuration File Editor, used for adapter configuration or converting an existing application's adapter-configurations.xml file to HOCON format.
Instance Type radio button Select Global to specify that this operator executes in the global execution environment of the containing EventFlow module, which is defined in the module's configuration file.

Select Local to specify that this operator executes in the constrained environment defined by a Python Instance operator present in this EventFlow module.

Local Instance ID text Only active when Instance Type specifies Local. In this case, enter the canvas name of a Python Instance operator in this EventFlow module that defines a local Python execution environment.
Global Instance ID drop-down list Only active when Instance Type specifies Global. In this case, the global Python execution environment is defined in this module's configuration file. Use the dropdown control to select the name of a global instance defined in configuration.
Asynchronous check box When selected, the operator executes the Python script using a non-blocking call. This allows long operations to be executed without suspending the processing of the containing module. Make sure module invariants are preserved around the call. Note that, in contrast to the concurrent parallel execution feature of StreamBase, this operator does not allocate additional threads and uses lightweight job scheduling.
Log Level INFO Controls the level of verbosity the adapter uses to issue informational traces to the console. This setting is independent of the containing application's overall log level. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE.

Script Tab

Property Type Description
Enable control port check box When selected, the Python operator gains a control port on the canvas. Connect an Input Stream to this port whose schema contains two string fields named Command and Script. When the control port exists, a new Python script can be uploaded to a running Python operator, replacing a script loaded at startup using either the File or Script text options described next.

To use the control port, send in a tuple with the Command field containing the single supported command, Load, and send an entire Python script in the Script field, with line breaks and indentation correctly honored.

When testing the control port with the Manual Input view in StreamBase Studio, you can copy a script and paste it into the single Script field. This appears to accept only one line, but the entire script is actually accepted. Use up-arrow and down arrow (or Ctrl+up and down-arrow) to see the separate lines of the pasted script.

Script source radio button Select File to specify that the initial executable script comes from the specified Python file.

Select Script text to specify using the code specified in the Script field as the initial executable script.

Script file resource chooser Choose a valid Python file to load. Use the Choose button to browse the current module's Resource Search Path for Python files. This browse dialog opens in the project's src/main/resources folder.
Script multiline text Specifies a script of known working Python code to be executed for each incoming tuple.

Output Tab

Property Type Description
Output variables schema definition Definition for the expected output variables. Each field defined for the schema corresponds to a Python session variable expected to be stored by this operator's script, or any previous call. The output variables must be of a type castable to a StreamBase field type, as shown in Data Type Conversion above.

AMS Tab

Use the AMS tab to specify that an artifact, such as a model file, is to be pulled from a running TIBCO Artifact Management Server, which is a separately installed product available from Spotfire Streaming download sites.

Note

If you deploy an artifact from the AMS system, it first checks your list of artifacts to match the specified path. If matched, AMS uses the specified model name. If the path is not matched, then the artifact's file basename is used. For example, pytf/audit.index would resolve to a model name of audit.

Property Type Description
Required On Startup check box When enabled, the specified artifacts are requested from AMS at initialization of the EventFlow module that contains this Python operator, and the system waits until all artifacts are loaded.
Artifact's Name String The first value of the path is the AMS project name followed by the full path to the artifact. For example: pythontf/checkpoint-1999.index
Artifact's Version int If a version is not specified, the latest version is assumed.

Cluster Aware Tab

Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.

Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.

Input and Output Port

The input port transparently accepts any incoming tuple, which can optionally contain a field of type tuple with the reserved name inputVars. The outgoing tuple contains a field of type tuple named outputVars.

  • inputVars — optional tuple containing variables to be set in the Python session.

  • outputVars — tuple whose structure is defined in the Output tab of the Properties view, containing variables read from the Python session.

All other incoming fields are transparently passed. The inputVars field is not propagated to the output. The outputVars field is not allowed in the input port.

Python Versions Supported

This section describes the Python interpreters tested for use with the Spotfire Streaming Python operators. Python versions are described as of April, 2019.

In general:

  • Your system can have as many Python versions, from as many vendors, as you require, as long as the name or path to the Python executables are different. Thus, a single system can have python, python3, pypy, and pypy3 installed at the same time, if necessary.

  • The topmost Python in your shell's PATH is the dominant Python version for Python called at the shell prompt.

  • However, the Spotfire Streaming Python operators use the Python version specified in the adapter's configuration files, src/main/resources/adapter-configurations.xml or src/main/configurations/sbengine.conf.

  • The Python operator only calls for Python with the name python on the system PATH as a fallback, if there is no Python version specified in configuration.

  • For Windows, use forward slashes in path names.

Windows: Python.org

Download 64-bit Python 2.7 or Python 3.7 from python.org and run the installer provided. Users can also download Anaconda's python distribution for Python 3.7 from https://repo.anaconda.com/archive/Anaconda3-2020.07-Windows-x86_64.exe. Python.org provides:

Python Command Version Installation path to specify in configuration files Notes
python.exe 2.7.16 C:/Python27/python.exe  
python.exe 3.7.3 C:/Program Files/Python37/python.exe When installed for all users using the Advanced options.
python.exe 3.7.3 C:/Users/sbuser/AppData/Local/Programs/Python/Python37/python.exe When installed for the current user.

(Replace sbuser in the Python 3 path with your Windows login name.)

Pythons from python.org come with a minimal set of included Python packages. This means there is less you need to update, compared to ActiveState, but also means you have the freedom to install only the packages you need. Both Python versions include pip and/or pip3 commands.

Windows: Activestate Python

Download Python 2.7 or Python 3 from activestate.com. Activestate includes the pip and/or pip3 commands, and a large collection of Python packages with their Python downloads, including many data science packages. At this writing, Activestate encourages the use of their Python 3.7.4 edition for the best data science support. They do provide a Python 3.7 installer, but it does not include the full set of packages provided by their 3.6 installer.

Download the most recent Anaconda installer that includes Python 3.5 (Anaconda 4.2.0) or Python 3.6 (Anaconda 5.2.0).

Pypy Command Version Installation path to specify in configuration files
python 2.7.14 C:/Python27/python.exe (or python2.7.exe or python2.exe)
python3 3.5.4 C:/Python35/python.exe (or python3.5.exe)
python3 3.6.6 C:/Python36/python.exe (or python3.6.exe)
python3 3.7.4 C:/Python37/python.exe (or python3.7.exe)

Because Activestate bundles data science packages with their installer, including TensorFlow and OpenCV, you must update those packages to the latest release after installing Activestate Python 3.5. Use commands like the following: 

python3 -m pip install --update tensorflow
python3 -m pip install --update opencv-python

Windows: Pypy

Pypy is an alternative Python implementation that claims a significant speed advantage over CPython-based Python implementations. Download either a Python 2.7 or 3.x equivalent from pypy.org. Only 32-bit editions are available for Windows. Pypy provides:

Python Command Version Python Equivalent Installation path to specify in configuration files
pypy 7.1.0 2.7.13 Delivered as zip files with no default installation path. Unzip the contents and arrange PATH and PYTHONPATH as you require. For example:

  • C:/Pypy2/pypy.exe

  • C:/Pypy3/pypy3.exe

pypy3 7.0.0 3.5.3
pypy3 7.1.0 3.6.1

There are the pip and pip3 equivalents specific for Pypy; getting those installed and using them to download packages to pypy's site-packages directory can be daunting.

MacOS: Default Python 2

MacOS ships with Python 2.7.10 as of macOS Sierra 10.12, High Sierra 10.13, and Mojave 10.14. You can install pip for the macOS Python 2.7 with the following command: 

sudo /usr/bin/easy_install pip

Use the following path to specify the macOS-shipped Python in the Spotfire Streaming Python operator's configuration files.

Python Command Version Installation path to specify in configuration files
python 2.7.10 /usr/bin/python

MacOS versions since 10.11 El Capitan have included a feature called System Integrity Protection, or SIP. SIP restricts the ability of processes to replace, upgrade, or overwrite commands that ship with macOS, and that includes /usr/bin/python. SIP also makes it difficult to use pip to install or upgrade certain packages already in place as part of the macOS-provided Python 2.7.10 installation.

This feature can prevent you from installing complex data science packages such as TensorFlow, which has many dependencies. The best workarounds are to either use Homebrew's Python 3 with TensorFlow, or to install Homebrew's python2 package to bypass the Python 2 installation included in macOS.

MacOS: Python from Homebrew

You can obtain Python from Homebrew. Use the following command to install Python 3; at this writing, the latest version is 3.7.4: 

brew install python

Use this command to install Homebrew's alternative Python 2: 

brew install python@2

Use the following paths to specify the Homebrew-installed Python in the Spotfire Streaming Python operator's configuration files.

Python Command Version Installation path to specify in configuration files
python or python2 2.7.16 /usr/local/bin/python (or python2)
python3 3.7.4 /usr/local/bin/python3

Download the most recent Anaconda installer that includes Python 3.5 (Anaconda 4.2.0) or Python 3.6 (Anaconda 5.2.0).

Recent versions of Homebrew's Python 3 includes the pip3 command. If you installed an earlier version of Homebrew's Python 3 but do not have pip3, run these commands: 

brew update
brew upgrade
brew postinstall python3

MacOS: Pypy from Homebrew

Homebrew makes Pypy available for macOS in two releases installed with these commands: 

brew install pypy
brew install pypy3

Use the following paths to specify the Homebrew-installed Pypy in the Spotfire Streaming Python operator's configuration files.

Python Command Version Python Equivalent Installation path to specify in configuration files
pypy 7.1.0 2.7.13 /usr/local/bin/pypy
pypy3 7.0.0 3.6.1 /usr/local/bin/pypy3

Each Pypy version from Homebrew installs with its own pip_pypy or pip_pypy3 command.

To use Pypy on macOS, make sure you have configured your locale setting correctly, either in the shell environment inherited by Studio, or explicitly set in the Environment tab of the Run Configuration for any EventFlow module that includes a Python operator. In particular, you may need to set the LANG environment variable equal to en_US.UTF-8 (or to the equivalent setting for your locale). See locale(1) and the LANG environment variable in your platform's reference documentation.