Python Operator Samples

About the Samples

These samples illustrate how to use the Python operator, and how to configure a local Python execution environment with the Python Instance operator.

The Python operator allows you to execute any valid Python code within an EventFlow module. The Python operator and its companion Python Instance operator allow Python-centric teams to reuse existing Python code in an event processing context without requiring major rewrites to that code. The Python operators allow the execution of Python-based statistical modeling, data science processing, and machine learning produced with Python packages such as SciPy and TensorFlow.

Importing This Sample into StreamBase Studio

In StreamBase Studio, import this sample with the following steps:

  • From the top-level menu, select File>Load StreamBase Sample.

  • Enter python to narrow the list of options.

  • Select Using the Python operator from the Large Data Storage and Analysis category.

  • Click OK.

StreamBase Studio creates a single project containing the sample files.

Samples Setup

Before running the samples, you must edit two configuration files to specify the full path to the Python executable of your installed and preferred version of Python:

  • Edit src/main/resrouces/adapter-configurations.xml to specify the Python version for the global instance of Python to use in EventFlow modules as a whole. The file affects the P01-Version sample.

  • Edit src/main/configurations/sbengine.xml to specify the Python version for local instances defined by Python Instance operators. This file affects the P02-Local-Instance and P03-TensorFlow samples.

Example Python executable paths for supported Python versions are shown in Python Versions Supported in the StreamBase Authoring Guide.

Running the P01-Version Sample

This sample demonstrates performing a very basic Python call and returning a result.

  1. Make sure you have edited this sample's adapter-configurations.xml configuration file to specify the path to the Python version you want to use for the global instance.

  2. In the Project Explorer view, open the sample you just loaded.

    If you see red marks on a project folder, wait a moment for the project to load its features.

    If the red marks do not resolve themselves after a minute, select the project, right-click, and select Maven>Update Project from the context menu.

  3. Open the src/main/eventflow/packageName folder.

  4. Open the P01-version.sbapp file and click the Run button. This opens the SB Test/Debug perspective and starts the module.

  5. The sample runs itself one time on startup by means of a Once adapter. Look in the Output Streams view for a report of the configured Python source and version number.

  6. To run the sample again, click Send Data in the Manual Input view. This sends an empty tuple that populates another row in the Output Streams view.

  7. When done, press F9 or click the Terminate EventFlow Fragment button.

Running the P02-Local-Instance Sample

This sample demonstrates two things:

  • Asking Python to perform a simple calculation of the Fibonacci numbers that are less than a provided integer.

  • Making that Python calculation in a local instance defined by a Python Instance operator.

Note

On macOS, this sample runs with pypy or pypy3 only if you set LANG=en_US.UTF-8 in the Environment tab of the Run Configuration for this module. (Or set LANG to the appropriate value for your locale.)

Follow these steps:

  1. Make sure you have edited this sample's sbengine.xml configuration file to specify the path to the Python version you want to use for this sample's local instance.

  2. In the Project Explorer view, open the sample you just loaded.

    If you see red marks on a project folder, wait a moment for the project to load its features.

    If the red marks do not resolve themselves after a minute, select the project, right-click, and select Maven>Update Project from the context menu.

  3. Open the src/main/eventflow/packageName folder

  4. Open the P02-local-instance.sbapp file and click the Run button. This opens the SB Test/Debug perspective and starts the module.

  5. In the Manual Input view, the InputStream stream is preselected.

    Enter any integer into the fib field, such as 100, 3000, or 11000.

    Click Send Data.

  6. In the Output Streams view, look for an OutputStream tuple that show a list of the members of the Fibonacci series that are less than the integer you sent.

  7. Enter another integer in the fib input field and press Send Data.

  8. Now, in the Manual Input view, select the Input Stream named zControl.

  9. Enter stop in the command field and press Send Data.

  10. Select the InputStream stream again, and send another integer to Python. This time, the result is an Adapter Exception error message.

    This is because the stop command disabled the local Python execution environment defined by the Python Instance operator. Notice that the name of this operator is ThisInstance, and notice that in the Properties view for the Python operator that its Instance Type is Local and its Local Instance Id is ThisInstance.

    This means the Python operator is configured to operate in the local execution environment named ThisInstance, but in steps 7 and 8, you disabled ThisInstance.

  11. Re-enable the local execution environment by selecting the zControl input stream and sending the command start.

  12. Now return to the InputStream stream and send in more integers, which succeed this time.

  13. When done, press F9 or click the Terminate EventFlow Fragment button.

Running the P03-TensorFlow Sample

This sample demonstrates using Google's open source TensorFlow machine learning framework to run a model in multiple thread instances, to evaluate model input, and to receive model output. This sample requires that the Python installation you wish to use is configured with the TensorFlow and OpenCV-Python packages. In addition, there are limitations on the Python vendors and versions you can use with TensorFlow and with this sample.

The S03-TensorFlow sample was tested with TensorFlow 1.6.0 and 1.7.0 on Windows, macOS, and Linux. On all platforms, the 1.7.0 release shows Console view warnings about deprecated modules, but otherwise, the two versions run the same.

TensorFlow Limitations

The TensorFlow package, and also therefore this sample, has the following limitations:

  • On Windows, TensorFlow requires Python 3.5 or 3.6. It is not supported with Python 2.7 on Windows.

  • On Windows, ActiveState provides both Python 3.5 and 3.6 installers, but recommends using their Python 3.5 installer.

    • The P03-TensorFlow sample does not run with ActiveState Python 3.6.

    • ActiveState Python 3.5 installs a large set of data science Python packages, including TensorFlow and OpenCV-Python. However, these bundled packages are old versions and must be upgraded with commands like the following:

      pip3 install --update tensorflow
      pip3 install --update opencv-python
  • On macOS and Linux, the TensorFlow web site claims support for TensorFlow as a Python 2.7 package. However, the pip commands associated with Python 2.7 on these platforms do not locate or download the TensorFlow packages. You may be able to manually install TensorFlow for use with Python 2.7.

  • This sample does not run with Pypy because no Pypy-compatible TensorFlow package is available.

Installing TensorFlow and OpenCV-Python Packages

The usual way to install Python packages is with the pip or pip3 command. Make sure you have installed a compatible pip command to match your Python version and vendor, as described in Python Versions Supported in the StreamBase Authoring Guide.

With standard installations of Python and Pip as described in that section, Windows and macOS users should need only the following commands:

pip3 install tensorflow
pip3 install opencv-python

On Linux, preface the same commands with sudo -H:

sudo -H pip install tensorflow
sudo -H pip install opencv-python

Use the pip or pip3 command associated with your Python version to confirm that these packages were installed for that Python version:

pip3 list

or

pip list

Steps to Run

The P03-TensorFlow sample uses a worker module to allow eight operations to run simultaneously. The worker module is run with a StreamBase concurrency setting of multiplicity=8. Each simultaneous module has its own instance of Python and initializes the model within each instance. This occurs in worker-only.sbapp, which does not run on its own, but only when called by P03-tensorflow.sbapp.

  1. In the Project Explorer view, open the sample you just loaded.

    If you see red marks on a project folder, wait a moment for the project to load its features.

    If the red marks do not resolve themselves after a minute, select the project, right-click, and select Maven>Update Project from the context menu.

  2. Open the src/main/eventflow/packageName folder.

  3. Open the P03-tensorflow.sbapp file and click the Run button. This opens the SB Test/Debug perspective and starts the module, which is self-running by means of a Once adapter.

  4. In the Output Streams view, select the stream named OutputStream in the Stream control. After a moment of operation, this stream shows a list of image files and their result strings.

  5. If you wish to rerun the model processing, send any tuple (including a null tuple) on the Restart input stream, and click Send Data.

  6. When done, press F9 or click the Terminate EventFlow Fragment button.

Sample Location

When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.

Important

Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.

Using the workspace copy of the sample avoids permission problems. The default workspace location for this sample is:

studio-workspace/sample_python

See Default Installation Directories for the default location of studio-workspace on your system.