Contents
These samples illustrate how to use the Python operator, and how to configure a local Python execution environment with the Python Instance operator.
The Python operator allows you to execute any valid Python code within an EventFlow module. The Python operator and its companion Python Instance operator allow Python-centric teams to reuse existing Python code in an event processing context without requiring major rewrites to that code. The Python operators allow the execution of Python-based statistical modeling, data science processing, and machine learning produced with Python packages such as SciPy and TensorFlow.
In StreamBase Studio, import this sample with the following steps:
-
From the top menu, select
→ . -
In the search field, enter
python
to narrow the list of samples. -
Select
Using the Python operator
from the Large Data Storage and Analysis category. -
Click
.
StreamBase Studio creates a single project containing the sample files.
Before running the samples, open the sample project's sbd.sbconf
file. You must edit the configuration file in two places to point to the path of your installed version of Python:
-
Edit the
<adapter-configurations>
section to specify the Python version for the global instance of Python to use in EventFlow modules as a whole. The section affects the P01-Version sample. -
Edit the
<operator-paramters>
section to specify the Python version for local instances defined by Python Instance operators. This section affects the P02-Local-Instance and P023-TensorFlow samples.
The sbd.sbconf
file for this project includes extensive comments that show example path names. You can also consult Python Versions Supported in the StreamBase Authoring Guide.
This sample demonstrates performing a very basic Python call and returning a result.
-
Make sure you have edited this sample's configuration file to specify the path to the Python version you want to use.
-
In the Package Explorer view, double-click to open
P01-version.sbapp
. Make sure the application is the currently active tab in the EventFlow Editor. -
Click the Run button. This opens the SB Test/Debug perspective and starts the application.
-
The sample runs itself one time on startup by means of a Once adapter. Look in the Application Output view for a report of the configured Python source and version number.
-
To run the sample again, click Manual Input view. This sends an empty tuple that populates another row in the Application Output view.
in the -
When done, press F9 or click the Stop Running Application button.
This sample demonstrates two things:
-
Asking Python to perform a simple calculation of the Fibonacci numbers that are less than a provided integer.
-
Making that Python calculation in a local instance defined by a Python Instance operator.
Note
On macOS, this sample runs with pypy3 only if you set LANG=en_US.UTF-8
in the Environment tab of the Run Configuration for this module. (Or set LANG to the appropriate value for your locale.)
Follow these steps:
-
Make sure you have edited this sample's configuration file to specify the path to the Python version you want to use.
-
In the Package Explorer view, double-click to open
P02-local-instance.sbapp
. Make sure the application is the currently active tab in the EventFlow Editor. -
Click the Run button. This opens the SB Test/Debug perspective and starts the application.
-
In the Manual Input view, the
InputStream
stream is preselected.Enter any integer into the
fib
field, such as90
,500
, or1100
.Click
. -
In the Application Output view, look for an OutputStream tuple that show a list of the members of the Fibonacci series that are less than the integer you sent.
-
Enter another integer in the
fib
input field and press . -
Now, in the Manual Input view, select the Input Stream named
zControl
. -
Enter
stop
in the command field and press . -
Select the
InputStream
stream again, and send another integer to Python. This time, the result is an Adapter Exception error message.This is because the
stop
command disabled the local Python execution environment defined by the Python Instance operator. Notice that the name of this operator isThisInstance
, and notice that in the Properties view for the Python operator that its Instance Type is Local and its Local Instance Id isThisInstance
.This means the Python operator is configured to operate in the local execution environment named
ThisInstance
, but in steps 7 and 8, you disabledThisInstance
. -
Re-enable the local execution environment by selecting the
zControl
input stream and sending the commandstart
. -
Now return to the
InputStream
stream and send in more integers, which succeed this time. -
When done, press F9 or click the Stop Running Application button.
This sample demonstrates using Google's open source TensorFlow machine learning framework to run a model in multiple thread instances, to evaluate model input, and to receive model output. This sample requires that the Python installation you wish to use is configured with the TensorFlow and OpenCV-Python packages. In addition, there are limitations on the Python vendors and versions you can use with TensorFlow and with this sample.
The S03-TensorFlow sample was tested with TensorFlow 1.13.1 on Windows, macOS, and Linux.
The TensorFlow package, and also therefore the P03-TensorFlow sample, has the following limitations:
-
On Windows, TensorFlow requires Python 3.5 or 3.6. It is not supported with Python 2.7 on Windows.
-
On Windows, ActiveState provides both Python 3.5 and 3.6 installers, but recommends using their Python 3.5 installer.
-
The P03-TensorFlow sample does not run with ActiveState Python 3.6.
-
ActiveState Python 3.5 installs a large set of data science Python packages, including TensorFlow and OpenCV-Python. However, these bundled packages are old versions and must be upgraded with commands like the following:
pip3 install --update tensorflow pip3 install --update opencv-python
-
-
On Linux and macOS, TensorFlow supports Python 2.7, 3.4, 3.5, or 3.6.
-
This sample does not run with Pypy because no Pypy-compatible TensorFlow package is available at the time of this writing in April, 2019.
The usual way to install Python packages is with the pip or pip3 command. Make sure you have installed a compatible pip command to match your Python version and vendor, as described in Python Versions Supported in the StreamBase Authoring Guide.
With standard installations of Python and Pip as described in that section, Windows and macOS users should need only the following commands:
pip3 install tensorflow pip3 install opencv-python
On Linux, preface the same commands with sudo -H
:
sudo -H pip install tensorflow sudo -H pip install opencv-python
Use the pip or pip3 command associated with your Python version to confirm that these packages were installed for that Python version:
pip3 list
or
pip list
The P03-TensorFlow sample uses a worker module to allow eight operations to run simultaneously. The worker module is run with
a StreamBase concurrency setting of multiplicity=8
. Each simultaneous module has its own instance of Python and initializes the model within each instance. This occurs in worker-only.sbapp
, which does not run on its own, but only when called by P03-tensorflow.sbapp
.
-
Make sure you have edited this sample's configuration file to specify the path to the Python version you want to use.
-
In the Package Explorer view, double-click to open
P03-tensorflow.sbapp
. Make sure the module is the currently active tab in the EventFlow Editor. -
Click the Run button. This opens the SB Test/Debug perspective and starts the module, which is self-running by means of a Once adapter.
-
As the server starts, StreamBase Studio switches to the SB Test/Debug perspective.
-
In the Application Output view, select the stream named
OutputStream
in the Stream control. After a moment of operation, this stream shows a list of image files and their result strings. -
If you wish to rerun the model processing, send any tuple (including a null tuple) on the
Restart
input stream, and click . -
When done, press F9 or click the Terminate EventFlow Fragment button.
This sample demonstrates three ways to load the Python script:
-
From Python file.
-
From script text box.
-
Enable control port and load script when running.
Follow these steps:
-
Edit this sample's configuration file to specify the path to the Python version you want to use.
-
In the Package Explorer view, double-click to open
P04-script-resource.sbapp
. Make sure the application is the currently active tab in the EventFlow Editor. -
Double-click the Python adapter and open the Script tab. The
Script text
is checked as Script source. -
Save the module and click the Run button. This opens the SB Test/Debug perspective and starts the application.
-
In the Manual Input view, Use the Stream control to select the
InputStream
stream.Enter any integer into the
fib
field, such as90
,500
, or1100
.Click
. -
In the Application Output view, select the result tuple to see it in the details section below. Look for a tuple field named
fibresult
that show a list of the members of the Fibonacci series that are less than the integer you sent. Notice that this field is in the top row of three, with the other two fields empty. -
Press F9 or click the Stop Running Application button. (We do this to clear the Application Output view of previous results.)
-
Double-click the Python adapter icon and open the Script tab. In the Script source control, select
File
.In the Script file control, make sure
fib.py
is selected. If not, use the button to search the resource path for this file. -
Save the module and click the Run button again.
-
In the Manual Input view, Select the
InputStream
stream, enter an integer into thefib
field, and click . -
In the Application Output view, look for an output tuple field named
fibresultFromFile
that shows the Fibonacci series again. Notice that this output fills the middle field this time, with top and bottom fields empty. -
Press F9 or click the Stop Running Application button.
-
Double-click the Python adapter icon again. With either
File
orScript text
selected, make sure Enable control port is selected. (When this check box is enabled and the control port is used, it overrides either setting of Script source.) -
Save the module and click the Run button again.
-
To take advantage of the control stream, you must send a replacement script to the running module. To do this, in the Manual Input view, select
yControlStream
from the Stream control. Then:-
In the Command field, enter
Load
. -
In the Script field, copy and paste the following script. Be careful not to change the indent levels.
from __future__ import print_function def fib2(n): result = [] a, b = 0, 1 while b < n: result.append(b) a, b = b, a+b return result fibresultFromControlStream = fib2(fib) print(fibresultFromControlStream) print('')
Once pasted, you can only see the top line of the script. Use the up and down arrow keys to scroll through the pasted script in the
Script
control to make sure the line breaks and indentation pasted correctly. -
Click Send Data. There is no feedback for this step.
-
-
In the Manual Input view, select the
InputStream
stream, enter an integer into thefib
field, and click . -
In the Application Output view, look for a tuple this time named
fibresultFromControlStream
that shows the results of the Fibonacci calculation as before. This time, the output tuple fills the bottom field of the three, with the top and middle fields empty. -
When done, press F9 or click the Stop Running Application button.
When you load the sample into StreamBase Studio, Studio copies the sample project's files to your Studio workspace, which is normally part of your home directory, with full access rights.
Important
Load this sample in StreamBase Studio, and thereafter use the Studio workspace copy of the sample to run and test it, even when running from the command prompt.
Using the workspace copy of the sample avoids permission problems. The default workspace location for this sample is:
studio-workspace
/sample_python
See Default Installation Directories for the default location of studio-workspace
on your system.