Using the External Process Operator

This topic explains how to use the External Process operator and describes the configuration settings you can make in the operator's Properties view.

Introduction

The External Process operator is a Java operator that provides a way for StreamBase applications to run arbitrary operating system commands as if typed at the shell command prompt for the current operating system. This feature is especially useful in StreamBase high availability design patterns, where an application in one container might need to send an sbadmin command to an application in another container or on another StreamBase.

The External Process operator is a member of the Java Operator group in the Palette view in StreamBase Studio. Select the External Process operator from the Insert an Operator or Adapter dialog, which you invoke with one of the following methods:

  • Drag the Adapters, Java Operators token from the Operators and Adapters drawer of the Palette view to the canvas.

  • Click in the canvas where you want to place the operator, and invoke the keyboard shortcut O V

  • From the top-level menu, invoke Insert>Operator>Java.

From the Insert an Operator or Adapter dialog that opens, select External Process and double-click or press OK.

Running External Commands

The External Process operator incurs operating system overhead and should not be considered for any portion of a StreamBase application that requires fast performance. The operator should never be called repeatedly to perform a task. This operator is designed to be called on rare occasions, such as in a failover event in a highly available StreamBase Server cluster.

You can use this operator to run any command on the PATH of the StreamBase Server instance that is running the operator. You can specify command switches, command arguments, and environment variables for the command you want to run, and you can modify the switches, arguments, and environment variables at runtime while calling the specified command. You can specify variables as part of the command to be run, and have those variables resolved at runtime.

Caution

This operator has the same permissions as the containing StreamBase Server to run any operating system command in the PATH and environment of that server, and by default, the working directory of the operator is the same as the working directory of the containing server. Use caution in specifying the command for this operator to run.

The specified operating system command must be locatable on the PATH of the containing StreamBase Server, or it can be specified by absolute path or a path relative to the containing StreamBase Server's current working directory.

Properties View Settings

This section describes the properties you can set for an External Process operator, using the various tabs of the Properties view in StreamBase Studio.

General Tab

This section describes the properties on the General tab in the Properties view for the External Process operator.

Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.

Operator: A read-only field that shows the formal name of the operator.

Class name: Shows the fully qualified class name that implements the functionality of this operator. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.

Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this operator starts.

Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.

Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.

Command Arguments Tab

This section describes the properties on the Command Arguments tab in the Properties view for the External Process operator.

Operator Type

Select either CommandLine or Arguments to specify how you will enter the command to be run by this operator. Select CommandLine to enter the command as a StreamBase expression. Specify Arguments to enter the command in table form, with one row for each command, switch, and option.

Command line as expression

If you selected CommandLine as the Operator Type, use this field to specify the full command to be run, expressed in the form of a StreamBase expression. You must surround the entire command line with single quotes to ensure that the expression is evaluated as an expression, even if your expression contains only literal strings.

The command expression can contain the name of one or more fields in the incoming command tuple on the input port. In this case, the value of each specified field is substituted when the command is run.

Using an expression to specify the command to be run allows you to evaluate variables at runtime, instead of passing the variables to the operator in the input command tuple.,

When the command line you want to run contains single quotes, escape them with backslash-double-quote to prevent their interpretation as the end of the expression.

Include the command line in the tuple output

If you select this checkbox, the schema sent to the output port is extended with a field containing the exact command line run, with all variables resolved. Use this checkbox for debugging while developing your operator.

Command output type

Specify either blob or string (the default). This setting specifies the data type of three fields of the output tuple: stdout, stderr, and (if enabled) cmdline.

Command Arguments

If you selected Arguments as the Operator Type, use the Command Arguments table to specify the command to be run and its arguments. Specify the command name in row 1, using a relative or absolute path to the command. In subsequent rows, you can optionally add the exact switches and arguments with which to run the command, split into separate rows on whitespace boundaries. That is, if you want to run the command sbadmin setLeadershipStatus NON_LEADER, specify the three parts of the command on three rows. To run the same command except specifying a StreamBase URI, use five rows as shown in this example:

sbadmin
-u
sb://remotesbhost:10001/appcontainer
setLeadershipStatus
LEADER

Any row of the Command Arguments section can contain the name of a field in the incoming input tuple. In this case, the contents of that field are substituted when the command is run. For example, your input command tuple might contain a field named serverURI that contains the programmatically determined URI of one member of a StreamBase Server cluster. In this case, specify five rows in the Arguments section, containing:

sbadmin
-u
serverURI
setleadershipStatus
NON_LEADER

If you have a name collision with a field in the input tuple, you can use quotes to escape the interpretation as a field name. For example, if you wanted the operator to run the command cat file1, but there happens to be a field in the input tuple also named file1, enter the following in the command arguments table:

cat
"file1"
Arguments field prefix

If you selected Arguments as the Operator Type, and if you enable the Include arguments and environment variables in output option on the Advanced tab, then the string you specify in this field is prepended to the contents of any command argument fields sent to the output port. The default value is arg_.

Environment Tab

This section describes the properties on the Environment tab in the Properties view for the External Process operator.

Environment

Use the Environment table to specify any environment variables needed by the command you want to run, that are not already in the environment of the containing StreamBase Server, or whose values you want to change for the purpose of running the specified command. For each row, specify the variable name, an equals sign, and value. For example:

Environment variables field prefix

If you specify any environment variables, and if you enable the Include arguments and environment variables in output option on the Advanced tab, then the string you specify in this field is prepended to the contents of any environment variable fields sent to the output port. The default value is env_.

Advanced Tab

This section describes the properties on the Advanced tab in the Properties view for the External Process operator.

Include input tuple in output

If you select this checkbox, the schema sent to the output port is extended with the fields of the command tuple received at the input port.

Input tuple field prefix

If you use the checkbox above, the string you specify in this field is prepended to the contents of each input tuple field sent to the output port. The default value is input_.

Include arguments and environment variables in output

If you select this checkbox, the schema sent to the output port is extended with fields containing the path of the command run, the switches and arguments run, and any environment variables specified in the operator.

Working Directory

Use this field to specify a working directory for the command to be run. The default working directory is that of the StreamBase Server that runs this operator. Specify as a StreamBase expression the full absolute path, or a path relative to the working directory of the StreamBase Server that runs this operator. You must surround the entire expression with single quotes to ensure that the expression is evaluated as an expression, even if your expression contains only literal strings.

Run this command asynchronously

Select this checkbox to run the specified command without holding any locks. This allows you to call sbadmin shutdown and other such commands that attempt to place locks, without causing a deadlock situation. With this option enabled, this operator's output tuple occurs asynchronously, similar to marking the operator to run in parallel threads in the Concurrency tab. The difference is that with the parallel threads setting, the operator's command can still place a lock.

Caution

Use this feature with care and consideration of its consequences. If you are unsure, do NOT select this checkbox.

Number of async worker threads

This option is enabled when the previous option is selected. The default value of 1 means that the operator's commands are performed serially. This is ideal behavior when using the operator to perform commands that must be run in sequence, such as Add Container, then Modify Container. Positive values specify a limit on the number of threads to spawn to perform the operator's requested operations. Negative or 0 means the operator spawns an unlimited number of threads.

Cluster Aware Tab

Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.

Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.

Concurrency Tab

Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.

Caution

Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.

Ports

By default, the External Process operator has one input port and one output port:

  • Input Port. The operator takes commands from the input tuple. The schema of the input command tuple is application-specific, but typically includes string fields containing switches, arguments, or parameters for the command to be run. Each operator runs its specified command on receipt of a tuple on the input port, optionally using fields in the incoming tuple to specify switches or parameters for the command.

  • Output Port. The operator sends a tuple on the output port whenever it runs the specified command. By default, the schema of the output tuple is the following:

    Field Name Data Type Description
    stdout blob Contains the standard output of the command that was run.
    stderr blob Contains the standard error of the command that was run.
    exitcode int Contains the exit code value returned from the command that was run.

    The schema of the output tuple can be optionally extended by using one or more of the following options, which are described in the indicated sections.

    Include the command line in the tuple output Command Arguments Tab
    Include arguments and environment variable in output Advanced Tab and Environment Tab
    Include input tuple in output Advanced Tab

You can also add an optional Error Output port, which outputs a StreamBase error tuple for any error thrown by the operator, as described in General Tab.