Intermediate Stream Dequeuing

< Previous		Next >

Introduction

For debugging purposes only, you can dequeue data from intermediate streams in your application, not only from output streams. This debugging feature allows you to examine the value of a tuple at an intermediate point in the stream of tuples flowing left to right, before that tuple has been fully processed, and before it has arrived on an output stream. This feature is not to be used for inserting new data into the middle of an application.

Caution

When intermediate stream dequeuing is enabled, the StreamBase Server instance hosting your application consumes considerably more memory, to the point that you might not be able to successfully launch large applications.

Intermediate stream dequeuing (ISD) is disabled by default. You can enable it for a particular run or debug session as described below in Enabling Full ISD.

ISD Compared to Other Features

Do not confuse intermediate stream dequeuing with the always expose feature of input and output streams.

Always Expose Stream Feature: The General tab of the Properties view for input and output streams has a checkbox labeled Always expose stream for enqueue or dequeue, respectively. This feature is an attribute of explicit input and output streams that allows you to designate a stream to be enqueueable or dequeuable no matter how deeply its containing module is nested. This feature only affects input and output streams.
Intermediate Stream Dequeue: ISD is a property setting of the hosting server, and affects all modules in an application. This feature allows you to dequeue not only from output streams, but from the output ports of Map, Filter, Union, or other operators in any module. You can limit the number of operators exposed for dequeuing by specifying a regular expression to match against the names of intermediate streams.

For another debugging approach, consider using runtime tracing in place of, or in addition to, ISD. Runtime tracing is described in Interpreting Trace Files, and includes a comparison table in Tracing Versus Dequeuing.

What Streams Are Exposed

The following table clarifies what streams are exposed for dequeuing for various settings of intermediate stream dequeuing.

ISD Setting	Regular Expression Setting	Dequeuable Streams
ISD disabled (the default for run and debug modes)	N/A	For the top-level module only: All explicit output streams. Any sub-module output stream individually exposed for dequeuing in its Properties view.
ISD enabled	None specified.	For the top-level module and all sub-modules: All explicit input and output streams. All explicit error input and error output streams. All intermediate streams in all modules.
ISD enabled	Regular expression substring specified.	For the top-level module: All explicit input and output streams. All explicit error input and error output streams. For all sub-modules: Any input, output, error, or intermediate stream whose name matches the supplied regular expression substring. Any sub-module output stream individually exposed for dequeuing in its Properties view.

ISD Setting

Regular Expression Setting

Dequeuable Streams

ISD disabled (the default for run and debug modes)

N/A

For the top-level module only:

All explicit output streams.
Any sub-module output stream individually exposed for dequeuing in its Properties view.

ISD enabled

None specified.

For the top-level module and all sub-modules:

All explicit input and output streams.
All explicit error input and error output streams.
All intermediate streams in all modules.

ISD enabled

Regular expression substring specified.

For the top-level module:

All explicit input and output streams.
All explicit error input and error output streams.

For all sub-modules:

Any input, output, error, or intermediate stream whose name matches the supplied regular expression substring.
Any sub-module output stream individually exposed for dequeuing in its Properties view.

Note

Error output streams are considered intermediate streams for purposes of dequeuing. To allow dequeuing directly from an error output stream, you must enable ISD. Note that this is useful only for debugging: in a well-designed application, your application does not need to read the tuples output on error streams, which are passed automatically to the next higher module, or to the hosting Server.

Enabling Full ISD in Run Configuration

Enable intermediate stream dequeuing via the Advanced tab in a run configuration by checking the Enable Intermediate Stream Dequeue option and leaving the filter unspecified.

For example, the output of sbc list stream for the StreamBase Best Bids and Asks sample is as follows:

stream  BestAsks
stream  BestBids
stream  NYSE_Feed
stream  out:IsNewBestAsk_1
stream  out:IsNewBestBid_1
stream  out:Update_Bids_and_Asks_1

Enabling Partial ISD in Run Configuration

You can restrict ISD to the components that match a regular expression you specify via the Advanced tab in a run configuration.

For example, the normal output of sbc list stream without ISD for the StreamBase Best Bids and Asks sample is as follows:

stream  BestAsks
stream  BestBids
stream  NYSE_Feed

Now use the Intermediate Stream Dequeue option to restrict ISD to the streams that match a regular expression (for example, .IsNew.):

The results from sbc list stream are now: [verify]

stream  BestAsks
stream  BestBids
stream  NYSE_Feed
stream  out:IsNewBestAsk_1
stream  out:IsNewBestBid_1

Enabling Full ISD with System Properties

Enable intermediate stream dequeuing by setting the streambase.codegen.intermediate-stream-dequeue property in a HOCON configuration file of type c.t.e.streambase.c.sbengine, in the StreamBaseEngine root object.

If you set that property = "true", you enable full ISD, where all available intermediate streams are enabled for dequeuing. For example:

name = "sbengine"
version = "1.0.0"
type = "com.tibco.ep.streambase.configuration.sbengine"
configuration = {
  StreamBaseEngine = {
    systemProperties =
    {
     "streambase.codegen.intermediate-stream-dequeue" = "true"
    }
  }
}

Enabling Partial ISD with System Properties

You can restrict ISD to the components that match a regular expression you specify with the streambase.codegen.intermediate-stream-dequeue-regex system property.

For example, the normal output of sbc list stream without ISD for the StreamBase Best Bids and Asks sample is as follows:

stream  BestAsks
stream  BestBids
stream  NYSE_Feed

If you enable full ISD with the configuration file from the previous section, the same command emits the following:

stream  BestAsks
stream  BestBids
stream  NYSE_Feed
stream  out:IsNewBestAsk_1
stream  out:IsNewBestBid_1
stream  out:Update_Bids_and_Asks_1

Now use the following configuration file to restrict ISD to the streams that match the regular expression .IsNew.:

name = "sbengine"
version = "1.0.0"
type = "com.tibco.ep.streambase.configuration.sbengine"
configuration = {
  StreamBaseEngine = {
    systemProperties =
    {
     "streambase.codegen.intermediate-stream-dequeue" = "true"
     "streambase.codegen.intermediate-stream-dequeue-regex" = ".IsNew."
    }
  }
}

The results from sbc list stream are now:

stream  BestAsks
stream  BestBids
stream  NYSE_Feed
stream  out:IsNewBestAsk_1
stream  out:IsNewBestBid_1

Intermediate Stream Naming Convention

With intermediate stream dequeuing enabled, you can dequeue data from any output port of any operator using a stream name of the form:

out:operatorName_N

where operatorName is the name of the operator and N is the number of the output port for that operator.

Consider the Split.sbapp application, which is one of the operator samples installed with the StreamBase kit:

To verify the intermediate stream names, run sbc list when the application is running with ISD enabled. The following example shows the intermediate streams in boldface:

container   default
container   system
stream      INTRUSION_TooManyIPsForUser
stream      INTRUSION_TooManyUsersForIP
stream      IPandUserLogin
stream      out:CheckIPsForAUser_1
stream      out:CheckUsersInAnIP_1
stream      out:ProcessIPFirst_1
stream      out:ProcessIPFirst_2
schema      schema:IPandUserLogin
operator    CheckIPsForAUser
operator    CheckUsersInAnIP
operator    IPCountExceeded
operator    ProcessIPFirst
operator    UserCountExceeded

You can also run epadmin display stream with a --servicename or --adminport global parameter to see the intermediate streams in a fragment running with ISD enabled. The output of this command is verbose, with several lines of information per stream. Intermediate streams are marked as such in the Type line. For example:

[sample_operator_Split.sbuser] Engine = com_tibco_sb_sample_operator_Split2
[sample_operator_Split.sbuser] Path = default.out:CheckIPsForAUser_1
[sample_operator_Split.sbuser] Type = intermediate
[sample_operator_Split.sbuser] Enqueue = true
[sample_operator_Split.sbuser] Dequeue = true
[sample_operator_Split.sbuser] Total Tuples = 0
[sample_operator_Split.sbuser] Schema = (UserID int, CountOfIPs int)

Unconventional Intermediate Stream Names

Intermediate stream names can be found that do not appear to follow the naming convention described in the previous section. When determining the name of intermediate streams from which to dequeue, you must confirm the names as actually used in your application. Confirm with an sbc list or epadmin display stream command.

In some cases, the name of an intermediate stream can vary from the default convention, depending on the history of edits to the application.