Client Access to Capture Fields

Building applications with capture fields is relatively straightforward, and once data is in a capture field, you can easily pass it between operators and access it in tables. You can access streams that include capture fields from either a client connection or from a Java operator, but doing so requires transforming the capture fields into an accessible format. There are two strategies for transforming capture fields: FLATTEN and NEST, each having different trade-offs.

The next topic illustrates typechecking issues that capture fields can have due to schema conflicts. The example uses the default FLATTEN strategy. For descriptions of how the two strategies work, see Runtime Strategies for Capture Fields.

The following modules illustrate how modules that pass through capture fields can fail to typecheck when they add fields having names that duplicate the names of fields mapped to the capture via the calling module.

Here is an EventFlow application, capture_inner.sbapp, which defines a capture field r of whose data type name is remainder. It adds a field called area that goes to output stream intermediate but is removed prior to output stream outstream.

The application capture_inner.sbapp has an input stream instream whose schema consists of two ints, x and y, and a capture, r, containing data which the user has given the data type @remainder. The AddArea Map operator multiples x and y, adding the result as a new field called area. AddArea's output schema is then {x int, y int, r @remainder, area int}. That stream connects directly to output stream intermediate, and also connects to Map operator RemoveArea, which removes the field area from the stream before passing the tuple to output port outstream. The application has no information about the contents of capture field r.

Here is the EventFlow application capture_outer_conflict.sbapp, which passes tuples to the module capture_inner.sbapp. The module's output port has been assigned to output stream intermediate in the Port Associations table in the inner module reference's Output Ports tab. As the EventFlow diagram shows,this association creates a typechecking error.

The capture_outer_conflict example illustrates what can happen when an application provides capture_inner with an input stream whose schema includes fields in addition to x and y. In the example, input stream instream has an additional field, area, which is captured to the r capture field by module inner (capture_inner.sbapp). When the application restores capture field r at its output port, two fields named area exist in the stream intermediate. By connecting to the inner module reference's intermediate output port, the application introduces a typecheck error, as shown above.

If instead we assign the module's output port to output stream outstream in the Port Associations table instead of to the stream named intermediate, there is no typechecking error. The conflicting field area added by Map operator AddArea has been deleted by Map operator RemoveArea, enabling the input field area to be reconstituted, as the following diagram illustrates.

Now consider a slightly different outer module, capture_outer_no_conflict.sbapp, whose input schema includes fields called name and color instead of the field called area: {x int, y int, name string, color string}. When capture_inner.sbapp is referenced from capture_outer_no_conflict.sbapp, no typechecking error occurs when connecting to inner's output port. The resulting output stream outstream has a schema {x int,y int, name string, color string, area int}, as illustrated below.

In the Java Client Library, the StreamBaseClient.getStreamProperties method was extended with the argument CaptureTransformStrategy captureStrategy that allows you specify the capture field behavior you want. The default capture strategy is FLATTEN.

In the C++ Client Library, the StreamBaseClient::getStreamProperties method was extended with the captureStrategy argument (of type StreamProperties::CaptureTransformStrategy) that allows you to specify the capture field behavior you want. The default is FLATTEN.

Similarly, the StreamBase Python Client Library includes methods that apply capture strategies: Client.getSchemaForStream(string entity, CaptureTransformStrategy strategy = FLATTEN) returns the runtime schema of a stream using the specified strategy. Also, Client.getStreamProperties(string entity, CaptureTransformStrategy strategy = FLATTEN) returns a description of a stream using the specified strategy.

The Operator base class for all StreamBase operators and adapters was extended with a method setCaptureStrategy(CaptureTransformStrategy captureStrategy). Call this method during the typecheck phase of your operator or adapter to specify your preferred capture field behavior. If you do not call this method, the default FLATTEN strategy is used. In addition, the methods getRuntimeInputSchema, getRuntimeOutputSchema, getTypecheckInputSchema, and getTypecheckOutputSchema were added to the Operator class. The existing methods getInputSchema and getOutputSchema are still present, with new behavior. These methods function as shown in the following table:

The method getSchemaForCapture is still present in the API and still works, but operators and adapters are not likely to need to call it.

Capture fields make modules more reusable by allowing inner modules to accept supersets of their input schemas. However, as the above example illustrates, in some reuse contexts the names of captured fields can conflict with names of fields that sub-modules generate and output. Should applications potentially have this problem, outer modules can further encapsulate fields captured by inner modules at run time by reconstituting them as subfields of a field of type tuple. That is, when they exit an inner module, they are nested within a placeholder field.

Nesting captured fields is a runtime operation that you choose as a capture transform strategy. The default capture transform strategy, called FLATTEN, is not to nest captured tuples in this manner, as nesting is usually unnecessary and complicates schemas. The FLATTEN strategy expands capture fields at the same level as non-captured fields. The other capture transform strategy, NEST, encapsulates and expands captured fields as type Tuple. Use NEST to avoid duplication of field names, at the cost of a more layered schema. The two strategies are also settable properties of binary and CSV input and output file adapters, and several others.

To nest capture fields within an application, you can use the Runtime Schema Cast operator, which casts its input schema to its configured target output schema at run time. The Capture Fields for Generic Data Store sample included with StreamBase uses this Java operator to demonstrate how to use operator API methods to place data from streams into capture fields.

See also: