Wombat MAMA Input Adapter

Introduction

The Spotfire Streaming Input Adapter for Wombat MAMA allows StreamBase applications to subscribe to a set of symbols from a single Wombat MAMA source. An instance of the Wombat MAMA Input Adapter subscribes to a specified set of symbols on a single Wombat MAMA source using a specified messaging middleware. After a subscription request has completed, the Wombat MAMA Adapter receives messages from each subscription. Each message is converted to a StreamBase tuple and is then enqueued in the order received.

In Wombat MAMA, a symbol is analogous to a topic in most publish-subscribe infrastructures. In MAMA documentation, the terms symbol and subject are often used interchangeably and do not necessarily correspond to the symbol for the securities whose information is published using the symbol, although there is frequently a one-to-one correspondence. (An exception to this correspondence is a Group Subscription. In a Group Subscription, subscription to the group's Symbol typically results in the receipt of messages for many individual items, depending on how the group has been configured.) Symbol subscriptions can be specified using a comma-separated list, loaded from a symbol file, dynamically specified using input streams to the adapter, or a combination of all three.

All subscription requests are initiated before the adapter finishes initializing. However, as is usual with MAMA, these subscription requests are queued and sent to the source at the rate specified by the Throttle property. Thus, subscription processing for all symbols may not be completed before message processing for some subscriptions begin.

Notes

This document assumes familiarity with the MAMA infrastructure. Terms and contexts are used that assume familiarity with the Wombat Universal Feed Handler Suite, and much of the material discussed in the MAMA Developers Guide from Wombat. In addition, the Wombat-specific functionality of the adapter corresponds fairly closely with the functionality implemented in Wombat’s mamalisten programs (especially the Java version, com.wombat.mama.examples.MamaListen).

StreamBase does not license or distribute Wombat libraries or other Wombat-provided runtime artifacts. These must be obtained from Wombat and installed prior to installation of the adapter.

The Wombat MAMA Input Adapter releases are built against specific versions of the mamajni.jar file that is targeted for a particular messaging middleware. For information about supported configurations, please contact Support team.

Tip

As a practical note for users and installers of this adapter, it is a good idea to make sure you can get com.wombat.mama.examples.MamaListen to work in the target environment and with the data of interest independent of your StreamBase installation before attempting to use the StreamBase Wombat MAMA Input Adapter.

Making the MAMA Middleware Available

As described earlier, the MAMA middleware you will be using is not provided by StreamBase. Before you can run a StreamBase application that calls this adapter, you must obtain an appropriate middleware package from Wombat.

You must then make sure the Wombat MAMA middleware JAR file, (and any other files the JAR file depends on), are accessible to StreamBase Server when it loads and runs. You do this as described in the next section.

You can drop a Wombat adapter icon into the EventFlow canvas of your application without any prerequisites. You can configure your Wombat connection in the Properties view of the adapter's icon without requiring the presence of the mama.jar middleware file. It is only when trying to run and use the adapter that the middleware JAR file must be on the server's classpath.

Running Applications With The Wombat MAMA Adapter

Running From StreamBase Studio

In order to run a StreamBase application containing one or more instances of the Wombat MAMA Adapter from within StreamBase Studio, you must ensure that the following configuration steps have been taken:

  • Add mamajni.jar (found in your Wombat MAMA's lib directory) to your project's list of referenced libraries: in StreamBase Studio's Project Explorer, right-click your project's top-level node and select Build Path>Add External Archives, and select your copy of mamajni.jar from the file dialog.

  • In the StreamBase Server configuration file (usually, sbengine.conf), add a <java-vm/library> property for the location of your Wombat MAMA native libraries. On Windows, this will be under the bin/dynamic subdirectory of the MAMA installation location. On UNIX, it will be under the lib subdirectory. For example:

    On Windows:

    <library path="C:/MyWombatInstall/bin/dynamic"/>

    On Linux

    <library path="/opt/my_wombat_install/lib"/>

  • The WOMBAT_PATH environment variable must provide the path to the mama.properties file on your system.

Running From The Command Line

In order to run a StreamBase application containing one or more instances of the Wombat MAMA Adapter from the command line, you must ensure that the following configuration steps have been taken:

  • In the StreamBase Server configuration file (usually, sbengine.conf), add a <java-vm/library> property for the location of your Wombat MAMA native libraries. On Windows, this will be under the bin/dynamic subdirectory of the MAMA install location. On UNIX, it will be under the lib subdirectory. For example:

    On Windows:

    <library path="C:/MyWombatInstall/bin/dynamic"/>

    On Linux:

    <library path="/opt/my_wombat_install/lib"/>

  • Add mamajni.jar (found in your Wombat MAMA's lib directory) to your project's list of referenced JAR files by adding a <java-vm/jar> property to your StreamBase Server configuration file. For example:

    <jar file="/opt/my_wombat_install/lib/mamajni.jar"/>

  • The WOMBAT_PATH environment variable must provide the path to the mama.properties file on your system.

Adding the Wombat Input Adapter to an EventFlow Module

To add the adapter to an EventFlow application in StreamBase Studio:

  1. Create a new StreamBase project, choosing the option to create an EventFlow application.

  2. Drag the Wombat MAMA Input icon from the Operators and Adapter drawer in the Palette view to the EventFlow canvas.

  3. Click the Wombat adapter's icon to open its Properties view.

  4. Define properties for the Wombat adapter and add any input and output streams required, as described in the next sections.

Configuring Input and Output Streams

In addition to subscription data from the MAMA feed, the Wombat MAMA Input Adapter can be configured to receive control data on up to three input and output streams, and to emit data on either one to four output streams. The use of each stream is predefined for a particular purpose. In an EventFlow, StreamBase Studio may add or remove streams automatically when you modify the appropriate settings. For more information, see Adapter Properties Settings, which documents all the adapter settings.

Input Streams

The three input streams for the Wombat MAMA Input Adapter are used to provide the following types of control data. In EventFlow modules, each is enabled using the setting indicated:

Adding subscriptions

If the Enable Add Symbol Stream setting is selected, the symbols received on this stream are added to the subscriptions on the MAMA feed.

Deleting subscriptions

If the Enable Delete Symbol Stream setting is selected, the symbols received on this stream are ignored by the adapter, even if they are subscribed in the MAMA feed.

Displaying currently active subscriptions

If the Enable Query For Symbols setting is selected, any tuple received on this stream triggers an output tuple that contains symbols from all sources (the MAMA feed and optional input streams).

The number of input ports depends on which of these settings you enable, but they are always in this order:

  1. Add Symbols stream

  2. Delete Symbols stream

  3. Query Symbols stream

For example, if you enable an Add Symbols stream, it will always be on the adapter's first input port. If you enable both an Add Symbols and a Delete Symbols stream, then the Delete Symbols stream will be on the second input port. But If the Add Symbols stream is not configured, then the Delete Symbols stream will be on the first input port.

Output Streams

One to five output streams can be configured. They are related to your input stream configuration, and depending on your adapter settings they may or may not be available. However those that are enabled will always appear in the order in which they are presented here.

  1. The first output stream always exists: it sends out the processed MAMA messages.

  2. If you enabled the Output Data Dictionary setting, an output stream is configured as the second output port. It sends the contents of the Wombat field dictionary. The output stream has the fields shown in the following table:

    Field Name Data Type
    fid int
    type int
    typeName string
    name string
  3. If you enabled the Enable Query For Symbols setting, an output stream is configured as the next output port, and is used to send the results of the output query. The schema for this port is described in Stream Schemas.

  4. If you enabled the Output Errors setting, an output stream is configured on the next output port. Whenever an error occurs, a tuple is emitted on this port to describe the error. The output stream has the fields shown in the following table:

    Field Name Data Type Description
    operator string Names the adapter that received this error.
    timestamp timestamp Marks the time at which the error was received.
    errorCategory string Denotes the broad category in which this error falls. As of this writing, only one category is valid: "Subscriptions", which represents subscription errors.
    errorType string Designates the exact error that occurred within the category. When errorCategory is "Subscriptions" (that is, a subscription error has occurred), valid values for errorType are "Bad Symbol", "Expired" and "Timeout".
    errorData string Contains error-specific information. In the case of all three currently supported errors in the Subscription category, this field contains the symbol for the error that occurred.
    errorDescription string Provides a human-readable description of the error.
  5. If you enabled the Enable Queue Monitoring setting, an output stream is configured on the next output port. This will cause the adapter to issue a tuple whenever the number of MAMA messages waiting to be read by the adapter goes above the specified High Watermark (see High Watermark), or below the specified Low Watermark (see Low Watermark).

    Note

    This functionality is only available when using MAMA version 4.x and above.

    Note

    Depending on the transport type you are using, specifying a Low Watermark value may not be supported and will have no effect (i.e. no tuples will be emitted when the low threshold is reached).

    The output stream has the fields shown in the following table:

    Field Name Data Type Description
    queueName string The name of the queue to which this event applies.
    eventType string Either highWatermark or lowWatermark.
    queueSize int The current number of awaiting MAMA messages in this queue.

Stream Schemas

Each of the control streams must have a certain StreamBase schema, so that the adapter knows what data to expect.

  • The Add and Delete Symbols streams have the same schema, which contains just one string field named symbol. If the Include Subscribe Pass-Thru Field option in the Subscriptions tab is turned on then you must add a PassThru field of any type.

  • The Query Symbols input stream can have any schema. The schema for this control port is added to the schema for the corresponding output port. That is, any fields on the Query Symbols input schema are also created on the output port schema, and the values enqueued in those fields are copied to the output stream.

    For example, the Query Symbols stream could be created with a field named id, which might serve to mark any output tuples with the ID of the query that led to their existence.

  • The schema for the Query Symbols output stream includes an additional field named symbol. This field should not be on the schema for the input port. When a tuple is sent to mark the completion of a symbol query, it includes a null value for the symbol field.

Dynamic Symbol Subscription

As mentioned above you can use the Add Symbol and Delete Symbol streams to dynamically subscribe to and unsubscribe from symbols from within your application.

If a given subscription times out, an error tuple is emitted on the Errors output stream and the subscription is automatically deleted. Thus, there is no need for you to delete it explicitly via the Delete Symbol stream. You may attempt to re-subscribe to the symbol again at any time.

Adapter Properties Settings

Settings are used to control most of the behavior of the adapter. The settings are grouped under several tabs in the Properties view in StreamBase Studio.

Note

The summary descriptions here are provided as a convenience, but are not intended to be definitive. They are based on Wombat MAMA 3.3 with the LBM messaging middleware, which was used for testing and development, and may not necessarily reflect the Wombat behaviors for the middleware's software version and platform. For up-to-date and precise descriptions of these properties, read the Wombat documentation.

General Tab

Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.

Adapter: A read-only field that shows the formal name of the adapter.

Class name: Shows the fully qualified class name that implements the functionality of this adapter. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.

Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this adapter starts.

Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.

Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.

Adapter Properties Tab

MAMA Version

Default: 4.x and above

The version of MAMA to use. This is a required setting.

Middleware

Default: lbm

The middleware MAMA should use. This setting is only meaningful when MAMA Version is set to 4.x and above. The possible values are:

Setting Middleware
lbm 29 West's LBM
tibrv Tibco's Rendezous
WMW Wombat's TCP Middleware
Transport

Default: internal

The name of the transport, defined in mama.properties, for this adapter to use. Examples: lbm_tport or tibrv_tport. This is a required setting.

Source

Default: Empty

The name of the Wombat MAMA source for this adapter instance. A single adapter instance can subscribe to only one MAMA source. Available source names are defined by the local environment and typically represent a feed handler. Examples are CTA and utp. This is a required setting.

Throttle

Default: 1000

Outbound subscription throttle: the maximum number of subscription requests per second to send to the feed handler.

Message Dispatch Threads

Default: 3

The number of threads MAMA uses to dispatch messages.

Enable Add Symbol Stream

Default: false

Lets you optionally subscribe to symbols from an input stream, in addition to subscribed symbols from the MAMA feed.

If you use this field, you must attach an input stream to the adapter's first input port. The input stream's schema must include a string field named symbol that will receive symbols.

Enable Delete Symbol Stream

Default: false

Lets you optionally unsubscribe symbols received on an input stream, overriding their subscriptions from the MAMA feed for this instance.

If you use this field, you must attach an input stream to the second adapter input port. The input stream's schema must include a string field named symbol that will receive symbols.

Enable Query For Symbols

Default: false

Lets you optionally trigger an output tuple that contains symbols from all sources (the MAMA feed and optional input streams).

If you use this field, you must attach an input stream to the third adapter input port, and an output stream for the query result. The input stream's schema can contain any field. A tuple arriving on this input stream triggers the release of symbols on the output stream.

Enable Entitlement Tracing

Default: false

Trace entitlements to the adapter log. An entitlement server verifies whether a user is allowed to use the adapter.

Enable Queue Monitoring

Default: false

When this option is selected, a new output port will be added on which High and Low watermark warning tuples will be issued when those thresholds are reached in the incoming MAMA queue(s).

Note

This functionality is only available when using MAMA version 4.x and above.

High Watermark

Default: 10000

The number of incoming MAMA messages in a queue over which a warning tuple will be issued on the Queue Monitoring port (see 5).

Low Watermark

Default: 1000

The number of incoming MAMA messages in a queue under which a warning tuple will be issued on the Queue Monitoring port (see 5). This will only occur after a High watermark has been previously reached for this queue.

Note

Depending on the transport type you are using, specifying a Low Watermark value may not be supported and will have no effect (i.e. no tuples will be emitted when the low threshold is reached).

Enable Passthrough Cache

Default: false

This option should only be turned on if running in a republishing mode with a Wombat Output Adapter. The passthrough cache optimizes the republishing of messages by limiting the amount of fields that go through the StreamBase application.

Note

This functionality is only available when using MAMA version 4.x and above.

Passthrough ID field name

Default: empty

If passthrough cache is enabled above, this field will need to be defined in the message output schema as a long. It stores the id of the message with which it was saved in the cache. The Wombat Output Adapter will look for this field in its message input schema.

Passthrough Cache Time To Live

Default: 600

If passthrough cache is enabled above, this field specifies the number of seconds messages will be allowed to stay in the cache. Any messages older than this period will be deleted. This cleanup of old messages is not desirable, it is recommended that any messages not published by Wombat Output Adapter be discarded. The Wombat Output Adapter can be configured to have a second input port for messages that should be discarded, which will immediately remove them from the passthrough cache.

Log Level

Default: INFO

Choose one of the following standard Sun logging levels: SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL, OFF. The log level setting for the adapter is independent of the containing application's overall log level.

Dictionary Tab

Use Dictionary

Default: true

Specifies whether the adapter retrieves a Wombat field dictionary at startup, and whether the adapter uses that dictionary for fieldname or FID translation. This property will almost always be true in practice, and is exposed as an adapter property primarily for debugging or for extraordinary situations.

Load Dictionary From

Default: Load From Feed Handler

Specifies where the dictionary is loaded from. If set to Load From Feed Handler the adapter retrieves a Wombat field dictionary at startup from the Wombat source. If set to Load From File the adapter retrieves a Wombat field dictionary from a local file specified in Dictionary File Path

Dictionary Source

Default: WOMBAT

The name of the Wombat source to use for the dictionary for this adapter instance.

Dictionary Transport

Default: Empty

The name of the Wombat messaging middleware, defined in mama.properties, for this adapter to use to access the data dictionary (for example, LBM or tibrv). If this setting is left blank, the same transport is used as specified in the Transport setting in the Adapter Properties tab.

Dictionary Wait Timeout

Default: 30000

The timeout interval to use when trying to obtain a data dictionary. Values represent milliseconds.

Dictionary File Path

Default: empty

If Load Dictionary From File is selected, specifies that path of the file where dictionary can be loaded from.

Output Data Dictionary

Default: false

Outputs the contents of the Wombat field dictionary (retrieved at adapter initialization) once to the second adapter instance output port.

Subscriptions Tab

Symbol(s)

Default: Empty

A comma-separated list of one or more stock symbols to subscribe to when the adapter is initialized. You can use either or both of the Symbol and Symbol List File Name properties. A symbol is only subscribed once.

Symbol List File Name

Default: Empty

The name of a text file that contains a list of one or more stock symbols to subscribe to. The file format is one symbol per line. The file must be created or imported into your project. A list file can be used in addition to, or instead of, entries in the preceding Symbol field.

Timeout

Default: 10

Subscription timeout, in seconds. When no response is received from the Wombat infrastructure (for example, the Source's feed handler), this is how long the MAMA API is to wait before retrying a subscription request.

Retries

Default: 1

Specifies the number of times the MAMA API is to retry a subscription request before notifying the adapter that the subscription request has timed out. From the adapter's perspective, the total time elapsed before it considers the subscription request to have failed is Timeout * Retries.

Require Initial

Default: true

Specifies whether subscriptions require an initial message to be sent on subscription creation. For most market data situations, the appropriate setting is true.

Recover Gaps

Default: true

When enabled, the adapter attempts to recover from sequence number gaps by requesting a recap.

Snapshot

Default: false

If true, each subscription causes an initial message to be sent for the symbol, but no updates.

Quality for All

Default: true

If false, then setInvokeQualityForAllSubscs(false) is invoked on the adapter instance's messaging middleware.

Group Subscription

Default: false

If this property is true, all subscriptions for this adapter instance are group subscriptions.

Output Subscription Errors

Default: false

Emits a tuple describing errors as they occur. The third output port is used.

Include Subscribe Pass-Thru Field

Default: false

When set, a field named PassThru is expected in the schema of the Subscribe input port, and a field of the same type is added to the MarketData output port. There are no constraints on the PassThru field's type. The value received in this field from the subscription tuple is placed in each output tuple emitted as a result of that subscription.

Special Fields Tab

Symbol Field

Default: Empty

The name of destination schema field in which to place the result of MamaMsg.getSymbol() for each received message. In this context, the symbol corresponds to the symbol/subject for a subscription.

Source Field

Default: Empty

The name of destination schema field in which to place the configured Wombat source.

Quality Field

Default: Empty

The name of the destination schema field in which to place the most recently received quality value for the subscription corresponding to the received message. Subscription quality indications are those received via the MamaSubscription onQuality() callback. The values placed in the Quality field are int values that correspond to MAMA API MamaQuality constants. See the Data Quality description for an interpretation of the meaning of those values.

For the version of MAMA against which this adapter was tested (4.x) the int values and their corresponding MamaQuality constants are listed in the table below. These are Wombat-specified values, and StreamBase makes no assertion that the meaning of these values will not change when used against other versions of the Wombat MAMA API. These values are provided here for convenience.

Quality field value MamaQuality constant
0 OK
1 MaybeStale
2 PartialStale
3 Stale
MdMsgType Field

Default: Empty

The name of destination schema field in which to place the result of com.wombat.mama.MamaMsgType.typeForMsg(), converted to a StreamBase int, on each received message.

Note

MdMsgType values are defined by Wombat and may be updated independently of this adapter. See the Wombat MAMA constant-values.html Javadoc page section for the MamaMsgType class for the values supported by the version of MAMA used at your site. However, as a convenience, the following table provides a list of such values as documented by Wombat at the time this adapter document was written.

Value Meaning
0 General update
1 Initial value
2 Trade cancel
3 Trade error
4 Trade correction
5 Closing summary
6 Refresh or recap of some or all fields
7 Deleted symbol
8 Expired security (such as option, future, and so on)
9 Full-record snapshot (no subsequent updates)
10 No subscribers (unexpected subscription)
11 Bad (unknown) symbol
12 Pre-opening summary (includes Wombat-generated morning roll)
13 Quote update
14 Trade update
15 Order update
16 Order book Initial value (complete book)
17 Order book delta
18 Clear order book
19 Order book recap
20 Order book snapshot (complete book)
21 Not permissioned (on feed)
22 Not present now, but might show up later
23 Security status update
24 Corporate action
100 Miscellaneous
101 Stale data
IsDefinitelyDelayedField, IsDefinitelyDuplicateField, IsPossiblyDelayedField, IsPossiblyDuplicateField, IsOutOfSequenceField

Default: Empty

The name of the destination schema field in which to place the result of invoking the corresponding MamaMsg method on each received message.

Edit Schemas Tab

In this tab, specify the output tuple fields schema that is used to convert received messages into StreamBase tuples, and outputs the tuples into the application stream. Only the fields specified in the output schema are included in the tuples. Any other fields in incoming Wombat messages are ignored.

Wombat data types are converted into StreamBase data types according to the Data Type Mappings section. The destination schema's field names must correspond to field names in the Wombat dictionary with the following exceptions:

  • Field names beginning with an underscore character are assumed to specify Wombat FIDs. The field value is retrieved from the MamaMsg by dropping the underscore and converting the rest of the destination field name to an integer FID.

  • Field names specified in an adapter property that represent special fields not passed in the MamaMsg as data but are retrieved per message using other parts of MAMA API or adapter properties. For example, if the Source Field property is populated with mySource, the value of the adapter’s Source property is placed in the mySource field of all tuples enqueued from the adapter.

The value of fields whose names in the destination schema have no corresponding MamaMsg field or special adapter field become null in all tuples enqueued from the adapter.

For general instructions on using the Edit Schemas tab, see Defining Input Streams.

Data Type Mappings

Incoming Wombat message field values are converted from Wombat types to StreamBase types according to the following table. If the destination type for the incoming Wombat field type is the Preferred StreamBase type, the conversion is done directly (the most efficient conversion). Otherwise, the adapter attempts to convert an Object-wrapped form of the incoming value via the generic Tuple.setField(name, Object) method.

Any fields that have Wombat data types that are not supported are set to null in the converted tuple. Any Wombat data types not listed in the table below are not supported.

Wombat MamaFieldDescriptor Data Type Preferred StreamBase DataType
BOOL bool
CHAR string
I8 int
U8 int
I16 int
U16 int
I32 int
U32 long
I64 long
U64 long
F32 double
F64 double
STRING string
TIME timestamp, string or double
DATETIME timestamp, string or double
PRICE double
OPAQUE blob
MSG tuple
I8ARRAY list<int>
U8ARRAY list<int>
I16ARRAY list<int>
U16ARRAY list<int>
I32ARRAY list<int>
U32ARRAY list<long>
I64ARRAY list<long>
U64ARRAY list<long>
F32ARRAY list<double>
F64ARRAY list<double>
VECTOR_STRING list<string>
VECTOR_MSG list<tuple>

Cluster Aware Tab

Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.

Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.

Concurrency Tab

Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.

Caution

Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.

Note

The Concurrency tab's option to Run this component in a parallel region applies to this output adapter in the same way as it does to other Spotfire Streaming adapters and operators. That is, running the adapter in its own parallel region provides an additional level of queuing to absorb traffic peaks inside StreamBase Server, provided it is suitable for your application. However, be aware that the Wombat event queue also provides a separate means of message buffering which cannot be eliminated with this adapter implementation.

Deploying an Application with the Wombat Input Adapter

To deploy an application with the Wombat MAMA Input Adapter outside of StreamBase Studio, follow the guidelines in Deploy Applications with epadmin in the Concepts Guide. In particular, edit the HOCON configuration files, and make the mamaversion.jar and any native libraries specific to the messaging middleware accessible to the adapter code at runtime as described in Using External JAR Files and in Making the MAMA Middleware Available.

Related Topics