This topic introduces concepts that provide the foundation for using TIBCO StreamBase® and TIBCO® Live Datamart.
If you are familiar with streaming data concepts, skip ahead to About TIBCO StreamBase Studio.
TIBCO StreamBase® is a program that processes and analyzes structured, streaming data.
The word structured in this definition is important. You may be familiar with a streaming video service such as Netflix, or a streaming audio service. The data streamed by those services is in the form of video or audio files, which are unstructured by nature. However, if a streaming media service also provides metadata about each movie or song, chances are good that the metadata would be provided in structured form.
A structured stream of data is in a format that is known or discoverable, where that format is repeated for each unit of data. A structured stream might be a sequence of name-value pairs, with a defined start and stop keyword at the beginning and end. The name-value pairs between each start and stop keyword define one unit of data in that stream. Another stream might be a complex mix of hierarchical data that include list subunits with no fixed length, or where one unit of data can contain subunits of data with a different structure than the containing unit.
The twenty-first century has many examples of streams of structured data:
Stock market transactions are reported in structured streams from one brokerage to its peers, or reported from markets such as NASDAQ or the New York Stock Exchange to the public.
Sports statistics are streamed while each game is played to news sites and to the public.
In the petroleum industry, each oil or gas well reports a stream of operating statistics to a central administration site.
Sensors stream reports of seismic activity at their location to a monitoring site.
Commercial web sites stream purchase information from customers to back-end servers that process each order and return real-time results back to those customers.
A TIBCO StreamBase application can be developed to ingest each of these stream types and many others in order to analyze the incoming data in many ways. StreamBase applications then emit the same stream, or a modified stream, or any number of alternative streams as outputs.
StreamBase has an internal data structure called a tuple to encapsulate the incoming stream of data. The tuple format is very flexible and can manage both simple name-value pair streams as well as multi-level, hierarchical data streams with multiple levels of subunits.
The format of each tuple is described in a schema in the same way that a database table's structure is described as a schema. The difference with StreamBase schemas is that they can be very complex and multi-level to support incoming data streams that require them.
The schema describes a single unit of the streamed data. Each subsequent unit is expected to find its data fully described by the same schema, even if some portions of the next incoming tuple are left blank or null.
Each schema is organized into one or more named fields. In a name-value pair, one field represents the name portion. Each incoming tuple has its own value for each field.
In general, a StreamBase application has three parts:
It ingests one or more incoming streams of data in components call Input Streams.
It processes and analyzes the incoming data, tuple by tuple, in thousands of possible ways.
It emits tuple data in components called Output Streams.
The three-part structure in that list can be summarized as:
Input stream to internal processing to output stream.
This is the simplest possible StreamBase application format, but it presumes that the incoming data streams are already in the StreamBase tuple format.
More likely is that the incoming data stream is in its own format and needs to be adapted to use the tuple format. StreamBase components called adapters are responsible for this transformation:
Incoming data streams are transformed by input adapters that transform a public or proprietary streaming data format to the SteamBase tuple format.
Outgoing data streams transform the StreamBase tuple format to another appropriate data format.
The addition of adapters allows for three more structures for StreamBase applications:
Input adapter to internal processing to output stream.
Input streams to internal processing to output adapter.
Input adapter to internal processing to output adapter.
Of course, StreamBase applications are not restricted to a single Input Stream or input adapter. One application can ingest several streams from different sources, including from JDBC databases. Likewise, an application can emit data in several Output Streams or output adapters, or can write to one or more JDBC databases.
TIBCO StreamBase EventFlow™ is the name of the programming language that underlies every StreamBase application.
EventFlow is stored on disk in an XML format, but that XML format is not exposed or documented. In fact, direct editing of the underlying EventFlow XML is not supported by TIBCO.
Instead, TIBCO StreamBase includes TIBCO StreamBase Studio™, an Eclipse-based development environment that lets you specify your application logic by means of a graphical, drag and drop component system that validates the generated EventFlow code at every step. StreamBase Studio is described on the next page.