Detecting Patterns

Pattern matching on a single stream is only really useful if you include a predicate that is satisfied by the events happening in a particular order. When you want to detect patterns within tuples, or events arriving on a single stream, you can use aliasing to indicate how many events are included in the template. For example, if you want to consider pairs of events, the template will reference the same stream twice applying a different alias to each reference. The following StreamSQL template indicates that the pattern involves two events arriving in a specified order on the stream named InputStream:

InputStream AS stream1 
    THEN InputStream AS stream2

For patterns involving a greater number of events, the template becomes more complex. This StreamSQL template describes a pattern involving three events:

(InputStream AS s1 AND InputStream AS s2)
    THEN InputStream AS s3

Note how parentheses are used to specify the pattern; the AND operator indicates that the order of the first two events is not specified, but both of these events must occur before the third event, which follows the THEN operator.

To complete either of these pattern specifications, you must specify the period of time within which the pattern must be detected and apply constraints that would further limit the events that satisfy the pattern. Without these additional entries, the template itself has little meaning.

Let's more fully flush out the details of an example, first using StreamSQL and then an EventFlow diagram.

CREATE INPUT STREAM InputStream
  (stock string, price double, shares int);

CREATE OUTPUT STREAM Out AS
  SELECT s1.stock AS stock1, s1.price AS price1,
         s1.shares AS shares1, s2.shares AS shares2
  FROM PATTERN InputStream AS s1 THEN
               InputStream AS s2 THEN
               InputStream AS s3
  WITHIN 20 TIME
  WHERE s1.stock=s2.stock AND s2.stock=s3.stock AND
        ((s3.price>s2.price) OR (s3.price>s1.price));

A single input stream is referenced three times in the FROM PATTERN clause; consequently, each stream must later be referred to through its alias named. The WITHIN clause specifies that the desired pattern be detected within 20 seconds; fractional seconds, for example, 20.5, are also acceptable. Finally, the WHERE clause specifies that the pattern involves three events with identical values in the stock field, where the value in the price field of the third event is greater than the price field in either the first or second event. The target list following the SELECT keyword lists the field values that will be included in the tuple emitted by this statement when a successful pattern match has been identified.

The following EventFlow application produces the same result.

Two tabs within the Pattern operator's StreamBase Properties view are used to configure this operator. On the Pattern Settings tab, enter the template, size value, and predicate (which correspond to the FROM PATTERN, WITHIN, and WHERE clauses in the StreamSQL statement).

On the Output Settings tab specify the fields that will be included in the emitted tuple. Since an arc in an EventFlow application does not have a name, in the following figures the incoming stream is referred to using a name derived from its corresponding input port. The single input port in this example is named input1.

Start either version of this application and then enqueue the following tuples. (You must submit all of the tuples within the specified time period. If you are uncomfortable with this constraint, increase the value.)

After enqueuing the third tuple, nothing was emitted on the output stream. This is correct as s3.price was not greater than the price value in either of the first two tuples.

After enqueuing the fourth tuple, three tuples are emitted.

Since all the tuples were enqueued during the specified time duration, the first emitted tuple results from the first, second and fourth tuples satisfying the pattern and the second emitted tuple results from the first, third and fourth tuples satisfying the pattern. Finally, the third tuple results from the second, third and fourth tuples satisfying the pattern.

As a further example, you should enqueue the following eight tuples.

If you work quickly enough so that all eight tuples are enqueued within the specified time duration, no tuples are emitted until the eighth tuple is enqueued. Then, 21 tuples are emitted. Review the values in the shares1 and shares2 fields, which illustrates that multiple matching patterns have been detected.

Now, investigate the effect of changing the value of the stock field in some of the tuples. Just make sure you enqueue at least three tuples with the same stock field value, and make sure the price field in the third tuple is greater than the price field values in either the first or second tuple.