Designing Applications for High Availability

This page builds on information in Primer on Clustering and HA Using StreamBase, especially the list in StreamBase Components That Support Clustering.

Before beginning your high availability (HA) application's design, establish which HA problem you are tasked with solving. Have in mind exactly which problem you're seeking to solve with HA techniques. See What is StreamBase Clustering? for a discussion of the goals of several different HA design patterns.

There are several components used to assemble StreamBase clustering and HA solutions, as listed in StreamBase Components That Support Clustering. Each solution uses these components in a different way. Remember that making a high-availability application is a complex event processing problem. Take advantage of StreamBase's stream processing fundamentals to find creative ways to implement a high-availability application that meets your site's design goals.

In StreamBase, there are common design components for clustering and HA applications. This page describes these HA design features as patterns for your application to build on.

The use of containers is essential in an HA application. Containers allow you to separate your main application's logic from the HA-related logic. You can start and stop the application in a container dynamically, and you can dynamically start and stop enqueuing and dequeuing of individual ports.

This means your HA container can send commands to your application container to enable and disable processing based on events such as a change in leadership status, or missed heartbeat from the other member of a cluster pair.

The following diagram illustrates the separation of application and HA logic into containers. The system container is always present in every sbd instance.

Use the HA Heartbeat adapter in your HA container's application. The adapter is used in pairs, with one member of the pair designated as heartbeat server. The HA Heartbeat adapter in the other StreamBase Server's HA container is designated the client of the heartbeat server, and does the work of connecting to the heartbeat server, and of reporting failure to connect.

See HA Heartbeat Input Adapter for instructions. See the High Availability Sample for an example implementation.

Event tuples are sent on the system.control output stream to announce any change in leadership status of the containing sbd server, and any container startup event. The application in your HA container can monitor the system.control output stream for these events and take appropriate action.

Instead of using explicit HA programming logic and the sbadmin command, you can access system.control events directly in your HA applications and use the configuration file for each HA server to specify how to handle changes in status.

Establish a container connection to connect system.control as the tuple source for the control port's input stream for each HA application in each HA container. You can use any of ways to make a container connection described in Container Connections, but the simplest is to use the Container connection property of the input stream:

The use of container connections imposes some restrictions on the startup order of containers, as described in Container Start Order.

When you enable automatic HA processing, StreamBase creates a container named ha on StreamBase Server. This newly created ha container has an application that tracks the leadership status between two specified servers. See Automatic HA, Using Control Stream Features and High Availability Sample.

The External Process operator provides a way for a StreamBase application to invoke a command-line utility at the operating system shell level. You can take advantage of this operator to let the HA logic in an HA container send sbadmin or sbc commands to the application logic in the application container of either server. See Using the External Process Operator.

To implement a hot-warm template means having the secondary server's application logic running and processing tuples, but not sending output tuples downstream. (See Hot-Warm Server Pair Template for more on the hot-warm template.)

To suspend output tuples, the output streams in the secondary server's application logic can be started with the --dequeue=DROP_TUPLES option. Or to drop messages on input, use the --enqueue=DROP_TUPLES option. On a failover event, send an sbadmin modifyContainer command to re-enable enqueuing or dequeuing with the ENABLED status. See sbadmin in the Reference Guide for more on this feature.

Shared disk access does not necessarily mean that your application allows two separate StreamBase Server instances on separate hosts to write to the same disk Query Table at the same time. Best practice is to put all application logic that implements disk Query Table access into its own container in each server. In this case, when a failover event is detected, such as a change in leadership status, or a failure to detect an HA heartbeat, the HA logic on the secondary server can send an sbadmin command to start the container that holds the disk Query Table. In this way, you can ensure that only one application is writing to the shared disk Query Table at a time.

To minimize startup time when restarting an sbd server, use precompiled applications. See Precompiled Application Archives.

Most of the design patterns on this page are illustrated in High Availability Sample and High Availability Shared Disk Access Sample. Study the sample applications thoroughly before beginning your own application design.

To ramp up to speed quickly, you can use High Availability Sample as a starting point, substituting your site's stream processing StreamBase application in place of the sample's app.ssql file. Or use High Availability Shared Disk Access Sample as your base, substituting your stream processing application in place of that sample's qtapp.sbapp. The Query Table Replication Sample shows a more complex design pattern.