Persistence of Query Tables

Contents

In-Memory or On-Disk Query Tables
Setup For On-Disk Query Tables
On-Disk Query Table Parameters
Related Topics

Query Tables are containers for sharing data and maintaining state within a StreamBase application. They accomplish this by storing tuple values outside of any stream. At runtime, a Query Table can have its values modified by one or more Query operators that write to, read from, delete from, or update their associated Query Table.

StreamBase applications can share query tables within modules, between nested modules, and across containers. Query Tables can be associated with multiple Query operators, but each Query operator can be associated with only one Query Table.

By default, Query Tables are ephemeral. That is, they are stored in system memory that is released when the StreamBase application that created the tables exits. You can however, put a Query Table in permanent storage, in which case it can persist between sessions that use it. The two types of tables are termed memory-based and disk-based, respectively. Throughout this topic, remember that disk-based query tables are a feature that you can use only if your StreamBase license enables it.

In-Memory or On-Disk Query Tables

As stated above, any data stored at runtime in a memory-resident table is not saved after the hosting StreamBase Server is shut down. For disk-based Query Tables, the data can persist indefinitely. For example, you could store data for a StreamBase application that runs only for several hours a day, and requires data from the previous run during initialization each day. Another possible use of persistent Query Tables tables is to maintain state information for a StreamBase application.

In the following figure, the Table Settings tab of the Properties view for a Query Table data construct is shown with all default values. Notice that the selected Query Table is declared to be in-memory.

The Type control is a drop-down menu that has two options: In memory and On disk (requires extra license).

For a local table, you can write over the default selections to specify the name of a parameter in the form ${param-name}. This allows you to pass in the choice of table type at runtime.

The following example sets the Type control to the parameter ${tabletype}. The module that contains this table must contain or be passed a parameter of that name with either disk or memory as its value.

Setup For On-Disk Query Tables

In the case of memory-resident Query Tables (and disk-based Materialized Windows), data stored in the table at runtime is not saved after StreamBase Server shuts down. In the case of disk-based Query Tables, the data stored in the tables persists between StreamBase Server sessions.

Note

The use of disk-based Query Tables (or disk-based Materialized Windows) is not available when using a StreamBase evaluation or developer license. The disk-based Query Table option must be enabled in your StreamBase license. Contact TIBCO for further information on this licensing option.

To make disk-based Query Table data persist between StreamBase Server sessions, follow these steps:

  1. In the Table Settings tab of the Query Table's Properties view, set the Type property to On disk.

  2. Set the Truncate on startup property to No.

  3. Specify a file system location in which the application is to store its on-disk query tables using one of the following techniques:

    • On the sbd command line, use the --datadir option.

    • Use the STREAMBASE_DATA environment variable.

    • In the StreamBase Server configuration file, define a datadir parameter in the <server> element. You must specify a closing backslash for Windows to indicate a directory path. For example: 

      <server>
  <param name="datadir" value="C:\Apps\qtstorage\" />
</server>

      Use a closing slash for UNIX paths:

      <server>
  <param name="datadir" value="/home/sbuser/apps/qtstorage/" />
</server>

    If a data directory is not specified, or if the special value +TEMP+ is defined, StreamBase creates a temporary directory on server startup, which is deleted on server shutdown.

On-Disk Query Table Parameters

StreamBase also provides options for transactional updates of disk-based Query Tables, adding new configuration parameters for the <server> element of a server configuration file. For example: 

<server>
  <param name="datadir" value="C:\myapps\streambase\datadir\" />
  <param name="disk-querytable-transaction_mode" value="1" />
  <param name="disk-querytable-flush-interval-ms" value="5000" />    
</server>

See StreamBase Server Configuration File XML Reference for the XML syntax of server configuration files.

If your application uses disk-based Query Tables, you may be able to improve performance by setting a value for the following parameter for the <server> element of the configuration file: 

<param name="disk-querytable-cache" value="nnn">

The value="nnn" attribute sets the amount of main memory that will be used to cache data from disk-based Query Table operations. The units are in MB of main memory and must be a power of 2. When unspecified, the default value is 1 MB. Use caution when setting this parameter, because too high a value may consume more memory than needed, and could negatively impact other resources that require memory during the execution of the StreamBase application or other applications. As with any resource setting, establish baseline performance metrics and then test the effect of increasing or decreasing values under normal and peak loads.

Related Topics

Back to Top