IPFIX Adapter

Introduction

The Spotfire Streaming Adapter for IPFIX allows a StreamBase application to parse IPFIX messages to produce template, option template, and data record tuples.

Adapter Properties

This section describes the properties you can set for this adapter, using the various tabs of the Properties view in StreamBase Studio.

General Tab

Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.

Adapter: A read-only field that shows the formal name of the adapter.

Class name: Shows the fully qualified class name that implements the functionality of this adapter. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.

Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this adapter starts.

Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.

Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.

Adapter Properties Tab

Property Description
Dictionary File The dictionary file that specifies that enterprise entities. This file is a JSON formatted file that must conform to the JSON schema described in this section
Data Field The field of the input tuple that contains the binary IPFIX payload.
Output Template With Data If enabled, the template or option template associated with the data records is set in the output tuple.
Enabled Pass Through Fields If enabled, the input tuple is copied to a field in the output tuple.
Pass Through Output Field Name The name of the field to be added to the output tuple which will contain the input tuples data.
Capture Transform Strategy The strategy to use when transforming capture fields for this operator: FLATTEN or NEST.
Log Level Controls the level of verbosity the adapter uses to send notifications to the console. This setting can be higher than the containing application's log level. If set lower, the system log level is used. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE.

Cluster Aware Tab

Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.

Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.

Concurrency Tab

Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.

Caution

Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.

Data Input Port

The data input port must contain at least one blob field that includes the IPFIX binary information to parse. You must also specify the field name in the Data Field adapter property.

Data Output Port

Use the data output port to output parsed IPFIX sets. Sets can contain one or more of Templates, Options Templates, or Data records. The data output port schema is:

Field Name Field Type Description
Header tuple The parsed packet header with the following fields:

  • Version (int)—Version of IPFIX to which this Message conforms.

  • Length (int)—Total length of the IPFIX Message, measured in octets, including Message Header and Set(s).

  • ExportTime (timestamp)—Time at which the IPFIX Message Header leaves the Exporter.

  • SequenceNumber (long)—Incremental sequence counter of all IPFIX Data Records sent in the current stream from the current Observation Domain by the Exporting Process.

  • ObservationDomainId (long)—An identifier of the Observation Domain that is locally unique to the Exporting Process.

Sets list(tuple) A Set is a collection of records that have a similar structure, prefixed by a header. In an IPFIX Message, zero or more Sets follow the Message Header. There are three different types of Sets: Template Sets, Options Template Sets, and Data Sets. The sets list tuple contains the following fields:

  • Header (tuple)—The set header tuple which contains the following tuple schema:

    • Id (int)—Identifies the Set. A value of 2 is reserved for Template Sets. A value of 3 is reserved for Options Template Sets. Values from 4 to 255 are reserved for future use. Values 256 and above are used for Data Sets. The Set ID values of 0 and 1 are not used, for historical reasons

    • Length (int) - Total length of the Set, in octets, including the Set Header.

  • TemplateRecords (list(tuple))—The Template Records contained in the set which contains the following fields:

    • Header (tuple)—The Template Record Header which contains the following tuple schema:

      • TemplateId (int)—Each Template Record is given a unique Template ID in the range 256 to 65535.

      • FieldCount (int)—Number of fields in this Template Record.

    • Field Specifiers (list(tuple)) — The list of field specifier tuples that describe the template record and which contains the following tuple schema:

      • Enterprise (boolean)—If false, the entity id identifies an entity in [IANA-IPFIX], and the Enterprise Number will be null. If true, the entity id identifies an enterprise-specific Information Element, and the Enterprise Number field will be present.

      • EntityId (int)—A numeric value that represents the Entity Id Information Element. Refer to [IANA-IPFIX]

      • FieldLength (int)—The length of the corresponding encoded Entity.

      • EnterpriseNumber (long) — IANA enterprise number [IANA-PEN] of the authority defining the Entity Information Element identifier in this Template Record

      • Scope (boolean)—Always false for Template Records.

  • DataRecords (list(tuple))—The data records as defined by the given JSON dictionary file. Fields specified in the dictionary that are not present in the current data record are ignored and left as null.

  • OptionsTemplateRecords (list(tuple)) — The Template Records contained in the set which contains the following fields:

    • Header (tuple)—The Template Record Header which contains the following tuple schema:

      • TemplateId (int)—Each Template Record is given a unique Template ID in the range 256 to 65535

      • FieldCount (int)—Number of fields in this Template Record

      • ScopeFieldCount (int) — Number of scope fields in this Options Template Record

    • Field Specifiers (list(tuple)) — The list of field specifier tuples that describe the template record which contains the following tuple schema:

      • Enterprise (boolean)—If false, the entity id identifies an entity in [IANA-IPFIX], and the Enterprise Number will be null. If true, the entity id identifies an enterprise-specific Information Element, and the Enterprise Number field will be present.

      • EntityId (int)—A numeric value that represents the Entity Id Information Element. Refer to [IANA-IPFIX]

      • FieldLength (int)—The length of the corresponding encoded Entity.

      • EnterpriseNumber (long) — IANA enterprise number [IANA-PEN] of the authority defining the Entity Information Element identifier in this Template Record

      • Scope (boolean)—True if a field is considered to be a scope field.

Dictionary Output Port

The dictionary output port will output the parsed dictionary in tuple form, for reference.

The schema for the dictionary output port is:

Field Name Field Type Description
Enterprises list(tuple)

A list of all the enterprises in the dictionary with the following schema:

  • Id (long)—The enterprise Id

  • Organization (string)—The enterprise organization name.

  • Contact (string)—The enterprise contact name.

  • Email (string)—The enterprise contact email.

  • Entities (list(tuple))—The list of entities (Information Element Identifiers) contained in the enterprise with the following tuple schema:

    • Id (int)—The entity Id

    • Name (string)—The entity name

    • DataType (string)—The entity data type

    • DataTypeSemantics (string)—The entity data type semantics

    • Units (string)—The entity units

Status Output Port

The status output port will output tuples giving relevant information.

The schema for the status output port is:

Field Name Field Type Description
Status String A string describing the current status of the adapter.
Time timestamp The timestamp that the status occurred.
Info list(tuple) A list of name value pairs of data to got more context to the status.

The following is a list of status outputs and the associated objects:

Status Info Elements
Parse Error
  • Error—The parse error human readable string to describe the type of parsing error.

  • SequenceNumber—The packet sequence number that has the error. This field may or may not be present depending on the type of error.

  • Length—The packet length. This field may or may not be present depending on the type of error.

  • Remaining—The number of bytes remaining in the buffer. This field may or may not be present depending on the type of error.

  • HeaderSize—The complete packet header size. This field may or may not be present depending on the type of error.

  • Template—The template associated with the error. This field may or may not be present depending on the type of error.

Field Parse Error
  • Error—The field parse error human readable string to describe the type of field parsing error.

  • Element—The field element that failed to parse.

  • EnterpriseId—The Enterprise Id associated with the field element.

  • EntityId—The Information Element Identifier entity associated with the field element.

Missing Dictionary Field
  • EnterpriseId—The enterprise number given by the incoming template which is not contained in the given JSON dictionary file. This status occurs only once for each missing element.

  • EntityId—The Information Element Identifier entity given by the incoming template which is not contained in the given JSON dictionary entities file under the associated enterprise id. This status occurs only once for each missing element.

The Enterprise and Entity Dictionary

The dictionary is used to determine what fields are parsed from the IPFIX packets.

Dictionary JSON Schema

The following is the JSON schema required for the dictionary file.

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "properties": {
        "includeDictionary": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "file": {
                        "type": "string"
                    }
                },
                "required": [
                    "file"
                ]
            }
        },
        "enterprises": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "id": {
                        "type": "integer"
                    },
                    "organization": {
                        "type": "string"
                    },
                    "contact": {
                        "type": "string"
                    },
                    "email": {
                        "type": "string"
                    },
                    "entities": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "properties": {
                                "id": {
                                    "type": "integer"
                                },
                                "name": {
                                    "type": "string"
                                },
                                "dataType": {
                                    "type": "string"
                                },
                                "dataTypeSemantics": {
                                    "type": "string"
                                },
                                "units": {
                                    "type": "string"
                                }
                            },
                            "required": [
                                "id",
                                "name",
                                "dataType"
                            ]
                        }
                    }
                },
                "required": [
                    "id"
                ]
            }
        }
    },
    "required": [
        "enterprises"
    ]
}

Dictionary Layout

The following sections list the major categories of the dictionary file and provide an overview of their usage.

includeDictionary

The includeDictionary section is an array of files that allows one dictionary file to include another dictionary file. This enables you to organize your dictionaries and combine them into a single dictionary.

  • file—The relative path to the linked dictionary file to include.

enterprises

The enterprises section allows you to add enterprise values.

  • id—A long value representing the enterprise.

    organization—(Optional) The organization name.

    contact—(Optional) The enterprise contact.

    email—(Optional) The enterprise contact email.

    entities—An array of entities associated with this enterprise:

    • id—The integer item value representing the item. This value is left 0 padded and used to match the packets item type.

    • name—(Optional) The description of this item.

    • dataType—The data type of this entity which is used to determine the associated StreamBase data type when creating the output schema, this value is also used to determine how to parse the entity field from the binary data. Allowed values are:

      • boolean—Maps to a StreamBase boolean field

      • dateTimeMicroseconds—Maps to a StreamBase tuple containing a DateTimeSeconds timestamp field and a Fractions long field

      • dateTimeMilliseconds—Maps to a StreamBase timestamp field

      • dateTimeNanoseconds—Maps to a StreamBase tuple containing a DateTimeSeconds timestamp field and a Fractions long field

      • dateTimeSeconds—Maps to a StreamBase timestamp field

      • float32—Maps to a StreamBase double field

      • float64—Maps to a StreamBase double field

      • ipv4Address—Maps to a StreamBase string field

      • ipv6Address—Maps to a StreamBase string field

      • macAddress—Maps to a StreamBase string field

      • octetArray—Maps to a StreamBase blob field

      • signed16—Maps to a StreamBase int field

      • signed32—Maps to a StreamBase int field

      • signed64—Maps to a StreamBase long field

      • signed8—Maps to a StreamBase int field

      • unsigned16—Maps to a StreamBase int field

      • unsigned32—Maps to a StreamBase long field

      • unsigned64—Maps to a StreamBase long field

        Note

        Java does not fully support unsigned data types, and as such StreamBase may display any unsigned64 with a length of 8 as a negative value.

      • unsigned8—Maps to a StreamBase int field

    • dataTypeSemantics—The data type semantics associated with this entity. This value is not used internally and is only output on the dictionary output port for reference.

    • units—The units associated with this entity. This value is not used internally and is only output on the dictionary output port for reference.

Suspend and Resume Behavior

When suspended, the adapter no longer processes tuples.

When resumed, the adapter outputs the current dictionary file on the dictionary output port, and then starts processing input tuples.