Syslog Input Adapter

Introduction

The Spotfire Streaming Syslog Input Adapter allows a StreamBase application to act as a Syslog collector for receiving syslog messages.

The adapter has the following capabilities:

  • Listens for syslog messages sent to the given port number.

  • When such a message is received, outputs a tuple describing the values contained in the message.

  • Optionally receives commands from the StreamBase application to connect (start listening) and disconnect (stop listening).

  • Optionally issues status tuples on a dedicated to inform the application of interesting events such as connection and disconnection.

The adapter exposes a set of input and output ports, the fields for which are defined later in this document. In addition, the adapter exposes a set of adapter properties as described in Adapter Properties.

Adapter Properties

Settings are used to control most of the behavior of the adapter. The settings are grouped under several tabs in the Properties view in StreamBase Studio.

In the table in this section, the Property column shows each property name as found in the one or more adapter properties tabs of the Properties view for this adapter.

Input Adapter Properties

General Tab

Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.

Adapter: A read-only field that shows the formal name of the adapter.

Class name: Shows the fully qualified class name that implements the functionality of this adapter. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.

Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this adapter starts.

Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.

Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.

Adapter Properties Tab

Property Data Type Default Description
UDP port int 514 The port number to use, on which the adapter will listen for incoming messages from Syslog agents.

Note

This setting is required.

Parse Incoming Messages boolean (check box) true (selected) When selected, the adapter attempts to parse and interpret incoming syslog messages in order to extract meaningful information (such as the hostname of the machine originating the message, structured data elements, etc). Otherwise the entirety of the message will be placed in the rawMessage field (along with a timestamp in the receiptTime field) and all other fields will be left null. See Message Parsing Rules for details on the parsing rules applied by the adapter.
Connect On Startup boolean (check box) true (selected) When selected, the adapter attempts to open the specified UDP port and start listening for incoming messages immediately after starting. If cleared, the adapter remains idle after starting and waits for a Connect command as described in Command Input Port.
Add Command Input Port boolean (check box) false (cleared) When selected, the adapter adds an input port for the app to issue commands to the adapter, for example to make it start or stop listening for incoming messages. The schema for this port is described in Command Input Port.
Add Status Output Port boolean (check box) false (cleared) When selected, the adapter adds an output port to issue events that may be of interest to the application, for example to inform the app that the adapter has started or stopped listening for incoming traps. The schema for this port is described in Status Output Port Schema.
Log Level drop-down list INFO Controls the level of verbosity the adapter uses to send notifications to the console. This setting can be higher than the containing application's log level. If set lower, the system log level will be used. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE.

Cluster Aware Tab

Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.

Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.

Concurrency Tab

Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.

Caution

Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.

Adapter Ports And Schemas

Input Port Overview

The adapter has at most one input port, depending on its configuration (see Add Command Input Port in Adapter Properties Tab).

Command Input Port

The (optional) Command input port is used to instruct the adapter to perform certain actions, such as connect or disconnect from the UDP port on which it listens for traps.

The schema for the Command Input port is expected to contain the following fields:

Field Name Field Type Field Description
command string Contains the name of the command to execute. This field is required to be non-null for all supported commands. See below for a list of the available commands.
params list(string) All-purpose list of string parameters that may be of use to specify additional information to apply the command. Currently, this field is not used by any of the supported commands and is only here to allow for future expansion.

This field may be left null or empty if no parameters are required or needed.

Available commands include:

  • Connect: Start listening for syslog messages on the configured UDP port. Required fields:

    • command: Should contain the value Connect. Case is unimportant.

  • Disconnect: Stop listening for syslog messages. Required fields:

    • command: Should contain the value Disconnect. Case is unimportant.

Output Port Overview

The adapter has at least one output port, and optionally a second one:

  • The first port issues a tuple for each syslog message received, and describes its content.

  • The second (optional) port routes status messages, such as connection and disconnection events.

The schema for each output port is described in the following sections:

Syslog Message Output Port Schema

The first output port routes incoming syslog messages to the StreamBase application.

The schema for this port is pre-set by the adapter, and is described in the following table.

Note

Only the rawMessage and receiptTime fields are guaranteed to be non-null in all cases. The validity of the other fields in the schema will be dependent on whether the adapter is configured to attempt further parsing of the message, and the extent to which this parsing was successful. The parsing rules (and the fields they affect) are described in Message Parsing Rules.

Field Name Field Type Description
rawMessage string The entire syslog message, verbatim. This field will never be null.
receiptTime timestamp The time at which the message was received by the adapter. This field will never be null.
priority int The priority value of this message. This number is used to derive the originating facility as well as the severity of the message (specifically: facility * 8 + severity).
facility int The facility that generated this message. Facilities are defined (in RFC 5424, "The Syslog Protocol" (March 2009)) as follows:
0 - kernel messages
1 - user-level messages
2 - mail system
3 - system daemons
4 - security/authorization messages
5 - messages generated internally by syslogd
6 - line printer subsystem
7 - network news subsystem
8 - UUCP subsystem
9 - clock daemon
10 - security/authorization messages
11 - FTP daemon
12 - NTP subsystem
13 - log audit
14 - log alert
15 - clock daemon
16 - local use 0
17 - local use 1
18 - local use 2
19 - local use 3
20 - local use 4
21 - local use 5
22 - local use 6
23 - local use 7
facility int The severity of this message. Severities are defined as follows:
0 - Emergency: system is unusable
1 - Alert: action must be taken immediately
2 - Critical: critical conditions
3 - Error: error conditions
4 - Warning: warning conditions
5 - Notice: normal but significant condition
6 - Informational: informational messages
7 - Debug: debug-level messages
timestamp timestamp The time at which this event was generated. If the message conforms to RFC 5424, the string from which the timestamp is derived includes correct timezone information; if the message only conforms to RFC 3164, the string describing the timestamp did not include timezone information (since the spec makes no allowance for it) and the timestamp is interpreted as local time.
hostname string The host name or IP address of the originating system.
appname string The name of the device or program that generated the message.
procID string The ID of the process that originated the message.
msgID string The type of message.
msg string The actual message portion of this syslog entry.
structuredData list<tuple> Messages conforming to RFC 5424 may contain "structured data". This field contains the list of entries for such data.
^ id string Case-sensitive identifier for this entry.
^ params list<tuple> Contains the list of name/value pairs in this entry.
^ ^ name string The name of this parameter.
^ ^ value string The value of this parameter.

Status Output Port Schema

The second output port routes Status messages to the StreamBase application, provided the adapter has been configured to expose this optional port (see Add Status Output Port in Adapter Properties Tab).

The schema for this port is pre-set by the adapter, and is described in the following table:

Field Name Field Type Field Description
type string Describes the type of status being emitted. Currently the only value supported is Event.
action string The action that triggered this status tuple. One of:
  • Connected - Connection has been established and the adapter is now listening for incoming traps.

  • Disconnected - Adapter has disconnected from the UDP port and has stopped listening for traps.

info list(string) Contains extra information pertaining to the event. May be null or empty depending on the type of event being described.

Message Parsing Rules

Syslog messages do not have to conform to a hard and fast format. The only real requirement is that only one message be present in a given UDP packet. However over the years some conventions have emerged and some specifications have even been published, with varying degrees of adoption.

When the adapter receives a syslog message it will at a minimum set the rawMessage field of the outgoing tuple to contain the entirety of this message, and the receiptTime field will contain a time stamp indicating the exact time at which the message was received by the adapter (this is not to be confused with the timestamps often contained within the message itself, which represent the time at which the message was sent). These two fields are the only ones that are guaranteed to be non-null and meaningful in all cases, regardless of whether further interpretation has been attempted and was successful.

When the adapter is configured to attempt further parsing of the messages (see Adapter Properties Tab), incoming syslog messages are examined in order to extract meaningful information (such as the hostname of the machine originating the message, RFC 5424 "structured data elements", etc). If identified in the parsing process, these discrete elements are placed in dedicated fields in the output schema to aid the application in interpreting and characterizing the message.

Several parsing attempts are made, according to the following rules:

  1. Parse using the specification outlined in RFC 5424, "The Syslog Protocol" (March 2009).

  2. If the message's format does not conform to the above, parse it using the specification outlined in RFC 3164, "The BSD Syslog Protocol" (August 2001).

  3. If the message still cannot be parsed successfully, attempt to at least extract the message's priority by checking if it begins with the pattern <NNN> where NNN is an integer number. If successful, the priority is extracted and the rest of the message is treated as a single string.

  4. Failing all these attempts (or if Parse Incoming Messages is unchecked) the entirety of the message is placed in the rawMessage field and all other fields are left null.

The complete list of fields in the output schema describing a syslog message is given in Syslog Message Output Port Schema. The table below describes the circumstances in which a given field's value may be set. Note of course that even when a message conforms to a given format some fields may still be null if they had no value in the message (for example an RFC 5242 message may leave its 'appname' component unset while still providing meaningful values for other fields).

Field name Parsing rules from the above list that may set its value
rawMessage 1, 2, 3, 4
receiptTime 1, 2, 3, 4
priority 1, 2, 3
facility 1, 2, 3
severity 1, 2, 3
timestamp 1, 2
hostname 1, 2
appname 1
procID 1
msgID 1
msg 1, 2, 3
structuredData 1

Client System Configuration

Devices and programs generating syslog messages do so by sending them to the system's Syslog daemon (syslogd). This daemon is the program responsible for properly routing the messages to local logs file and/or collectors such as the adapter. The syslogd daemon needs to be explicitly told to forward messages to your StreamBase application in order for the adapter to receive them. This is done by modifying the /etc/syslog.conf configuration file. To forward all syslog messages to your application, ensure the following line is included (replacing the host name by that of the machine running StreamBase):

*.* @logserv.example.com

(You will need root access to make this change.)

Afterwards, restart syslogd with the command:

service syslogd restart

From then on the client machine will forward all syslog entries to your StreamBase application. Refer to syslogd(8) and syslog.conf(5) documentation for instructions on configuring finer-grained forwarding rules (e.g. only forwarding messages of a certain priority).

Typechecking and Error Handling

The adapter uses typecheck messages to help you configure the adapter in your StreamBase application. In particular, the adapter generates typecheck messages for the following reasons:

  • A required property is missing.

  • One or more required fields in the Command input port is missing or is of the wrong type or size.

Suspend and Resume Behavior

When suspended, the adapter remains connected to the configured UDP port if it was already connected, but all incoming traps are dropped.

When resumed, the adapter starts routing incoming traps to the StreamBase application again.

Related Topics