HTTP Client Input Adapter

Contents

Introduction
HTTP Client Properties
URL Encode And Decode Types
Input Control Port
Data Output Port
Status Output Port
Suspend and Resume Behavior
Related Topics

Introduction

The Spotfire Streaming HTTP Client adapter performs HTTP requests and emits the contents via the Data output port.

The adapter has multiple samples, described in HTTP Client Samples. These samples demonstrate how to perform various HTTP request types including server-sent events and chunked response.

Almost all properties can be overridden via the input control tuple at runtime. Options such as Output Headers, Output Cookies, Output Input Tuple, and Executor threads which would affect the schema or run system cannot be overridden at runtime.

REST, SOAP, WEB, SSE (Server-Sent Events), CHUNKED, GET, POST, PUT

HTTP Client Properties

This section describes the properties you can set for this adapter, using the various tabs of the Properties view in StreamBase Studio.

General Tab

Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.

Adapter: A read-only field that shows the formal name of the adapter.

Class name: Shows the fully qualified class name that implements the functionality of this adapter. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.

Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this adapter starts.

Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.

Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.

Adapter Properties Tab

Property Description
URL The URL endpoint to connect to. This is the default value used when the input tuple's URL field is null. If this value is empty and the input tuple URL field is empty, an error message is emitted on the status port.
Request Method The method of request to send. This contains the default value used when the input tuple's settings.requestMethod field is missing or empty.
Request Type The type of request to start. This contains the default value used when the input tuple's settings.requestType field is missing or empty. The Request Type can be one of:

  • Basic, Used for standard HTTP Requests that emit the results via the data field.

  • File Download, Used for standard HTTP Requests but when you want to store the contents as a binary payload or file. If the Download Path is set via the property or input tuple the request results are stored in that file and the data field contains the full path to the file. If no Download Path is specified the results are output via the binary field of the output tuple.

  • Server-Sent Events, Used to request server-sent events streams from the HTTP server.

  • Chunked Response, Used to request chunked response long poll requests from the HTTP server.

Download Path If the Download Path is set via the property or input tuple the request results are stored in that file and the data field contains the full path to the file. If no Download Path is specified the results are output via the binary field of the output tuple.
Asynchronous If enabled, the HTTP request is done asynchronously. This option has no effect when the request type is sever-sent events or chunked.
Log Level Controls the level of verbosity the adapter uses to send notifications to the console. This setting can be higher than the containing application's log level. If set lower, the system log level is used. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE.

Proxy Tab

Property Description
Use Proxy Use a proxy server in processing the HTTP GET request.
Proxy Host The proxy server host name or IP address.
Proxy Port The proxy server TCP port number.
Proxy User The username to use with the proxy if required.
Proxy Password The password to use with the proxy if required.

Advanced Tab

Property Description
Connect timeout The time in milliseconds to wait for the connection to establish, set to 0 to wait for ever.
Read timeout The time in milliseconds to wait for a read operation to complete, set to 0 to wait for ever.
Reconnect timeout Sets a specified timeout value, in milliseconds, to wait for reconnect attempts when using server-sent events. A timeout of zero is interpreted as an unlimited timeout.
Poll Frequency The time in milliseconds to wait between requesting, set to 0 to disable polling, note this option has no effect when the request type is sever-sent events or chunked response.
Ignore certificate errors If enabled, any errors produced by invalid TLS/SSL certificates are ignored and the website is processed as normal. Warning! This can lead to man in the middle attacks.
Executor Threads The number of threads to use for pooling connections.

Output Tab

Property Description
Output Input Tuple If enabled, the input tuple is emitted with every data output tuple.
Output Headers If enabled, the headers are emitted with every data output tuple.
Output Cookies If enabled, the cookies are emitted with every data output tuple.
Output Tuple Per Line If enabled, response from the server emits one tuple per line received. This option is only valid with basic requests.
Output Blank Lines This option sends blank tuples when a blank line is received. This option is only available when emitting tuples per line.
Output Null Tuple On Completion This option sends a tuple with all fields set to null when reading is complete. This option is only available when emitting tuples per line.
Output Connecting Status If enabled, a status message is emitted for each attempt to connect for a request.
URL Decode Data If enabled, any string data returned is URL decoded before being output.
URL Decode Data Type The type of URL Decode operation to perform; see Encode Types.
Use Default Charset If selected, specifies whether the Java platform default character set is to be used. If cleared, a valid character set name must be specified for the Character Set property.
Character Set The name of the character set encoding that the adapter is to use to read input or write output.

Auth Tab

Property Description
Auth Type The authentication type to use with the request

  • None, No authentication is performed.

  • Basic, Basic HTTP authentication is performed using the given username and password.

  • Basic Non-Preemptive. In non-preemptive mode the authentication information is added only when server refuses the request with 401 status code and then the request is repeated with authentication information.

  • Digest, HTTP digest authentication. Does not require usage of SSL/TLS.

  • OAuth 2 Client Credentials. A client ID and secret are used for authentication to an OAuth 2 token URL.

Username The username to be sent to the server.
Password The password to be sent to the server.
OAuth 2 Token URL The Token URL of an OAuth 2 server.
OAuth 2 Client ID The client ID to be provided to the OAuth 2 Token URL.
OAuth 2 Client Secret The client secret to be provided to the OAuth 2 Token URL.
OAuth 2 Scope The OAuth 2 scope to provide to the token URL.
OAuth 2 Additional Token Parameters Additional parameters to provide to the token URL separated by &. For example, "audience=...&state=...".

Cluster Aware Tab

Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.

Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.

Concurrency Tab

Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.

Caution

Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.

URL Encode And Decode Types

Name Description
AUTHORITY The URI authority component type.
FRAGMENT The URI fragment component type.
HOST The URI host component type.
MATRIX_PARAM The URI path component type that is a matrix parameter.
PATH The URI path component type.
PATH_SEGMENT The URI path component type that is a path segment.
PORT The URI port component type.
QUERY The URI query component type.
QUERY_PARAM The URI query component type that is a query parameter, space character is encoded as +.
QUERY_PARAM_SPACE_ENCODED The URI query component type that is a query parameter, space character is encoded as %20.
SCHEME The URI scheme component type.
UNRESERVED ALPHA / DIGIT / "-" / "." / "_" / "~" characters.
USER_INFO The URI user info component type.

Input Control Port

The HTTP Client adapter's input control port, all values are options with the exception of the url field. The schema is defined as:

Field Data Type Description
command string The command tells the adapter what action to take. If this field is missing or empty, the default value of connect is used.
url string The URL endpoint to connect to. If this value is empty, the default property value is used.
data list(tuple) This list of tuples can be used for POST and PUT operations to input the data that should be sent as the payload field. If multiple entries are given, the payload is sent as a multipart/mixed payload. If any one entry specifies more than one data type field, a multipart/mixed is also sent.
data.text string The payload as text to send with this request. If this field is missing or null, it is ignored. The type of this payload is defined by the data.mediaType field which defaults to text/plain if the data.mediaType property is missing or empty.
data.binary blob The payload in binary format to send with this request. Binary payloads ar2e always sent as a multipart/mixed payload. The type of this payload is defined by the data.mediaType field, which defaults to application/octet-stream if the data.mediaType property is missing or empty.
data.filePath string The path to a file to send with this request. Binary payloads are always sent as a multipart/mixed payload. The type of this payload is defined by the data.mediaType field, which defaults to application/octet-stream if the data.mediaType property is missing or empty.
data.mediaType string The media type to use with this payload, such as text/plain or application/json. If this field is missing or empty, the default is defined by whichever field has a payload.
data.name string If data is being sent as a binary payload, this name is sent to the server as the name part of the payload. If this field is missing or null, data is used.
data.urlEncode boolean This only applies to the data.text payload field. If true, the payload is URL-encoded before sending.
data.urlEncodeType string This only applies to the data.text payload field. If data.urlEncode is true, this value specifies the type of encoding; valid values are listed in Encode Types. The default value is QUERY_PARAM_SPACE_ENCODED.
headers list(tuple) A list of names to list(value) strings that represent the headers to send with this request.
headers.name string The header name. If the field is empty, the header is ignored.
headers.values list(string) A list of string values for this header. If this field is null or empty, the header is ignored.
queryParameters list(tuple) A list of query parameters to send with the request.
queryParameters.name string The name of the query parameter. If this field is empty, the parameter is ignored.
queryParameters.value string The value of the query parameter. Null values are defaulted to empty strings.
queryParameters.urlEncode string If true, the parameter is URL-encoded before sending. If this field is missing or null, the default is true.
queryParameters.urlEncodeType string This queryParameters.urlEncode is true, this value specifies the type of encoding; valid values are listed in Encode Types. The default value is QUERY_PARAM_SPACE_ENCODED.
cookies list(tuple) A list of cookies to send with the request.
cookies.name string The name of the cookie. If this field is empty, the cookie is ignored.
cookies.value string The value of the cookie. Null values are defaulted to empty strings.
settings tuple Settings that can be changed per tuple at runtime.
settings.requestMethod string The request method to use; valid values are GET, POST, PUT, HEAD, DELETE, and OPTIONS. If this field is missing, empty, or null, the default is the currently selected Request Method property of the adapter.
settings.requestType string The request type to use; valid values are Basic, File Download, Server-Sent Events, and Chunked Response. If this field is missing, empty, or null, the default is the currently selected Request Type property of the adapter.
settings.downloadPath string The path to store files during a File Download request. If this field is missing, empty, or null, the default is the currently selected Download Path property of the adapter.
settings.asynchronous boolean Determines whether this request should be sent synchronously or asynchronously. If this field is missing, empty, or null, the default is the Asynchronous property of the adapter.
settings.proxy tuple A tuple containing the proxy information to use for this request. If this field is missing, empty, or null, the default is the set of values in the Proxy properties tab.
settings.proxy.useProxy boolean Determines whether a proxy should be used for this request. If this field is missing, empty, or null, the default is the Use Proxy property of the Proxy properties tab.
settings.proxy.host string The proxy host to use for this request. If this field is missing, empty, or null, the default is the Proxy Host property of the Proxy properties tab.
settings.proxy.port int The proxy port to use for this request. If this field is missing, empty, or null, the default is the Proxy Port property of the Proxy properties tab.
settings.proxy.user string The proxy user to use for this request. If this field is missing, empty, or null, the default is the Proxy User property of the Proxy properties tab.
settings.proxy.pass string The proxy password to use for this request. If this field is missing, empty, or null, the default is the Proxy Password property of the Proxy properties tab.
settings.advanced tuple A tuple containing the advanced settings to use for this request. If this field is missing, empty, or null, the default is the set of values in the Advanced properties tab.
settings.advanced.connectTimeout int The connection timeout to use for this request. If this field is missing, empty, or null, the default is the Connect Timeout property of the Advanced properties tab.
settings.advanced.readTimeout int The read timeout to use for this request. If this field is missing, empty, or null, the default is the Read Timeout property of the Advanced properties tab.
settings.advanced.reconnectTimeout int The reconnect timeout to use for this request. This value is only used with server-sent event requests. If this field is missing, empty, or null, the default is the Reconnect Timeout property of the Advanced properties tab.
settings.advanced.pollFrequency int The poll frequency to use for this request. This value is only used with basic and file download requests. If this field is missing, empty, or null, the default is the Poll Frequency property of the Advanced properties tab.
settings.advanced.ignoreCertificateError boolean If true, TLS/SSL certificate errors are ignored for this request. If this field is missing, empty, or null, the default is the Ignore Certificate Errors property of the Advanced properties tab.
settings.auth tuple A tuple containing the authentication settings to use for this request. If this field is missing, empty, or null, the defaults are the values in the Auth properties tab.
settings.auth.authType string The authentication type to use for this request. Valid values are None, Basic, Basic Non-Preemptive, Digest, and OAuth 2. If this field is missing, empty, or null, the default is the Auth Type property of the Auth properties tab.
settings.auth.user string The authentication user to use for this request. If this field is missing, empty, or null, the default is the Username property of the Auth properties tab.
settings.auth.pass string The authentication password to use for this request. If this field is missing, empty, or null, the default is the Password property of the Auth properties tab.
settings.auth.oAuth2 tuple A tuple containing the OAuth 2 settings to use for this request. If this field is missing, empty, or null, the defaults are the values in the Auth properties tab. Ignored if Auth Type is not set to OAuth 2.
settings.auth.oAuth2.tokenUrl string The OAuth 2 token URL to use for this request. If this field is missing, empty, or null, the defaults are the values in the Auth properties tab. Ignored if Auth Type is not set to OAuth 2.
settings.auth.oAuth2.clientId string The OAuth 2 client ID to provide to the token URL for this request. If this field is missing, empty, or null, the defaults are the values in the Auth properties tab. Ignored if Auth Type is not set to OAuth 2.
settings.auth.oAuth2.clientSecret string The OAuth 2 client secret to provide to the token URL for this request. If this field is missing, empty, or null, the defaults are the values in the Auth properties tab. Ignored if Auth Type is not set to OAuth 2.
settings.auth.oAuth2.scope string The OAuth 2 scope to provide to the token URL for this request. If this field is missing, empty, or null, the defaults are the values in the Auth properties tab. Ignored if Auth Type is not set to OAuth 2.
settings.auth.oAuth2.additionalTokenParams string Additional parameters to provide to the token URL separated by &. For example, "audience=...&state=...". If this field is missing, empty, or null, the defaults are the values in the Auth properties tab. Ignored if Auth Type is not set to OAuth 2.
settings.output tuple A tuple containing the output settings to use for this request. If this field is missing, empty, or null the defaults are the values in the Output properties tab.
settings.output.outputTuplePerLine boolean If true, response from the server emits a tuple per line received. This option is only valid with basic requests. If this field is missing, empty, or null, the default is the Output Tuple Per Line property of the Output properties tab.
settings.output.outputBlankLines boolean If true, empty length response strings are output. This option is only available when emitting tuples per line. If this field is missing, empty, or null, the default is the Output Blank Lines property of the Output properties tab.
settings.output.outputNullTupleOnCompletion boolean If true, a status message is output for each attempt to connect for a request. If this field is missing, empty, or null, the default is the Output Connecting Status property of the Output properties tab.
settings.output.outputConnectingStatus boolean If true, the request sends a tuple with all fields set to null when reading is complete. This option is only available when emitting tuples per line. If this field is missing, empty, or null, the default is the Output Null Tuple On Completion property of the Output properties tab.
settings.output.useDefaultCharset boolean If true, the default character set is used. If this field is missing, empty, or null, the default is the Use Default Charset property of the Output properties tab.
settings.output.charset string The character set to use. If this field is missing, empty, or null, the default is the Charset property of the Output properties tab.
settings.output.urlDecode boolean If true, any string data returned is URL-decoded before being emitted. If this field is missing, empty, or null, the default is the URL Decode Data property of the Output properties tab.
settings.output.urlDecodeType string The type of URL Decode operation to perform; for valid values, see Encode Types. If this field is missing, empty, or null, the default is the URL Decode Data Type property of the Output properties tab.

Note

If the following fields change between tuples, it can cause performance loss due to creation of a new connection builder:

  • settings.proxy.useProxy

  • settings.proxy.host

  • settings.proxy.port

  • settings.proxy.user

  • settings.proxy.pass

  • settings.auth.authType

  • settings.auth.user

  • settings.auth.pass

  • settings.auth.oAuth2.tokenUrl

  • settings.auth.oAuth2.clientId

  • settings.auth.oAuth2.clientSecret

  • settings.auth.oAuth2.scope

  • settings.auth.oAuth2.additionalTokenParams

  • settings.advanced.ignoreCertificateError

  • settings.advanced.connectTimeout

  • settings.advanced.readTimeout

Data Output Port

Field Data Type Description
id string Used only with server-sent events, this is the ID of the current event.
type string Used only with server-sent events, this is the type of the current event.
data string This field contains the data from the request. If this request is a File Download, this is the full path of the resulting lcoal file that contains the contents.
binary blob Used only with File Download requests. This field contains the binary data from the request. If this request is a File Download and no download path is specified, this field contains the binary data.
comment string Used only with server-sent events, this is the comment of the current event.
statusCode int The response status code.
mediaType string The response media type, such as text/html or application/json.
length int The response length.
headers list(tuple) A list of name to list(value) strings that represent the headers from the response. This field is only present when the Output Headers option is enabled in the Output properties tab.
headers.name string The header name.
headers.values list(string) A list of string values for this header.
cookies list(tuple) A list of cookies from the response. This field is only present when the Output Cookies option is enabled in the Output properties tab.
cookies.name string The name of the cookie.
cookies.value string The value of the cookie.
cookies.comment string The comment of the cookie.
cookies.domain string The domain of the cookie.
cookies.expiry timestamp When the cookie will expire.
cookies.httpOnly boolean Returns true if this cookie contains the HttpOnly attribute.
cookies.maxAge int The maximum age of the cookie in seconds.
cookies.path string The path of the cookie.
cookies.secure boolean Whether the cookie will only be sent over a secure connection.
cookies.version string The version of the cookie.
inputTuple tuple A copy of the input tuple that caused this output. If a URL is set in the Adapter Properties Tab, then this field contains the generated tuple created on startup. This field is only present when the Output Input Tuple option is enabled in the Output properties tab.

Status Output Port

Field Data Type Description
status string The name of the status.
time timestamp The date and time the status information was produced.
info list(name string, value string) A list of name-value pairs of details for the status, such as database name or error message.

All status messages that may be produced and the info list they provide:

  • Input Error—A field of the input tuple was invalid. Info list contains:

    • Input Tuple—The input tuple in CSV format.

    • Message—A human readable error message.

  • Error—An error occurred running the request. Info list contains:

    • URL—The URL of the request that failed.

    • Input Tuple—The input tuple in CSV format.

    • Message—A human readable error message.

  • File Not Found—A file given for the request was not found. Info list contains:

    • URL—The URL of the request that failed.

    • Input Tuple—The input tuple in CSV format.

    • File—The file that could not be found.

    • Message—A human readable error message.

  • Response Error—An error occurred processing the response. Info list contains:

    • URL—The URL of the request that failed.

    • Input Tuple—The input tuple in CSV format.

    • Status Code—The response status code.

    • Message—A human readable error message.

    • Response—A human readable response from the server.

  • Connect Failed—The request failed to connect. Info list contains:

    • URL—The URL of the request that failed.

    • Input Tuple—The input tuple in CSV format.

    • Message—A human readable error message.

  • Connecting—The given URL has initiated a connection. Only output for Chunked Response and Server-Sent Events requests. Info list contains:

    • URL—The URL of the request that failed.

    • Input Tuple—The input tuple in CSV format.

  • Connected—The given URL has connected. Only output for Chunked Response and Server-Sent Events requests. For Chunked Response connections, this message is not output until the first chunk is received and the response code can be confirmed. Info list contains:

    • URL—The URL of the request that failed.

    • Input Tuple—The input tuple in CSV format.

  • Disconnected—The given URL has disconnected. Info list contains:

    • URL—The URL of the request that failed.

    • Input Tuple—The input tuple in CSV format.

Suspend and Resume Behavior

When suspended, the adapter stops processing.

When resumed, the adapter once again starts processing web pages.

Related Topics