Contents
The Spotfire Streaming HTTP Client adapter performs HTTP requests and emits the contents via the
Data
output port.
The adapter has multiple samples, described in HTTP Client Samples. These samples demonstrate how to perform various HTTP request types including server-sent events and chunked response.
Almost all properties can be overridden via the input control tuple at runtime. Options such as Output Headers, Output Cookies, Output Input Tuple, and Executor threads which would affect the schema or run system cannot be overridden at runtime.
REST, SOAP, WEB, SSE (Server-Sent Events), CHUNKED, GET, POST, PUT
This section describes the properties you can set for this adapter, using the various tabs of the Properties view in StreamBase Studio.
Name: Use this required field to specify or change the name of this instance of this component. The name must be unique within the current EventFlow module. The name can contain alphanumeric characters, underscores, and escaped special characters. Special characters can be escaped as described in Identifier Naming Rules. The first character must be alphabetic or an underscore.
Adapter: A read-only field that shows the formal name of the adapter.
Class name: Shows the fully qualified class name that implements the functionality of this adapter. If you need to reference this class name elsewhere in your application, you can right-click this field and select Copy from the context menu to place the full class name in the system clipboard.
Start options: This field provides a link to the Cluster Aware tab, where you configure the conditions under which this adapter starts.
Enable Error Output Port: Select this checkbox to add an Error Port to this component. In the EventFlow canvas, the Error Port shows as a red output port, always the last port for the component. See Using Error Ports to learn about Error Ports.
Description: Optionally, enter text to briefly describe the purpose and function of the component. In the EventFlow Editor canvas, you can see the description by pressing Ctrl while the component's tooltip is displayed.
Property | Description |
---|---|
URL | The URL endpoint to connect to. This is the default value used when the input tuple's URL field is null. If this value is empty and the input tuple URL field is empty, an error message is emitted on the status port. |
Request Method | The method of request to send. This contains the default value used when the input tuple's settings.requestMethod field is missing or empty. |
Request Type | The type of request to start. This contains the default value used when the input tuple's settings.requestType field is missing
or empty. The Request Type can be one of:
|
Download Path | If the Download Path is set via the property or input tuple the request results are stored in that file and the data field contains the full path to the file. If no Download Path is specified the results are output via the binary field of the output tuple.
|
Asynchronous | If enabled, the HTTP request is done asynchronously. This option has no effect when the request type is sever-sent events or chunked. |
Log Level | Controls the level of verbosity the adapter uses to send notifications to the console. This setting can be higher than the containing application's log level. If set lower, the system log level is used. Available values, in increasing order of verbosity, are: OFF, ERROR, WARN, INFO, DEBUG, TRACE. |
Property | Description |
---|---|
Use Proxy | Use a proxy server in processing the HTTP GET request. |
Proxy Host | The proxy server host name or IP address. |
Proxy Port | The proxy server TCP port number. |
Proxy User | The username to use with the proxy if required. |
Proxy Password | The password to use with the proxy if required. |
Property | Description |
---|---|
Connect timeout | The time in milliseconds to wait for the connection to establish, set to 0 to wait for ever. |
Read timeout | The time in milliseconds to wait for a read operation to complete, set to 0 to wait for ever. |
Reconnect timeout | Sets a specified timeout value, in milliseconds, to wait for reconnect attempts when using server-sent events. A timeout of zero is interpreted as an unlimited timeout. |
Poll Frequency | The time in milliseconds to wait between requesting, set to 0 to disable polling, note this option has no effect when the request type is sever-sent events or chunked response. |
Ignore certificate errors | If enabled, any errors produced by invalid TLS/SSL certificates are ignored and the website is processed as normal. Warning! This can lead to man in the middle attacks. |
Executor Threads | The number of threads to use for pooling connections. |
Property | Description |
---|---|
Output Input Tuple | If enabled, the input tuple is emitted with every data output tuple. |
Output Headers | If enabled, the headers are emitted with every data output tuple. |
Output Cookies | If enabled, the cookies are emitted with every data output tuple. |
Output Tuple Per Line | If enabled, response from the server emits one tuple per line received. This option is only valid with basic requests. |
Output Blank Lines | This option sends blank tuples when a blank line is received. This option is only available when emitting tuples per line. |
Output Null Tuple On Completion | This option sends a tuple with all fields set to null when reading is complete. This option is only available when emitting tuples per line. |
Output Connecting Status | If enabled, a status message is emitted for each attempt to connect for a request. |
URL Decode Data | If enabled, any string data returned is URL decoded before being output. |
URL Decode Data Type | The type of URL Decode operation to perform; see Encode Types. |
Use Default Charset | If selected, specifies whether the Java platform default character set is to be used. If cleared, a valid character set name must be specified for the Character Set property. |
Character Set | The name of the character set encoding that the adapter is to use to read input or write output. |
Property | Description |
---|---|
Auth Type | The authentication type to use with the request
|
Username | The username to be sent to the server. |
Password | The password to be sent to the server. |
OAuth 2 Token URL | The Token URL of an OAuth 2 server. |
OAuth 2 Client ID | The client ID to be provided to the OAuth 2 Token URL. |
OAuth 2 Client Secret | The client secret to be provided to the OAuth 2 Token URL. |
OAuth 2 Scope | The OAuth 2 scope to provide to the token URL. |
OAuth 2 Additional Token Parameters | Additional parameters to provide to the token URL separated by &. For example, "audience=...&state=...". |
Use the settings in this tab to enable this operator or adapter for runtime start and stop conditions in a multi-node cluster. During initial development of the fragment that contains this operator or adapter, and for maximum compatibility with releases before 10.5.0, leave the Cluster start policy control in its default setting, Start with module.
Cluster awareness is an advanced topic that requires an understanding of StreamBase Runtime architecture features, including clusters, quorums, availability zones, and partitions. See Cluster Awareness Tab Settings on the Using Cluster Awareness page for instructions on configuring this tab.
Use the Concurrency tab to specify parallel regions for this instance of this component, or multiplicity options, or both. The Concurrency tab settings are described in Concurrency Options, and dispatch styles are described in Dispatch Styles.
Caution
Concurrency settings are not suitable for every application, and using these settings requires a thorough analysis of your application. For details, see Execution Order and Concurrency, which includes important guidelines for using the concurrency options.
Name | Description |
---|---|
AUTHORITY | The URI authority component type. |
FRAGMENT | The URI fragment component type. |
HOST | The URI host component type. |
MATRIX_PARAM | The URI path component type that is a matrix parameter. |
PATH | The URI path component type. |
PATH_SEGMENT | The URI path component type that is a path segment. |
PORT | The URI port component type. |
QUERY | The URI query component type. |
QUERY_PARAM | The URI query component type that is a query parameter, space character is encoded as +. |
QUERY_PARAM_SPACE_ENCODED | The URI query component type that is a query parameter, space character is encoded as %20. |
SCHEME | The URI scheme component type. |
UNRESERVED | ALPHA / DIGIT / "-" / "." / "_" / "~" characters. |
USER_INFO | The URI user info component type. |
The HTTP Client adapter's input control port, all values are options with the exception of the url field. The schema is defined as:
Field | Data Type | Description |
---|---|---|
command | string | The command tells the adapter what action to take. If this field is missing or empty, the default value of connect is used.
|
url | string | The URL endpoint to connect to. If this value is empty, the default property value is used. |
data | list(tuple) | This list of tuples can be used for POST and PUT operations to input the data that should be sent as the payload field. If multiple entries are given, the payload is sent as a multipart/mixed payload. If any one entry specifies more than one data type field, a multipart/mixed is also sent. |
data.text | string | The payload as text to send with this request. If this field is missing or null, it is ignored. The type of this payload is
defined by the data.mediaType field which defaults to text/plain if the data.mediaType property is missing or empty.
|
data.binary | blob | The payload in binary format to send with this request. Binary payloads ar2e always sent as a multipart/mixed payload. The
type of this payload is defined by the data.mediaType field, which defaults to application/octet-stream if the data.mediaType property is missing or empty.
|
data.filePath | string | The path to a file to send with this request. Binary payloads are always sent as a multipart/mixed payload. The type of this
payload is defined by the data.mediaType field, which defaults to application/octet-stream if the data.mediaType property is missing or empty.
|
data.mediaType | string | The media type to use with this payload, such as text/plain or application/json . If this field is missing or empty, the default is defined by whichever field has a payload.
|
data.name | string | If data is being sent as a binary payload, this name is sent to the server as the name part of the payload. If this field
is missing or null, data is used.
|
data.urlEncode | boolean | This only applies to the data.text payload field. If true, the payload is URL-encoded before sending. |
data.urlEncodeType | string | This only applies to the data.text payload field. If data.urlEncode is true, this value specifies the type of encoding; valid
values are listed in Encode Types. The default value is QUERY_PARAM_SPACE_ENCODED .
|
headers | list(tuple) | A list of names to list(value) strings that represent the headers to send with this request. |
headers.name | string | The header name. If the field is empty, the header is ignored. |
headers.values | list(string) | A list of string values for this header. If this field is null or empty, the header is ignored. |
queryParameters | list(tuple) | A list of query parameters to send with the request. |
queryParameters.name | string | The name of the query parameter. If this field is empty, the parameter is ignored. |
queryParameters.value | string | The value of the query parameter. Null values are defaulted to empty strings. |
queryParameters.urlEncode | string | If true, the parameter is URL-encoded before sending. If this field is missing or null, the default is true. |
queryParameters.urlEncodeType | string | This queryParameters.urlEncode is true, this value specifies the type of encoding; valid values are listed in Encode Types. The default value is QUERY_PARAM_SPACE_ENCODED .
|
cookies | list(tuple) | A list of cookies to send with the request. |
cookies.name | string | The name of the cookie. If this field is empty, the cookie is ignored. |
cookies.value | string | The value of the cookie. Null values are defaulted to empty strings. |
settings | tuple | Settings that can be changed per tuple at runtime. |
settings.requestMethod | string | The request method to use; valid values are GET, POST, PUT, HEAD, DELETE, and OPTIONS. If this field is missing, empty, or null, the default is the currently selected Request Method property of the adapter. |
settings.requestType | string | The request type to use; valid values are Basic, File Download, Server-Sent Events, and Chunked Response. If this field is missing, empty, or null, the default is the currently selected Request Type property of the adapter. |
settings.downloadPath | string | The path to store files during a File Download request. If this field is missing, empty, or null, the default is the currently selected Download Path property of the adapter. |
settings.asynchronous | boolean | Determines whether this request should be sent synchronously or asynchronously. If this field is missing, empty, or null, the default is the Asynchronous property of the adapter. |
settings.proxy | tuple | A tuple containing the proxy information to use for this request. If this field is missing, empty, or null, the default is the set of values in the Proxy properties tab. |
settings.proxy.useProxy | boolean | Determines whether a proxy should be used for this request. If this field is missing, empty, or null, the default is the Use Proxy property of the Proxy properties tab. |
settings.proxy.host | string | The proxy host to use for this request. If this field is missing, empty, or null, the default is the Proxy Host property of the Proxy properties tab. |
settings.proxy.port | int | The proxy port to use for this request. If this field is missing, empty, or null, the default is the Proxy Port property of the Proxy properties tab. |
settings.proxy.user | string | The proxy user to use for this request. If this field is missing, empty, or null, the default is the Proxy User property of the Proxy properties tab. |
settings.proxy.pass | string | The proxy password to use for this request. If this field is missing, empty, or null, the default is the Proxy Password property of the Proxy properties tab. |
settings.advanced | tuple | A tuple containing the advanced settings to use for this request. If this field is missing, empty, or null, the default is the set of values in the Advanced properties tab. |
settings.advanced.connectTimeout | int | The connection timeout to use for this request. If this field is missing, empty, or null, the default is the Connect Timeout property of the Advanced properties tab. |
settings.advanced.readTimeout | int | The read timeout to use for this request. If this field is missing, empty, or null, the default is the Read Timeout property of the Advanced properties tab. |
settings.advanced.reconnectTimeout | int | The reconnect timeout to use for this request. This value is only used with server-sent event requests. If this field is missing, empty, or null, the default is the Reconnect Timeout property of the Advanced properties tab. |
settings.advanced.pollFrequency | int | The poll frequency to use for this request. This value is only used with basic and file download requests. If this field is missing, empty, or null, the default is the Poll Frequency property of the Advanced properties tab. |
settings.advanced.ignoreCertificateError | boolean | If true, TLS/SSL certificate errors are ignored for this request. If this field is missing, empty, or null, the default is the Ignore Certificate Errors property of the Advanced properties tab. |
settings.auth | tuple | A tuple containing the authentication settings to use for this request. If this field is missing, empty, or null, the defaults are the values in the Auth properties tab. |
settings.auth.authType | string | The authentication type to use for this request. Valid values are None, Basic, Basic Non-Preemptive, Digest, and OAuth 2. If this field is missing, empty, or null, the default is the Auth Type property of the Auth properties tab. |
settings.auth.user | string | The authentication user to use for this request. If this field is missing, empty, or null, the default is the Username property of the Auth properties tab. |
settings.auth.pass | string | The authentication password to use for this request. If this field is missing, empty, or null, the default is the Password property of the Auth properties tab. |
settings.auth.oAuth2 | tuple | A tuple containing the OAuth 2 settings to use for this request. If this field is missing, empty, or null, the defaults are the values in the Auth properties tab. Ignored if Auth Type is not set to OAuth 2. |
settings.auth.oAuth2.tokenUrl | string | The OAuth 2 token URL to use for this request. If this field is missing, empty, or null, the defaults are the values in the Auth properties tab. Ignored if Auth Type is not set to OAuth 2. |
settings.auth.oAuth2.clientId | string | The OAuth 2 client ID to provide to the token URL for this request. If this field is missing, empty, or null, the defaults are the values in the Auth properties tab. Ignored if Auth Type is not set to OAuth 2. |
settings.auth.oAuth2.clientSecret | string | The OAuth 2 client secret to provide to the token URL for this request. If this field is missing, empty, or null, the defaults are the values in the Auth properties tab. Ignored if Auth Type is not set to OAuth 2. |
settings.auth.oAuth2.scope | string | The OAuth 2 scope to provide to the token URL for this request. If this field is missing, empty, or null, the defaults are the values in the Auth properties tab. Ignored if Auth Type is not set to OAuth 2. |
settings.auth.oAuth2.additionalTokenParams | string | Additional parameters to provide to the token URL separated by &. For example, "audience=...&state=...". If this field is missing, empty, or null, the defaults are the values in the Auth properties tab. Ignored if Auth Type is not set to OAuth 2. |
settings.output | tuple | A tuple containing the output settings to use for this request. If this field is missing, empty, or null the defaults are the values in the Output properties tab. |
settings.output.outputTuplePerLine | boolean | If true, response from the server emits a tuple per line received. This option is only valid with basic requests. If this field is missing, empty, or null, the default is the Output Tuple Per Line property of the Output properties tab. |
settings.output.outputBlankLines | boolean | If true, empty length response strings are output. This option is only available when emitting tuples per line. If this field is missing, empty, or null, the default is the Output Blank Lines property of the Output properties tab. |
settings.output.outputNullTupleOnCompletion | boolean | If true, a status message is output for each attempt to connect for a request. If this field is missing, empty, or null, the default is the Output Connecting Status property of the Output properties tab. |
settings.output.outputConnectingStatus | boolean | If true, the request sends a tuple with all fields set to null when reading is complete. This option is only available when emitting tuples per line. If this field is missing, empty, or null, the default is the Output Null Tuple On Completion property of the Output properties tab. |
settings.output.useDefaultCharset | boolean | If true, the default character set is used. If this field is missing, empty, or null, the default is the Use Default Charset property of the Output properties tab. |
settings.output.charset | string | The character set to use. If this field is missing, empty, or null, the default is the Charset property of the Output properties tab. |
settings.output.urlDecode | boolean | If true, any string data returned is URL-decoded before being emitted. If this field is missing, empty, or null, the default is the URL Decode Data property of the Output properties tab. |
settings.output.urlDecodeType | string | The type of URL Decode operation to perform; for valid values, see Encode Types. If this field is missing, empty, or null, the default is the URL Decode Data Type property of the Output properties tab. |
Note
If the following fields change between tuples, it can cause performance loss due to creation of a new connection builder:
-
settings.proxy.useProxy
-
settings.proxy.host
-
settings.proxy.port
-
settings.proxy.user
-
settings.proxy.pass
-
settings.auth.authType
-
settings.auth.user
-
settings.auth.pass
-
settings.auth.oAuth2.tokenUrl
-
settings.auth.oAuth2.clientId
-
settings.auth.oAuth2.clientSecret
-
settings.auth.oAuth2.scope
-
settings.auth.oAuth2.additionalTokenParams
-
settings.advanced.ignoreCertificateError
-
settings.advanced.connectTimeout
-
settings.advanced.readTimeout
Field | Data Type | Description |
---|---|---|
id | string | Used only with server-sent events, this is the ID of the current event. |
type | string | Used only with server-sent events, this is the type of the current event. |
data | string | This field contains the data from the request. If this request is a File Download, this is the full path of the resulting lcoal file that contains the contents. |
binary | blob | Used only with File Download requests. This field contains the binary data from the request. If this request is a File Download and no download path is specified, this field contains the binary data. |
comment | string | Used only with server-sent events, this is the comment of the current event. |
statusCode | int | The response status code. |
mediaType | string | The response media type, such as text/html or application/json .
|
length | int | The response length. |
headers | list(tuple) | A list of name to list(value) strings that represent the headers from the response. This field is only present when the Output Headers option is enabled in the Output properties tab. |
headers.name | string | The header name. |
headers.values | list(string) | A list of string values for this header. |
cookies | list(tuple) | A list of cookies from the response. This field is only present when the Output Cookies option is enabled in the Output properties tab. |
cookies.name | string | The name of the cookie. |
cookies.value | string | The value of the cookie. |
cookies.comment | string | The comment of the cookie. |
cookies.domain | string | The domain of the cookie. |
cookies.expiry | timestamp | When the cookie will expire. |
cookies.httpOnly | boolean | Returns true if this cookie contains the HttpOnly attribute. |
cookies.maxAge | int | The maximum age of the cookie in seconds. |
cookies.path | string | The path of the cookie. |
cookies.secure | boolean | Whether the cookie will only be sent over a secure connection. |
cookies.version | string | The version of the cookie. |
inputTuple | tuple | A copy of the input tuple that caused this output. If a URL is set in the Adapter Properties Tab, then this field contains the generated tuple created on startup. This field is only present when the Output Input Tuple option is enabled in the Output properties tab. |
Field | Data Type | Description |
---|---|---|
status | string | The name of the status. |
time | timestamp | The date and time the status information was produced. |
info | list(name string, value string) | A list of name-value pairs of details for the status, such as database name or error message. |
All status messages that may be produced and the info list they provide:
-
Input Error—A field of the input tuple was invalid. Info list contains:
-
Input Tuple—The input tuple in CSV format.
-
Message—A human readable error message.
-
-
Error—An error occurred running the request. Info list contains:
-
URL—The URL of the request that failed.
-
Input Tuple—The input tuple in CSV format.
-
Message—A human readable error message.
-
-
File Not Found—A file given for the request was not found. Info list contains:
-
URL—The URL of the request that failed.
-
Input Tuple—The input tuple in CSV format.
-
File—The file that could not be found.
-
Message—A human readable error message.
-
-
Response Error—An error occurred processing the response. Info list contains:
-
URL—The URL of the request that failed.
-
Input Tuple—The input tuple in CSV format.
-
Status Code—The response status code.
-
Message—A human readable error message.
-
Response—A human readable response from the server.
-
-
Connect Failed—The request failed to connect. Info list contains:
-
URL—The URL of the request that failed.
-
Input Tuple—The input tuple in CSV format.
-
Message—A human readable error message.
-
-
Connecting—The given URL has initiated a connection. Only output for Chunked Response and Server-Sent Events requests. Info list contains:
-
URL—The URL of the request that failed.
-
Input Tuple—The input tuple in CSV format.
-
-
Connected—The given URL has connected. Only output for Chunked Response and Server-Sent Events requests. For Chunked Response connections, this message is not output until the first chunk is received and the response code can be confirmed. Info list contains:
-
URL—The URL of the request that failed.
-
Input Tuple—The input tuple in CSV format.
-
-
Disconnected—The given URL has disconnected. Info list contains:
-
URL—The URL of the request that failed.
-
Input Tuple—The input tuple in CSV format.
-
When suspended, the adapter stops processing.
When resumed, the adapter once again starts processing web pages.