Http

The http sink connects and sends http(s) requests to configured downstream HTTP(s) server over single connection per worker thread. If server doesn't close the connection, it will be reused for subsequent requests.

If server is not available, sink will keep on trying to re-connect each time new event arrives. In other words, if there are not events in the pipeline, this sink will not try to connect.

This sink can execute only POST, PUT, PATCH or DELETE requests.

JAVAYAML

.consumeFromSource(...)
.terminateWithSink(HttpSinkConfigBuilder.builder()
    .withHttpClient(value)
    .withRetryConfiguration(builder -> builder
        .withRetries(value)
        .withBackoffDelay(value)
        .withMaxBackoffDelay(value)
    )
    .withExceptionHandler(value)
)

sink:
  http:
    httpClient: value
    retryConfiguration:
      retries: value
      backoffDelay: value
      maxBackoffDelay: value
    exceptionHandler: value

Java dependency management

Add this declaration to your dependency management system to access the configuration DSL for this plugin in Java.

MavenGradle

<dependency>
    <groupId>org.voltdb</groupId>
    <artifactId>volt-stream-plugin-http-api</artifactId>
    <version>1.8.0</version>
</dependency>

implementation group: 'org.voltdb', name: 'volt-stream-plugin-http-api', version: '1.8.0'

Properties

httpClient

Http client reference Required.

Type: object

retryConfiguration

Retry configuration of failed request. Type: object

Fields of retryConfiguration:

retryConfiguration.retries

Number of retry attempts after a request failure. Type: number

Default value: 3

retryConfiguration.backoffDelay

Initial delay before the first retry attempt. Type: object

Default value: PT0.2S

retryConfiguration.maxBackoffDelay

Maximum delay between consecutive retry attempts. Type: object

Default value: PT3S

exceptionHandler

Custom exception handler enabling interception of all errors related to this sink.

Type: object

JSON Schema

You can validate or explore the configuration using its JSON Schema.

Usage Examples

JAVAYAML

 stream
        .withName("Read data from HTTP server and print to stdout")
        .configureResource("a-http-client", HttpClientResourceConfigBuilder.builder()
               .withAddress(HostAndPort.fromParts("localhost", 9090))
               ...
               )
        .consumeFromSource(...)
        .processWith(event -> new HttpRequest(...))
        .terminateWithSink(HttpSinkConfigBuilder.builder()
               .withHttpClientName("a-http-client")
               .withHeaders(Map.of("Content-Type", "application/json"))
        );

There is also a way to configure http client resource inline directly.

 stream
        .withName("Read data from HTTP server and print to stdout")
        .consumeFromSource(...)
        .processWith(event -> new HttpRequest(...))
        .terminateWithSink(HttpSinkConfigBuilder.builder()
               // the name of the resource is `httpClient`
               .withHttpClientBuilder(builder -> {
                     builder.withAddress(HostAndPort.fromParts("localhost", 9090))
               })
               .withHeaders(Map.of("Content-Type", "application/json"))
        );

resources:
- name: a-http-client
  http-client:
    address: localhost:32112
    ...
source:
  ...
sink:
  http:
    httpClientReference: a-http-client
    headers:
      Content-Type: application/json

the http-client resource can be configured inline like

source:
  ...
sink:
  http:
    httpClient:
      address: localhost:32112
      ...
    headers:
      Content-Type: application/json

Metrics

Http metrics

Metric enum: org.voltdb.stream.plugin.http.HttpMetric

Prometheus name	Type	Description
voltsp_http_buffer_capacity_total	gauge	Remaining HTTP source message buffer capacity.
voltsp_http_connection_attempt_total	counter	Number of HTTP sink connection attempts.
voltsp_http_connection_error_total	counter	Number of HTTP sink connection errors.
voltsp_http_connection_timeout_total	counter	Number of HTTP sink connection timeouts.
voltsp_http_exceeded_payload_size_total	counter	Number of HTTP requests rejected for exceeding the configured payload size.
voltsp_http_idle_timeout_total	counter	Number of HTTP idle connection timeouts.
voltsp_http_internal_errors_total	counter	Number of HTTP source internal errors.
voltsp_http_invalid_media_type_total	counter	Number of HTTP requests rejected for invalid media type.
voltsp_http_invalid_path_total	counter	Number of HTTP requests rejected for invalid path.
voltsp_http_invalid_request_method_total	counter	Number of HTTP requests rejected for invalid method.
voltsp_http_received_bytes	counter	Number of HTTP request body bytes received by the source.
voltsp_http_received_requests_total	counter	Number of HTTP requests received by the source.
voltsp_http_request_acknowledged_total	counter	Number of HTTP sink requests acknowledged.
voltsp_http_request_failed_total	counter	Number of failed HTTP sink requests.
voltsp_http_response_code_2xx_total	counter	Number of HTTP sink responses with a 2xx status code.
voltsp_http_response_code_3xx_total	counter	Number of HTTP sink responses with a 3xx status code.
voltsp_http_response_code_4xx_total	counter	Number of HTTP sink responses with a 4xx status code.
voltsp_http_response_code_5xx_total	counter	Number of HTTP sink responses with a 5xx status code.
voltsp_http_request_round_trip_time_seconds	histogram	HTTP sink request round-trip duration.