S3¶

The S3 sink writes data to an Amazon S3 bucket or compatible object storage service. It uploads incoming byte arrays as objects to the specified bucket and prefix.

Each object is stored with a unique key generated from the current timestamp and a UUID. The content type of the objects can be configured.

JAVAYAML

.consumeFromSource(...)
.terminateWithSink(S3SinkConfigBuilder.builder()
    .withS3Resource(value)
    .withBucketName(value)
    .withPrefix(value)
)

sink:
  s3:
    s3Resource: value
    bucketName: value
    prefix: value

Java dependency management¶

Add this declaration to your dependency management system to access the configuration DSL for this plugin in Java.

MavenGradle

<dependency>
    <groupId>org.voltdb</groupId>
    <artifactId>volt-stream-plugin-aws-api</artifactId>
    <version>1.8.0</version>
</dependency>

implementation group: 'org.voltdb', name: 'volt-stream-plugin-aws-api', version: '1.8.0'

Properties¶

`s3Resource`¶

Reference to the S3 client resource. Required.

Type: object

`bucketName`¶

The name of the S3 bucket to write objects to. Required.

Type: string

`prefix`¶

The prefix to prepend to object keys. This can be used to organize objects in the bucket. Type: string

JSON Schema¶

You can validate or explore the configuration using its JSON Schema.

Usage Examples¶

JAVA

stream
       .withName("Write data to S3")
       .configureResource("s3-client", S3ResourceConfigBuilder.class, resourceBuilder -> {})
       .consumeFromSource(source)
       .terminateWithSink(
               S3SinkConfigBuilder.builder()
                       .withBucketName("my-bucket")
                       .withPrefix("output/")
                       .withS3ResourceName("s3-client")
       );

This configuration writes data to the "output/" prefix in the "my-bucket" bucket with the content type set to "application/json".

Metrics¶

S3 metrics¶

Metric enum: org.voltdb.stream.plugin.s3.S3Metric

Prometheus name	Type	Description
`voltsp_s3_create_event_total`	`counter`	Number of S3 object create events observed.
`voltsp_s3_delete_error_total`	`counter`	Number of failed S3 delete requests.
`voltsp_s3_delete_event_total`	`counter`	Number of S3 object delete events observed.
`voltsp_s3_delete_total`	`counter`	Number of successful S3 delete requests.
`voltsp_s3_get_error_total`	`counter`	Number of failed S3 get requests.
`voltsp_s3_get_total`	`counter`	Number of successful S3 get requests.
`voltsp_s3_list_error_total`	`counter`	Number of failed S3 list requests.
`voltsp_s3_list_total`	`counter`	Number of successful S3 list requests.
`voltsp_s3_put_error_total`	`counter`	Number of failed S3 put requests.
`voltsp_s3_put_size_bytes`	`counter`	S3 put object size.
`voltsp_s3_put_total`	`counter`	Number of successful S3 put requests.
`voltsp_s3_update_event_total`	`counter`	Number of S3 object update events observed.
`voltsp_s3_delete_time_seconds`	`histogram`	S3 delete request duration.
`voltsp_s3_get_size_bytes`	`histogram`	S3 get object size.
`voltsp_s3_get_time_seconds`	`histogram`	S3 get request duration statistics.
`voltsp_s3_list_time_seconds`	`histogram`	S3 list request duration.
`voltsp_s3_put_time_seconds`	`histogram`	S3 put request duration statistics.

S3 tags¶

Tag enum: org.voltdb.stream.plugin.s3.S3Tag

Metrics reported by this component may include these tags. Not every metric includes every tag; tags are present only when they are relevant to the measurement.

Tag	Description
`region`	AWS region used by the S3 or SQS client.

S3¶