S3

The s3 resource configures and exposes the Amazon S3 client instance.

Advanced users that need direct access to Amazon S3 do not need to configure and instantiate it manually. VoltSP uses built-in mechanisms to provide a production quality setup.

User can relay on VoltSP standard configuration options to customize this resource.

This resource provides access to Amazon S3 operations through the AWS SDK for Java.

JAVAYAML

.configureResource(S3ResourceConfigBuilder.builder()
    .withRegion(value)
    .withEndpoint(value)
    .withCredentials(builder -> builder
        .withAccessKey(value)
        .withSecretKey(value)
        .withProfileName(value)
        .withFromContainer(value)
        .withFromInstanceProfile(value)
        .withRoleArn(value)
        .withRoleSessionName(value)
    )
)

resource:
  s3:
    region: value
    endpoint: value
    credentials:
      accessKey: value
      secretKey: value
      profileName: value
      fromContainer: value
      fromInstanceProfile: value
      roleArn: value
      roleSessionName: value

Java dependency management

Add this declaration to your dependency management system to access the configuration DSL for this plugin in Java.

MavenGradle

<dependency>
    <groupId>org.voltdb</groupId>
    <artifactId>volt-stream-plugin-aws-api</artifactId>
    <version>1.8.0</version>
</dependency>

implementation group: 'org.voltdb', name: 'volt-stream-plugin-aws-api', version: '1.8.0'

Properties

region

The AWS region where the S3 bucket is located (e.g., us-east-1). Type: string

Default value: us-east-1

endpoint

The S3 endpoint URL. This is optional and can be used to specify a custom endpoint for S3 compatible services. Type: string

credentials

Optional credential configuration for s3 client.

VoltSP supports following ways of retrieving authentication credentials: - Using access key and secret key. This is the most basic way of authentication. It is also the least secure way due to the need to store secret key in the VoltSP configuration. - Using a profile from the AWS credentials file. - Using EC2 instance profile credentials. - Using container credentials (for ECS). - Using web identity token credentials (for AWS STS).

If no credentials are configured then VoltSP will use the default credential provider chain as defined by AWS Java SDK. This will look for credentials in the following order: 1. Environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) 2. Java system properties (aws.accessKeyId and aws.secretKey) 3. Web Identity Token credentials from the environment or container 4. Credentials file at the default location (~/.aws/credentials) 5. ECS container credentials 6. EC2 Instance profile credentials

Type: object

Fields of credentials:

credentials.accessKey

The AWS access key ID for authentication when using basic credentials. Type: string

credentials.secretKey

The AWS secret access key for authentication when using basic credentials. Type: string

credentials.profileName

The AWS profile name to use when using profile based authentication. This loads credentials from a profile file. Type: string

credentials.fromContainer

Credentials will be read from the container metadata service Type: boolean

credentials.fromInstanceProfile

Credentials will be read from the instance metadata service Type: boolean

credentials.roleArn

The role ARN to assume when using web identity token based authentication. Type: string

credentials.roleSessionName

The role session name to use when using web identity token based authentication. Type: string

JSON Schema

You can validate or explore the configuration using its JSON Schema.

Usage Examples

JAVAYAML

S3ResourceConfigBuilder.builder()
   .withRegion("us-east-1")
   .withEndpoint("https://s3.amazonaws.com")
   .withCredentials(credentialsBuilder -> credentialsBuilder
       .withAccessKey("your-access-key")
       .withSecretKey("your-secret-key"))
   .build();

resources:
    - name: s3-client
      s3:
        region: "us-east-1"
        endpoint: "https://s3.amazonaws.com"
        credentials:
          access-key: "your-access-key"
          secret-key: "your-secret-key"

source:
    stdin: {}

sink:
    stdout: {}

Metrics

S3 metrics

Metric enum: org.voltdb.stream.plugin.s3.S3Metric

Prometheus name	Type	Description
voltsp_s3_create_event_total	counter	Number of S3 object create events observed.
voltsp_s3_delete_error_total	counter	Number of failed S3 delete requests.
voltsp_s3_delete_event_total	counter	Number of S3 object delete events observed.
voltsp_s3_delete_total	counter	Number of successful S3 delete requests.
voltsp_s3_get_error_total	counter	Number of failed S3 get requests.
voltsp_s3_get_total	counter	Number of successful S3 get requests.
voltsp_s3_list_error_total	counter	Number of failed S3 list requests.
voltsp_s3_list_total	counter	Number of successful S3 list requests.
voltsp_s3_put_error_total	counter	Number of failed S3 put requests.
voltsp_s3_put_size_bytes	counter	S3 put object size.
voltsp_s3_put_total	counter	Number of successful S3 put requests.
voltsp_s3_update_event_total	counter	Number of S3 object update events observed.
voltsp_s3_delete_time_seconds	histogram	S3 delete request duration.
voltsp_s3_get_size_bytes	histogram	S3 get object size.
voltsp_s3_get_time_seconds	histogram	S3 get request duration statistics.
voltsp_s3_list_time_seconds	histogram	S3 list request duration.
voltsp_s3_put_time_seconds	histogram	S3 put request duration statistics.

S3 tags

Tag enum: org.voltdb.stream.plugin.s3.S3Tag

Metrics reported by this component may include these tags. Not every metric includes every tag; tags are present only when they are relevant to the measurement.

Tag	Description
region	AWS region used by the S3 or SQS client.