Volt-tumbling-count-window¶

The volt-tumbling-count-window creates a partitioned, distributed tumbling count window. In this implementation VoltDB acts as a storage for the window aggregate. The aggregate is emitted using VoltDB stream feature - so this implementation expects a topic to be defined in Volt's deployment file.

The volt-tumbling-count-window, during configuration phase, generates code for VoltSP side to extract fields necessary for aggregate update. It also automatically generates and loads schema and procedures into VoltDB. For now, only deployment file needs to be manually updated.

Based on VoltAggregateBuilder each event is transformed into a request. Lastly a partition key is extracted, and payload is routed to the right partition in VoltDB.

In case of downstream system error this volt-tumbling-count-window will re-process events.

JAVAYAML

// configure resource
stream.configureResource("primary-cluster", VoltDBResourceConfigBuilder.class,
    cb -> cb.addToServers("localhost", 21212));

stream
  .aggregateWithWindow(VoltWindows.<Event>partitionedTumblingCountWindow()
      .withMaxEventsPerWindow(500)
      .withPartitionKeyExtractor(Event::getUserId)
      .withVoltClientResourceName("primary-cluster")
      .withAggregateDefinition(builder -> builder
          .count()
          .sum("amount", Event::getAmount)
          .min("amount", Event::getAmount)
          .max("amount", Event::getAmount)
          .distinct("id", (EventToStringExtractor<Event>) Event::getId)
      )
      .withExceptionHandler((records, context, throwable) -> {
          // handle error
      })
  )
  .emit(WindowTrigger.atWindowEnd)
  ...

resources:
  - name: "primary-cluster"
    voltdb-client:
      servers: "localhost:21212"
emitter:
  volt-tumbling-count-window:
    maxEventsPerWindow: 500
    voltClientResource: "primary-cluster"
    partitionKeyExtractor: "com.example.Event::getUserId"
    aggregateDefinition:
      from: "com.example.Event"
      builder:
        - type: count
        - type: sum (amount)
        - type: min (amount)
        - type: max (amount)
        - type: distinct (id)
  triggers:
    - trigger: org.voltdb.stream.plugin.window.api.WindowTrigger#atWindowEnd
      ...

Java dependency management¶

Add this declaration to your dependency management system to access the configuration DSL for this plugin in Java.

MavenGradle

<dependency>
    <groupId>org.voltdb</groupId>
    <artifactId>volt-stream-plugin-volt-api</artifactId>
    <version>1.8.0</version>
</dependency>

implementation group: 'org.voltdb', name: 'volt-stream-plugin-volt-api', version: '1.8.0'

Properties¶

`maxEventsPerWindow`¶

Defines how many events this window can accumulate before it gets closed. Must be greater than 0 Type: number

`partitionKeyExtractor`¶

The partition is determined by hashing the value extracted from the message using this class. Required.

Type: object

`aggregateDefinition`¶

The aggregate is defined by the provided builder Required.

Type: object

`voltClientResource`¶

Client resource reference to be used when connecting to VoltDb cluster Required.

Type: object

`servers`¶

A set of host and port addresses for connecting to the VoltDB cluster. Only one address is sufficient for cluster topology discovery. Type: array

`batchSize`¶

The maximum number of records to include in a single batch before inserting data into VoltDB. Higher values can improve throughput but will increase memory usage. Type: number

Default value: 100000

`flushInterval`¶

The time interval after which batch is flushed to VoltDB, even if the desired batch size is not reached. Type: object

Default value: 1s

`client`¶

Configuration settings for the VoltDB client, including authentication, retry policies, and performance limits. Type: object

Fields of client:

`client.retry`¶

Configuration for retrying failed operations, including the number of retries and backoff delays. Type: object

Fields of client.retry:

`client.retry.retries`¶

Number of retry attempts after a request failure. Type: number

Default value: 3

`client.retry.backoffDelay`¶

Initial delay before the first retry attempt. Type: object

Default value: PT0.2S

`client.retry.maxBackoffDelay`¶

Maximum delay between consecutive retry attempts. Type: object

Default value: PT3S

`client.maxTransactionsPerSecond`¶

The maximum number of transactions allowed per second to control the rate of operations. Type: number

`client.maxOutstandingTransactions`¶

The maximum number of outstanding transactions allowed, limiting concurrent operations. Type: number

`client.requestTimeout`¶

The timeout duration for client requests to VoltDB, after which a request is considered failed. Type: object

`client.auth`¶

Client credentials for authenticating to the VoltDB cluster. Type: object

Fields of client.auth:

`client.auth.username`¶

Username used for authentication. Type: string

`client.auth.password`¶

Password used for authentication. Type: string

`client.ssl`¶

Client configuration for secure transport to the VoltDB cluster. Type: object

Fields of client.ssl:

`client.ssl.trustStoreFile`¶

Truststore file or trusted CA certificate; supported formats include JKS, PKCS#12, or PEM. Type: string

`client.ssl.trustStorePassword`¶

Truststore password. Type: string

`client.ssl.keyStoreFile`¶

Keystore file; supported formats include JKS, PKCS#12, or PEM Type: string

`client.ssl.keyStorePassword`¶

Keystore password. Type: string

`client.ssl.keyPassword`¶

Private key password. Optional — if not set, the key store password will be used. Type: string

`client.ssl.insecure`¶

If set to true, disables SSL certificate and hostname validation. Intended for debugging purposes only. Doesn't work with mTLS.

Type: boolean

`client.ssl.hostnameVerifier`¶

Custom hostname verifier for SSL connections. If not specified and 'insecure' is true, hostname verification will be disabled.

Type: object

`exceptionHandler`¶

Custom exception handler enabling interception of all errors related to this emitter. Type: object

Available Triggers¶

`org.voltdb.stream.plugin.window.api.WindowTrigger.atWindowEnd`¶

Triggered after window has been closed.

`org.voltdb.stream.plugin.window.api.WindowTrigger.atAggregate`¶

Triggered after each event has been aggregated. An emitted aggregate is a snapshot of the aggregate after the event has been applied.

JSON Schema¶

You can validate or explore the configuration using its JSON Schema.

Metrics¶

Volt metrics¶

Metric enum: org.voltdb.stream.plugin.volt.metrics.VoltMetric

Prometheus name	Type	Description
`voltsp_volt_affinity_reads_total`	`counter`	Number of VoltDB affinity reads.
`voltsp_volt_affinity_writes_total`	`counter`	Number of VoltDB affinity writes.
`voltsp_volt_procedure_bytes_read_bytes`	`counter`	Number of bytes read by the VoltDB client.
`voltsp_volt_procedure_bytes_written_bytes`	`counter`	Number of bytes written by the VoltDB client.
`voltsp_volt_procedure_invocation_aborts_total`	`counter`	Number of VoltDB procedure invocation aborts.
`voltsp_volt_procedure_invocation_completed_total`	`counter`	Number of completed VoltDB procedure invocations.
`voltsp_volt_procedure_invocation_errors_total`	`counter`	Number of VoltDB procedure invocation errors.
`voltsp_volt_procedure_invocation_retry_total`	`counter`	Number of VoltDB procedure invocation retries.
`voltsp_volt_procedure_invocation_timeouts_total`	`counter`	Number of VoltDB procedure invocation timeouts.
`voltsp_volt_roundrobin_reads_total`	`counter`	Number of VoltDB round-robin reads.
`voltsp_volt_roundrobin_writes_total`	`counter`	Number of VoltDB round-robin writes.
`voltsp_volt_procedure_invocation_latency_seconds`	`histogram`	VoltDB procedure invocation latency.
`voltsp_volt_sink_info`	`info`	Metadata describing a VoltDB sink.

Window metrics¶

Metric enum: org.voltdb.stream.plugin.window.WindowMetric

Prometheus name	Type	Description
`voltsp_aggregations_unique_keys_total`	`gauge`	Current number of unique aggregate keys held by a local window.
`voltsp_aggregations_created_total`	`counter`	Number of aggregate windows created.
`voltsp_aggregations_emitted_total`	`counter`	Number of aggregate snapshots emitted by a local window.
`voltsp_emitted_total`	`counter`	Number of events emitted by a local window. This can be errors, original events, late events, etc.
`voltsp_idle_keys_total`	`counter`	Number of idle aggregate keys removed by a local window.
`voltsp_late_event_total`	`counter`	Number of late events observed by a local window.

Volt tags¶

Tag enum: org.voltdb.stream.plugin.volt.metrics.VoltTag

Metrics reported by this component may include these tags. Not every metric includes every tag; tags are present only when they are relevant to the measurement.

Tag	Description
`connection_id`	VoltDB client connection id.
`partition_id`	VoltDB partition id.
`procedure_name`	VoltDB procedure name.
`table_name`	VoltDB table name.
`volt_batch_size`	Configured VoltDB batch size.

Volt-tumbling-count-window¶

Java dependency management¶

Properties¶

maxEventsPerWindow¶

partitionKeyExtractor¶

aggregateDefinition¶

voltClientResource¶

servers¶

batchSize¶

flushInterval¶

client¶

client.retry¶

client.retry.retries¶

client.retry.backoffDelay¶

client.retry.maxBackoffDelay¶

client.maxTransactionsPerSecond¶

client.maxOutstandingTransactions¶

client.requestTimeout¶

client.auth¶

client.auth.username¶

client.auth.password¶

client.ssl¶

client.ssl.trustStoreFile¶

client.ssl.trustStorePassword¶

client.ssl.keyStoreFile¶

client.ssl.keyStorePassword¶

client.ssl.keyPassword¶

client.ssl.insecure¶

client.ssl.hostnameVerifier¶

exceptionHandler¶

Available Triggers¶

org.voltdb.stream.plugin.window.api.WindowTrigger.atWindowEnd¶

org.voltdb.stream.plugin.window.api.WindowTrigger.atAggregate¶

JSON Schema¶

Metrics¶

Volt metrics¶

Window metrics¶

Volt tags¶

`maxEventsPerWindow`¶

`partitionKeyExtractor`¶

`aggregateDefinition`¶

`voltClientResource`¶

`servers`¶

`batchSize`¶

`flushInterval`¶

`client`¶

`client.retry`¶

`client.retry.retries`¶

`client.retry.backoffDelay`¶

`client.retry.maxBackoffDelay`¶

`client.maxTransactionsPerSecond`¶

`client.maxOutstandingTransactions`¶

`client.requestTimeout`¶

`client.auth`¶

`client.auth.username`¶

`client.auth.password`¶

`client.ssl`¶

`client.ssl.trustStoreFile`¶

`client.ssl.trustStorePassword`¶

`client.ssl.keyStoreFile`¶

`client.ssl.keyStorePassword`¶

`client.ssl.keyPassword`¶

`client.ssl.insecure`¶

`client.ssl.hostnameVerifier`¶

`exceptionHandler`¶

`org.voltdb.stream.plugin.window.api.WindowTrigger.atWindowEnd`¶

`org.voltdb.stream.plugin.window.api.WindowTrigger.atAggregate`¶