2.3. Configuring the Database

Documentation

VoltDB Home » Documentation » VoltDB Kubernetes Administrator's Guide

2.3. Configuring the Database

In addition to configuring the environment VoltDB runs in, there are many different characteristics of the database itself you can control. These include mapping network interfaces and ports, selecting and configuring database features, and identifying the database schema, class files, and security settings.

The network settings are defined through the cluster.serviceSpec properties, where you can choose the individual ports and choose whether to expose them through the networking service. For example, the following YAML file disables exposure of the admin port and assigns the externalized client port to 31313:

cluster:
  serviceSpec:
    type: NodePort
      adminPortEnabled: false
      clientPortEnabled: true
      clientNodePort: 31313

The majority of the database configuration options for VoltDB are traditionally defined in an XML configuration file. When using Kubernetes, these options are declared using YAML and Helm properties. The Helm properties follow the same structure as the XML configuration, beginning with "cluster.config". So, for example, where the number of sites per host is defined in XML as:

<deployment>
   <cluster sitesperhost="{n}"/>
</deployment>

It is defined in Kubernetes as:

cluster:
  config:
    deployment:
      cluster:
        sitesperhost: {n}

The following sections give examples of defining common database configurations options using YAML. See Section B.6, “VoltDB Database Configuration Options” for a complete list of the Helm properties available for configuring the database.

2.3.1. Configuring High Availability (K-Safety and Placement Groups)

Volt Active Data provides high availability through K-safety, where copies of each partition are distributed to different nodes in the database cluster. If a node fails, the database can continue to operate because there are still copies of every partition within the cluster. The amount of durability depends on the K factor. So a K factor of one means that the cluster is guaranteed to survive one node (or pod) failing, a factor of two guarantees two nodes, and so on. (See the chapter on Availability in the Using VoltDB manual for more information on how K-safety works.)

You set the K-safety factor using the cluster.config.deployment.cluster.kfactor property when configuring your database. For example, the following YAML sets the K-safety factor to two:

cluster:
  clusterSpec:
    replicas: 6
  config:
    deployment:
      cluster:
        sitesperhost: 8
        kfactor: 2

Note that the number of replicas must be at least as large as the K factor plus one (K+1) and K-safety is most effective if the number of replicas times the number of sites per host is a multiple of K+1.

The combination of K-safety and Kubernetes provides an automated, self-healing system where K-safety ensures the cluster survives individual nodes failing and Kubernetes manages the automated recreation of the pods when they fail so the database can be restored to a full complement of nodes as soon as possible. However, to take full advantage of this capability you need to ensure the Kubernetes infrastructure is configured correctly to distribute the Volt servers evenly and that Volt uses information about the configuration to manage the distribution of partitions within the database. The following sections explain how to use Kubernetes configuration options, such as affinity and spread constraints, and Volt placement groups to achieve maximum availability.

2.3.1.1. Configuring Kubernetes Clusters for High Availability (Spread Constraints and Affinity)

K-safety ensures the database cluster can survive at least a certain number of node failures. However, to reduce the risk of larger scale outages, you need to make sure that the Volt servers are distributed in such a way to minimize the impact of external outages. In particular, you want to ensure that each Volt server pod runs on a separate Kubernetes node (so that a Kubernetes node failure cannot impact multiple pods) and that the pods are, as much as possible, evenly distributed among the availability zones in use.

By default, the Volt Operator establishes Kubernetes affinity and anti-affinity rules such that no two Volt server pods can run on the same Kubernetes node. So, normally, you do not need to take any actions to make this happen. However, if you are overriding the Operator's default configurations, you will need to make sure your custom Kubernetes configuration includes this behavior.

When using multiple availability zones, you should also adjust the Kubernetes configuration — specifically the spread constraints — so that the Volt server pods are evenly distributed among the zones. This makes it possible to avoid the database failing due to the loss of any one zone that contains an unbalanced and excessive number of Volt server processes. You can define the distribution of server pods within your Helm configuration using the cluster.clusterSpec.topologySpreadConstraints property. The following example demonstrates how to do this, using the label selector to identify the Volt server processes.

cluster:
  clusterSpec:
    topologySpreadConstraints:
    - topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      maxSkew: 1
      labelSelector:
        matchLabels:
          name: voltdb-cluster

If you are running multiple databases within a single namespace, you should consider replacing the last line of the configuration, "name: voltdb-cluster", with an identifier that is specific to the cluster being configured. For example, if the cluster release name is mydb, the last line of the configuration should read "voltdb-cluster-name: mydb-voltdb-cluster".

2.3.1.2. Cloud Native Placement Groups

K-safety guarantees the minimum number of nodes that can fail without stopping the database. Configuring Kubernetes affinity and spread constraints to evenly distribute the database server pods reduces the overall threat of external failures taking down the database. However, to fully maximize the availability, Volt needs to use knowledge about the Kubernetes configuration to intelligently distribute the individual copies of the partitions among those servers.

The cluster may survive more failures than just the minimum guaranteed by K-safety depending on how the partitions are distributed and which nodes fail. Placement groups are a mechanism for providing more context concerning the hardware environment to improve the likelihood of the cluster surviving multiple failures. For example, if you tell Volt certain nodes are in the same region and zone (i.e. in the same placement group), it avoids placing all copies of any partition on those nodes, so if the zone fails, the database can survive.

Because you do not control exactly where each pod is created in Kubernetes, Volt can use its knowledge of the Kubernetes availability zones and regions[2] to automate the placement groups and minimize the potential of an infrastructure failure taking the database down with it. You enable cloud native placement groups in Kubernetes by setting the property cluster.clusterSpec.useCloudNativePlacementGroup to "true". For cloud native placement groups to be effective, the cluster configuration must meet the following requirements:

  • The cluster must be distributed over three or more regions or availability zones.

  • The number of nodes (or replicas) must be a multiple of the number of availability zones.

  • The number of availability zones must be a multiple of K+1.

For example, the following configuration assumes the cluster is distributed across four availability zones:

cluster:
  clusterSpec:
    replicas: 8
    useCloudNativePlacementGroup: true
  config:
    deployment:
      cluster:
        sitesperhost: 8
        kfactor: 1

Once the database is running, you can use the @Statistics system procedure with the HOST selector to determine where each node is running and what partitions are running on that node. In addition, if one or more nodes go down, the "SAFETOSTOP" column lets you know which of the remaining nodes could safely be stopped without endangering the cluster as a whole.

$ sqlcmd
1> execute @Statistics HOST;
TIMESTAMP     HOST_ID HOSTNAME              PARTITIONS              LEADERS     PLACEMENTGROUP SAFETOSTOP REGION ZONE
------------- ------- --------------------- ----------------------- ----------- -------------- ---------- ------ ----
1677777171869       0 mydb-voltdb-cluster-0 24,25,26,27,28,29,30,31 25,27,29,31 east--zone4    true       east   zone4
1677777171870       1 mydb-voltdb-cluster-1 8,9,10,11,12,13,14,15   8,10,12,14  east--zone1    true       east   zone1
1677777171870       2 mydb-voltdb-cluster-2 8,9,10,11,12,13,14,15   9,11,13,15  east--zone2    true       east   zone2
1677777171870       3 mydb-voltdb-cluster-3 16,17,18,19,20,21,22,23 16,18,20,22 east--zone3    true       east   zone3
1677777171870       4 mydb-voltdb-cluster-4 16,17,18,19,20,21,22,23 17,19,21,23 east--zone4    true       east   zone4
1677777171870       5 mydb-voltdb-cluster-5 24,25,26,27,28,29,30,31 24,26,28,30 east--zone3    true       east   zone3
1677777171870       6 mydb-voltdb-cluster-6 0,1,2,3,4,5,6,7         0,2,4,6     east--zone1    true       east   zone1
1677777171870       7 mydb-voltdb-cluster-7 0,1,2,3,4,5,6,7         1,3,5,7     east--zone2    true       east   zone2
(Returned 8 rows in 0.01s)
TIMESTAMP     PLACEMENTGROUP SAFETOSTOP
------------- -------------- ----------
1677777171882 east--zone3    true
1677777171882 east--zone2    true
1677777171882 east--zone1    true
1677777171882 east--zone4    true

2.3.2. Configuring Command Logging

Command logging provides durability of the database content across failures. You can control the level of durability as well as the length of time required to recover the database by configuring the type of command logging and size of the logs themselves. In Kubernetes this is done with the cluster.config.deployment.commandlog properties. The following example enables synchronous command logging and sets the log size to 3,072 megabytes and the frequency to 1,000 transactions:

cluster:
  config:
    deployment:
      commandlog:
        enabled: true
        synchronous: true
        logsize: 3072
        frequency:
          transactions 1000

2.3.3. Configuring Export

Export simplifies the integration of the VoltDB database with external databases and systems. You use the export configuration to define external "targets" the database can write to. In Kubernetes you define export targets using the cluster.config.deployment.export.configurations property. Note that the configurations property can accept multiple configuration definitions. In YAML, you specify a list by prefixing each list element with a hyphen, even if there is only one element. The following example defines one export target, eventlog, using the file export connector:

cluster:
  config:
    deployment:
      export:
        configurations:
           - target: eventlog
             type: file
             properties:
                type: csv
                nonce: eventlog                 


[2] Placement groups depend on the Kubernetes labels topology.kubernetes.io/region and topology.kubernetes.io/zone, which are defined automatically by most commercial cloud providers. If you are using a custom cloud deployment, you will need to make sure these labels are declared appropriately before enabling cloud native placement groups.