Kubernetes is an orchestration tool for managing applications and services run in containers. It is designed to automate the management of distributed applications, with a particular focus on microservices. VoltDB is not a microservice; there is more interaction between a VoltDB cluster's individual nodes than in most microservices. However, you can still use Kubernetes to run and manage VoltDB clusters.

This document describes one approach for running VoltDB in a Kubernetes environment. It is automated by scripts provided in the VoltDB kit starting with VoltDB V8.3.3. The process utilizes four key aspects of Kubernetes:

Before starting the process described in this document, make sure your development system and account meet the following requirements:

In addition to the preceding requirements for your development system, your Kubernetes hosting environment must also have access to the Docker repository you are using (or another repository the compiled Docker image will be copied to) and have the ability to instantiate persistent storage volumes to retain the VoltDB root directories and their contents for when pods are deleted and recreated.

VoltDB, like all memory-intensive applications, is sensitive to base system settings that control access to assets such as memory, CPU, and networking. Memory configuration options such as swapping and Transparent Huge Pages (THP) can dramatically impact the performance of a VoltDB database. In extreme cases, THP can result in unnecessary allocations of memory to the point where the server process runs out of memory. This is why VoltDB recommends turning off THP.

The VoltDB Administrator's Guide contains a chapter on how to prepare a server for optimal use with VoltDB, including disabling THP. However, in a virtualized environment — and particularly when using containers — you do not have direct access to the base system's core settings from within the virtual machine (VM) or container. Therefore, to ensure proper and consistent operation of VoltDB in such environments, you must make the necessary adjustments on the base system itself.

If you are running VMs on hardware you manage, this can be done by configuring the servers yourself. If, on the other hand, you are using a hosted environment, you should consider other alternatives. You can negotiate with the hosting provider to ensure the servers are configured correctly. Alternately, you can use privileged mode VMs (essentially, dedicated servers reserved for your account) and Kubernetes features to modify the base system before any containers are run.

The following are some resources that can help you make the appropriate decision about your configuration choices:

The first step is to create a Kubernetes configuration file. You do this by copying the configuration template in the /tools/kubernetes folder where VoltDB is installed. For example:

$ cd ~/voltdb/tools/kubernetes
$ cp config_template.cfg productDB.cfg

By default, the name of the configuration file is used for all output files, the Docker image, and the Kubernetes cluster itself. Alternately, you can specify a tag name in the configuration file that is used to name the artifacts.

Once you create a copy of the configuration file, edit it to define the properties you want for your database cluster, including deployment attributes and runtime assets such as the schema and procedure classes. The scripts ensure your assets are available to the cluster from Kubernetes ConfigMaps and that they and the software are copied to the appropriate locations during startup.

See the README file in the /tools/kubernetes folder for specifics on what properties you can set in the configuration file.

The scripts in the Docker image look for an existing database root directory. If they do not find it, they allocate Kubernetes persistent storage and initialize a new root directory. The use of persistent storage means the root directory is preserved, even when individual pods or the cluster as a whole goes down. This supports VoltDB availability features, such as command logging and automated snapshots, that save and restore the database contents across sessions.

Once you edit the configuration file, you can create the Docker image that Kubernetes uses to start the cluster. For example:

$ cd ~/voltdb/tools/kubernetes
$ bash voltdb-k8s-utils.sh -B

The next step is to load the assets needed to configure and start VoltDB into Kubernetes ConfigMaps. These assets may include the VoltDB configuration file, database schema, procedure classes, and license file, as defined by Kubernetes configuration created in Section 2.1, “Creating the Configuration File”. For example:

$ cd ~/voltdb/tools/kubernetes
$ bash voltdb-k8s-utils.sh -M

Finally, you create the definition of the Kubernetes stateful set (that is, the YAML file) and start the cluster using the -C and -S arguments to the voltdb-k8s-utils.sh script. For example:

$ cd ~/voltdb/tools/kubernetes
$ bash voltdb-k8s-utils.sh -C
$ bash voltdb-k8s-utils.sh -S

Once the cluster starts, Kubernetes takes care of monitoring and restarting the pods as needed to keep the database running. The voltdb-k8s-utils.sh script provides other arguments to perform management actions manually, such as shutting down the cluster and deleting Kubernetes resources. See the README for details.

To establish DR between two VoltDB clusters, the nodes of the consumer cluster must be able to create a network connection to the IP addresses of all the producer cluster nodes. In passive DR, this means the replica cluster must be able to resolve the IP addresses of the master cluster. XDCR clusters act as both consumer and producer, so all clusters must be able to resolve the IP addresses of other clusters in the XDCR relationship.

When setting up DR between clusters in different Kubernetes clusters or between Kubernetes and another production environment, it is important that your network topology meets this requirement; for example, by using a virtual private cloud (VPC) , a virtual Local area network (VLAN), or port remapping as described in Section 4, “Connecting to a VoltDB Database in Kubernetes”. Which of these features is available to you and how you instantiate them is dependent on your Kubernetes provider and your agreement with them.

Establishing the DR connection at runtime occurs in two steps:

The IP address used in step #1 needs to be static so it can be defined in the configuration file before the clusters start. On the other hand, the IP addresses Kubernetes assigns to individual pods used in step #2 are temporal and can change.

Fortunately, the addresses used in the configuration file to find the producer cluster and the addresses returned by the producer at runtime do not have to be the same. One way to bridge this difference is to use a Kubernetes load balancer to connect a static IP address to the producer cluster. The consumer cluster can configure its DR connection using the static IP address of the load balancer and then resolve the actual IP addresses returned at runtime using a virtual network.

When using port remapping (rather than VPC or VLANs), it is important that in step #1 the producer cluster responds the with remapped IP addresses rather than its internal addresses. You can ensure this happens by specifying the remapped address (and, optionally, port) in the --drpublic argument in the VOLTDB_START_ARGS property of the Kubernetes configuration file..

Finally, it is important to note that performance of the network dramatically impacts the performance of data replication and VoltDB itself. For best performance, you must use networking resources that can support the peak bandwidth requirements of the replication stream. Hosting services often provide networking features with different performance tiers. Inadequate network bandwidth can result in bottlenecks, backlogs, and ultimately, if not addressed, the DR connection itself may be broken. So use of high bandwidth cross-zone or cross-region networking is recommended when supporting DR.