K-safety ensures the database cluster can survive at least a certain number of node failures. However, to reduce the risk of larger scale outages, you need to make sure that the Volt servers are distributed in such a way to minimize the impact of external outages. In particular, you want to ensure that each Volt server pod runs on a separate Kubernetes node (so that a Kubernetes node failure cannot impact multiple pods) and that the pods are, as much as possible, evenly distributed among the availability zones in use.

By default, the Volt Operator establishes Kubernetes affinity and anti-affinity rules such that no two Volt server pods can run on the same Kubernetes node. So, normally, you do not need to take any actions to make this happen. However, if you are overriding the Operator's default configurations, you will need to make sure your custom Kubernetes configuration includes this behavior.

When using multiple availability zones, you should also adjust the Kubernetes configuration — specifically the spread constraints — so that the Volt server pods are evenly distributed among the zones. This makes it possible to avoid the database failing due to the loss of any one zone that contains an unbalanced and excessive number of Volt server processes. You can define the distribution of server pods within your Helm configuration using the cluster.clusterSpec.topologySpreadConstraints property. The following example demonstrates how to do this, using the label selector to identify the Volt server processes.

cluster:
  clusterSpec:
    topologySpreadConstraints:
    - topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      maxSkew: 1
      labelSelector:
        matchLabels:
          name: voltdb-cluster

If you are running multiple databases within a single namespace, you should consider replacing the last line of the configuration, "name: voltdb-cluster", with an identifier that is specific to the cluster being configured. For example, if the cluster release name is mydb, the last line of the configuration should read "voltdb-cluster-name: mydb-voltdb-cluster".

K-safety guarantees the minimum number of nodes that can fail without stopping the database. Configuring Kubernetes affinity and spread constraints to evenly distribute the database server pods reduces the overall threat of external failures taking down the database. However, to fully maximize the availability, Volt needs to use knowledge about the Kubernetes configuration to intelligently distribute the individual copies of the partitions among those servers.

The cluster may survive more failures than just the minimum guaranteed by K-safety depending on how the partitions are distributed and which nodes fail. Placement groups are a mechanism for providing more context concerning the hardware environment to improve the likelihood of the cluster surviving multiple failures. For example, if you tell Volt certain nodes are in the same region and zone (i.e. in the same placement group), it avoids placing all copies of any partition on those nodes, so if the zone fails, the database can survive.

Because you do not control exactly where each pod is created in Kubernetes, Volt can use its knowledge of the Kubernetes availability zones and regions^[2] to automate the placement groups and minimize the potential of an infrastructure failure taking the database down with it. You enable cloud native placement groups in Kubernetes by setting the property cluster.clusterSpec.useCloudNativePlacementGroup to "true". For cloud native placement groups to be effective, the cluster configuration must meet the following requirements:

For example, the following configuration assumes the cluster is distributed across four availability zones:

cluster:
  clusterSpec:
    replicas: 8
    useCloudNativePlacementGroup: true
  config:
    deployment:
      cluster:
        sitesperhost: 8
        kfactor: 1

Once the database is running, you can use the @Statistics system procedure with the HOST selector to determine where each node is running and what partitions are running on that node. In addition, if one or more nodes go down, the "SAFETOSTOP" column lets you know which of the remaining nodes could safely be stopped without endangering the cluster as a whole.

$ sqlcmd
1> execute @Statistics HOST;
TIMESTAMP     HOST_ID HOSTNAME              PARTITIONS              LEADERS     PLACEMENTGROUP SAFETOSTOP REGION ZONE
------------- ------- --------------------- ----------------------- ----------- -------------- ---------- ------ ----
1677777171869       0 mydb-voltdb-cluster-0 24,25,26,27,28,29,30,31 25,27,29,31 east--zone4    true       east   zone4
1677777171870       1 mydb-voltdb-cluster-1 8,9,10,11,12,13,14,15   8,10,12,14  east--zone1    true       east   zone1
1677777171870       2 mydb-voltdb-cluster-2 8,9,10,11,12,13,14,15   9,11,13,15  east--zone2    true       east   zone2
1677777171870       3 mydb-voltdb-cluster-3 16,17,18,19,20,21,22,23 16,18,20,22 east--zone3    true       east   zone3
1677777171870       4 mydb-voltdb-cluster-4 16,17,18,19,20,21,22,23 17,19,21,23 east--zone4    true       east   zone4
1677777171870       5 mydb-voltdb-cluster-5 24,25,26,27,28,29,30,31 24,26,28,30 east--zone3    true       east   zone3
1677777171870       6 mydb-voltdb-cluster-6 0,1,2,3,4,5,6,7         0,2,4,6     east--zone1    true       east   zone1
1677777171870       7 mydb-voltdb-cluster-7 0,1,2,3,4,5,6,7         1,3,5,7     east--zone2    true       east   zone2
(Returned 8 rows in 0.01s)
TIMESTAMP     PLACEMENTGROUP SAFETOSTOP
------------- -------------- ----------
1677777171882 east--zone3    true
1677777171882 east--zone2    true
1677777171882 east--zone1    true
1677777171882 east--zone4    true