You configure autoscaling using the cluster.clusterSpec.autoScaling.* properties. First, you must set the cluster.clusterSpec.autoScaling.enabled property to true. Next, you select one or more metrics to measure as triggers for autoscaling. At the same time, you define the upper and lower thresholds using the .scaleUp and .scaleDown properties. There are currently four metrics you can choose from:

You must define at least one metric as a trigger, but you can define more than one if you like. For example, the following Helm chart enables autoscaling based on both TPS and CPU, using 50K TPS or 75% CPU usage as the upper threshold and 10K TPS or 25% CPU usage as the lower threshold:

cluster:
  clusterSpec:
    autoScaling:
      enabled: true
      metrics:
        tps:
          scaleUp: 50000
          scaleDown: 10000
        cpu:
          scaleUp: 75
          scaleDown: 25

Once autoscaling begins, it takes time for the cluster to either elastically expand or shrink, which can vary significantly depending on the size of the data, the current workload, and other circumstances. During this period, it is a good idea to monitor the process to ensure the elastic operations are proceeding as expected.

The operator periodically updates the status of the operation in the operator log, the Kubernetes cluster status, and as Kubernetes events. You can see the operator logs using the kubectl logs command. For example, if the Helm release name is mydb:

$  kubectl logs -f deploy/mydb-voltdb-operator    

To see the Kubernetes events associated with the operator, you can use the kubectl events command:

$  kubectl events --for deploy/mydb-voltdb-operator    

Or you can see both the cluster status and Kubernetes events using the kubectl describe command:

$  kubectl describe voltdbcluster mydb-voltdb-cluster
 [ . . . ]
Status:
  Cluster State:
    Auto Scaling:                                
      Desired Replicas:  3                       
      Metrics:                                   
        Tps:
      State:  Monitoring auto-scaling metrics    

You can use whichever method you find most useful. However, the following examples use the cluster status because it is the easiest to read. In the preceding example, where a three node, K=1 cluster has autoscaling enabled, the status display tells you:

If a metric crosses an associated threshold, the status changes to indicate which metric it is and when the threshold was crossed:

Status:
  Cluster State:
    Auto Scaling:
      Desired Replicas:  3                                          
      Metrics:
        Tps:
          Direction:               ScaleUp                          
          Last Value:              22981
          Time Threshold Crossed:  2024-02-02T19:27:05.870Z
      State:                       Monitoring auto-scaling metrics    

The number of replicas (1) remains the same, because the stabilization window is in effect. But the metrics section (2) now includes information on which threshold has been triggered and when. While the status line still reports it is monitoring the metrics to make sure it stays over the threshold for the duration of the stabilization window.

If the metrics stays above (or below) the threshold when the stabilization window expires, the autoscaling event begins and the status changes to reflect the new state:

Status:
  Cluster State:
    Auto Scaling:
      Desired Replicas:  5                                         
      Direction:         ScaleUp
      Metrics:
        Tps:
          Direction:               ScaleUp                         
          Last Value:              28617
          Time Threshold Crossed:  2024-02-02T19:38:03.763Z
      State:                       Scaling cluster                 
      Time Scaling Notified:       2024-02-02T19:38:34.695Z
      Time Scaling Started:        2024-02-02T19:38:34.695Z  

The target number of replicas (1) increases by K+1, to 5. Note this is the desired number of nodes, not the current number. It also reports the direction the cluster is resizing (in this case, scaling up). The metrics information (2) continues to report the current value and when the threshold was crossed triggering the autoscale activity. Finally, the status line (3) now reports that the cluster is elastically resizing to expand the cluster.

The cluster status will continue to report this information throughout the elastic resizing, which can take minutes, or even hours, depending on how much data must be moved and how busy the database is processing transactions. But once the resize is complete, the status returns to its initial stage, except the number of desired replicas now matches the new cluster size, 5.

Status:
  Cluster State:
    Auto Scaling:
      Desired Replicas:  5
      Metrics:
        Tps:
      State:  Monitoring auto-scaling metrics

The autoScaling.maxRetries and autoScaling.retryTimeout properties give the Volt operator some flexibility in recovering from issues that might arise during an autoscaling event. However, not all failures are transitory. For example, there may be provisioning or configuration issues that are stopping the cluster from expanding. Which it is why it is important to know what to do if autoscaling does not operate correctly.

The first step is to monitor the autoscaling activity — either proactively or by reviewing the logs and status after the fact. If the maximum number of retries is exceeded or a scaling operation fails to restart, the cluster status is changed to indicate that autoscaling has stopped:

Status:
  Cluster State:
    Auto Scaling:
      Desired Replicas:  3
      Direction:         ScaleDown
      Metrics:
        Tps:
          Direction:               ScaleDown
          Time Threshold Crossed:  2024-02-02T21:24:17.690Z
      State:                       Auto-scaling stopped by failure

At this point autoscaling has stopped. But more importantly, the elastic resizing has not completed. So it requires human intervention to either complete or revert the resizing before autoscaling can be resumed. Again, the first step is to determine what is causing the resize operation to fail and, if at all possible, correcting the situation. This is particularly true when resizing to reduce the size of the cluster because the only reliable way to recover the cluster is to complete the resizing operation.

To do this, you want to shutdown the cluster, then restart it specifying the number of replicas before the resize operation began. For example, if the cluster was resizing from 5 nodes to 3, you will want to restart the cluster with the number of replicas set to 5:

$ helm upgrade mydb voltdb/voltdb --resuse-values \
  --set cluster.clusterSpec.replicas=5

Finally, there are a few basic rules about what not to do with the cluster while using autoscaling:

In all cases, it is safest to wait for all resizing to complete, then disable autoscaling by setting cluster.clusterSpec.autoScaling.enabled to false before performing any of the preceding actions.