Elastic scaling is the ability to resize the cluster as needed, without having to shutdown the database. Elastic scaling supports both increasing and decreasing the size of the cluster. For example, you might want to increase the size of the cluster ahead of an important announcement that will drive additional traffic — and subsequently require additional capacity. Similarly, you may want to reduce the size for the cluster during slow periods to limit the number of resources that would be under utilized.
Adding and removing nodes using elastic scaling are each handled separately because increasing the size of the cluster requires adding new nodes to the cluster first. While when decreasing the size of the cluster, the nodes are already part of the cluster and VoltDB decides which node are most advantageous to remove based on the distribution of partitions within the cluster.
To add nodes to the cluster you start the additional nodes using the voltdb start --add command. To remove nodes from the cluster, you use the voltadmin resize command and the cluster decides which nodes to remove.
But in both cases, the correct number of nodes must be added or removed at the same time. The number of nodes added or removed must result in the resized cluster meeting the requirements for a K-safe cluster based on the K-safety value and number of sites per host (as described in Section 10.2.2, “Calculating the Appropriate Number of Nodes for K-Safety”). So for a cluster with no K-safety (K=0), nodes can be added and removed individually. For K-safe clusters, K+1 nodes must be added or removed at a time. For example, with K=1 two nodes must be added at a time. While in the case of reducing the size of the cluster, two nodes must be removed but the resulting cluster must also meet the requirement that the total number of partitions (sites per host X number of nodes) is divisible by K+1.
Finally, resizing the cluster "on the fly" does require both time and some amount of resources while the data and partitions are rebalanced. The length of time required to complete the rebalancing depends on the amount of data present and the current workload. Similarly, the performance impact of resizing on the ongoing operation of the cluster depends on how much additional capacity the cluster has to assign to rebalance tasks.
The following sections describe how to:
Add nodes using elastic scaling
Remove nodes using elastic scaling
Control the time and performance impact of elastic scaling by configuring the rebalance workoad
When you are ready to extend the cluster by adding one or more nodes, you simply initialize and start the VoltDB
database process on the new nodes using the voltdb init command to initialize and the voltdb
start command to start with the --add argument, specifying the name of one or more of the
existing cluster nodes as the hosts. For example, if you are adding node ServerX to a cluster where ServerA is already a
member, you can execute the following commands on ServerX:
$ voltdb init --config=configuration.yaml $ voltdb start --add --host=ServerA
Once the elastic add action is initiated, the cluster performs the following tasks:
The cluster acknowledges the presence of a new server.
Copies of the current schema and configuration settings are sent to the new node.
Once sufficient nodes are added, copies of all replicated tables and their share of the partitioned tables are sent to the new nodes.
As the data is redistributed (or rebalanced), the added nodes begin participating as full members of the cluster.
There are some important notes to consider when expanding the cluster using elastic scaling:
You must add a sufficient number of nodes to create an integral K-safe unit. That is, K+1 nodes. For example, if the K-safety value for the cluster is two, you must add three nodes at a time to expand the cluster. If the cluster is not K-safe (in other words it has a K-safety value of zero), you can add one node at a time.
When you add nodes to a K-safe cluster, the nodes added first will complete steps #1 and #2 above, but will not complete steps #3 and #4 until the correct number of nodes are added, at which point all nodes rebalance together.
While the cluster is rebalancing (Step #3), the database continues to handle incoming requests. However, depending on the workload and amount of data in the database, rebalancing may take a significant amount of time.
Once elastic scaling is complete, your database configuration has changed. If you shutdown the database and then
restart, you must specify the new server count in the --count argument to the voltdb
start command.
When you want to reduce the size of your cluster, you use the voltadmin resize command to start the resizing process. First, as with any significant maintenance activity, it is a good idea to take a snapshot of the database contents before you begin, just in case you need to restore it later. The next step is to test to make sure the cluster can be reduced. You do this using the voltadmin resize --test command:
$ voltadmin resize --test
The --test qualifier verifies that there are sufficient nodes and partitions to reduce the cluster while maintaining the K-safety and sitesperhost settings. If not, the command will report that the cluster cannot be reduced in size. If resizing is possible, the command reports which nodes will be removed when resizing begins.
Once you are ready to begin the resizing process, you use the voltadmin resize command:
$ voltadmin resize
The command repeats the test phase, reports which nodes will be removed and starts the resizing process.
Once resizing begins, the process cannot be canceled. Even if the cluster stops, resizing will continue once the cluster restarts (and you must restart all of the original nodes so the resize operation can complete). So be sure you want to reduce the cluster size before you issue the voltadmin resize command.
The length of time it takes for resizing to complete depends on the amount of data in the database and the current workload. You can adjust parameters that affect resizing (as described in Section 9.3.3, “Configuring How VoltDB Rebalances Nodes During Elastic Scaling”). However, increasing the duration or throughput for resizing will likely have a corresponding inverse impact on the performance of ongoing database activities. Use the voltadmin status to check on the current status of the resizing operation, or use the @Statistics system procedure with the REBALANCE selector for details.
Finally, if an unexpected event causes the resize process to fail — which will be reported in the server logs — you can restart the resize operation using the voltadmin resize --restart command.
As you add or remove nodes using elastic scaling, VoltDB rebalances the cluster by rearranging data within the partitions. During elastic expansion, as soon as you add the necessary number of nodes (based on the K-safety value), VoltDB rebalances the cluster, moving data from existing partitions to partitions on the new nodes. During elastic contraction, before the nodes are removed, VoltDB rebalances the cluster by moving data from partitions that are being removed to partitions that will remain.
During the rebalance phase, the database remains available and actively processing client requests. How long the rebalance operation takes is dependent on two factors: how often rebalance tasks are processed and how much data each transaction moves.
Rebalance tasks are fully transactional, meaning they operate within the database's ACID-compliant transactional model. Because they involve moving data between two or more partitions, they are also multi-partition transactions. This means that each rebalance work unit can incrementally add to the latency of pending client transactions.
You can control how quickly the rebalance operation completes versus how much rebalance work impacts ongoing client
transactions using the subproperties of deployment.systemsettings.elastic:
The duration property sets a target value for the length of time each rebalance
transaction will take, specified in milliseconds. The default is 50 milliseconds.
The throughput property sets a target value for the number of megabytes per
second that will be processed by the rebalance transactions. The default is 2 megabytes.
When you change the target duration, VoltDB adjusts the amount of data that is moved in each transaction to reach the target execution time. If you increase the duration, the volume of data moved per transaction increases. Similarly, if you reduce the duration, the volume per transaction decreases.
When you change the target throughput, VoltDB adjusts the frequency of rebalance transactions to achieve the desired volume of data moved per second. If you increase the target throughout, the number of rebalance transactions per second increases. Similarly, if you decrease the target throughout, the number of transactions decreases.
For example, the following configuration file sets the target duration to 15 milliseconds and the target throughput to 1 megabyte per second before starting the database:
Documentationdeployment:
systemsettings:
elastic:
duration: 15
throughput: 1