Archive
Archive
As new versions of VoltDB become available, you will want to upgrade the VoltDB software on your database cluster. The simplest approach for upgrading recent versions of VoltDB — V6.8 or later — is to perform an orderly shutdown saving a final snapshot, upgrade the software on all servers, then re-start the database. (If you are upgrading from earlier versions of the software, you can still upgrade using a snapshot. But you will need to perform the save and restore operations manually.)
However, upgrading using snapshots involves downtime while the software is being updated. An alternative is to use database replication (DR) — either passive DR or cross data center replication (XDCR) — to upgrade with minimal or no downtime.
Using passive DR you can copy the active database contents to a new cluster, then switch the application clients to point to the new server. The advantage of this process is that the only downtime the business application sees is the time needed to promote the new cluster and redirect the clients.
Using cross data center replication (XDCR), it is possible to perform an online upgrade, where there is no downtime and the database is accessible throughout the upgrade operation. If two or more clusters are already active participants in an XDCR environment, you can shutdown and upgrade the clusters, one at a time, to perform the upgrade leaving at least one cluster available at all times.
The following sections describe five approaches to upgrading VoltDB software:
Upgrading the VoltDB software on a single database cluster is easy. All you need to do is perform an orderly shutdown saving a final snapshot, upgrade the VoltDB software on all servers in the cluster, then restart the database. The steps to perform this procedure are:
Shutdown the database and save a final snapshot (voltadmin shutdown --save).
Upgrade VoltDB on all cluster nodes.
Restart the database (voltdb start).
This process works for any recent (V6.8 or later) release of VoltDB.
To upgrade older versions of VoltDB software (prior to V6.8), you must perform the save and restore operations manually. The steps when upgrading from older versions of VoltDB are:
Place the database in admin mode (voltadmin pause).
Perform a manual snapshot of the database (voltadmin save --blocking).
Shutdown the database (voltadmin shutdown).
Upgrade VoltDB on all cluster nodes.
Re-initialize the root directory on all nodes (voltdb init --force).
Start a new database in admin mode (voltdb start --pause).
Restore the snapshot created in Step #2 (voltadmin restore).
Return the database to normal operations (voltadmin resume).
When upgrading the VoltDB software in a production environment, it is possible to minimize the disruption to client applications by upgrading across two clusters using passive database replication (DR). To use this process you need a second database cluster to act as the DR replica and you must have a unique cluster ID assigned to the current database.
The basic process for upgrading the VoltDB software using DR is to:
Install the new VoltDB software on the secondary cluster
Use passive DR to synchronize the database contents from the current cluster to the new cluster
Pause the current database and promote the new cluster, switching the application clients to the new upgraded database
The following sections describe in detail the prerequisites for using this process, the steps to follow, and — in case there are any issues with the updated database — the process for falling back to the previous software version.
The prerequisites for using DR to upgrade VoltDB are:
A second cluster with the same configuration (that is, the same number of servers and sites per host) as the current database cluster.
The current database cluster must have a unique cluster ID assigned in its deployment file.
The cluster ID is assigned in the <dr> section of the deployment file
and must be set when the cluster starts. It cannot be added or altered
while the database is running. So if you are considering using this
process for upgrading your production systems, be sure to add a
<dr> tag to the deployment
and assign a unique cluster ID when starting the database, even if you
do not plan on using DR for normal operations.
For example, you would add the following element to the deployment file when starting your primary database cluster to assign it the unique ID of 3.
<dr id="3">
An important constraint to be aware of when using this process is that you must not make any schema changes during the upgrade process. This includes the period after the upgrade while you verify the application's proper operation on the new software version. If any changes are made to the schema, you may not be able to readily fall back to the previous version.
The procedure for upgrading the VoltDB software on a running
database using DR is the following. In the examples, we assume the
existing database is running on a cluster with the nodes
oldsvr1 and oldsvr2 and the new
cluster includes servers newsvr1 and
newsvr2. We will assign the clusters unique IDs 3
and 4, respectively.
Install the new VoltDB software on the secondary cluster.
Follow the steps in the section "Installing VoltDB" in the Using VoltDB manual to install the latest VoltDB software.
Start the second cluster as a replica of the current database cluster.
Once the new software is installed, create a new database on the secondary server using the voltdb init --force and voltdb start commands and including the necessary DR configuration to create a replica of the current database. For example, the configuration file on the new cluster might look like this:
<dr id="4" role="replica"> <connection source="oldsvr1,oldsvr2"/> </dr>
Once the second cluster starts, apply the schema from the current database to the second cluster. Once the schema match on the two databases, replication will begin.
Wait for replication to stabilize.
During replication, the original database will send a snapshot of the current content to the new replica, then send binary logs of all subsequent transactions. You want to wait until the snapshot is finished and the ongoing DR is processing normally before proceeding.
First monitor the DR statistics on the new cluster. The DR consumer state changes to "RECEIVE" once the snapshot is complete. You can check this in the Monitor tab of the Volt Management Center or from the command line by using sqlcmd to call the @Statistics system procedure, like so:
$ sqlcmd --servers=newsvr1 1> exec @Statistics drconsumer 0;
Once the new cluster reports the consumer state as "RECEIVE", you can monitor the rate of replication on the existing database cluster using the DR producer statistics. Again, you can view these statistics in the Monitor tab of the Volt Management Center or by calling @Statistics using sqlcmd:
$ sqlcmd --servers=oldsvr1 1> exec @Statistics drproducer 0;
What you are looking for on the producer side is that the DR latency is low; ideally under a second. Because the DR latency helps determine how long you will wait for the cluster to quiesce when you pause it and, subsequently, how long the client applications will be stalled waiting for the new cluster to be promoted. You determine the latency by looking at the difference between the statistics for the last queued timestamp and the last ACKed timestamp. The difference between these values gives you the latency in microseconds. When the latency reaches a stable, low value you are ready to proceed.
Pause the current database.
The next step is to pause the current database. You do this using the voltadmin pause --wait command:
$ voltadmin pause --host=oldsvr1 --wait
The --wait flag tells voltadmin to wait until all DR and export queues are flushed to their downstream targets before returning control to the shell prompt. This guarantees that all transactions have reached the new replica cluster.
If DR or export are blocked for any reason — such as a network outage or the target server unavailable — the voltadmin pause --wait command will continue to wait and periodically report on what queues are still busy. If the queues do not progress, you will want to fix the underlying problem before proceeding to ensure you do not lose any data.
Promote the new database.
Once the current database is fully paused, you can promote the new database, using the voltadmin promote command:
$ voltadmin promote --host=newsvr1
At this point, your database is up and running on the new VoltDB software version.
Redirect client applications to the new database.
To restore connectivity to your client applications, redirect them from the old cluster to the new cluster by creating connections to the new cluster servers newsvr1, newsvr2, and so on.
Shutdown the original cluster.
At this point you can shutdown the old database cluster.
Verify proper operation of the database and client applications.
The last step is to verify that your applications are operating properly against the new VoltDB software. Use the Volt Management Center to monitor database transactions and performance and verify transactions are completing at the expected rate and volume.
Your upgrade is now complete. If, at any point, you decide there is an issue with your application or your database, it is possible to fall back to the previous version of VoltDB as long as you have not made any changes to the underlying database schema. The next section explains how to fall back when necessary.
In extreme cases, you may find there is an issue with your application and the latest version of VoltDB. Of course, you normally would discover this during testing prior to a production upgrade. However, if that is not the case and an incompatibility or other conflict is discovered after the upgrade is completed, it is possible to fall back to a previous version of VoltDB. The basic process for falling back is to the following:
If any problems arise before Step #6 (redirecting the clients) is completed, simply shutdown the new replica and resume the old database using the voltadmin resume command:
$ voltadmin shutdown --host=newsvr1 $ voltadmin resume --host=oldsvr1
If issues are found after Step #6, the fall back procedure is basically to repeat the upgrade procedure described in Section 4.4.3.2, “The Passive DR Upgrade Process” except reversing the roles of the clusters and replicating the data from the new cluster to the old cluster. That is:
Update the configuration file on the new cluster to enable DR as a master, removing the <connection> element:
<dr id="4" role="master"/>
Shutdown the original database and edit the configuration file to enable DR as a replica of the new cluster:
<dr id="3" role="replica"> <connection source="newsvr1,newsvr2"/> </dr>
Re-initialize and start the old cluster using the voltdb init --force and voltdb start commands.
Follow steps 3 through 8 in Section 4.4.3.2, “The Passive DR Upgrade Process” reversing the roles of the new and old clusters.
It is also possible to upgrade the VoltDB software using cross data center replication (XDCR), by simply shutting down, upgrading, and then re-initalizing each cluster, one at a time. This process requires no downtime, assuming your client applications are already designed to switch between the active clusters.
Use of XDCR for upgrading the VoltDB software is easiest if you are already using XDCR because it does not require any additional hardware or reconfiguration. The following instructions assume that is the case. Of course, you could also create a new cluster and establish XDCR replication between the old and new clusters just for the purpose of upgrading VoltDB. The steps for the upgrade outlined in the following sections are the same. But first you must establish the cross data center replication between the two (or more) clusters. See the chapter on Database Replication in the Using VoltDB manual for instructions on completing this initial step.
Once you have two clusters actively replicating data with XCDCR (let's call them clusters A and B), the steps for upgrading the VoltDB software on the clusters is as follows:
Pause and shutdown cluster A (voltadmin pause --wait and shutdown).
Clear the DR state on cluster B (voltadmin dr reset).
Update the VoltDB software on cluster A.
Start a new database instance on A, making sure to use the old deployment file so the XDCR connections are configured properly (voltdb init --force and voltdb start).
Load the schema on Cluster A so replication starts.
Once the two clusters are synchronized, repeat steps 1 through 4 for cluster B.
Note that since you are upgrading the software, you must create a new instance after the upgrade (step #3). When upgrading the software, you cannot recover the database using just the voltdb start command; you must use voltdb init --force first to create a new instance and then reload the existing data from the running cluster B.
Also, be sure all data has been copied to the
upgraded cluster A after step #4 and before proceeding to upgrade the
second cluster. You can do this by checking the @Statistics system
procedure selector DRCONSUMER on cluster A. Once the DRCONSUMER
statistics State column changes to "RECEIVE", you
know the two clusters are properly synchronized and you can proceed to
step #5.
In extreme cases, you may decide after performing the upgrade that you do not want to use the latest version of VoltDB. If this happens, it is possible to fall back to the previous version of VoltDB.
To "downgrade" from a new version back to the previous version, follow the steps outlined in Section 4.4.4, “Performing an Online Upgrade Using Multiple XDCR Clusters” except rather than upgrading to the new version in Step #2, reinstall the older version of VoltDB. This process is valid as long as you have not modified the schema or deployment to use any new or changed features introduced in the new version.
The sections describing the upgrade process for passive DR and active XDCR both explain how to fall back to the previous version of VoltDB in case of emergency. This section explains how to fall back, or downgrade, when using the standard save and restore process described in Section 4.4.1, “Upgrading VoltDB Using Save and Restore”.
The following process works if you are reverting between two recent versions of VoltDB and you do not use any new features between the upgrade and the downgrade. There are no guarantees an attempt to downgrade will succeed if the two software versions are more than one major version apart or if you utilize a new feature from the higher version software prior to downgrading.
With those caveats, the most reliable way to fall back to a previous VoltDB version is:
Extract the database schema and stored procedure classes
Pause the database, save a snapshot, and shutdown
Re-install the previous version of VoltDB
Initialize a new database root directory, using the extracted schema and classes
Start the new database instance (in pause mode) using the older version of VoltDB
Manually restore the data from the snapshot created in Step #2
Resume normal operations
This process ensures that only the schema, stored procedures, and data are returned to the older version of the software, and new software features will not impact your restore process. For example:
$ voltdb get schema -D ~/db/new --output=/tmp/mydb.sql $ voltdb get classes -D ~/db/new --output=/tmp/mydb.jar $ voltadmin pause $ voltadmin save /tmp mydata $ voltadmin shutdown [ downgrade VoltDB software . . . ] $ voltdb init -f -D ~/db/old --schema=/tmp/mydb.sql --classes=/top/mydb.jar $ voltdb start -D ~/db/old --pause & $ voltadmin restore /tmp mydata $ voltadmin resume
V12.x Documentation