4.4. Upgrading VoltDB Software

Documentation

VoltDB Home » Documentation » Administrator's Guide

4.4. Upgrading VoltDB Software

Finally, as new versions of VoltDB become available, you will want to upgrade the VoltDB software on your database cluster. The simplest approach for upgrading a VoltDB cluster is to perform an orderly shutdown saving a final snapshot, upgrade the software on all servers, then re-start the database.

However, this method involves downtime while the software is being updated. An alternative is to use database replication (DR) — either passive DR or cross data center replication (XDCR) — with minimal or no downtime.

Using passive DR you can copy the active database contents to a new cluster, then switch the application clients to point to the new server. The advantage of this process is that the only downtime the business application sees is the time needed to promote the new cluster and redirect the clients.

Using cross data center replication (XDCR), where both clusters are active participants, you can shutdown and upgrade the clusters, one at a time, to perform the upgrade leaving at least one cluster available at all times.

The following sections describe all three approaches:

4.4.1. Upgrading VoltDB on a Single Database Cluster

Upgrading the VoltDB software on a single database cluster is easy. All you need to do is perform an orderly shutdown saving a final snapshot, upgrade the VoltDB software on all servers in the cluster, then restart the database. The steps to perform this procedure are:

  1. Shutdown the database and save a final snapshot (voltadmin shutdown --save).

  2. Upgrade VoltDB on all cluster nodes.

  3. Restart the database (voltdb start).

This process works for any recent (V6.8 or later) release of VoltDB.

4.4.2. Upgrading Older Versions (Pre-V6.8) of VoltDB

To upgrade older versions of VoltDB software, you must perform the save and restore operations manually. The steps when upgrading from older versions of VoltDB are:

  1. Place the database in admin mode (voltadmin pause).

  2. Perform a manual snapshot of the database (voltadmin save --blocking).

  3. Shutdown the database (voltadmin shutdown).

  4. Upgrade VoltDB on all cluster nodes.

  5. Re-initialize the root directory on all nodes (voltdb init --force).

  6. Start a new database in admin mode (voltdb start --pause).

  7. Restore the snapshot created in Step #2 (voltadmin restore).

  8. Return the database to normal operations (voltadmin resume).

4.4.3. Using a DR Replica to Upgrade VoltDB With Reduced Downtime

When upgrading the VoltDB software in a production environment, it is possible to minimize the disruption to client applications by upgrading across two clusters using passive database replication (DR). To use this process you need a second database cluster to act as the DR replica and you must have a unique cluster ID assigned to the current database.

The basic process for upgrading the VoltDB software using DR is to:

  1. Install the new VoltDB software on the secondary cluster

  2. Use passive DR to synchronize the database contents from the current cluster to the new cluster

  3. Pause the current database and promote the new cluster, switching the application clients to the new upgraded database

The following sections describe in detail the prerequisites for using this process, the steps to follow, and — in case there are any issues with the updated database — the process for falling back to the previous software version.

4.4.3.1. Prerequisites for Upgrading with Passive DR

The prerequisites for using DR to upgrade VoltDB are:

  • A second cluster with the same configuration (that is, the same number of servers and sites per host) as the current database cluster.

  • The current database cluster must have a unique cluster ID assigned in its deployment file.

The cluster ID is assigned in the <dr> section of the deployment file and must be set when the cluster starts. It cannot be added or altered while the database is running. So if you are considering using this process for upgrading your production systems, be sure to add a <dr> tag to the deployment and assign a unique cluster ID when starting the database, even if you do not plan on using DR for normal operations.

For example, you would add the following element to the deployment file when starting your primary database cluster to assign it the unique ID of 3.

<dr id="3">

Important

An important constraint to be aware of when using this process is that you must not make any schema changes during the upgrade process. This includes the period after the upgrade while you verify the application's proper operation on the new software version. If any changes are made to the schema, you may not be able to readily fall back to the previous version.

4.4.3.2. The Passive DR Upgrade Process

The procedure for upgrading the VoltDB software on a running database using DR is the following. In the examples, we assume the existing database is running on a cluster with the nodes oldsvr1 and oldsvr2 and the new cluster includes servers newsvr1 and newsvr2. We will assign the clusters unique IDs 3 and 4, respectively.

  1. Install the new VoltDB software on the secondary cluster.

    Follow the steps in the section "Installing VoltDB" in the Using VoltDB manual to install the latest VoltDB software.

  2. Start the second cluster as a replica of the current database cluster.

    Once the new software is installed, create a new database on the secondary server using the voltdb init --force and voltdb start commands and including the necessary DR configuration to create a replica of the current database. For example, the configuration file on the new cluster might look like this:

    <dr id="4" role="replica">
       <connection source="oldsvr1,oldsvr2"/>
    </dr>

    Once the second cluster starts, apply the schema from the current database to the second cluster. Once the schema match on the two databases, replication will begin.

  3. Wait for replication to stabilize.

    During replication, the original database will send a snapshot of the current content to the new replica, then send binary logs of all subsequent transactions. You want to wait until the snapshot is finished and the ongoing DR is processing normally before proceeding.

    • First monitor the DR statistics on the new cluster. The DR consumer state changes to "RECEIVE" once the snapshot is complete. You can check this in the Monitor tab of the VoltDB Management Center or from the command line by using sqlcmd to call the @Statistics system procedure, like so:

      $ sqlcmd --servers=newsvr1
      1> exec @Statistics drconsumer 0;
    • Once the new cluster reports the consumer state as "RECEIVE", you can monitor the rate of replication on the existing database cluster using the DR producer statistics. Again, you can view these statistics in the Monitor tab of the VoltDB Management Center or by calling @Statistics using sqlcmd:

      $ sqlcmd --servers=oldsvr1
      1> exec @Statistics drproducer 0;

      What you are looking for on the producer side is that the DR latency is low; ideally under a second. Because the DR latency helps determine how long you will wait for the cluster to quiesce when you pause it and, subsequently, how long the client applications will be stalled waiting for the new cluster to be promoted. You determine the latency by looking at the difference between the statistics for the last queued timestamp and the last ACKed timestamp. The difference between these values gives you the latency in microseconds. When the latency reaches a stable, low value you are ready to proceed.

  4. Pause the current database.

    The next step is to pause the current database. You do this using the voltadmin pause --wait command:

    $ voltadmin pause --host=oldsvr1 --wait

    The --wait flag tells voltadmin to wait until all DR and export queues are flushed to their downstream targets before returning control to the shell prompt. This guarantees that all transactions have reached the new replica cluster.

    If DR or export are blocked for any reason — such as a network outage or the target server unavailable — the voltadmin pause --wait command will continue to wait and periodically report on what queues are still busy. If the queues do not progress, you will want to fix the underlying problem before proceeding to ensure you do not lose any data.

  5. Promote the new database.

    Once the current database is fully paused, you can promote the new database, using the voltadmin promote command:

    $ voltadmin promote --host=newsvr1

    At this point, your database is up and running on the new VoltDB software version.

  6. Redirect client applications to the new database.

    To restore connectivity to your client applications, redirect them from the old cluster to the new cluster by creating connections to the new cluster servers newsvr1, newsvr2, and so on.

  7. Shutdown the original cluster.

    At this point you can shutdown the old database cluster.

  8. Verify proper operation of the database and client applications.

    The last step is to verify that your applications are operating properly against the new VoltDB software. Use the VoltDB Management Center to monitor database transactions and performance and verify transactions are completing at the expected rate and volume.

Your upgrade is now complete. If, at any point, you decide there is an issue with your application or your database, it is possible to fall back to the previous version of VoltDB as long as you have not made any changes to the underlying database schema. The next section explains how to fall back when necessary.

4.4.3.3. Falling Back to a Previous Version

In extreme cases, you may find there is an issue with your application and the latest version of VoltDB. Of course, you normally would discover this during testing prior to a production upgrade. However, if that is not the case and an incompatibility or other conflict is discovered after the upgrade is completed, it is possible to fall back to a previous version of VoltDB. The basic process for falling back is to the following:

  • If any problems arise before Step #6 (redirecting the clients) is completed, simply shutdown the new replica and resume the old database using the voltadmin resume command:

    $ voltadmin shutdown --host=newsvr1
    $ voltadmin resume --host=oldsvr1
  • If issues are found after Step #6, the fall back procedure is basically to repeat the upgrade procedure described in Section 4.4.3.2, “The Passive DR Upgrade Process” except reversing the roles of the clusters and replicating the data from the new cluster to the old cluster. That is:

    1. Update the configuration file on the new cluster to enable DR as a master, removing the <connection> element:

      <dr id="4" role="master"/>
    2. Shutdown the original database and edit the configuration file to enable DR as a replica of the new cluster:

      <dr id="3" role="replica">
         <connection source="newsvr1,newsvr2"/>
      </dr>
    3. Re-initialize and start the old cluster using the voltdb init --force and voltdb start commands.

    4. Follow steps 3 through 8 in Section 4.4.3.2, “The Passive DR Upgrade Process” reversing the roles of the new and old clusters.

4.4.4. Using Cross Data Center Replication (XDCR) to Upgrade VoltDB Without Downtime

It is also possible to upgrade the VoltDB software using cross data center replication (XDCR), by simply shutting down, upgrading, and then re-initalizing each cluster, one at a time. This process requires no downtime, assuming your client applications are already designed to switch between the two active clusters.

Use of XDCR for upgrading the VoltDB software is easiest if you are already using XDCR because it does not require any additional hardware or reconfiguration. The following instructions assume that is the case. Of course, you could also create a new cluster and establish XDCR replication between the old and new clusters just for the purpose of upgrading VoltDB. The steps for the upgrade outlined in the following sections are the same. But first you must establish the cross data center replication between the two clusters. See the chapter on Database Replication in the Using VoltDB manual for instructions on completing this initial step.

Once you have two clusters actively replicating data with XCDCR (let's call them clusters A and B), the steps for upgrading the VoltDB software on the clusters is as follows:

  1. Pause and shutdown cluster A (voltadmin pause --wait and shutdown).

  2. Clear the DR state on cluster B (voltadmin dr reset).

  3. Update the VoltDB software on cluster A.

  4. Start a new database instance on A, making sure to use the old deployment file so the XDCR connections are configured properly (voltdb init --force and voltdb start).

  5. Load the schema on Cluster A so replication starts.

  6. Once the two clusters are synchronized, repeat steps 1 through 4 for cluster B.

Note that since you are upgrading the software, you must create a new instance after the upgrade (step #3). When upgrading the software, you cannot recover the database using just the voltdb start command; you must use voltdb init --force first to create a new instance and then reload the existing data from the running cluster B.

Also, be sure all data has been copied to the upgraded cluster A after step #4 and before proceeding to upgrade the second cluster. You can do this by checking the @Statistics system procedure selector DRCONSUMER on cluster A. Once the DRCONSUMER statistics State column changes to "RECEIVE", you know the two clusters are properly synchronized and you can proceed to step #5.

4.4.4.1. Falling Back to a Previous Version

In extreme cases, you may decide after performing the upgrade that you do not want to use the latest version of VoltDB. If this happens, it is possible to fall back to the previous version of VoltDB.

To "downgrade" from a new version back to the previous version, follow the steps outlined in Section 4.4.4, “Using Cross Data Center Replication (XDCR) to Upgrade VoltDB Without Downtime” except rather than upgrading to the new version in Step #2, reinstall the older version of VoltDB. This process is valid as long as you have not modified the schema or deployment to use any new or changed features introduced in the new version.