As mentioned before, throughput and latency are not abstract numbers. They are a consequence of the schema, stored procedures, application, and server configuration in use. As with any VoltDB solution, to affectively benchmark performance you must start with a well-designed application. Specifically:

See Chapter 2, Proof of Concept for more information on writing effective VoltDB applications.

It is common practice to start benchmarking with the proof of concept, since the POC includes an initial schema and the key transactions required by the solution. If you decide to proceed with development, it is also a good idea to benchmark the application periodically throughout the process to make sure you understand how the additional functions impact your overall performance targets.

However, before benchmarking the POC, it is important to decide how you are going to measure performance. The following sections provide useful information for determining what and how to measure performance in a VoltDB application.

Latency and throughput are related. To measure throughput, your test application can make stored procedure calls as fast as it can and then measure how many are completed in a second (TPS). However, if the test application invokes more transactions that the server can complete in that time frame, the additional invocations must wait in the queue before being processed. The time these transactions wait in the queue will result in the latency measurement suddenly spiking.

There is an optimal throughput for any given application and system configuration. As the rate of invocations from the client application increases, there is a direct increase in TPS while latency stays relatively flat. However, as the invocation rate approaches and surpasses the optimal throughput (the limit of what the current configuration can process), latency increases dramatically and TPS may even drop, as shown in Figure 5.1, “Determining Optimal Throughput and Latency”.

Figure 5.1. Determining Optimal Throughput and Latency

If your test application "fire hoses" the database server — that is, it sends invocations as fast as it can — all you can measure is the misleading throughput and latency on the right side of the preceding chart. To determine the optimal rate, you need to be able control the rate at which your benchmark application submits transaction requests. This process is called rating limiting.

At its simplest, rate limiting is constraining the number of invocations issued by the application. For example, the following program loop, constrains the application to invoking the SignIn stored procedure a maximum of 10 times per millisecond, or 10,000 times per second.

boolean done = false;
long maxtxns = 10;
while (!done) {
  long txns = 0;
  long millisecs = System.currentTimeMillis();
  while (millisecs == System.currentTimeMillis()) {
      if ( txns++ < maxtxns } {
         myClient.callProcedure(new SignInCallback(),
                                "SignIn",
                                id, millisecs);
      }
  }
}

You could use a command line argument to parameterize the rate limit value maxtxns and then use multiple runs of the application to create a graph similar to Figure 5.1, “Determining Optimal Throughput and Latency”. An even better approach is to use a limit on latency to automatically control the invocation rate and let the application close in on the optimal throughput value.

How rate limiting based on latency is done is to have a variable for the target latency as well as a variable for maximum allowable throughput (such as maxtxns in the preceding example). The application measures both the average throughput and latency for a set period (every 1 to 5 seconds, for example). If the average latency exceeds the goal, reduce the maximum transactions per second, then repeat. After the same period, if the latency still exceeds the goal, reduce the maximum transaction rate again. If the latency does meet the goal, incrementally increase the maximum transaction rate.

By using this mix of rate limiting and automated adjustment based on a latency goal, the test application will eventually settle on an optimal throughput rate for the current configuration. This is the method used by the sample applications (such as voter and Voltkv) that are provided with the VoltDB software.