Chapter 5. Monitoring VoltDB Databases

http://voltserver:8080/

The Volt Management Center provides a graphical display of key aspects of database performance, including throughput, memory usage, query latency, and partition usage. To use the Management Center, connect to one of the cluster nodes using a web browser, specifying the HTTP port (8080 by default) as shown in the example URL above. The Management Center shows graphs for cluster throughput and latency as well as CPU and memory usage for the current server. You can also use the Management Center to examine the database schema and to issue ad hoc SQL queries.

VoltDB provides callable system procedures that return detailed information about the usage and performance of the database. In particular, the @Statistics system procedure provides a wide variety of information depending on the selector keyword you give it. Some selectors that are particularly useful for monitoring include the following:

When using the @Statistics system procedure with the PROCEDUREPROFILE selector for monitoring, it is a good idea to set the second parameter of the call to "1" so each call returns information since the last call. In other words, statistics for the interval since the last call. Otherwise, if the second parameter is "0", the procedure returns information since the database started and the aggregate results for minimum, maximum, and average execution time will have little meaning.

When calling @Statistics with the MEMORY or TABLE selectors, you can set the second parameter to "0" since the results are always a snapshot of the memory usage and table volume at the time of the call. For example, the following Python script uses @Statistics with the MEMORY and PROCEDUREPROFILE selectors to check for memory usage and latency exceeding certain limits. Note that the call to @Statistics uses a second parameter of 1 for the PROCEDUREPROFILE call and a parameter value of 0 for the MEMORY call.

import sys
from voltdbclient import *

nano = 1000000000.0
memorytrigger = 4 * (1024*1024)     # 4gbytes
avglatencytrigger = .01 * nano      # 10 milliseconds
maxlatencytrigger = 2 * nano        # 2 seconds

server = "localhost"
if (len(sys.argv) > 1): server = sys.argv[1]

client = FastSerializer(server, 21212)
stats = VoltProcedure( client, "@Statistics", 
   [ FastSerializer.VOLTTYPE_STRING, 
     FastSerializer.VOLTTYPE_INTEGER ] )

# Check memory
response = stats.call([ "memory", 0 ])
for t in response.tables:
   for row in t.tuples:
      print 'RSS for node ' + row[2] + "=" + str(row[3])
      if (row[3] > memorytrigger):
         print "WARNING: memory usage exceeds limit."

# Check latency
response = stats.call([ "procedureprofile", 1 ])
avglatency = 0
maxlatency = 0
for t in response.tables:
   for row in t.tuples:
      if (avglatency < row[4]): avglatency = row[4]
      if (maxlatency < row[6]): maxlatency = row[6]
print 'Average latency= ' + str(avglatency) 
print 'Maximum latency= ' + str(maxlatency)
if (avglatency > avglatencytrigger):
   print "WARNING: Average latency exceeds limit."
if (maxlatency > maxlatencytrigger):
   print "WARNING: Maximum latency exceeds limit."

client.close()

The @Statistics system procedure is the the source for many of the monitoring options discussed in this chapter. Two other system procedures, @SystemCatalog and @SystemInformation, provide general information about the database schema and cluster configuration respectively and can be used in monitoring as well.

The system procedures are useful for monitoring because they let you customize your reporting to whatever level of detail you wish. The other advantage is that you can automate the monitoring through scripts or client applications that call the system procedures. The downside, of course, is that you must design and create such scripts yourself. As an alternative for custom monitoring, you can consider integrating VoltDB with existing third party monitoring applications, as described in Section 5.3, “Integrating VoltDB with Prometheus”. You can also set the database to automatically pause if certain system resources run low, as described in the next section.

In addition to monitoring database activity on a "as needed" basis, you can enable VoltDB to proactively send Simple Network Management Protocol (SNMP) alerts whenever important events occur within the cluster. SNMP is a standard for how SNMP agents send messages (known as "traps") to management servers or "management stations".

SNMP is a lightweight protocol. SNMP traps are sent as UDP broadcast messages in a standard format that is readable by SNMP management stations. Since they are broadcast messages, the sending agent does not wait for a confirmation or response. And it does not matter, to the sender, whether there is a management server listening to receive the message or not. You can use any SNMP-compliant management server to receive and take action based on the traps.

When you enable SNMP in the deployment file, VoltDB operates as an SNMP agent sending traps whenever management changes occur in the cluster. You enable SNMP with the <snmp> element in the deployment file. You configure how and where VoltDB sends SNMP traps using one or more of the attributes listed in Table 5.1, “SNMP Configuration Attributes”.

SNMP is enabled by default when you include the <snmp> element in the deployment file. Alternately, you can explicitly enable and disable SNMP using the enabled={true|false} attribute to the element. For example, the following deployment file entry enables SNMP alerts, sending traps to mgtsvr.mycompany.com using SNMP V3 with the username "voltdb":

<snmp enabled="true"
      target="mgtsvr.mycompany.com"
      username="voltdb"
/>

Once SNMP is enabled, VoltDB sends alerts for the events listed in Table 5.2, “SNMP Events”.

For the latest details about each event trap, see the VoltDB SNMP Management Information Base (MIB), which is installed with the VoltDB server software in the file /tools/snmp/VOLTDB-MIB in the installation directory.