NTP (Network Time Protocol) is a protocol and a set of system tools that help synchronize time across servers. The actual purpose of NTP is to keep an individual node's clock "accurate". This is done by having the node periodically synchronize its clock with a reference server. You can specify multiple servers to provide redundancy in case one or more time servers are unavailable.
The important point to note here is that VoltDB doesn't care whether the cluster view of time is "correct" from a global perspective, but it does care that they all have the same view. In other words, it is important that the nodes all synchronize to the same reference time and server.
To manage time effectively on a VoltDB cluster you must:
Start NTP on each node
Point each instance of NTP to the same set of reference servers
You start NTP by starting the NTP service, or daemon, on your system. On most systems, starting the NTP daemon happens automatically on startup. You do not need to perform this action manually. However, if you need to make adjustments to the NTP configuration, it is useful to know how to stop and start the service. For example, the following command starts the daemon:
$ service ntp start -x
You specify the time server(s) in the NTP configuration file (usually
/etc/ntp.conf). You can
specify multiple servers, one server per line. For example:
The configuration file is read when the NTP service starts. So, if you change the configuration file after NTP is running, stop and restart the service to have the new configuration options take affect.
In many cases, the preceding basic configuration is sufficient. However, there are issues that can arise time varies within the cluster.
If you are unsure whether a difference between the clocks in your cluster is causing performance issues for your database, the first step is to determine how much clock skew is present. When the VoltDB server starts it reports the maximum clock skew as part of its startup routine. For example:
INFO - HOST: Maximum clock/network skew is 12 milliseconds (according to leader)
If the skew is greater than 200 milliseconds, the cluster refuses to start. But even if the skew is around 100 milliseconds, the difference can delay certain operations and the nodes may drift farther apart in the future. The most common issues when using NTP to manage time are:
Time drifts between adjustments
Different time servers reporting different times
The NTP daemon checks the time servers periodically and adjusts the system clock to account for any drift between the local clock and the reference server (by default, somewhere between every 1 to 17 minutes). If the local clock drifts too much during that interval, it may never be able to fully correct itself or provide a consistent time value to VoltDB.
You can reduce the polling interval by setting the minpoll and maxpoll arguments as part of the server definition in the NTP configuration file. By setting minpoll and maxpoll to a low value (measured as exponential values of 2 seconds), you can ensure that the VoltDB server checks more frequently. For example, setting minpoll and maxpoll to 4 (that is, 16 seconds), you ensure the daemon polls the reference server approximately every minute.
It is also possible that the poll does not get a response. When this happens, the NTP daemon normally waits for the next interval before checking again. To increase the likelihood of receiving a new reference time — especially in environments with network fluctuations — you can use the burst and iburst arguments to increase the number of polls during each internal.
By combining the burst, iburst, minpoll, and maxpoll arguments, you can increase the frequency that the NTP daemon synchronizes and thereby reduce the potential drift of the local server's clock. However, you should not use these arguments with public servers, such as the ones included in the NTP configuration file by default. Excessive polling of public servers is considered impolite. Instead, you should only use these arguments with a private server (as described in Section 10.2.4, “Example NTP Configuration”). For example, the ntp.conf entry might look like the following:
server myntpsvr iburst burst minpoll 4 maxpoll 4
Even if your system synchronizes with an NTP server, there can be skew between the reference servers themselves. Remember, the goal of NTP is to synchronize your system with a reference time source, not necessarily to reduce the skew between multiple local systems. Even if the polling frequency is improved for each node in a VoltDB cluster, the skew between them may never reach an acceptable value if they are synchronizing against different reference servers.
This situation is made worse by the fact that the most common host names for reference servers (such as ntp.ubuntu.com) are not actual IP addresses, but rather front ends to a pool of servers. So even if the VoltDB nodes have the same NTP configuration file, they might not end up synchronizing against the same physical reference server.
You can determine what actual servers your system is using to synchronize by using the NTP query tool (ntpq) with the -p argument. The tool displays a list of the servers it has selected, with an asterisk (*) next to the server currently in use and plus signs (+) next to alternatives in case the primary server is unavailable. For example:
$ ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== +dns3.cit.cornel 220.127.116.11 2 u 14 1024 377 32.185 2.605 0.778 -louie.udel.edu 18.104.22.168 2 u 297 1024 377 22.060 3.643 0.920 gilbreth.ecn.pu .STEP. 16 u - 1024 0 0.000 0.000 0.000 *otc2.psu.edu 22.214.171.124 2 u 883 1024 377 29.776 1.963 0.901 +europium.canoni 126.96.36.199 2 u 1017 1024 377 90.207 2.741 0.874
Note that NTP does not necessarily choose the first server on the list and that the generic host names are resolved to different physical servers.
So, although it is normal to have multiple servers listed in the NTP configuration file for redundancy, it can introduce differences in the local system clocks. If the maximum skew for a VoltDB cluster is consistently outside of acceptable values, you should take the following steps:
Change from using generic host names to specific server IP addresses (such as otc2.psu.edu or 188.8.131.52 in the preceding example)
List only one NTP server to ensure all VoltDB nodes synchronize against the same reference point
Of course, using only one reference server for time introduces a single point of failure to your environment. If the reference server is not available, the database nodes receive no new reference values for time. The nodes continue to synchronize as best they can, based on the last valid reference time and historical information about skew. But over time, the clock skew within the cluster will start to drift.
You can provide both redundancy and maintain a single source for time synchronization, by creating your own NTP server.
NTP assumes a hierarchy (or strata) of servers, where each level of server synchronizes against servers one level up and provides synchronization to servers one level down. You can create your own reference server by inserting a server between your cluster nodes and the normal reference servers.
For example, assume you have a node myntpsvr that uses the default NTP configuration for setting its own clock. It can list multiple reference servers and use the generic host names, since the actual time does not matter, just that all cluster nodes agree on a single source.
Then the VoltDB cluster nodes list your private NTP server as their one and only reference node. By doing this, all the nodes synchronize against a single source, which has strong availability since it is within the same physical infrastructure as the database cluster.
Of course, there is always the possibility that access to your own NTP server could fail, in which case the database nodes need a fallback to ensure they continue to synchronize against the same source. You can achieve this by:
Adding all of the cluster nodes as peers of the current node in the NTP configuration file
Adding the current node (localhost) as its own server and setting it as a low level stratum (for example, stratum 10)
By listing the nodes of the cluster as peers, you ensure that when the reference server (myntpsvr in this example) becomes unavailable, the nodes will negotiate between themselves on an alternative source. At the same time, listing localhost (127.127.0.1) as a server tells the node that it can use itself as a reference server. In other words, the cluster nodes will agree among themselves to use one of their own as the reference server for synchronizing time. Finally, by using the fudge statement to set the stratum of localhost to 10, you ensure that the cluster will only pick one of its own members as a reference server for NTP if the primary server is unavailable.
Example 10.1, “Custom NTP Configuration File” shows what the resulting NTP configuration file might look like. This configuration
can be the same on all nodes of the cluster, since
peer entries referencing the current node are
Example 10.1. Custom NTP Configuration File
server myntpsvr burst iburst minpoll 4 maxpoll 4 peer voltsvr1 burst iburst minpoll 4 maxpoll 4 peer voltsvr2 burst iburst minpoll 4 maxpoll 4 peer voltsvr3 burst iburst minpoll 4 maxpoll 4 server 127.127.0.1 fudge 127.127.0.1 stratum 10
 The name of the NTP service varies from system to system. For Debian-based operating systems, such as Ubuntu, the service name is "ntp". For Red Hat-based distributions, such as CentOS, the service name is "ntpd".
 Use of the -x option is recommended. This option causes NTP to "slew" time — slowly increasing or decreasing the clock to adjust time — instead of making one-time jumps that could create sudden changes in clock skew for the entire cluster.
 The default values for minpoll and maxpoll are 6 and 10, respectively. The allowable value for both is any integer between 4 and 17 inclusive.