15.10. The RabbitMQ Export Connector

Documentation

VoltDB Home » Documentation » Using VoltDB

15.10. The RabbitMQ Export Connector

The RabbitMQ connector fetches serialized data from the export streams and writes it to a RabbitMQ message exchange. RabbitMQ is a popular message queueing service that supports multiple platforms, multiple languages, and multiple protocols, including AMQP.

Before using the RabbitMQ connector, we strongly recommend reading the RabbitMQ documentation and becoming familiar with the software, since you will need to set up a RabbitMQ exchange, queues, and routing key filters to make use of VoltDB's RabbitMQ export functionality. The instructions in this section assume a working knowledge of RabbitMQ and the RabbitMQ operational model.

You must also install the RabbitMQ Java client library before you can use the VoltDB connector. To install the RabbitMQ Java client library:

  1. Download the client library version 3.3.4 or later from the RabbitMQ website (http://www.rabbitmq.com/java-client.html).

  2. Copy the client JAR file into the lib/extension/ folder where VoltDB is installed for each node in the cluster.

When the RabbitMQ connector receives data from the VoltDB export streams, it establishes a connection to the RabbitMQ exchange as a producer. It then writes records to the service using the optional exchange name and routing key suffix. RabbitMQ uses the routing key to identify which queue the data is sent to. The exchange examines the routing key and based on the key value (and any filters defined for the exchange) sends each message to the appropriate queue.

Every message sent by VoltDB to RabbitMQ contains a routing key that includes the name of the export stream. You can further refine the routing by appending a suffix to the stream name, based on the contents of individual stream columns. By default, the value of the export stream's partitioning column is used as a suffix for the routing key. Alternately, you can specify a different column for each stream by declaring the routing.key.suffix property as a list of stream and column name pairs, separating the stream from the column name with a period and separating the pairs with commas. For example:

<export>
   <configuration target="queue" enabled="true" type="rabbitmq">
     <property name="broker.host">rabbitmq.mycompany.com</property>
     <property name="routing.key.suffix">
         voter_export.state,contestants_export.contestant_number
     </property>
  </configuration>
</export>

The important point to remember is that it is your responsibility to configure a RabbitMQ exchange that matches the name associated with the exchange.name property (or take the default exchange) and create queues and/or filters to match the routing keys generated by VoltDB. At a minimum, the exchange must be able to handle routing keys starting with the export stream names. This can be achieved by using a filter for each export stream. For example, using the flight example in Section 15.2, “Planning your Export Strategy”, you can create filters for EXPORT_FLIGHT.*, EXPORT_CUSTOMER.*, and RESERVATION_FINAL.*.

Table 15.5, “RabbitMQ Export Properties” lists the supported properties for the RabbitMQ connector.

Table 15.5. RabbitMQ Export Properties

PropertyAllowable ValuesDescription
broker.host*stringThe host name of a RabbitMQ exchange server.
broker.portintegerThe port number of the RabbitMQ server. The default port number is 5672.
amqp.uristringAn alternate method for specifying the location of the RabbitMQ exchange server. Use of amqp.uri allows you to specify additional RabbitMQ options as part of the connection URI. Either broker.host or amqp.uri must be specified.
virtual.hoststringSpecifies the namespace for the RabbitMQ exchange and queues.
usernamestringThe username for authenticating to the RabbitMQ host.
passwordstringThe password for authenticating to the RabbitMQ host.
exchange.namestringThe name of the RabbitMQ exchange to use. If you do not specify a value, the default exchange for the RabbitMQ server is used.
routing.key.suffix{stream}.{column}[,...]

Specifies which stream columns to use as a suffix for the RabbitMQ routing key. The routing key always starts with the stream name, in uppercase. A suffix is then appended to the stream name, separated by a period.

By default, the value of the stream's partitioning column is used as the suffix. Using this property you can specify a list of stream column names, where the stream name and column name are separated by a period and the list of stream references is separated by commas. This syntax allows you to specify a different routing key suffix for each stream.

queue.durabletrue, falseWhether the RabbitMQ queue is durable. That is, data in the queue will be retained and restarted if the RabbitMQ server restarts. If you specify the queue as durable, the messages themselves will also be marked as durable to enable their persistence across server failure. The default is true.
binaryencodinghex, base64Specifies whether VARBINARY data is encoded in hexadecimal or BASE64 format. The default is hexadecimal.
skipinternalstrue, falseSpecifies whether to include six columns of VoltDB metadata (such as transaction ID and timestamp) in the output. If you specify skipinternals as true, the output contains only the exported stream data. The default is false.
timezonestringThe time zone to use when formatting the timestamp. Specify the time zone as a Java timezone identifier. The default is GMT.

*Required