The custom formatter must contain at least two Java classes: one that implements the org.voltdb.importer.formatter.Formatter interface and one that extends the org.voltdb.importer.formatter.AbstractFormatterFactory interface.

For the sake of example, let's assume the custom formatter classes are called MyFormatter and MyFormatterFactory. When the associated import connector is initialized, the VoltDB importer infrastructure calls the classes' methods in the following order:

In many cases, the easiest way to create custom class is to modify an existing example. And VoltDB provides an example formatter that you can use as a base for your customizations in the VoltDB github at the following URL:

The next sections describe how to modify this example — or how to create a custom formatter from scratch, if you wish.

package myformatter;

import org.voltdb.importer.formatter.AbstractFormatterFactory;

public class MyFormatterFactory extends AbstractFormatterFactory {
    /**
     * Creates and returns the formatter object.
     */
    @Override
    public MyFormatter create() {
        MyFormatter formatter = new MyFormatter(m_formatName, m_formatProps);
        return formatter;
    }
}

package myformatter;

import java.util.Properties;
import org.voltdb.importer.formatter.FormatException;
import org.voltdb.importer.formatter.Formatter;

public class MyFormatter implements Formatter {

    String column_width = "";

    MyFormatter (String formatName, Properties prop) {
        column_width = prop.getProperty("column_width");
    }

@Override
public Object[] transform(ByteBuffer payload) throws FormatException {

   String buffer = new String(payload.array());
   ArrayList<Object> list = new ArrayList<Object>();

   int position = 0;
   while (position < buffer.length()) {
      int endpoint = Math.min(position+column_width, buffer.length());
      list.add(buffer.substring(position,endpoint));
      position += column_width;
   }
   return list.toArray();
}

Once the custom formatter source code is complete, you are ready to compile and package the formatter as an OSGi bundle.

When compiling the source code, be sure to include the VoltDB JAR files in the Java classpath. For example, if VoltDB is installed in the folder /opt/voltdb, you will need to include /opt/voltdb/voltdb/* and /opt/voltdb/lib/* in the classpath.

You will also need to include a number of OSGi-specific attributes in the final JAR file manifest. For example, you must include the Bundle-Activator attribute pointing to the FormatterFactory class. To ensure all the necessary properties are set, it is easiest to use the ant utility and an ant build file. The following is an example build.xml file, with the items that you must modify highlighted in bold text:

<project default="build">
   <path id='project.classpath'>
      <!-- Replace this with the path to the VoltDB jars -->
      <fileset dir='/opt/voltdb'>
        <include name='voltdb/*.jar' />
        <include name='lib/*.jar' />
      </fileset>
  </path>

  <target name="build" depends="clean, dist, formatter"/>

  <target name="clean">
    <delete dir="obj"/>
    <delete file="myformatter.jar"/>
  </target>

  <target name="dist">
    <mkdir dir="obj"/>
    <javac srcdir="src" destdir="obj">
      <classpath refid="project.classpath"/>
    </javac>
  </target>

  <target name="formatter">
    <jar destfile="myformatter.jar" basedir="obj">
      <include name="myformatter/MyFormatter.class"/>
      <include name="myformatter/MyFormatterFactory.class"/>
      <manifest>
        <attribute name="Bundle-Activator" 
                   value="myformatter.MyFormatterFactory" />
        <attribute name="Bundle-ManifestVersion" value="2" />
        <attribute name="Bundle-Name" value="My Formatter OSGi Bundle" />
        <attribute name="Bundle-SymbolicName" value="MyFormatter" />
        <attribute name="Bundle-Version" value="1.0.0" />
        <attribute name="DynamicImport-Package" value="*" />
      </manifest>
    </jar>
  </target>
</project>

Once you have built and packaged the custom formatter, you are ready to install and use it in your VoltDB infrastructure.

$ cp obj/myformatter.jar /opt/voltdb/bundles/

deployment:
  import:
    configuration:
    - nickname: mycustomimport
      type: kafka
      format: myformatter.jar/MyFormatter
     [ . . . ]

deployment:
  import:
    configuration:
    - nickname: mycustomimport
      type: kafka
      format: myformatter.jar/MyFormatter
      property:
      - name: brokers
        value: kafka.myorg.org:9092
      - name: topics
        value: customer
      - name: procedure
        value: CUSTOMER.insert
      format-property:
      - name: column_width
        value: 15

You can also use custom formatters with the standalone kafkaloader utility. To use a custom formatter with kafkaloader you must:

The environment variables define the paths to the formatter JAR file and the Apache ZooKeeper libraries, respectively. (Note that ZooKeeper does not need to be running, but you must have a copy of the standard ZooKeeper libraries installed and accessible via the ZK_LIB environment variable.)

The formatter properties file must contain, at a minimum, a "formatter" property that is assigned to the formatter class of the custom formatter. It can contain other properties required by the formatter. The following is the properties file for kafkaloader that matches the example given in the previous section to configure the custom formatter using the built-in importer infrastructure:

formatter=MyFormatter
column_width=15

If both your formatter and the ZooKeeper libraries are in a folder myformatter under your home directory, along with the preceding properties file, you could start the kafkaloader utility with the following commands to use the custom formatter:

$ export FORMATTER_LIB="$HOME/myformatter/"
$ export ZKLIB="$HOME/myformatter/"
$ kafkaloader --formatter=$HOME/myformatter/formatter.config \
  --topic=customer --zookeeper=kafkahost:2181