csvloader — Imports the contents of a CSV file and inserts it into a VoltDB table.
csvloader table-name [arguments]
csvloader -p procedure-name [arguments]
The csvloader command reads comma-separated values and inserts each valid line of data into the specified table in a VoltDB database. The most common way to use csvloader is to specify the database table to be loaded and a CSV file containing the data, like so:
$ csvloader employees -f acme_employees.csv
Alternately, you can use standard input as the source of the data:
$ csvloader employees < acme_employees.csv
In addition to inserting all valid content into the specified database table, csvloader creates three output files:
Error log — The error log provides details concerning any errors that occur while processing the input file. This includes errors in the format of the input as well as errors that occur attempting the insert into VoltDB. For example, if two rows contain the same value for a column that is declared as unique, the error log indicates that the second insert fails due to a constraint violation.
Failed input — A separate file contains the contents of each line that failed to load. This file is useful because it allows you to correct any formatting issues and retry just the failed content, rather than having to restart and reload the entire table.
Summary report — Once all input lines are processed, csvloader generates a summary report listing how many lines were read, how many were successfully loaded and how long the operation took.
All three files are created, by default, in the current working directory using "csvloader" and the table name as prefixes. For example, using csvloader to insert contestants into the sample voter database creates the following files:
It is possible to use csvloader to load text files other than CSV files, using the
--escape flags. Note that csvloader uses Python to process the
command line arguments. So to enter certain non-alphanumeric characters, you must use the appropriate escaping mechanism for
Python command lines. For example, to use a tab-delimited file as input, you need to use the
flag, escaping the tab character like so:
$ csvloader --separator=$'\t' \ -f employees.tab employees
It is also important to note that, unlike VoltDB native clients, when interpreting string values for TIMESTAMP columns, csvloader evaluates the value in the local timezone. That is, the timezone set by the local system. To have string values interpreted as Greenwich Mean Time, set the system variable TZ to "GMT" prior to invoking the csvloader. For example:
$ export TZ=GMT; csvloader employees -f employees.csv
Specifies the number of rows to submit in a batch. If you do not specify an insert procedure, rows of input are
sent in batches to maximize overall throughput. You can specify how many rows are sent in each batch using the
--batch flag. The default batch size is 200. If you use the
--procedure flag, no
batching occurs and each row is sent separately.
Specifies what to do with missing values in the input. By default, if a line contains a missing value, it is
interpreted as a null value in the appropriate datatype. If you do not want missing values to be interpreted as nulls,
you can use the --blank argument to specify other behaviors. Specifying
--blank error results in an
error if a line contains any missing values and the line is not inserted. Specifying
returns the corresponding "empty" value in the appropriate datatype. An empty value is interpreted as the
Zero for all numeric columns
Zero, or the Unix epoch value, for timestamp columns
An empty or zero-length string for VARCHAR and VARBINARY columns
Specifies the character set of the input file. The default character set is UTF-8.
Specifies the maximum size of quoted column input, in bytes. Mismatched quotation marks in the input can cause csvloader to read all subsequent input — including line breaks — as part of the column. To avoid excessive memory use in this situation, the flag sets a limit on the maximum number of bytes that will be accepted as input for a column that is enclosed in quotation marks and spans multiple lines. The default is 16777216 (that is, 16MB).
Specifies the escape character that must precede a separator or quotation character that is supposed to be interpreted as a literal character in the CSV input. The default escape character is the backslash (\).
Specifies the location of a CSV file to read as input. If you do not specify an input file, csvloader reads input from standard input.
Specifies that the first line of the CSV file is a header row, containing the names of the columns. The column names must match columns in the VoltDB table. However, by using --header, the columns can appear in a different order in the CSV file from the order in the database schema. Note that you must specify all of the table column names in the header. The arguments --header and --procedure are mutually exclusive.
Specifies the maximum number of rows to be read from the input stream. This argument (along with --skip) lets you load a subset of a larger CSV file.
Specifies the target number of errors before csvloader stops processing input. Once csvloader encounters the specified number of errors while trying to insert rows, it will stop reading input and end the process. Note that, since csvloader performs inserts asynchronously, it often attempts more inserts before the target number of exceptions are returned from the database. So it is possible more errors could be returned after the target is met. This argument lets you conditionally stop a large loading process if more than an acceptable number of errors occur.
Disables the interpretation of quotation characters in the CSV input. All input other than the separator character and line break will be treated as literal input characters.
Specifies that the CSV input must not contain any whitespace between data values and separators. By default, csvloader ignores extra space between values, quotation marks, and the value separators. If you use this argument, any input lines containing whitespace will generate an error and not be inserted into the database.
Specifies the password to use when connecting to the database. You must specify a username and password if security is enabled for the database. If you specify a username with the --user argument but not the --password argument, VoltDB prompts for the password. This is useful when writing shell scripts because it avoids having to hardcode passwords as plain text in the script.
Specifies the network port to use when connecting to the database. If you do not specify a port, csvloader uses the default client port 21212.
Specifies a stored procedure to use for loading each record from the data file. The named procedure must exist in the database schema and must accept the fields of the data record as input parameters. By default, csvloader uses a custom procedure to batch multiple rows into a single insert operation. If you explicitly name a procedure, batching does not occur.
Specifies the quotation character that is used to enclose values. By default, the quotation character is the double quotation mark (").
Specifies the directory where csvloader writes the three output files. By default, csvloader writes output files to the current working directory. This argument lets you redirect output to an alternative location.
Specifies the network address of one or more nodes of a database cluster. By default, csvloader attempts to insert the CSV data into a database on the local system (localhost). To load data into a remote database, use the --servers argument to specify the database nodes the loader should connect to.
Specifies the character used to separate individual values in the input. By default, the separator character is the comma (,).
Specifies the number of lines from the input stream to skip before inserting rows into the database. This argument (along with --limitrows) lets you load a subset of a larger CSV file.
Specifies the use of SSL encryption when communicating with the server. Only necessary if the cluster is configured to use SSL encryption for the external ports. See Section D, “Using CLI Commands with SSL” for more information.
Specifies that if connections to all of the VoltDB servers are broken, the loader will stop. Normally, if the connection to the database is lost, csvloader periodically attempts to reconnect until the servers come back online and it can complete the loading process. However, you can use this argument to have the loader process stop if the VoltDB cluster becomes unavailable.
Specifies that all values in the CSV input must be enclosed in quotation marks. If you use this argument, any input lines containing unquoted values will generate an error and not be inserted into the database.
Specifies that existing records with a matching primary key are updated, rather than being rejected. By default, csvloader attempts to create new records. The --update flag lets you load updates to existing records — and create new records where the primary key does not already exist. To use --update, the table must have a primary key.
Specifies the username to use when connecting to the database. You must specify a username and password if security is enabled for the database.
The following example loads the data from a CSV file,
languages.csv, into the helloworld table
from the Hello World example database and redirects the output files to the ./logs subfolder.
$ csvloader helloworld -f languages.csv -r ./logs
The following example performs the same function, providing the input interactively.
$ csvloader helloworld -r ./logs "Hello", "World", "English" "Bonjour", "Monde", "French" "Hola", "Mundo", "Spanish" "Hej", "Verden", "Danish" "Ciao", "Mondo", "Italian" CTRL-D