Introducing TypeDB Loader: bulk-load CSV data into TypeDB

Until now, bulk-loading data into TypeDB always required custom logic, generally writing your own tooling to generate one or more huge files of insert statements. We’ve replaced that with TypeDB Loader, our new command-line tool to load data 3-5x faster without you having to write any code.

Using a CSV file as input to  a bulk query (powered by the new given stage), it loads the data into TypeDB with batching, parallelizing, and checkpointing built-in. Compared to the hand-rolled approach, TypeDB Loader is:

• Easier. One query template replaces thousands of generated insert statements.
• Faster. Rows are batched into single server calls, with optional concurrency – 3–5x faster than the typical loading script.
• More reliable. Failed rows are recorded and skipped rather than aborting the run, and an interrupted load can be resumed from a checkpoint.

One template, many rows

Instead of writing a query per row, you write the query once and declare which values come from the data using a given stage:

given $name: string, $age: integer?, $active: boolean,
      $balance: decimal, $birthday: date?, $joined-at: datetime-tz;
insert
  $p isa person,
    has name == $name,
    has active == $active,
    has balance == $balance,
    has joined-at == $joined-at;
  try { $p has age == $age; };
  try { $p has birthday == $birthday; };

The given stage declares one variable per CSV column, with its value type. For each row, the loader parses the cells into typed values, binds them to the variables, and takes them as inputs to the pipeline. Values are passed to the server as values, not spliced into query text, preventing escaping issues.

Two details worth noting:

• Typed inputs. All of TypeDB’s value types are supported: boolean, integer, double, decimal, string, date, datetime, datetime-tz, and duration. Each cell is validated against its declared type before being sent to the server.
• Optional inputs. $age: integer? marks the input as optional. A null cell (empty by default; configurable with --null-values) produces an empty binding, and the matching try { ... }; block becomes a no-op — the row loads without that attribute.

The template is a full TypeQL pipeline, so it isn’t limited to inserting entities: it can match existing data to insert relations like loading a friendships CSV that references people already in the database.

A complete example

The full example, complete with all the files here, is in our examples repository: a small social network of people and the friendships between them, loaded from two CSV files.

The schema (schema.tql):

define

attribute name, value string;
attribute age, value integer;
attribute active, value boolean;
attribute balance, value decimal;
attribute birthday, value date;
attribute joined-at, value datetime-tz;

relation friendship,
  relates friend @card(2);

entity person,
  owns name @key,
  owns age,
  owns active,
  owns balance,
  owns birthday,
  owns joined-at,
  plays friendship:friend;

The query template above (insert-people.tql) loads the people (people.csv):

name,age,active
user_1,68,true
user_2,,false
user_3,73,true

Run the loader from your TypeDB distribution directory:

./typedb loader \
  --query=insert-people.tql \
  --schema-file=schema.tql \
  --database=social-network \
  --create-db \
  --data=data.csv \
  --header \
  --username=admin \
  --address=localhost:1729

This one command creates the database (--create-db), applies the schema (--schema-file), matches CSV columns to given variables by header name, and loads the rows. The loader prompts for your password and connects over TLS by default; add --tls-disabled for a local development server. While it runs, it reports rows processed, batches committed, rows rejected, and throughput.

The second load shows the match-then-insert case. Its template (insert-friendships.tql) looks up the two people named in each row of friendships.csv and inserts a friendship relation between them:

given $name-a: string, $name-b: string;
match
  $a isa person, has name == $name-a;
  $b isa person, has name == $name-b;
insert
  friendship (friend: $a, friend: $b);

The database and schema already exist, so the second command is shorter:

./typedb loader \
  --query=insert-friendships.tql \
  --database=social-network \
  --data=friendships.csv \
  --header \
  --batch-rows=2000 \
  --parallel-batches=4 \
  --username=admin \
  --address=localhost:1729

In this case, we specify batch-rows and parallel-batches explicitly. This configures the parallelism of the operations – in this case allowing up to four batches of up to 2000 rows to be inserted at the same time.

Batching and parallelism

Two flags control performance:

• --batch-rows (default 1000): how many rows are submitted and committed together in one write transaction. Batching amortizes commit and query parsing overhead and accounts for most of the speedup over plain inserts. Larger batches mean fewer commits, but a failed commit rejects the whole batch.
• --parallel-batches (default 1): how many batches may be in flight at once. Raising this improves throughput when network latency dominates such as loading into a remote deployment. Batches may then commit in any order, so you must make sure that your CSV rows are independent of load order.

Errors and rejects

When a row fails to parse or a batch fails to commit, the loader records the offending rows in rejects.csv (with the original header, so the file can be fixed and re-loaded directly) and the reason for each in rejects.log, then continues. To fail fast instead, use --stop-on-error to abort on the first failure, or --max-rejects=<n> to abort after too many.

Checkpoint and resume

TypeDB Loader records all progress (rejects.csv, rejects.log, and checkpoint.json) to an output directory – defaulting to loader_<data-stem>_progress next to the data CSV, or specified by --output-dir. After each batch, the loader updates a checkpoint.json recording the run’s parameters, content hashes of the data file, query file, and live schema, and which batches have completed. If a run is interrupted, resume it by pointing it at the output directory:

./typedb loader --resume=loader_people_progress

The resumed run re-uses the checkpointed parameters (the password is never stored, so it is prompted again) and continues from the first uncommitted batch, allowing you to pick up an interrupted run exactly where it left off without any loss of data.

Getting started

TypeDB Loader is included in the typedb-all distribution as of 3.12.0 and available as a standalone typedb-loader download. Grab the complete loader example from our examples repository, run ./typedb loader --help for the full list of options, and join us on Discord to tell us how it works on your data.