New in 3.12: Batch querying in TypeQL with ‘given’

TypeDB 3.12 introduces the given stage into the TypeQL language, providing a natural way to run the same query for multiple inputs. This is particularly useful in batch-loading tasks where the same pattern must be created multiple times in the database. It also separates input values from the query string, preventing TypeQL injection attacks.

given $pnr: string, $flight-number: string, $seat-number: string;
match
  $p isa booking, has pnr == $pnr;
  $f isa flight, has flight-number == $flight-number;
insert 
  $_ isa seat-allocation, 
links (flight: $f, booking: $p),
has seat-number == $seat-number;

A natural solution for TypeQL

Considering a TypeQL query is a pipeline of stages, a natural way to implement [multiple] query inputs is to simply pipe them into the first stage. The given stage serves as an adapter by declaring the name & types of the input variables. The query can then be type-checked and compiled independent of the actual input values. Each input row is validated against the declaration at runtime. Here’s a Rust example:

let query = "given $n: string, $e: string;\
  insert $p isa employee, has name == $n, has email == $e;";

let mut given_rows = GivenRows::new(vec!["n", "e"], 2);
given_rows.push_row(vec!["Alice".into(), "alice@example.com".into()])?;
given_rows.push_row(vec!["Bob".into(), "bob@example.com".into()])?;

transaction.query_with_rows(query, given_rows).await?;

The equivalent in SQL with a JDBC driver would be to use a PreparedStatement with executeBatch.

String sql = "INSERT INTO employees (name, email) VALUES (?, ?)"; PreparedStatement ps = conn.prepareStatement(sql))
ps.setString(1, "Alice");
ps.setString(2, "alice@example.com");
ps.addBatch(); 
ps.setString(1, "Bob"); 
ps.setString(2, "bob@example.com");
ps.addBatch();
ps.executeBatch(); 

Accepted inputs

Since they are fed directly into the pipeline, input rows must contain TypeDB instances or values (types are not accepted). Concepts retrieved from the results of a previous query can be used as input. Each driver also has methods to create TypeDB value concepts from values in the language. 

In rust, the value concepts can be constructed like in the earlier example. In other languages, the value constructors are exposed via TypeDB.Concept. For example in Java:

String query = "given $x: person, $v: integer; insert $x has age == $v;";

List<Map<String, Object>> givenRows = List.of(
    java.util.Map.ofEntries(
        Map.entry("x", personEugene), # From previous query
        Map.entry("v", TypeDB.Concept.newInteger(12))
    ),
    java.util.Map.ofEntries(
        Map.entry("x", personFred),  # Retrieved previous query
        Map.entry("v", TypeDB.Concept.newInteger(34))
    )
);

QueryAnswer inserted = transaction.query(query, givenRows).resolve();

Since the HTTP API is meant for use in languages scenarios where you don’t have/want a full-fledged GRPC driver, the HTTP API also accepts unparsed values and IIDs as inputs.

{
  "query": "given $p: person, $dob: datetime, $name: string; insert $p has date-of-birth == $dob, has name == $name;",
  "given": [
    {
      "p": "0x1e00000000000000000000",
      "name": "John",
      "dob": {
        "kind": "value",
        "valueType": "datetime",
        value: "1999-09-20T16:40:05.00"
      }
    }
  ],
  //...
}

Minimal overheads for bulk-inserts

Before the given stage was introduced, concatenating multiple patterns into a single insert query was a way to amortize the network overhead. However, this approach meant the size of the query grew linearly with the amount of data to insert, and the gains disappeared as the time it took to parse & compile the large query approached the network round-trip-time. 

The given stage approach gives us a constant-sized query regardless of the number of input rows, shifting the bottleneck to the execution of the query operators – where it should be. 

This decoupling also allows TypeDB to cache the compiled queries, effectively eliminating the compilation overhead and allowing fresh batches of data to be executed with low latency. Overall, we’ve seen a 3x-5x increase in the throughput of batch-inserts.

Safety against TypeQL injection

The best-practice to protect against query injection is to separate the query input from the query-string, as most SQL implementations achieve through query-parameterisation. The given stage achieves the same separation since values are now passed in as input rows to the pipeline and don’t influence the query-string.

tx.query(
  query="given $e: string; match $p isa employee, has email == $e;",
  given_rows=[{"e": email_to_lookup}],
)

Since input variables are introduced at the beginning of the pipeline, they must be carried through all the stages till the point where they are needed. This is the main difference between using the given stage and having parametrised queries as other databases do. We don’t expect this difference to be of much practical relevance.

Chaining queries on the application side

Though a single TypeQL pipeline is often expressive enough to compose complex queries, there still may be cases where multiple queries are required, such as branching queries or ones where an external service has to be called. By introducing the ability to pass concepts as inputs to queries, the given stage greatly reduces the friction in chaining queries together. 

course_rows = tx.query("match $o isa course, links (teacher: $t), has hours-per-week $h;").resolve();
courses = [  Course(row["c"], row["t"], row["h"]) for row in course_rows.into_concept_rows() ]

timetable = plan_timetable(courses); # Fancy solver
given_rows = [ 
  {"i" : TypeDB.Concept.new_integer(slot), "c": course} 
    for (slot, course) in timetable
]

query = """
  given $i: integer, $c: course; 
  insert $t isa timeslot, links (course: $c), has slot-index == $i;
"""
tx.query(query, given_rows = given_rows).resolve(); # Insert all in one go.

In previous versions of TypeDB, The course would have to be matched anew using an iid constraint (or an identifying attribute).

Conclusion

The given stage brings the TypeDB database and client applications closer together by making it easier for results of one query to be used in later queries, and providing an ergonomic API for getting values from the driver language into TypeDB. It also allows inserting large batches of data in single queries –  cutting network, query-compilation times to enable efficient data-loading tools such as the new TypeDB loader. We believe it’s a very natural extension of the TypeQL language which makes writing applications using the driver more intuitive.

You can check our TypeQL & TypeDB Driver references for documentation. As always, we’re eager to hear your thoughts on it in our Discord server.