Replacing implicit optionals with a None type in TypeDB

Examine TypeDB's new explicit optionality handling semantics, and compare them to alternatives used by other programming languages and databases.

Joshua Send


TypeDB has implicitly handled optional variables since version 3.4. Auto-unwrapping every optional is convenient most of the time, but can hide bugs: making a variable optional silently changes your query’s answers. In addition, there is no way to condition on emptiness, making complex updates difficult to orchestrate. So, we changed our type system to include a native None type, forcing explicit handling – like Rust or Kotlin, but in a database.

In this article we’ll look at null-ability and optionality, examine implicit versus explicit handling patterns, and define TypeDB’s new optionality semantics.

Tldr; TypeDB 3.15 ships with a new set of constructs: an isset constraint for filtering unset variables, and if conditional operators for write clauses. These replace the automatic, implicit unwrap behaviour for enhanced control, safety, and maintainability. Now, making a variable optional forces you to handle it downstream rather than implicitly eliminating answers. And, we’ve formalized our optionals as a built-in None type that participates in TypeQL’s union-of-types model just like any other type. For example, a variable that could be polymorphically person, company, or an empty optional is typed as $x: person | company | None.

From null to types

Every language eventually has to say “this isn’t here.” A company may not have an office address, or an order may not have had its tax computed yet. The simplest approach is a sentinel: pick a value outside the data domain, like an empty string for a missing address, and treat it as “absent.” Of course, that breaks as soon as the sentinel turns out to be a real value. A tax of 0 is a real value in most systems.

As a result, most languages added a special null: one extra value that fits into every domain and can’t be confused with real data. The catch is that nothing tracks which variables might hold it, so you have to track that yourself, and every time you forget, it becomes a bug. Tony Hoare called it his “billion dollar mistake”.

SQL adds another layer: because NULL can be compared, you can get values that are neither true nor false – but UNKNOWN. Every predicate has to account for three-valued logic.

Functional languages took a different route. Instead of adding a special value to every domain, they introduced a type for absence:

-- Haskell

data Maybe a = Nothing | Just a
// Rust

enum Option<T> { None, Some(T) }

Now the compiler knows which variables might be empty, and it makes you handle that case before your code runs. The idea has worked so well it’s spread to mainstream languages, from std::optional in C++ to Optional in Java.

Optionality in databases

Databases are particularly interesting in that optionality can arise in two places: the storage of values, and the handling of values.

In SQL, everything is done in terms of tables and columns, and columns can contain nulls. The query language operates over tables exclusively. A computed NULL in SQL is interchangeable with a stored null.

For example, here we get one output table with both stored and computed NULLs:

INSERT INTO companies VALUES
(1, 'Acme',   '555-0100'),
(2, 'Globex', NULL);              -- stored NULL: no phone on record
INSERT INTO offices VALUES
(1, '1 Acme Way');                -- Globex has no office row
SELECT c.name, c.phone, o.address
FROM companies c
LEFT JOIN offices o ON o.company_id = c.id;
name  |  phone   |  address
-------+----------+------------
Acme   | 555-0100 | 1 Acme Way
Globex | NULL     | NULL

Other databases, such as MongoDB, separate stored data from variables, allowing you to have stored nulls, and variable bindings that are null independently.

In TypeDB, we operate a bit differently: you can never store the absence of entity, relation, or attribute instances, but you can have empty variables.  Absence is simply not storing anything:

insert
  $c isa company, has office_address "1 TypeDB Road";
  $d isa company;    # no office address is recorded

One way to think about this is by conceptualizing how a graph works: you can’t have links to nodes that don’t exist, so you just don’t create the link. As a natural extension of this idea, when you pattern-match in TypeDB, you’re searching over the patterns that do exist in the database. In other words, you end up with all-or-nothing matches.

As a result, in TypeDB 2, if you wanted to match companies with offices and companies without offices, you’d have to issue two queries:

## 1. those with offices
match $c isa company, has office_address $addr;

## 2. those without offices
match $d isa company; not { $d has office_address $address; };

This is clearly suboptimal: you have to issue multiple queries and you’re looking up overlapping data twice. And worse, this doesn’t compose well once you have many optional components you want to look up.

In TypeDB 3.4, we shipped optional matching:

match
  $c isa company;
  try { $c has office_address $addr; };

This returns a stream of rows containing a company $c, always present, paired with an address $addr which may or may not be present. We use None instead of null to describe an unset variable.

So – the storage layer doesn’t store nulls. We could still conceptualize variables to be bound to values with additional None value present in every domain… but as you can probably guess by the name on the tin – we went the other way and leaned into types.

In TypeDB, a query returning an empty variable is using a type-system level None, instead of an untyped value null. No three-valued logic or stored nulls, and type checked end to end for safety.

The closest existing comparisons are graph systems: property graphs (like Neo4j), and RDF stores (queried with SPARQL). Neo4j handles absence using null as a variable value – meaning you have to deal with 3 valued logic and consider it in every domain. In contrast, TypeDB’s None is an entirely different type, that you handle explicitly, and beyond which you just use regular logic. SPARQL has optional pattern matching, which may leave variables unbound as well. However, unlike TypeDB, SPARQL doesn’t type check queries – which is where the semantics of optionals and other types really come into play.

Handling optionals in TypeDB

TypeDB’s goal is to build a safer, stronger foundation for your data. As such, we always want our compiler to be able to do as much of the work for you as possible, catching bugs before your queries touch your users’ data and allowing you and your agents to fix bugs quickly. Pushing optionality into the type system is part of that promise.

That’s where the fun begins: what is the interpretation of an Optional variable in TypeDB? And how do you work with them?

In the query above, we could work out what the expected types are of every variable:

$c: company
$addr: office_address?

The things we have to define based on this are:

– What happens when you pattern match using an optional variable?

– What happens when you write data using an optional variable?

– How do you check if a variable is set to a value or empty?

Round 1: implicit optionality

TypeDB 3.4 shipped optionality that worked implicitly, trusting the user to be careful. You never had to explicitly handle optional variables anywhere.

As an example of how this manifests:

# Read as: take an input stream of rows containing office addresses and for each match the company with the indicated address

given $addr: office_address?;
match
  $c isa company, has office_address $addr;

We use the ? type annotation syntax to indicate an optional type.

There are two possible behaviours here:

1) Ignore any inputs where $addr is empty and don’t return any company

2) Error since the empty optional isn’t handled

We originally shipped option 1. We further extended this to write operations, only requiring that statements containing optional variables be wrapped in try, which is skipped if any of its variables are empty:

given $addr: office_address?;
insert
  $c isa company;
  try { $c has $addr; };

Most of the time this is the behaviour you want: just perform operations when the variables are set.

The downside of this approach is that all invocations of an optional variable immediately eliminate the entire row being processed, and that you couldn’t reference an optional outside of a try in write clauses. What if you want to change behaviours based on a missing value?

given $tax: tax?;
match
  $order isa order;
  { $order has tax $tax; }
  or {
    # how do we look up orders _without_ a tax set if we weren't given a tax value?
  };

Round 2: explicit optionality

As of TypeDB 3.15, we’re deprecating implicit optional handling in favor of explicit handling. Queries using the implicit system will log warnings encouraging migration.

Switching to explicit optionality handling has the side effect of improving long-term system correctness: making a non-optional variable optional forces you to handle that throughout the query. You don’t just silently get a different answer set that looks okay, without considering the consequences.

What does this look like?

1) Invoking an optional variable without handling the empty case explicitly is deprecated

2) We introduce syntax for asserting a variable is set in read patterns: isset

3) We introduce if gates to enable conditional write behaviours

4) We introduce syntax for short-circuiting optional variables in expressions

1. Deprecation of implicit optional handling

The server now logs warnings when using optional variables without explicit handling:

given $addr: office_address?;
match
  $c isa company, has office_address $addr;
WARN ir/pipeline/block.rs:276: Deprecated usage of optional variable or try blocks:
The optional variable 'addr' was used in a context where optionals are not permitted.
Use `isset` in the pattern to match only non-empty values.
Expressions can use `?` to propagate the empty value to the result.

2. Asserting a variable is not empty

To correct the deprecated behaviour above, we now require writing one additional pattern that strips the optionality from the variable’s type.

given $addr: office_address?;
match
  isset $addr;  # explicitly eliminate rows with the address unset
  $c isa company, has office_address $addr;

This assertion is order-agnostic, just like anything written in a match clause. The check fails for any row where $addr is None, converting the type from $addr: office_address? to just $addr: office_address.

You can negate this to answer the question we had previously, finding orders with the indicated tax if present, else finding orders without any tax:

given $tax: tax?;
match
  $order isa order;
  { isset $tax; $order has tax $tax; }
  or {
    not { isset $tax; }; not { $order has tax $_; };
  };

3. Handling optionality in writes

Write operations don’t read as elegantly as a match when using optionals. For example, saying insert isset $addr; $c isa company, has $addr; doesn’t make any sense!

By introducing if conditionals, we compose readably with isset constraint:

given $addr: office_address?;
insert
  if { isset $addr; } then {
    $c isa company, has $addr;
  }

It has a similar effect of changing the optionality of the $addr variable from office_address? to office_address – but this time just within the then clause.

This construct, which just matches the current try behaviour in writes, is nicely extensible to future else clauses. We’ll also add things like “downcasting”, for example: given $addr: address; ... insert if { $addr isa office_address; } then { < use $addr as an office_address specifically > }. We’re really looking forward building on this foundation.

4) Handling optionality in expressions

Expressions are a typical construct in queries. For example, you might use the total to compute a tax if a tax rate is known:

given $order_id: order_id, $tax_rate: double?;
match
  $order isa order, has $order_id, has line_items_sum $sum;
  let $tax = $sum * $tax_rate?;

This shorthand allows you to short-circuit the optionality up to the let assignment, setting $tax to None whenever $tax_rate is empty. Of course, you can always write the equivalent long form as well:

given $order_id: order_id, $tax_rate: double?;
match
  $order isa order, has $order_id, has line_items_sum $sum;
  try {
    isset $tax_rate;
    let $tax= $sum * $tax_rate;
  };

The semantics of optional variables

Finally, we come to defining the actual representation of optional variables in the theory. Previously, we hadn’t fully pinned it down, as it didn’t really matter. However, with the additional constructs like ‘downcasting’ and explicit None-elimination, it’s time to formalize it.

TypeDB’s type system is different to most programming and database languages in that it leans heavily on a union-of-types model (perhaps the closest analogy is TypeScript or Kotlin). This in particular arises from TypeDB’s deep polymorphic capabilities:

match
  entity $t;
  $e isa $t;

Here, the types of each variable are multiple. Let’s say our model has companies and people, we would end up with this typing:

$t: Type[company] | Type[person]
$e: company | person

How does the idea of optionals fit into this existing type system?

We contemplated three representations, the first of which is to use a more classical functional/Rust-like types:

# 1. Optional as a wrapper of a union
$e: Optional<company | person>
# 2. A union of Optional wrappers of a single type
$e: Optional<company> | Optional<person>
# 3. A union of types with a None type
$e: company | person | None

The issue with the first and second models is that they are a steep departure from our typical semantics and introduce more expressivity than we actually want. The second would permit a variable to contain both optionals and non-optionals, such as : $e : Optional<company> | person, which we don’t need. They could even allow things like Optional<Optional<...>> types, which we don’t foresee needing either.

The last is unconventional in databases, but cleanly composes with all of our existing structures. By treating the None as just another type, albeit a built-in one, all the existing union-of-types expansion and elimination type inference operations work as before. Now, binding variables in a try clause adds None to the set of possible types, isset eliminates it. And by simply updating our compiler to error when working with a possible None type in an unsupported step, we generate useful errors to help the user to update their queries consistently, and avoid three-valued logic altogether.

Looking forward

TypeDB will always push for safety and maintainability at the forefront of its language and implementation.

We believe that the changes we’ve made, making optionality handling required and extending our semantics to handle that, are a clear improvement towards that vision.

To facilitate the transition to the new semantics, we’re only emitting deprecation warnings when using the preexisting optionality handlers. And, since the new semantics are stricter than the old – if you miss the memo when it comes time to really switch, TypeDB will send you right to the places you need to update.

Let us know what you think! Our door is always open – leave us a message at community@typedb.com, or jump into our Discord server.

Share this article

TypeDB Newsletter

Stay up to date with the latest TypeDB announcements and events.

Subscribe to newsletter

Further Learning

Feedback