High availability is now live in TypeDB

A single database server is a single point of failure. A dead disk, a kernel upgrade, or a host retired by your cloud provider takes the database offline until the server comes back, and your application can no longer function.

Starting with TypeDB 3.14.1, TypeDB Cloud and TypeDB Enterprise remove that single point of failure. Your databases run on a cluster of servers that each hold a full copy of the data. When one of them fails, another takes over within seconds, and TypeDB drivers and Console find it without any change to your code.

Below, you’ll find what clustered TypeDB offers and what it guarantees, how failover looks from your application, and how to get there.

How a cluster works

A TypeDB cluster is a group of servers, called replicas, that agree on every change with the Raft consensus algorithm. At any time, one replica is the primary and the others are secondaries.

– The primary serves all client operations: schema and data writes, reads, database and user management, export and import. Keeping them on one server gives every change a single, strict order.

– The secondaries keep live copies. They receive every committed change, store it durably, and apply it to their own copy of the data, ready to take over.

– A change is committed once a majority of the replicas has stored it. Run three servers to survive one failure, five to survive two.

A cluster adds availability, not throughput. Today, the primary serves reads as well as writes, so the capacity of a cluster is the capacity of its primary, and you scale it by giving the primary more resources. Because every commit waits for a majority of replicas to store it, a commit also takes a network round trip longer than on a single server. Upcoming versions will allow reading from secondary replicas to scale read throughput horizontally.

Consistency guarantees

Clustered TypeDB is strongly consistent, so its behaviour exactly matches a single TypeDB server:

– Acknowledged commits are never lost as long as a majority of the replicas survives, evenincluding when the primary that accepted the commit fails a moment later.

– Every transaction sees every commit acknowledged before it was opened, whichever replica was the primary at the time. Read-only transactions included: a cluster never serves you data that a newer primary has already changed.

– Transactions remain fully ACID.

Failover in practice

When the primary becomes unreachable, the remaining replicas elect a new one from the up-to-date secondaries within seconds. Every transaction open on the affected replicas is closed, read transactions included, and the cluster carries on accepting new ones on the new primary.

From your application’s point of view:

– Connect to any replica. A driver takes one address or several, so that it can still connect while one server is down. It learns the rest of the cluster from the first server it reaches, finds the primary, and routes every transaction to it.

– Automatic failover. After a transition, TypeDB’s gRPC drivers and Console find the new primary and retry connecting and opening transactions on their own. Over the plain HTTP API, a secondary answers with the primary’s address, which lets you implement the failover manually. Or, use the TypeScript HTTP driver which automates the process as well.

– Handle interrupted transactions. A transaction that was open during a failover has to be opened again. The drivers retry single operations, such as opening a transaction or creating a database, on their own. A transaction, though, spans several requests (open, queries, commit), and the driver cannot safely replay them for you, so your code has to run it again.

In practice, that takes one driver option and one habit. The option, primary_failover_retries, sets how many times the driver retries finding the primary after a failed request before it gives up. Each retry follows the address that a secondary sends back, or polls the known replicas every two seconds. The default is a single retry, so give it more room if failover may take longer in your deployment. The habit is to run each transaction in a short retry loop, and to write it so that running it twice does no harm:

from typedb.driver import (TypeDB, Credentials, DriverOptions, DriverTlsConfig,
                           TransactionType, TypeDBDriverException)

driver = TypeDB.driver(
    ["db-1.example.com:1729", "db-2.example.com:1729", "db-3.example.com:1729"],
    Credentials("admin", "password"),
    DriverOptions(DriverTlsConfig.enabled_with_native_root_ca(), primary_failover_retries=5),
)


def write(queries: list[str], attempts: int = 3):
    for attempt in range(1, attempts + 1):
        try:
            with driver.transaction("shop", TransactionType.WRITE) as tx:
                for query in queries:
                    tx.query(query).resolve()
                    tx.commit()
            return
        except TypeDBDriverException:
            if attempt == attempts:
                raise


write('put $c isa customer, has email "ana@example.com";')

When the primary fails between transactions, the driver finds the new one on its own, and the next transaction simply takes a couple of seconds longer. When it fails while a transaction is open, that transaction ends with a connection error, and the loop runs it again on the new primary. A commit interrupted this way may or may not have been applied, which is why the query uses put: it inserts only what doesn’t exist yet, so running it twice still leaves one customer. The same option exists in all gRPC-based drivers.

Single operations don’t need a loop. The driver retries them on the new primary by the driver itself, within the same primary_failover_retries:

driver.databases.create("shop")
driver.users.create("alex", "password")

Automatic migration from TypeDB CE

If you run TypeDB CE today, you don’t need to export and re-import your data or rebuild anything. A TypeDB Enterprise server starting on your existing CE data directory with the --initialize.migrate-ce.enabled=true flag migrates it in place into the first replica of a new cluster, with every database and every user, including their passwords.

The migration re-imports your databases through the replicated log. The only requirement is free disk space of about twice the size of your data. An interrupted migration continues where it left off , and by default keeps the original data as a backup in case you want to roll back.

Once the first replica is ready, you can add the remaining cluster nodes, as described below.

On TypeDB Cloud, existing clusters move to 3.14.1 or later through a fully automated one-time migration, described on each cluster’s page.

Growing and shrinking a cluster

A TypeDB Enterprise cluster can resize while it keeps serving. You start a new server with empty storage and register it on the primary with the admin tool. The new replica first receives the whole history of the cluster and doesn’t vote or serve clients until it has caught up. You’ll see this once all the replicas are ready (by using servers status in the admin tool or server list in TypeDB Console:

  id | address       | role      | term | status
  -------------------------------------------------
  1  | 10.0.0.1:1729 | primary   | 1    | available
  2  | 10.0.0.2:1729 | secondary | 1    | available
  3  | 10.0.0.3:1729 | secondary | 1    | available

You can also deregister a replica, including the current primary, to remove it from the cluster. Keep an odd number of voting replicas: four servers tolerate the same single failure as three.

Operating a cluster

– Encryption. Client connections use TLS as on a single server. Replication between servers is encrypted separately with mutual TLS: every replica presents a certificate to its peers and verifies theirs against the cluster’s CA.

– Monitoring. Every replica’s diagnostics endpoint (port 4104 by default) exposes Prometheus metrics for its role, term, and applied log index (raft_role, raft_current_term, raft_last_applied_index), replication latencies, and backpressure (raft_apply_pending_bytes).

– Inspection. servers status in the admin tool, Console, and every driver show which replica is the primary, which are secondaries, and which are unavailable.

– Recovery. A replica that restarts rejoins and catches up from the primary’s log on its own. If a replica loses its storage, deregister it, wipe its data and clustering directories, and register it again. Never restore it from an older copy of its data!

A small Raft implementation story

Working through the Raft paper while building highly available TypeDB was a lot of fun. The paper is famous for being easy to understand, but what we came to admire most is how much it says in a few words. In a system where any mistake can cost you data, the hardest problems are rare corner cases, and more than once mistakes we made already had the answers hidden in a sentence or two in the paper.

As an example, one of these bugs hid in the first moments of a failover under load. Raft guarantees that a newly elected primary has every committed entry in its log, but applying entries to the data runs a step behind, so the old primary’s last commits may still be waiting their turn. On promotion, our implementation set up its allocator of internal entity IDs by scanning the stored data, just as a server does when it starts, so it missed the IDs of those pending commits. New inserts reused them, and two entities merged.

As with many such issues, the paper already had the solution: a new leader starts its term by committing an empty entry, and since entries apply in order, once that entry is applied, so is everything before it. In our updated implementation, a TypeDB server now becomes primary only from that point (and that’s one of the reasons the failover can take some time – so driver retries numbers might require being set to bigger numbers if you write a lot!).

Designing our own solutions is part of building a novel database, but there’s something special about solving a hard problem with a few compact concepts other expert designers left you. Raft is a beautiful example to follow.

Getting started

– Read how clusters work and how to run one in the clustering documentation.

– Deploy a cluster on TypeDB Cloud, or get in touch to run TypeDB Enterprise.

– Questions about high availability? Ask us on Discord.