Why LLM-written TypeQL can end up more accurate than SQL once you allow retries

Comparing failure modes across MySQL, Neo4j and TypeDB on the Reactome pathway database

When an LLM writes a query and gets it wrong, one of two things happens. Either the database rejects it (a syntax error, a type error, something the application can catch and act on), or the query runs, and a wrong answer comes back looking exactly like a right one. 

We recently benchmarked LLM query generation across MySQL, Neo4j and TypeDB, asking Claude Sonnet 5 and DeepSeek V4 Pro the same 42 natural-language questions against the Reactome pathway database in each store, and scoring the generated SQL, Cypher and TypeQL by executing it and checking the result. The full methodology and results are in the accompanying report. This post looks at how each language fails when the generated query is wrong. From our results, this turned out to depend more on the language than on the model that wrote it.

Unless stated otherwise, every number below is Claude Sonnet 5 provided with the language’s skill document and five worked examples in the prompt.

How the failures break down

The percentages below are of 117 runs on answerable questions (39 questions, three repetitions each). First attempts, before any retries:

On raw accuracy the three are close: SQL leads at 82%, Cypher at 74% and TypeQL at 75%. The failure rows are where they separate. When SQL goes wrong it goes wrong silently: 20 of its 21 failures execute without complaint and hand back a plausible number, most often an undercount, because Reactome’s relational schema spreads one attribute across many per-class tables and the generated query unions only some of them. When TypeQL goes wrong, 27 of 29 failures are visible errors. Most of them rejections from TypeDB’s type checker, where the schema is part of the query semantics: a wrong guess about structure (a type label that doesn’t exist, a variable used as both an attribute and a value, an illegal recursion) is a type error at the server rather than an empty result. Out of 117 TypeQL runs, only 2 ended in an undetected wrong answer.

One of those two assumed an ordering attribute started at 1, when in the data it starts at 0. This is a perfectly valid query, but it answers a slightly different question to the one asked. As a result, it wasn’t detectable as wrong. A stricter schema rules out some categories of mistakes, but there will always be ones it can’t catch.

Cypher sits in between, with 60% of its failures silent. The property graph is schema-optional: no declaration ties labels, relationship types, directions or properties to the data, so the server has nothing to check a pattern against, and a wrong structural guess is still a valid query that simply matches nothing. What keeps Cypher’s numbers respectable here is that Reactome is an idealized domain: professionally curated, with every label and property clearly and consistently named. The model can use these given names and conventions to guide its query generation, but many real-world databases can have cryptic column names or conventions that drift across teams and years. A schema-enforced database fails loudly regardless of naming discipline; on messier data we’d expect these gaps to widen.

As a demonstrative example we could consider the authorship edge in Reactome, which points from a person to something that person authored. During one failed run, the LLM attempted the following Cypher query:

MATCH (ie:InstanceEdit)-[:author]->(p:Person)
RETURN count(*)
// 0 rows; the opposite direction matches 160,516 author edges

Neo4j executes this without complaint, because there is no declaration to check the direction against. The same wrong guess in TypeQL puts the person and the edit in each other’s roles:

match
  $ie isa instance-edit; $p isa person;
  edit-authorship (edit-author: $ie, authored-edit: $p);

TypeDB rejects this before executing it:

[INF11] Type-inference was unable to find compatible types for the pair of variables 'ie' & '_anonymous' across a 'links' constraint. Types were:
- ie: [instance-edit]
- _anonymous: [edit-authorship:edit-author]

No type lets an instance-edit play the edit-author role. In TypeQL it comes back as an error message that the model is then able to act on, so the query never runs

Fixing visible failures

When our benchmark encounters a failure the harness can see (such as an error or a malformed response) we test feeding it back to the model for another attempt. A query that runs and returns a wrong answer is terminal, because nothing knows it’s wrong. That means a language’s retry ceiling is set by its failure visibility. Here is the first table again after a budget of four retries:

Each language gains roughly what it had in the visible-error row to spend. TypeQL, where nearly every failure announces itself, climbs 17 points, from 75% to 92% (the best result of any language and model in the benchmark). Cypher climbs 8, from 74% to 82%. SQL, with almost nothing visible to retry, climbs a single point: its errors burn down to zero, but 17% of runs still end in a wrong answer that nothing can detect, exactly where it started. 

With this retry loop in place, we can now see the clear importance of visible errors. After retries, the accuracy order inverts compared to first attempts, leaving SQL in last place and TypeQL in first.

We can also look at the way accuracy progresses as you add retry limits of varying sizes

The trajectory helps show the impact of these varying amounts of retries on the languages. The first retry does most of the work everywhere: SQL reaches 83% at a budget of one and gains nothing after that, Cypher collects 80% of its eventual 82% there, and TypeQL jumps 12 points to 87%. Past that point only TypeQL keeps moving, stepping up again to 92% at a budget of three, where it plateaus.

What this means for agentic systems

For an agentic system, a query error is useful input. The loop reads it, revises the query and tries again, and with TypeQL a larger retry budget buys more accuracy. A silent wrong answer is invisible by definition. No amount of retry budget, self-review or orchestration can react to a failure that doesn’t present itself.

This provides an additional metric to consider when asking which database to put behind an LLM. Raw first-shot accuracy favours the language with the most training data, but the silent-failure rate, which tells you how often an LLM has a chance to correct its inevitable mistakes, is enormously important. After retries we see only 5% silent-failure rate for TypeDB and 17% for MySQL, with TypeDB also leading on accuracy after retries. 

In short, a strong type system that produces visible errors instead of invisible failures is a useful guardrail for agents. The more errors that can be caught and made visible before being acted upon, the more reliable the agentic system will be overall. We’d anticipate these reliability benefits to compound in larger, more interconnected agentic systems such as agentic swarms, as well as to enable potential applications in recursive learning.