The power of giving agents context and the ability to learn from mistakes

The best query language for an LLM agent isn’t necessarily the one the model knows best. It’s the one that gives the agent the best path from a wrong query to a correct one.

This is a companion piece to our LLM benchmarks blog, linked here.

Over the last year, one question has come up more than once: “Given the huge corpus of knowledge around SQL and Cypher, how can TypeQL realistically compete in AI query generation?”

We decided to investigate.

For the last month, we’ve run a series of benchmarks comparing the three languages. The results were interesting. In many cases, SQL and Cypher performed extremely well, largely as you would expect given the enormous amount of existing knowledge available to LLMs through pre-training.

However, when we provided the LLMs with some additional context about the language and schema, that advantage over TypeQL almost completely disappeared. TypeQL was able to come very close to both SQL and Cypher. Once we enabled retries, allowing the models to learn from the errors returned by the database, the gap disappeared. On Claude Sonnet 5, TypeQL overtook both.

I could turn that into a fairly straightforward marketing claim: “TypeQL performs better than SQL and Cypher for AI query generation when given the right context and the ability to retry.”

I think that misses the more interesting part of the benchmark.

There are two observations I want to explore:

• In-context learning can compensate for a pre-training deficit.
• Meaningful errors give agents the ability to learn from their mistakes.

Both have implications well beyond TypeDB. If we can give agents the context they need at runtime, and if the systems they interact with can return errors that contain useful information, then the agent’s ability to operate effectively may depend less on what it already knows and more on the environment we give it to work in.

For anyone building robust, trustworthy agents, I think that’s a much more interesting question than which query language an LLM has seen most often.

The impact of in-context learning

We ran the benchmark with a query-writing skill in the prompt, a markdown document that teaches the language, and separately with five worked examples. Then both together.

The impact of providing this context is striking. While the additional context did provide meaningful improvements to SQL (+20 points) and Cypher (+16 points), it transformed how TypeQL performed (+50 points).

This suggests that a lack of pre-training does not necessarily have to be a fundamental disadvantage for an agent. The right context can compensate for knowledge the model does not have, particularly when that context is specific to the system the agent is operating in.

The power of machine actionable feedback

Retries work only if an LLM is able to determine it has made an error. An error goes back to the model and it can try again, using info given to self-correct. When a query runs and returns a wrong answer without producing an error, the database itself gives the agent no signal that it should reconsider the query.

Sonnet’s first attempts, with full context in the prompt:

Looking at each in turn, of the queries that were wrong, SQL errored 5% of the time, Cypher 40%, TypeQL 93%.

TypeQL’s type system means a wrong guess gets rejected before the query runs at all. Cypher is looser: an invented property doesn’t error, it just matches nothing, so some mistakes surface and others quietly don’t. SQL will happily join two tables that have no business being joined and give you a number back.

When we enable retries for the LLMs, we can clearly see the impact of this difference. Four retries took TypeQL from 75% to 92%, Cypher from 74% to 82%, and SQL from 82% to 83%. 

What this means in production

The expected tradeoff for the in-context resources and retries is additional token usage. TypeQL currently uses more tokens than SQL or Cypher across the benchmark. This is partly because the models spent longer reasoning about a language they’d have less experience of, and partly because more of its mistakes triggered retries. The first should shrink as models learn TypeQL. The second is a choice between accuracy and cost, and for systems where accuracy matters we’d expect most teams to pay it. 

Over time, models will improve at all three languages, and first-shot accuracy should converge somewhere similarly high for each of them. The genuinely hard problem is what happens when they make mistakes. Agents can only fix a mistakes they identify, and that’s where SQL and Cypher struggled in our benchmark, where most of their errors weren’t visible.

A database that rejects bad queries with visible errors gives an agent a path back to the right answer. For teams building production systems on top of agents, this capability is one worth considering seriously, and we think TypeDB provides it better than most.