Why your agentic AI stack has a context problem, and it's not your retrieval layer

It has been talked about since LLM-based agents first started appearing, and we are still consistently running into the same wall. Chatbots and agents are still delivering answers that are confidently wrong.

The immediate instinct we have when agents hallucinate or fail is to point at the LLMs and shrug, or we investigate the retrieval layer to find the breakage. While this may surface some small wins, it doesn’t actually solve the underlying problem.

A big part of the challenge is that these agents are unable to retrieve the necessary context that would allow them to answer questions accurately. To understand this, we need to look deeper to understand the big question of: Where does the intelligence for your Agent actually live?

The issue of missing context

The majority of modern applications have a broad data layer, comprising many different systems. These then feed into an application layer, where that data is assembled to create the business logic. The databases are being used as a system of record, with all of the actual thinking going on higher in the stack.

Now, we are feeding that same data layer to agents in the hope that they will be able to recreate similar business logic that is currently constructed in the application layer. Assuming the answer doesn’t require much interpretation, this can work reasonably well. 

However, for many tasks, these agents will have to contrive their own meaning from the data. Take revenue reconciliation as a common example: however the numbers are stored in the raw database, there is likely a massive, complex SQL query sitting in the application layer that actually defines “what revenue is” for the business.

If you instruct an agent to give you revenue across a period, it will look at the data layer and assemble its own version of revenue based on the raw tables it has available. Because it bypasses the application layer’s logic, the agent invents its own semantics and returns what will typically be a confidently wrong answer.

The half-way there property graph problem

The logical next step is to move the semantics directly into the data layer. This is where existing knowledge graphs come in, most commonly built on property graphs like Neo4j.

These are excellent for storing highly related data, and providing rapid and efficient traversal and mapping across a domain. The catch is that to understand what sits at each node, the agent has to run traversals across the graph to see what’s within it. This is because there is rarely an overlaying schema that explicitly tells the agent what the data structurally means, so it still has to assemble its own logic to find an answer.

Consider an enterprise agent answering this question: which analysts currently have read access to the payments platform, and should they? A standard property graph tells the agent that Person A has a relationship to System B. It typically cannot tell the agent whether that relationship is still valid given Person A’s recent promotion, whether a regional data jurisdiction restriction supersedes the access grant, or whether access was temporarily inherited through a project that ended last quarter. Without a schema enforcing these rules, the agent is left to invent its own rules, and we already know where that leads.

Improving context layer through strong typing

This is where a knowledge graph with a strongly typed schema changes things. If your data combines the power of a property graph with a strong native typing, you can provide a deeper semantic understanding for the agents.

In practice, such types carry real constraints, inheritance hierarchies, and valid relationship patterns enforced at write time. A contractor cannot be assigned a management role because the type system makes it structurally impossible. Relationships carry their own data, including when they were created, when they expire, and who approved them. The model can also derive facts it wasn’t explicitly given, working them out from what it already knows.

Now, when an agent is trying to find answers from the data layer, it has the context it needs to make much more informed decisions, reducing the number of times the verifier (human or agent) has to go back and forth in a loop with the worker agent, thus cutting on tokens, time, and ultimately cost.

The architect’s question

Semantic modelling requires discipline and a good deal of time investment. Schema design takes thought to construct well, and when business rules change the model has to change with them. So, these improvements don’t come without a cost. Over a longer time period, those upfront costs can rapidly pay for themselves, and help to ensure your agents are giving answers that are less often so confidently wrong.

As a result, a good starting point is to look at your existing data layer and take the time to see how well constructed it is for the agents you’re looking to deploy.

This typed structure is the underlying design philosophy behind TypeDB, which encodes constraints, relationships, and logical inference directly into the data model rather than leaving meaning to be reconstructed at inference time.