← All posts

Layer 2: Why the Next Unlock in AI Is Not a Better Model

Writing code is mostly automated now, which turns developers into architects. Architects want everything in front of them. A Layer 2 system, a store of pre-computed insight that a model draws on instead of re-reading the world, is what puts it there. With the numbers from our own benchmark.

By Amritpal Chera

For a while now, we have been reverting back to systems thinking in programming.

I think of the writing part of programming as pouring concrete to make a building. There are only so many ways you can pour concrete, and that part is now about 80% automated. Why only 80%? With concrete you still have to account for the environment: temperature, weather, the ground it sits on. With code you have to make sure the way it is written fits the specific use case, the deployment standards, and the best practices of your particular niche.

What is left after the pouring is automated
pouring the concrete: the writing, now automatedthe 20%the concrete: weather, temperature, the ground it sits onthe code: fit to the use case, deployment standards, the practices of the niche
A way of seeing it, not a measurement. The part that remains is the part that needs judgement.

Since 80% is taken care of, developers have to upgrade into architects: shape their taste, meet the requirements, and waste the least possible amount of resources doing it. And architects hate being grounded in paperwork and data gathering. We would rather have everything in front of us and do what we do best, which is make the best use of what is on hand by making decisions nobody else would.

Hold on to that. It is the reason the rest of this matters.

Beyond the foundational models

In this piece I want to explore the real-world use of AI that extends beyond the foundational models: ChatGPT, Claude, Gemini. I want to look at what I call Layer 2: systems built on top of foundational models, with extra springs and bolts, designed to do one specific thing exceptionally well.

If you already know AI you know about fine-tuning: feeding a model data on a specific task and correcting it until it gets accurate. Layer 2 is a different move. Fine-tuning changes the model. Layer 2 changes what the model is handed.

A Layer 2 system is an umbrella around one or more LLMs, fine-tuned or not, combined with tools that pre-compute the understanding of the data before the model ever sees it. The sketch below is the simplest way I can draw the difference.

Left: a basic agentic system, a data source feeding an LLM that holds a context window, sub-LLMs and tools, then an output. Right: a Layer 2 system, where data sources pass through data requirements, data filtering and data storage first, produce top-level data insights, and only then reach the LLM and its output.

On the left, a basic agentic system: the data source feeds the LLM directly, and the LLM (with its context window, sub-LLMs and tools) does everything. On the right, a Layer 2 system: the data passes through requirements, filtering and storage first, comes out as top-level insights, and the LLM starts from there.

Read the left side first. A data source, an LLM with its context window, some sub-LLMs and tools, and an output. Everything that happens to the data happens inside the model. Now the right side. The data sources go through a requirements step (what do we actually need from this?), a filtering step (drop what does not meet it), and into storage. What comes out of that block is a layer of top-level data insights. Only then does the LLM enter, with the same context window, sub-LLMs and tools as before, except that what it is handed is already an answer-shaped piece of the world rather than the world.

Two differences, then. The level of understanding of the data before the model touches it. And how much of the whole system the model is. In the basic system, the LLM is the main driver of operations. In the Layer 2 system, the LLM is maybe 30 to 40% of the equation: it takes pre-calculated insight and turns it into the specific thing you asked for.

How much of the system is the model
basic agentthe LLM: reads, reasons, remembers, answerstoolsevery question re-reads the data source through the LLMLayer 2 systemfilteringstoragetop-level insightsthe LLM: 30 to 40%the LLM receives computed insights and turns them into the answer
Illustrative proportions. In the basic agent the LLM is nearly the whole system; in a Layer 2 system it is the last step.

Why it matters: the numbers

Quality context matters. A window that holds only what is relevant to the answer lets the model reason faster and spend far fewer tokens than it would sifting through everything that is not. And that matters because real-world data changes fast. To analyze it as it changes, you need a system that can answer almost instantly and at a cost that does not climb with every question.

We measured this rather than asserting it. In our benchmark we gave a frontier model (Claude Fable 5) the same ad-library access MessCube draws from, asked it the same market questions, and kept asking. Thirty-six questions in one continuous session, every citation verified against the real data afterward. Here is what the time per answer did.

Seconds per answer across a 36-question session
0s200s400s600scontext compaction21018212836570s398sthe store: 0.08 to 0.38s per questionquestion number, one continuous session
Measured. Two waves: latency climbs to 570 seconds, the session compacts its context, then it climbs again to 398 seconds as the agent re-reads what the summary dropped. The store answered the same questions in 0.08 to 0.38 seconds.

I want to be precise about what this shows, because the obvious reading is wrong. The foundational model did not get worse. Its answers stayed grounded the whole way: 381 of 381 cited ads checked out. We had predicted decay, we measured none, and we published that. What climbed was time, in two waves, because on every question the model went back to the raw data and read it again. Base models do not degrade over a long session. They re-derive.

100%
of the agent's citations still verified at question 36. It did not decay; it re-derived.
570s
for a single answer at the peak of the first wave
~110×
marginal cost per question, re-deriving versus reading a store
0.08s
to 0.38s per question against the pre-computed store

The context tells the same story from the other side. Re-asking the agent a question cold costs it 57 to 86 thousand tokens of working context each time. Inside one session the growth was gentler, about 2.1 thousand tokens per question, and the session still compacted at 98 thousand, half the window on the spec sheet. Real sessions never get the window the spec sheet promises.

Working context per question
0k50k100k150k200kagent, fresh re-ask57k to 86k tokens, every questionagent, one sessiongrows ~2.1k per question, compacted at 98kLayer 2 storeanswer-sized: the market lives outside the modelthe 200k window on the spec sheet
Measured. The Layer 2 store keeps the market outside the model, so the window stays answer-sized at any question count.
What each further question costs
$0$25$50$7511020304050$71$1.42 of retrieval, every single questionstore built once, then queriedquestions asked about one market
Measured. The agent's line is its retrieval spend only; its model tokens and your waiting time sit on top. The store is built once.

The biggest change between the two systems is the quality of the data the model had access to. In the basic system the model had to reassess the world every time it answered. It probed through thousands of data points per question. In ours the data was already filtered for the use case and summarized into top-level insights the model could steer by, so it reached the answer faster and reached the same answer every time: five out of five trials, identical.

Answer quality is a function of whether a persistent store exists to reconstruct from. The differentiator is what that store costs to re-read: theirs at minutes per question, ours at milliseconds.

Caching does not fix this

The instinct is to reach for caching. It does not solve it. Caching makes re-reading cheaper. It does not make re-reading unnecessary. The model still starts each question from the raw material and still spends the tokens to reason across it; you have only shaved the price of loading it.

Combine the store with a stronger harness and the picture changes. The agent no longer has to probe the entire system again and again. It already knows what exists, what was done, what changed, and where it needs to be. Each question reads what is there and analyzes only what is new.

What one question costs, per system
What each question costs the agentfetch the raw data again$1.42 of retrievalread it into context57k to 86k tokensreason from zeroanswer29s climbing to 570snextWhat each question costs the storequery what already exists0.08 to 0.38shand the LLM the relevant partanswer-sized contextanswerseconds, identical 5 of 5 timeswhat changed? analyze only thatthe transcript ends, the work evaporatesthe store appreciates with use
The agent's column is measured from the benchmark session. The store's column is how the Layer 2 system answered the same questions.

Layer 2 is the biggest unlock of agentic systems

One very clear application is self-improving AI systems for complex tasks. Before this architecture, a system had to re-evaluate itself against its prior context and refresh its memory completely on every pass.

The clearest implementation of that is programming. Many companies now keep a top-level knowledge base for their agents, but the agents do not surface it as effectively as they should, so every session starts by rediscovering the codebase. With a Layer 2 shape, the knowledge base is the store: what exists, what was done, what changed. The agent reads the map, not the territory, scopes the change to where it needs to be, and records back what it changed. Each pass leaves the map more accurate than it found it.

The self-improving loop
knowledge basewhat exists · what was done · what changedthe agentreads the map, not the territorythe changescoped to where it needs to bethe codebasethe territorythe loop the harness closeseach pass leaves the map more accuratereadsactslandsrecords what changed
The knowledge base tells the agent what exists, what was done and what changed. The agent acts on the map, the change lands, and what changed is recorded back.

Which brings me back to the architect. The reason we hate paperwork is that it stands between us and the decision. A Layer 2 system is the thing that does the paperwork once, keeps it, and puts everything in front of us. It is now possible to build self-improving iteration loops that surpass anything we see today. I will cover the use case of self-improving codebases in my next paper.

Thank you for reading.

The benchmark behind these numbers, including the round the frontier model won and the decay prediction that did not hold, is published in full: the technical whitepaper and the business report.