Contact

The Bill Nobody Modelled

 

“The future of enterprise AI will not belong to the smartest agents running the most expensive loops, but to the systems disciplined enough to know when to stop thinking and start executing.”

Somewhere in a mid-sized software company this quarter, a finance leader is looking at a cloud invoice that has grown faster than revenue, headcount, or usage, and cannot find the line item responsible. The product still works. The customers are still happy. Yet the gross margin on the flagship feature has quietly inverted, and the most engaged users, the ones every playbook says to cherish, are now the ones losing the most money.

This is not a billing error. It is a structural consequence of how the industry has chosen to build with large language models. As reasoning capabilities matured, deployment shifted from single-turn text generation to multi-step, autonomous planning. Agents now plan, route to tools, and execute complex workflows in wealth management, tax accounting, and software operations with minimal human oversight. That shift carries a cost most organisations never modelled, a compounding and highly variable infrastructure burden best named for what it is: the planning tax.

The planning tax is the cost incurred when stochastic language models are repeatedly asked to plan tasks that are fundamentally repetitive. In traditional deterministic software, the marginal cost of running a compiled sequence approaches zero and scales linearly. Agentic systems break that contract. Their cost spikes unpredictably with task complexity and grows disproportionately with engagement. The purpose of this article is to make that tax visible, to show why it behaves the way it does, and to argue that it is not a pricing problem to be waited out but an architecture problem to be engineered away.

The Numbers That Reach the Boardroom

The planning tax is not abstract. It is measurable, and it is already disrupting enterprise budgets. Standard per-token pricing has stopped being a useful metric; organisations now have to reason about aggregate cost per feature and per workflow.

At the level of the individual user, the pattern is stark. Internal metrics from early GitHub Copilot deployments reportedly showed that while the service charged a flat ten dollars a month, the average developer cost roughly thirty dollars a month in compute, and the heaviest coders cost up to eighty, forcing pricing changes. In richer multi-agent developer environments such as Devin, SWE-agent, and AutoGPT, power users running long tasks routinely generate token bills above two hundred dollars a month. Executives at frontier labs have publicly acknowledged that flat-rate premium tiers often lose money on the users who run tens of thousands of queries.

At the enterprise level, the problem takes the shape of agent sprawl. Decentralised business units deploy coding assistants, service bots, and copilots without central financial governance, and engineering teams accumulate monthly token bills exceeding one hundred thousand dollars, sometimes shared on professional networks as a badge of technical scale, rarely accompanied by a clear account of the business value delivered for the money.

Then there is the cost nobody budgets for, the failure cost. A serious return-on-investment model for an agentic system must subtract development, run-time, and failure costs from the gross task value multiplied by success rate and volume. When an agent fails in production, the organisation pays for the wasted compute, the expensive engineering labour to diagnose and fix it, and the business impact of lost customer value. An enterprise processing a thousand tickets a day at a seventy per cent success rate produces three hundred daily failures. At a conservative ten dollars per failure in labour and recovery, that is ninety thousand dollars a month, a sum that in many deployments quietly exceeds the entire allocated compute budget.

 

Two Rooms, One Invoice

The industry argues about agents in two rooms that rarely share a whiteboard. In one, strategists and consultants see a productivity revolution that dwarfs the cost of compute. In the other, architects, finance analysts, and engineers watch margins erode under non-deterministic execution.

The margin-erosion room sees the same systems differently, and it is not wrong either. From the engineering perspective, current agentic architecture applies probabilistic tools to deterministic problems. The planning tax is measurable and actively eroding product margins, often invisible to the executive team until the billing cycle closes. The core critique is that organisations use extended-thinking models, which are appropriate for novel and genuinely complex cognition, as expensive overhead for basic, repetitive routing. Because a true agent decides its own pathing, tool selection, and retry logic at run-time, the identical prompt can produce a thirty-fold variance in token consumption across runs. For high-volume enterprise operations, that variance is intolerable. The conclusion follows plainly: the answer is not to wait for cheaper tokens, but to re-architect when the AI is permitted to plan at all.

Both rooms are describing the same invoice. One sees the value it buys, the other sees the waste it hides. The reconciliation is not to pick a side but to separate the work that genuinely needs intelligence from the work that only needs to run.

Recompiling on Every Click

Computer science settled this kind of question long ago, through the phase distinction between compile-time and run-time. At compile-time, a program is analysed, type-checked, and translated into an executable form; it need not satisfy any live invariants, it only needs to be structurally sound. At run-time, that compiled program executes rapidly, deterministically, and with near-zero marginal overhead.
The current generation of agents behaves, almost universally, as an interpreter running only at run-time. Asked to onboard a client, extract fields from a standard invoice, or navigate a known application to download a report, the agent assesses the environment, formulates a plan from scratch, selects tools, and validates its output on the fly. Ask the same agent to do the same task an hour later and it discards its previous cognitive labour and pays the full price to plan the sequence anew.

 

“Treating a frontier model as a real-time decision engine for repetitive work is the architectural equivalent of recompiling an application every time a user clicks a button.”

Database engineering solved precisely this with query optimisation. A database spends heavy resources at compile-time, during query planning, to determine the optimal path to the data, then executes that plan at run-time without re-deriving the decision tree on every fetch. The path out of the planning tax is the same move applied to AI: front-load the expensive decision-making to design-time, produce a deterministic plan, and execute that plan freely thereafter.

Moving the Intelligence Upstream

To eliminate the planning tax structurally, an enterprise architecture has to split the workflow into two guarded phases, a probabilistic planning phase that runs once, and a deterministic execution phase that runs repeatedly. A growing body of research and open tooling shows this is not theoretical.

The clearest statement of the principle is a research system named Compiled AI, which strictly separates planning from execution for well-specified, high-volume, compliance-sensitive workflows. Its thesis is that while many workflows need intelligence to design, they do not need intelligence to execute. The architecture rests on three commitments: the language model runs only at generation time to build the logic and never at transaction time; deployed workflows run as static, deterministic Python with no further model calls; and every generated artefact must pass rigorous validation before deployment. That validation runs four gates in sequence. A syntax gate confirms the code is well-formed. A security gate screens for prompt injection and unsafe code, reportedly achieving 96.7 per cent accuracy on injection detection and 87.5 per cent on static safety analysis with zero false positives across 135 cases. An execution gate runs the code in a sandbox to confirm stability. An accuracy gate checks semantic correctness against ground truth, catching the roughly four per cent of workflows that run cleanly yet produce wrong answers. The reported results make the case: a 96 per cent completion rate with all failures occurring safely at compile-time, execution at 4.5 milliseconds against 2,004 for direct LLM calls, a one-time generation cost near 9,600 tokens, break-even against run-time inference at about seventeen transactions, and by a thousand transactions a 57-fold reduction in tokens against direct execution and 84-fold against run-time multi-agent frameworks.

For workflows too variable for frozen code, where a minor interface change would break a brittle script, Stanford researchers proposed Agentic Plan Caching. Ordinary semantic caching matches a user’s text to a stored answer and works for chatbots but fails for agents, because the correct output depends on live data and shifting interfaces. The intent to “delete the top comment” is stable, but the coordinates of the button are not. Agentic Plan Caching works as a test-time memory for the plan-act loop: when an expensive planner completes a task, the system extracts a structured plan template capturing intent and sequence rather than the final output. On a new query, lightweight keyword extraction checks for a matching template, and a small, cheap model such as Llama 3.2 3B adapts the template to the live context. Across five agent workloads it reduced serving costs by roughly forty-seven to fifty per cent and latency by twenty-seven per cent, while preserving about ninety-six and a half per cent of the original performance.

The same instinct drives the DSPy framework, which treats prompts not as hand-crafted strings but as declarative programs that can be compiled and optimised. Developers define a task with signatures for inputs and outputs and a deterministic evaluation metric, and optimisers search for the best instructions and demonstrations to maximise it. The output of a compile step is a frozen, optimised pipeline, so the heavy cost of finding the optimal path is amortised at compile-time and the deployed application runs without guessing its instructions at run-time. The philosophy is programming, not prompting, and its economic payoff is the same amortisation the other systems chase.

The principle even reaches browser automation. The legacy approach lets a model read the page and decide what to click on every run, burning tokens and breaking on minor changes. OpenCLI inverts this, using a model exactly once to explore a site and synthesise a deterministic command-line adapter in plain JavaScript, then routing every later interaction through that static adapter. A hundred executions require one planning call instead of a hundred, and execution stability reaches full reliability.

The Discipline of Remembering

Compiling planning into deterministic execution only works if the single planning pass is trustworthy, which makes observability the precondition for the whole approach. The emerging discipline of AgentOps exists for this, and it goes well beyond monitoring latency. Platforms such as Langfuse, Arize Phoenix, and Databricks MLflow capture every step of an agent’s reasoning trajectory, its tool selection, the context it retrieved, the parameters it extracted, and the logic it applied, typically by ingesting OpenTelemetry traces.

Those traces serve two purposes. They enable precise financial auditing, letting finance teams see which steps, bloated system prompts, or untrimmed histories inflate the bill without improving the success metric. And they become the raw material for compilation. When an agent successfully handles a novel or difficult case, its trace is evaluated, by an automated judge or a human reviewer, and if the trajectory is accurate and compliant it is converted into a permanent deterministic template. The dynamic exploration is logged once and then remembered forever, which is the entire economic argument in a sentence.

A practical version of this appears in the Microsoft Agent Framework, in a pattern that confines non-deterministic AI to judgement, reasoning, and classification, while deterministic code handles routing, retrieval, and execution. In an incident-response workflow, a triage agent reads an ambiguous alert and produces a structured verdict, the affected subsystem, the severity, an initial hypothesis, and control then returns to conventional code. Routing to specialist sub-agents runs on hard-coded conditional edges rather than an LLM guessing the next step, and reasoning loops carry a deterministic cap, for instance a maximum of three diagnosis iterations, that forces the stochastic component to yield back to compiled rules. The pattern is easy to summarise: AI proposes, code decides. Its financial effect is to cap the maximum exposure of any workflow, which is exactly the guarantee a finance leader needs before signing off on autonomy at scale.

At AscentCore we have come to think of this as a discipline rather than a feature, the practice of deciding, deliberately, where a system is allowed to think and where it is required only to run. The reward for that discipline is the combination the industry has been promising and mostly failing to deliver: the reasoning of a frontier model joined to the exactness, reliability, and zero-marginal-cost economics of ordinary code.

References

Share This Post

MORE TO EXPLORE