Why AI Agent Token Costs Keep Climbing, and How to Cut the Overhead

If your agent's API bill keeps climbing faster than the work it produces, the tokens you're paying for probably aren't the ones doing the reasoning. A public analysis of a production Hermes deployment (issue #4379) put a hard number on it: roughly 73% of every API call is fixed overhead, about 13,900 tokens that ship on every single request regardless of what the agent is actually being asked to do.

One thing to get straight up front: 73% is the raw, uncached overhead: the full weight of the payload before any provider-side discount. The caveat: tool schemas are static, and providers cache them via prompt caching, so on repeated workloads a large share of that overhead is discounted after the first call. That caveat matters, and we'll come back to it. But the raw number is still the right place to start, because caching is a discount on the overhead, not a removal of it. Everything you do to shrink the overhead in the first place makes the cached footprint smaller too.

That number matters because it reframes the whole cost problem. Most teams try to cut their agent bill by switching to a cheaper model or trimming conversation history. Both help at the margin. Neither one touches the structural 73%.

Why the overhead is structural

Overhead is not waste in the sloppy sense. It is the price of making a stateless API call behave like a stateful agent. Every time a framework calls the model, it has to re-send the context the model needs to act correctly:

• The system prompt and role instructions that define how the agent behaves.
• The full JSON schema for every tool the agent might call.
• Governance, safety, and formatting rules the runtime injects.
• Enough conversation history for the model to stay coherent across turns.

The model keeps no memory between calls, so all of this gets shipped again on every turn. In the Hermes breakdown, tool definitions and the system prompt dominate the overhead. When an agent loads tools across shell, filesystem, web, browser, delegation, and other capabilities, the tool schemas alone can account for thousands of tokens per call before the agent has read a single word of the task.

Then it compounds. A real agent task is rarely one call. It is a loop: read, act, observe, act again. Twenty turns on a moderate task means that fixed overhead gets paid twenty times over. The variable part of the payload, the part carrying the actual work, is often the smallest line on the invoice. Long-running agents run a lot of turns, and every one of them re-pays the toll.

The evidence is no longer anecdotal

For most of the last year, "agents are expensive" was a feeling. This quarter it became a set of numbers, from two very different vantage points.

The ground-level view comes from operators posting their own token traces and converging on tool-schema bloat as the primary culprit (issue #4379): engineers showing their working, not a vendor making a claim.

The enterprise view landed at the same time. When Meta moved to cap internal AI token spending, the Hacker News thread that followed made the scale concrete. Meta employees reportedly burned 73.7 trillion tokens in about thirty days. Uber exhausted its 2026 AI coding budget in four months and capped spend at $1,500 per employee per tool. ServiceNow moved to daily per-employee usage monitoring. These are companies with serious budgets discovering that agent token consumption scales in a way headcount never did.

The two signals point at the same root cause from opposite ends. The Hermes issue explains the per-call mechanics. The enterprise caps show what happens when you run those mechanics at scale without anyone controlling the overhead.

What operators are actually trying

The workarounds discussed across these threads fall into a few buckets, roughly ordered by how much they move the number.

Let the provider cache the static parts. This is the first line of defence and, for repeated workloads, the highest-leverage one you can turn on today. Because tool schemas and the system prompt are the same on every call, providers can cache them and charge a fraction of the price on subsequent hits. If you're running the same agent configuration across many calls and you haven't confirmed prompt caching is active, that's the first thing to check: it directly discounts the biggest slice of the 73%.

Route the easy turns to a cheaper model. This cuts the per-token price but not the token count. Useful, and limited. Matching each agent to the right adapter and model tier is where that choice gets made.

Trim history aggressively. Summarising or truncating old turns keeps the variable part of the payload small. That is real savings on long tasks, and most mature setups already do it.

Shrink the fixed part itself. Caching discounts the overhead; it doesn't remove it, and it doesn't help the first call or workloads where the tool set keeps changing. The remaining lever is to send fewer tool schemas in the first place, which both cuts the un-cached cost and shrinks the footprint that gets cached. That is where the community's attention has moved next.

Caching and history-trimming are table stakes now. Reducing the schema footprint is the newer front.

Lazy tool loading: where the ecosystem is heading

The pattern the community is converging on for that last lever is lazy tool loading (proposed in issue #6839). It's still under discussion rather than shipped, but it targets exactly the portion of overhead that caching leaves on the table.

The idea is simple to state. Instead of injecting the full schema for every tool on every call, the runtime would hand the model a lightweight list of what exists. If the turn needs no tools, the model answers and the heavy schemas never ship. If the model asks for a specific tool by name, the runtime makes a second pass with just that tool's definition attached. You'd pay for the schema of the tool you use, not for the forty you don't. The schemas that never ship are schemas the provider never has to cache either.

For an agent with a wide toolset, this is the difference between hauling the entire toolbox to every job and grabbing the one wrench the job needs. The savings would land exactly where the overhead concentrates. Cheap, frequent tools like shell and filesystem could stay always-loaded, while heavy, occasional ones like browser control, code execution, and web search would load only on demand.

It isn't free as a design. It can add a round trip on the turns that do need a tool, and it asks the runtime to manage a two-phase call cleanly. That's part of why it's still being debated rather than switched on everywhere. But for the overhead-heavy, many-tool agents that dominate real deployments, it goes straight at the largest line item instead of nibbling the edges, which is why it keeps coming up.

Where Paperclip fits

The reason this matters for how you choose a runtime is that overhead is easiest to control when it's a setting, not a re-engineering project.

Paperclip exposes a per-agent token budget, a ceiling on how much an agent may spend before the runtime steps in. Setting that budget, along with the turn and concurrency caps that stop a runaway run, is part of wiring the company after import. For a cost-conscious operator that's the lever that turns "agent token consumption scales in a way headcount never did" from a surprise on the invoice into a number you set on purpose. It's the same discipline the enterprise teams above are now imposing by hand, available as a default you choose once rather than a cap you bolt on after the bill lands.

On the schema-footprint side, lazy tool loading is the direction the ecosystem is converging on, and Paperclip's runtime-level architecture, where tools and budgets are configuration rather than per-agent code, is positioned to adopt that discipline as it matures. A Blueprint can ship with cost-lean defaults already chosen, so the overhead awareness the Hermes thread is reaching for by hand is simply how the setup starts.

The takeaway

The 73% number is worth internalising because it changes where you look. Caching gets you a discount on the overhead; it doesn't erase it, and it doesn't touch the first call or a shifting tool set. The cheapest structural win in an agent stack is usually not a cheaper model — it's refusing to build 13,900 tokens of unused tool schemas into every one of a thousand calls in the first place. The frameworks are moving toward lazy loading and token-budget control because that is where the structural cost sits, and the enterprise teams getting throttled right now are the proof of what happens when nobody owns that number.

Subscribe below for more on running AI-native operations. Practical field notes on agents in production, straight to your inbox.