LLM Cost Control: Architecture Patterns That Cap Your Bill
Last updated October 2026
An LLM cost optimization architecture caps spend with four mechanisms: a token budget enforced on every request, prompts laid out so the stable part reads from cache at a tenth of the input price, a routing floor that sends routine work to a small model, and a batch lane at half price. Tips reduce cost. Structure caps it.
Why does LLM spend grow faster than usage?
Because the unit you are billed for is not the unit you ship. You ship resolved requests. You are billed for tokens, and the number of tokens behind one resolved request is set by decisions nobody revisits after the prototype: how much context gets stuffed into every call, how many times the agent loops before it answers, how large an output you allow, and which model handles the eighty percent of traffic that is routine.
A feature that cost $300 a month in a pilot with fifty users does not cost $6,000 at a thousand users. It costs more, because real users ask longer questions, attach bigger documents, and trigger the retry paths the pilot never hit. Nothing in the standard client libraries stops any of this. The default behaviour of every major SDK is to send what you give it and bill what it costs.
That is the gap in most writing on this topic. The guides list savings techniques, caching, routing, compression, and stop there. A technique reduces a bill. Only a mechanism caps one, and a cap is what a CFO is asking for when they ask what this will cost at ten times the volume.
What does an LLM cost optimization architecture actually price?
Start from the published list prices, because the spread between them is the single largest lever you have. These are the current rates from each provider’s own pricing documentation as of October 2026, per million tokens.
| Model | Input | Cached input read | Output |
|---|---|---|---|
| Claude Opus 5.5 | $4.00 | $0.20 (0.05x) | $20.00 |
| Claude Sonnet 5.5 | $2.00 | $0.20 (0.1x) | $10.00 |
| Claude Haiku 4.5 | $1.00 | $0.10 (0.1x) | $5.00 |
| GPT-5.6 Sol | $5.00 | $0.50 | $30.00 |
| GPT-5.6 Luna | $0.20 | $0.02 | $1.20 |
| Gemini 3.8 Flash | $0.75 | $0.075 (context caching) | $3.75 |
Anthropic’s model pricing documentation lists Opus 5.5 at $4 per million input tokens and $20 per million output tokens, with Haiku 4.5 at $1 and $5. OpenAI’s API pricing page puts GPT-5.6 Sol at $5 input, $0.50 cached input and $30 output, against $0.20, $0.02 and $1.20 for GPT-5.6 Luna. Google’s Gemini API pricing lists Gemini 3.8 Flash at $0.75 input and $3.75 output with context caching at $0.075.
Read the output column across the table. The gap between the most expensive and the cheapest output token in that list is a factor of twenty five. No prompt compression trick recovers a twenty five times error in model selection, which is why routing belongs in the architecture rather than in a backlog ticket.
How much does prompt caching really save?
More than most teams assume, and only if the prompt is laid out for it. Anthropic’s prompt caching documentation prices a five minute cache write at 1.25x the base input price and a one hour write at 2x, with cache reads at 0.1x for most models and 0.05x for Opus 5.5. It also sets a floor: the minimum cacheable prefix is 512 tokens for Opus 5.5 and Sonnet 5.5, and shorter prompts are processed without caching and without an error, so a prompt that is too short to cache fails silently at full price.
The engineering consequence is a layout rule rather than a feature flag. Everything stable goes first, in a fixed order: system instructions, tool definitions, retrieved documents that do not change within a session, few-shot examples. Everything volatile goes last. Teams that interleave the user’s message between two blocks of static context get a cache miss on every call and conclude that caching does not work.
Why do agents blow the budget when single calls do not?
Because an agent resends its whole context on every step, so cost grows with the square of the conversation rather than with its length. The arithmetic is easy to do from the list prices above and worth doing before you ship.
Take an agent with a 20,000 token prefix of instructions and tool definitions that runs fifteen steps, each adding about 800 tokens of tool output. Billed input across the run is 15 times 20,000, plus 800 tokens accumulating across fourteen later steps, which is 384,000 input tokens. On Sonnet 5.5 at $2 per million that is about $0.77 per run, before output. Cache the prefix and the same run bills one write at 1.25x, fourteen reads at 0.1x, and only the incremental tokens at full rate: roughly $0.27. At ten thousand runs a day that is the difference between about $7,700 and about $2,700.
Reasoning tokens are the other half of the problem, because they are billed as output. Anthropic’s extended thinking documentation states that thinking tokens count toward the max_tokens limit for the turn and reports them in a thinking_tokens field, with a minimum budget of 1,024 tokens. Output is the expensive column in every row of the table above. An agent that thinks at length on every step of a fifteen step loop is buying output tokens fifteen times to answer one question.
The same compounding shows up in retrieval. Oversized chunks and a high top_k are a cost decision disguised as a quality decision, which is one of the failure modes we covered in RAG in production.
Which LLM cost optimization architecture patterns cap a bill?
Each of these is a mechanism, not a guideline. Each one refuses rather than degrades when it hits its limit, which is what makes the number on the invoice predictable.
- A budget object on every request. Four fields, set at the call site and enforced by your own client wrapper: maximum input tokens, maximum output tokens, maximum tool iterations, maximum retries. Count the input before sending and reject over budget calls with a typed error instead of truncating silently. A loop limit of eight to twelve steps catches almost every runaway agent without affecting legitimate work.
- A cache shaped prompt. A stable prefix above the 512 token minimum, assembled in a fixed order by one function that every call path uses. Log the cache read token count on every response. If that number is zero in production, your caching is decorative.
- A routing floor, not a routing hope. Classify the request first, then pick the model from a table, and make the small model the default rather than the fallback. Promote to the expensive model on an explicit signal, such as a failed validation or a confidence threshold, and log every promotion. The pattern only holds if you can prove quality survived it, which is what the frozen eval set in writing evals before you build is for.
- A batch lane. Anything with no user waiting on it moves to the asynchronous batch endpoint. Anthropic, OpenAI and Google all document a 50% discount on batch input and output, which makes the lane worth building for embeddings backfills, nightly summarization, evaluation runs and classification jobs.
Those four cover the structural cost. What is left is measurement, and the metric that matters is cost per resolved request rather than cost per token. A routing change that halves token spend while doubling the number of retries a user needs has made the product worse and the bill larger. We instrument delivery the same way on client work: our PULSE telemetry collects more than 400 data points per week per engagement, because a threshold with an owner is a control and a dashboard without one is decoration.
When should work move to a batch API?
Use a simple test: if no human is blocked on the response, the work is batch work. In practice that captures more of the bill than teams expect, because the obviously interactive feature is usually a small share of total tokens.
Four categories that almost always qualify, in the order they are worth moving:
- Embedding backfills and re-indexing. Large, predictable, and entirely offline.
- Evaluation runs. Every release scores the frozen eval set, and nobody is waiting on the result inside a request cycle.
- Bulk classification and enrichment. Tagging, routing, extraction over a backlog of records.
- Scheduled summarization. Nightly digests, weekly rollups, anything with a delivery window measured in hours.
The constraint is latency: these endpoints are asynchronous, and OpenAI documents a 24 hour processing window for its Batch API, so the lane needs its own queue, its own retry policy and its own failure handling. That is an afternoon of work that halves the price of a large share of your tokens, and it is one of the pieces we build into the first sprint rather than after it, following the sequencing in our pillar on shipping your first production AI feature in 30 days.
Book a Code Review
The teams with a surprise LLM bill are rarely the ones that chose the wrong model. They are the ones that never put a ceiling anywhere in the call path, so the cost of a request is whatever the request happens to need. The four patterns above take about a week to build and they turn an open ended variable into a number you can multiply by your traffic forecast.
If you want that built into the feature rather than retrofitted around it, our AI Integration Sprint is $24,500 over a fixed 30 day window, and the final milestone is waived if we miss day 30. The budget wrapper, the cache layout and the eval harness ship as part of the feature.
If something is already in production and the bill is the reason you are reading this, start with a read of the code. Book a code review and a senior engineer will walk your call path, price it against current rates, and reply within one business day.