Research brief · 2026-07-13
Cost optimisation for enterprise LLM deployments
The cost of a production LLM system rarely comes from the model list price. It comes from oversized default calls, long prompts, and running a frontier model where a small one would do. Five levers, in the order AIKm pulls them.
Cost levers, in order
| Lever | Typical saving | Quality risk |
|---|---|---|
| Model routing | 40–70% | Low, if the router has a fallback |
| Prompt discipline | 20–40% | Low |
| Retrieval hygiene | 10–30% | Low |
| Caching (semantic and exact) | 15–40% | Medium, needs staleness policy |
| Distillation or quantisation | 30–80% for high-volume routes | Medium, needs evaluation set |
1. Route by need, not by default
Most enterprise queries do not need a frontier model. Classify the request (lookup, summary, drafting, reasoning) and route accordingly. Keep a fallback to a stronger model when the router's confidence is low, and log every escalation so you can retrain the router.
2. Cut prompt bloat
System prompts grow. Templates duplicate examples. Retrieval returns too many passages. Measure token spend per route, and treat any prompt over a few thousand tokens as a defect to investigate.
3. Retrieval hygiene
Better retrieval reduces the number of passages you need to send to the model. Rerank aggressively, cap at the smallest k that still hits your groundedness target, and prefer smaller, denser chunks over long ones.
4. Cache what repeats
Enterprises ask the same questions repeatedly. Semantic caching keyed on a canonical form of the query, with an explicit time-to-live tied to the underlying data, is often the single highest-ROI change.
5. Distil or quantise the hot path
For a small number of very high-volume routes, distilling into a smaller task-specific model or serving a quantised open-weight model can pay back in weeks. This step only makes sense once you have an evaluation set (see the evaluation brief) that will tell you if quality slips.