LLM cost optimization is not simply a matter of choosing the cheapest model. This guide shows how to measure token usage, identify the requests that drive spend, and reduce avoidable inference work while preserving the answer quality your users actually need. You will find a repeatable audit workflow, cost formulas, practical assumptions, and worked examples covering prompt reduction, model routing, caching, batching, retrieval depth, and output limits.
Overview
AI application costs usually come from a combination of input tokens, output tokens, request volume, model selection, and provider-specific features. A useful cost review separates those drivers instead of treating the monthly invoice as one unexplained total.
The central principle is to optimize for cost per successful outcome, not cost per request. A shorter answer is not necessarily cheaper if it causes retries, human review, or an additional API call. Likewise, a more capable model may be economical for difficult cases if it prevents failed workflows. The goal is to remove waste while keeping reliability, latency, and answer quality within agreed limits.
Start by recording four operational metrics for each important workflow:
- Requests completed, including retries and fallbacks.
- Input and output tokens per request.
- Latency and failure rate.
- A quality measure relevant to the task, such as citation correctness, schema validity, resolution rate, or human acceptance.
Group these metrics by use case rather than averaging the entire application. A retrieval-augmented support assistant, a document summarizer, and a tool-calling workflow have different token patterns and should not receive the same optimization treatment. For background on comparing model quality, cost, latency, and context limits, see this AI model selection guide.
How to estimate
Use a simple request-level model before making architectural changes:
Estimated cost = requests × [(average input tokens × input rate) + (average output tokens × output rate)]
Keep the rates as variables in your spreadsheet or configuration rather than hard-coding them into documentation. Providers may use different billing units, cached-token categories, batch rates, or minimums, so confirm the current definitions in the provider documentation before applying the formula.
For a more realistic monthly estimate, extend the formula:
Monthly cost = traffic × cost per request + embedding cost + storage or retrieval cost + monitoring cost + retry and fallback cost
When a provider charges in units other than one token, convert the measured usage to that billing unit first. Also distinguish between attempted requests and successful requests. If ten percent of calls are retried, multiplying only successful traffic by the nominal request cost will understate spend.
A practical audit workflow is:
- Export usage for a representative period and group it by endpoint, feature, model, and outcome.
- Calculate median and high-percentile input and output tokens; averages can conceal unusually large documents or responses.
- Identify the largest cost contributors and the workflows with the most retries.
- Set a quality baseline using a fixed evaluation set or production review sample.
- Change one variable at a time, then compare cost, quality, latency, and failure rate.
- Roll out the change gradually and retain a rollback path.
This process prevents token reduction from becoming a blind compression exercise. If a change lowers usage but increases incorrect answers, measure the full workflow cost before accepting it.
Inputs and assumptions
Build a cost model with explicit, editable inputs. At minimum, include:
| Input | What to record | Why it matters |
|---|---|---|
| Traffic | Requests by workflow and time period | High-volume, low-cost requests may dominate total spend. |
| Input tokens | Median, upper percentile, and maximum | Long instructions, conversation history, and retrieved passages increase every call. |
| Output tokens | Typical and maximum completion size | Unbounded responses create both cost and latency risk. |
| Retries | Validation, timeout, and application retries | Failed attempts still consume resources in many designs. |
| Model mix | Model used for each request class | Routing can reserve stronger models for tasks that need them. |
| Cache rate | Eligible requests and successful cache hits | Repeated work may be avoidable, but stale results can reduce usefulness. |
| Retrieval depth | Documents or chunks sent to the model | More context is not automatically more relevant. |
Assumptions should be visible. State whether token counts include system prompts, whether conversation history is resent, how cache hits are priced, and whether batch processing has different service-level expectations. Keep illustrative calculations separate from live pricing. For production planning, replace every placeholder rate with the current rate for the selected provider, region, model, and billing mode.
Prompt token optimization often begins with duplicated instructions. Version system prompts, remove repeated formatting guidance, and pass only the fields a workflow needs. A maintenance process is useful here; the system prompt maintenance checklist covers versioning and validation practices.
For retrieval workflows, measure the marginal value of each additional chunk. If adding context rarely changes a correct answer but consistently increases input tokens, reduce retrieval depth, improve ranking, or summarize older context. A document-processing workflow may also benefit from chunking and citation controls, as described in this AI text summarizer tutorial.
Worked examples
Example 1: reducing repeated context
Assume a support workflow handles 20,000 requests in a planning period. Each request currently sends 6,000 input tokens and produces 700 output tokens. The application also retries 8 percent of requests. Treat these figures as illustrative inputs, not market benchmarks.
If a prompt revision, history compaction, and retrieval change reduce input to 4,200 tokens while output remains unchanged, input usage falls by 30 percent for the first attempts. The actual overall reduction will be smaller if retries still send the same large context, so measure both primary and retry calls. If the quality evaluation shows no meaningful loss, this is usually a stronger intervention than shortening every answer.
Example 2: output budgeting
Suppose a classification endpoint only needs a label, confidence field, and short explanation, but the model frequently returns several paragraphs. Add a structured output requirement, define a maximum useful explanation length, and enforce a server-side output limit appropriate to the task. Test cases should include ambiguous inputs, because an overly restrictive limit can remove information needed for a safe decision.
Compare four values before and after the change: average output tokens, schema-valid response rate, retry rate, and evaluator acceptance. If output tokens fall but schema failures rise, the limit is too aggressive or the prompt does not specify the required fields clearly. For tool workflows, validate arguments before execution and retry only the failed stage when possible. The tool calling tutorial provides a reliability-oriented framework for this pattern.
Example 3: model routing
Classify requests by difficulty, risk, and required capabilities. A smaller or faster model may handle routine extraction, normalization, or straightforward classification, while a more capable model handles ambiguous reasoning or escalation cases. Start with a conservative routing rule and evaluate borderline cases separately. The right comparison is not the rate of each model in isolation; it is the cost and success rate of the complete workflow, including fallbacks and human review.
Caching and batching can provide additional savings when the product requirements allow them. Cache stable results such as repeated document transformations, but define invalidation rules and avoid caching responses that depend on private, changing, or user-specific context. Batch offline jobs only when delayed completion is acceptable. Neither technique should be applied merely because it appears on an optimization checklist.
When to recalculate
Recalculate your LLM cost model whenever a pricing input, usage pattern, or quality requirement changes. At a minimum, review it on a regular quarterly cadence and after any major release. Trigger an earlier review when invoices increase unexpectedly, traffic shifts toward a different feature, conversation lengths grow, a new model becomes available, provider billing categories change, or retry rates rise.
Use a short production cost review:
- Refresh provider rates, model limits, cache rules, and batch assumptions from current documentation.
- Export recent usage and compare it with the forecast by workflow.
- Investigate the largest variance, not just the largest absolute bill.
- Rerun the quality set after prompt, routing, retrieval, or output changes.
- Update the cost sheet and record the decision, owner, date, and rollback condition.
Keep dashboards focused on cost per successful task, token distribution, retry percentage, latency, and quality. Revisit the prompt debugging guide when a token-saving change causes inconsistent behavior, and review prompt security testing before optimizing away instructions that protect data or tool use.
Effective AI inference cost reduction is an ongoing control loop: measure, change one input, test the outcome, and recalculate when the underlying assumptions move. That approach lowers avoidable token usage without confusing a smaller bill with a better system.
