Moving an LLM application to another provider or model family is more than replacing an endpoint and API key. This guide gives you a repeatable migration method for comparing API contracts, prompts, structured outputs, tool calls, performance, safety, and total operating cost before you switch production traffic.
Overview
An LLM application migration can fail in subtle ways. The new model may accept the same user request but interpret system instructions differently, produce valid-looking text instead of a required schema, call tools with different arguments, or respond with a latency profile that changes the user experience. A successful migration therefore treats the application as a system with measurable behavior—not as a single API integration.
Use this checklist to evaluate five layers:
- Transport: authentication, endpoints, SDKs, retries, timeouts, streaming, and error handling.
- Conversation contract: message roles, system instructions, content blocks, attachments, token limits, and conversation state.
- Capabilities: structured outputs, tool calling, vision or other input types, embeddings, batch processing, and model-specific features.
- Operational behavior: rate limits, latency, usage reporting, observability, safety controls, and incident procedures.
- Quality and economics: task accuracy, refusal behavior, consistency, cost per completed task, and the engineering effort required to maintain the integration.
Start by defining what must remain equivalent. Some teams need identical output schemas and tool behavior. Others only need the same business outcome and acceptable quality. This distinction determines how strict your regression tests and cutover criteria should be.
Before changing providers, inventory the current application. Record every model call, prompt version, input type, output format, tool, fallback, retry rule, and downstream dependency. A prompt version control process can make this inventory easier to maintain; see Prompt Version Control for Teams for a practical workflow.
How to estimate
Estimate a migration using separate scores for compatibility, quality, operating cost, and implementation effort. A single overall score can hide an unacceptable weakness, so keep the underlying values visible.
1. Estimate recurring usage cost
For each representative request type, calculate:
Monthly model cost = requests per month × average cost per request
If billing separates input and output usage, use:
Average cost per request = (average input units × input rate) + (average output units × output rate)
Keep the rates as variables in your worksheet rather than hard-coding them into application logic. Provider pricing and model-specific billing rules can change, and some applications also incur costs for tool calls, retrieval, embeddings, storage, or additional processing. Include those components when comparing complete workflows.
2. Estimate migration effort
Break the work into observable tasks instead of assigning one broad estimate:
- SDK or HTTP client changes
- Authentication and secret-management changes
- Message and content-format translation
- Prompt adaptation and versioning
- Structured-output and validation changes
- Tool schema and execution changes
- Streaming, retry, and error-handling work
- Regression-test creation and analysis
- Observability, dashboards, and rollback preparation
For each task, record an estimated engineering effort, a dependency, and a confidence level. Low-confidence items deserve an early spike. For example, testing whether a provider supports the exact output constraint your application needs is more useful than assuming compatibility from similar API terminology.
3. Compare quality using a fixed evaluation set
Create a test set that represents real traffic without exposing unnecessary sensitive data. Include routine requests, ambiguous inputs, long contexts, adversarial prompts, empty or malformed inputs, multilingual cases if relevant, and requests that should be refused or escalated.
Score each result against criteria appropriate to the application: factual correctness, instruction following, schema validity, citation or source behavior, tool-selection accuracy, tone, and safe handling. Measure pass rate and record failure categories. A model that has a similar average score but fails a critical workflow may not be a safe replacement.
For applications using retrieval, evaluate retrieval and generation separately. Changes in context limits, formatting, or instruction priority can alter how the model uses retrieved passages. A focused text summarizer and context-quality workflow can help isolate these effects.
Inputs and assumptions
Build a comparison table before implementation. The exact feature names vary by provider, so compare behavior and constraints rather than relying only on similarly named endpoints.
| Area | Questions to answer | Migration risk |
|---|---|---|
| Authentication | How are keys, projects, regions, permissions, and rotation handled? | Requests may fail or gain broader access than intended. |
| SDK and endpoint | Can the current client be adapted, or is a new adapter needed? | Retries, timeouts, and error types may change. |
| Message format | Are roles, content blocks, attachments, and conversation state represented the same way? | Instructions or user content may be interpreted differently. |
| System instructions | How are system-level rules prioritized and transmitted? | Prompt behavior may change without code errors. |
| Structured output | Are JSON, schemas, enums, or constrained responses supported as required? | Valid text may still fail downstream parsing. |
| Tool calling | Do tool names, schemas, argument formats, and call sequencing match? | Actions can be skipped, duplicated, or called with invalid arguments. |
| Streaming | What events arrive, in what order, and how are partial outputs terminated? | Interfaces may display incomplete or duplicated content. |
| Limits and quotas | What are the context, output, concurrency, and rate-limit assumptions? | Production traffic may be throttled or truncated. |
| Safety controls | Which application-level filters, moderation steps, and escalation rules are required? | Refusal and borderline-content behavior may differ. |
| Observability | Can you log request IDs, latency, usage, errors, model versions, and evaluation labels? | It becomes harder to diagnose quality drift. |
Use an adapter layer when practical. Keep business logic independent from provider-specific request construction, response parsing, and tool dispatch. The adapter should expose the application’s contract—for example, a typed response object and a normalized error model—while isolating provider differences behind it.
Do not copy a system prompt unchanged and assume portability. Review instruction ordering, delimiters, examples, output requirements, refusal handling, and assumptions about tool availability. Prompt changes should be versioned and tested alongside code. For security-sensitive applications, include the cases from this LLM prompt security testing checklist.
Worked examples
Example 1: Customer-support drafting
Suppose an application drafts support replies and routes complex cases to a human. The migration worksheet should track monthly requests, average input and output units, draft pass rate, escalation accuracy, schema-valid response rate, median and high-percentile latency, and the number of human edits per response.
Run the existing provider and candidate provider against the same de-identified ticket set. If the candidate produces acceptable drafts but changes escalation decisions, do not treat that as a prompt-only issue. Review the classification criteria, examples, output schema, and downstream routing code. The business outcome depends on all four.
For the cost estimate, calculate separate totals for drafting and classification if they use different model calls. Add expected retries and failed parses rather than assuming every request completes in one attempt. Customer-support prompt templates can provide a useful starting point for separating triage, drafting, escalation, and review tasks; see these support prompt templates.
Example 2: Retrieval-augmented question answering
For a RAG application, measure answer correctness, citation or evidence coverage, unsupported-claim rate, retrieval context size, time to first output, and total completion time. Keep the retrieved passages constant during the first comparison. Otherwise, a change in embedding model, chunking, or search ranking can obscure the effect of the generation model.
Next, test the full pipeline. Confirm that the candidate handles missing context, conflicting passages, long documents, and questions outside the knowledge base. A lower token cost is not an improvement if it creates more unsupported answers or human review work. For broader model trade-offs, use a structured AI model selection comparison.
Example 3: Tool-using automation
For an application that creates tickets or updates records, test tool selection, required arguments, validation failures, confirmation steps, and duplicate-call prevention. Use mocked tools first, then a controlled environment. Never make a provider migration depend on production side effects during initial testing.
Record whether the model asks for missing information, chooses the right tool, and stops when a tool returns an error. The tool calling tutorial covers design patterns for validation and secure execution.
When to recalculate
Revisit the migration estimate whenever an input changes, not only when the provider announces a major release. Recalculate after:
- pricing, billing units, quotas, or rate-limit terms change;
- the selected model is deprecated, replaced, or materially revised;
- your request mix, traffic volume, context length, or output length changes;
- new structured-output, tool-calling, streaming, or input requirements are added;
- quality benchmarks show a shift in pass rate, refusals, latency, or token usage;
- an incident exposes a weakness in retries, safety controls, logging, or rollback;
- your fallback provider or disaster-recovery plan changes.
Use a staged cutover. First, run offline regression tests. Next, send shadow traffic without exposing candidate outputs to users. Then release to a small, observable segment with a clear rollback switch. Compare quality, error rates, latency, usage, and business outcomes against the baseline. Expand only when the predefined acceptance criteria are met.
Keep the old integration available until you have verified rollback and reconciled any side effects from tools or asynchronous jobs. Track model, prompt, adapter, and evaluation-set versions in logs. An output-monitoring process for hallucinations, drift, latency, and feedback can help you detect problems after launch; see AI Output Monitoring.
Finally, maintain a migration worksheet with current assumptions, measured results, open risks, and the date of the last review. When prices, benchmarks, or traffic patterns move, update the inputs and rerun the same calculations. That simple habit turns provider switching from an emergency rewrite into a controlled engineering decision.