"Which is the best model?" is the most asked question and the least helpful. The model that leads the ranking today does not lead in six months, the price changes before that, and what is best for writing an answer is waste for classifying a request. Teams that pick one model and build everything on it discover the cost of that decision on the first switch, when they realize the switch is not possible.
The question that does have an answer is a different one: which model for each task, with what fallback, tested against which cases, and switchable at what cost. This text covers each part.
Per task, not per project
An agent in production executes tasks of different natures within the same conversation. Classifying the customer's intent. Deciding which specialist receives the request. Extracting fields from a text. Writing the final answer. Summarizing the history so the context does not overflow.
Classifying and routing are short decision tasks: a small, cheap model does them well, and does them fast. Writing the final answer to a customer is where quality shows, and where it is worth paying more. Using the most expensive model for everything multiplies the cost without improving the result where it does not matter; using the cheapest for everything saves where it should not. The right design routes: each task goes to the model that resolves it at the necessary quality for the lowest cost.
The six criteria, per task
| Criterion | Question |
|---|---|
| Quality on the specific task | On my real cases, not on a public benchmark, does it get right what it needs to? |
| Cost per call | With the context this task actually carries, what does each execution cost? |
| Latency | Is the customer waiting? Classification can take seconds; an answer on WhatsApp cannot. |
| Context window | Do the history and the documents fit? If not, the task needs summarization first. |
| Where it processes | Does the region meet my sector's requirements for personal data? Is there a no-retention option? |
| Availability | When it goes down, what happens? Is there a second model ready to take over? |
The first criterion is the most misleading. Public rankings measure a generic task; yours is specific, in your own language, with your own vocabulary. The only measure that counts is the accuracy rate on your cases, and that requires having the cases.
How to test before shipping
- Build a set of real cases. Fifty to a hundred examples taken from the operation, with the expected answer. Including the hard ones: the customer who writes everything in capitals, the order with two items, the empty field. It is the most valuable asset in the project and the one nobody wants to build.
- Run each candidate against the set. Accuracy rate, total cost, average latency. Three numbers per model, per task.
- Version the instructions together with the model. The same instructions behave differently on different models. The test is of the instruction-plus-model pair, and the pair needs a version.
- Test live with the real integrations before promoting. A model that gets it right with test data and wrong with the real return from the ERP has not passed.
- Keep the set alive. Every case that went wrong in production joins the set. That is how the next model switch is tested against what has already caused trouble.
A practical rule: if you cannot state the accuracy rate of your current model on your own cases, you have no basis for saying the next one is better. A ranking does not replace that.
Fallback is not optional
A model provider goes unavailable, degrades, caps volume. When that happens at 2pm on a Monday, an agent that depends on a single model stops serving. A production design has a second model ready for every critical task, and the switch happens on its own, by rule: cost, quality and availability.
That requires the second model to have been tested against the case set as well, and the instructions to work on both. It is more work at the start, and it is what separates an operation from a dependency.
The silent lock-in
Model lock-in rarely comes from a contract. It comes from instructions written and tuned over months against one model's quirks, from an output format only that model produces the expected way, from tools declared in one provider's dialect. When the time comes to switch, the team discovers that switching means rewriting, and does not switch.
Three decisions that keep the switch possible:
- An abstraction layer between agent and provider. The agent calls "the writing model", not a specific provider. Which one it is gets resolved by the router.
- Tools defined neutrally, with types, and translated into each provider's format by the platform. Standards like MCP exist for that.
- Instructions with versioning and rollback, so that adapting to a new model is a new, tested version rather than an edit on top of what is live.
How Runflow handles this
The SDK is multi-LLM: the agent defines the task, and the model is chosen by configuration, not in code. The LLM Hub routes between providers, with automatic fallback by cost, quality and availability, and cost control by team and by user, without every team having to sign up for an API of its own. You decide where the data goes: providers with no retention, or your own model on your infrastructure.
The Prompt Studio keeps the instructions versioned, with variables and rollback, without a code deployment, and live testing with the real connectors happens before promotion to production. That is what makes it possible to treat a model switch as a new tested version, rather than as a rewrite.
