The model selection decision is one of the most consequential choices in AI agent system design — and one of the most commonly underthought. Most teams default to the most capable frontier model they have access to, apply it uniformly across all their agents and tasks, and accept the cost as the price of capability. This works, but it leaves significant cost savings on the table and, paradoxically, sometimes produces worse results because a smaller, more constrained model would have followed instructions more reliably than a large model that is tempted to improvise.
The current LLM landscape divides roughly into three tiers, each with distinct capability and cost profiles.
Frontier models — the latest GPT, Claude, and Gemini variants — offer the strongest reasoning capability, the best instruction following on complex and ambiguous tasks, and the largest context windows. They also carry the highest per-token cost, typically ten to fifty times more expensive than mid-tier alternatives. These models excel at tasks requiring multi-step reasoning, synthesis across large bodies of information, nuanced judgment, and handling of edge cases that require general knowledge.
Mid-tier models offer strong performance on well-defined tasks at significantly lower cost. Models in this tier handle structured extraction, summarization, classification, and code generation reliably when the task is well-specified and the input is reasonably clean. They struggle with the same tasks when instructions are ambiguous or inputs contain significant noise.
Small and fast models — including many open-weight models deployable on modest hardware — are purpose-built for speed and cost at the expense of general capability. These models excel at narrow, well-defined tasks: classifying a document into one of ten categories, extracting structured fields from a standardized form, generating short formatted text from a template. They are not suitable for tasks requiring broad reasoning, but they are dramatically cheaper and faster for the tasks they handle well.
The right approach to model selection is to start with the task, not the model. For each agent you are building, characterize the task along three dimensions: reasoning complexity, output structure requirements, and edge case frequency.
Tasks with high reasoning complexity — evaluating a contract clause against a regulatory requirement, generating a personalized investment recommendation, synthesizing findings across ten research papers — require frontier models. The complexity of the reasoning path exceeds what smaller models can reliably execute. Using a smaller model to save cost here will degrade quality in ways that create downstream business problems.
Tasks with well-defined output structures and moderate reasoning complexity — extracting fields from a document into a schema, summarizing a support ticket into standard categories, converting unstructured data into structured records — often perform equivalently on mid-tier models compared to frontier models when the prompt is well-engineered. The structured output constraint reduces the space of possible failures, making the task tractable for a less capable model.
Tasks with minimal reasoning requirements — classifying an email into a routing category, detecting whether a document matches a predefined pattern, generating a short formatted response from a template — can often be handled by small models. When these tasks run at high volume, the cost savings from using a small model compound significantly.
The cost difference between model tiers is not incremental — it is an order of magnitude or more. Frontier models typically cost ten to fifty times more per token than mid-tier models, and mid-tier models cost ten to fifty times more than small models at the high end. For an agent running ten thousand tasks per day, the difference between routing those tasks through a frontier model versus a small model is the difference between hundreds of dollars per day and a few dollars per day.
The right frame is not "what is the cheapest model I can get away with" but "what is the minimum capability required for this task to meet quality requirements." The goal is to match capability to requirement — not to minimize cost at the expense of quality, and not to pay for capability you do not need.
The only reliable way to know which model tier is appropriate for a given task is to evaluate it empirically. Build a benchmark dataset for each agent task that includes a representative sample of real-world inputs — and importantly, a deliberately constructed set of edge cases that represent the hard instances your agent will encounter in production.
Average-case performance is easy. Every model looks good on average inputs. The performance differences emerge at the edges: ambiguous inputs, noisy data, unusual formatting, adversarial edge cases. Benchmark on edge cases, not just the median input, and you will find the quality floor of each model candidate.
Establish a quality threshold before evaluating models. Define what success looks like for each task — a specific output structure, a specific accuracy level on a labeled test set, a specific rate of flagging the right inputs for human review. Then find the cheapest model that clears that threshold. This gives you a principled answer rather than an intuitive one.
Model selection also involves latency tradeoffs that matter for user-facing agents. Frontier models typically have higher latency than mid-tier models, and mid-tier models have higher latency than small models. For a customer-facing agent where response time is visible to the user, a model that returns a response in two seconds is meaningfully better than one that takes eight seconds even if the quality is comparable.
Async agents — those running background processing pipelines, nightly batch jobs, or non-real-time workflows — can tolerate higher latency. For these agents, the extra seconds per call are irrelevant to the user experience, and the capability advantage of frontier models can be captured at the cost of slightly longer processing times.
The most sophisticated teams implement a routing layer that classifies each incoming task and directs it to the appropriate model tier. The classifier itself is typically a small, fast model that takes the task input and produces a complexity score or category. Simple tasks route to small models, moderate tasks route to mid-tier models, and complex tasks route to frontier models.
This architecture achieves near-frontier-quality average output at a fraction of frontier cost because the majority of tasks in most fleets are simple enough for smaller models to handle correctly. The frontier model handles the tasks that genuinely require it and is not consumed by tasks that a smaller model could have handled just as well.
For high-volume, well-defined tasks, fine-tuning a smaller base model on task-specific examples can close the quality gap between the small model and a larger model significantly. Fine-tuning makes sense when: you have a clear, stable task definition, you can collect several hundred to several thousand high-quality labeled examples, and the volume justifies the investment in creating and maintaining a fine-tuned model.
The maintenance burden of fine-tuning should not be underestimated. When the task evolves, the training data needs to evolve with it, and the model needs to be retrained. For rapidly evolving tasks, a well-prompted mid-tier model with no fine-tuning may be more practical than a fine-tuned small model that requires retraining every time requirements change.
The model landscape evolves fast. A model that was mid-tier six months ago may be competitive with frontier capabilities today. A small model released last month may outperform a mid-tier model from a year ago. The right approach is to treat model selection as an ongoing optimization rather than a one-time decision.
AgentCloud supports flexible LLM endpoint configuration, making it straightforward to swap models at the agent level and run parallel evaluations across model options. If you are building a multi-agent system and want to think through model selection with us, we would be glad to help.
Join the waitlist. Early access members get 3 months free.
Request Early Access