Choosing the Right Language Model for Your Agent
An agent is only as dependable as the language model making its decisions. Unlike a chatbot, it has to call tools, follow business rules and get several steps right in a row. So the question of which LLM to use is an architecture decision, not a matter of taste. It shapes cost, data protection and day-to-day operations. This guide sets out the criteria and a process for making the choice stick.
What does model selection mean for an AI agent?
Model selection means deciding, for each task your agent performs, which language model handles it, where that model runs and when you would replace it. For agents, fluent writing matters less than whether the model calls tools correctly, sticks to its instructions and asks when it is unsure instead of guessing.
A chatbot answers a question and a person reads the reply. An agent, by contrast, might create a draft quote in your ERP system, route a support ticket or pull the right inspection certificate from a document store. Each of those steps is a tool call with fixed parameters. A model that drops a required field in one case out of twenty is barely noticeable in a chat window. Inside an agent, it is an operational risk.
In practice, you are making three decisions:
- Which model: vendor, model family, size
- Where it runs: vendor API, a specific cloud region, your own servers
- How it is used: one model for everything, or different models for different steps
Which criteria should drive the choice?
Five criteria carry the decision: tool-use reliability, data control, cost per completed task, latency, and quality in the languages your business works in. Weight them against your process before you compare models. Otherwise the model with the loudest launch wins, not the one that fits.
Anthropic's guide to choosing a model names capabilities, speed and cost as the starting factors. For business agents, add data control and language coverage.
Tool-use reliability
This is the criterion that matters most for agents. Check whether the model picks the right function, fills every required parameter correctly, refrains from calling a tool when none fits, and stays consistent across several turns. Public benchmarks give a first signal. The Berkeley Function Calling Leaderboard, in its fourth version (page last updated April 2026), measures how accurately models call functions, including multi-turn dialogue and agentic tasks.
Repeatability tells you even more. The authors of τ-bench introduced a metric called pass^k, which measures how reliably an agent solves the same task across k repeated trials. The 2024 paper reported that even gpt-4o succeeded on fewer than half the tasks and scored below 25% on pass^8 in the retail domain. Those figures describe models of that time, but the lesson holds: one successful test run says nothing about steady-state operation.
Data control
Where are prompts processed, how long are they kept, and may the vendor use them for training? For API models, the answer sits in the vendor's contract and data documentation. As one example, OpenAI's documentation states that API data is not used for training by default, while abuse-monitoring logs are retained for up to 30 days; its European region, with storage and processing in the EEA and Switzerland, requires additional retention arrangements.
If data must not leave your network at all, such as engineering drawings, HR records or material under a non-disclosure agreement, open-weight models on your own hardware come into play. When that pays off is covered in Local AI for Business.
Cost per completed task
Price per million tokens is only part of the bill. An agent often calls the model several times per case, carries long context and retries after errors. Measure cost per successfully completed case instead. A cheap model that hands every fifth case back to a person can end up costing more than a pricier model that finishes almost everything.
Some vendors also offer controls within a single model. Anthropic, for instance, describes an effort setting that trades reasoning depth against latency and cost, and notes that tuning it is often a better lever than switching models (source).
Latency
If a person is waiting, say a field engineer on site, seconds count. If the agent runs in the background and prepares draft quotes overnight, latency hardly matters. Time the whole case including every tool call, not the response time of a single request.
Language quality
Many models are trained mainly on English text. If your agent writes to customers in German, French or Polish, or handles industry jargon, test that separately. A vendor claim such as support for "100+ languages and dialects" (from the Qwen3-32B model card) tells you little about how well a model summarises a technical report in Dutch.
What are the main options?
There are three broad routes: commercial models via an API, open-weight models on infrastructure you control, and models adapted through fine-tuning. Many production agents combine two of them, depending on the step and the sensitivity of the data involved.
API models
Commercial models from Anthropic, OpenAI, Google or Mistral are reached through an API. The upside: nothing to operate, fast access to new versions, and generally strong tool use. The downside: dependency on a vendor, data leaving your network, and models that get deprecated and must be replaced on the vendor's timetable.
Open-weight models
With open-weight models you download the weights and run them yourself, for example on an inference server such as vLLM. Read the licence. According to its model card, Qwen3-32B is released under Apache 2.0. The Llama 3.1 licence, by contrast, requires a visible "Built with Llama" notice when you distribute it and a separate licence from Meta for companies whose products exceeded 700 million monthly active users. Few mid-sized companies will hit that threshold, but it shows that open does not mean unconditional.
Tool use also needs more care with self-hosted models. vLLM converts the model's raw text into tool calls through a model-specific parser. Its documentation warns that without strict settings, arguments may occasionally be malformed, and that smaller Llama models often fail to produce the expected format.
Fine-tuning
Fine-tuning trains a model further on your own examples. It helps with fixed output formats, in-house terminology and repetitive classification. For up-to-date knowledge from your documents it is usually the wrong tool. Retrieval at runtime (RAG) fits better there, because you can change the content without retraining.
Mixing models
An agent does not have to rely on a single model. A common pattern lets a capable model handle planning and hard calls while smaller models do routine work: sorting emails, reading fields from delivery notes, shortening text. You can also route sensitive steps to a local model and the rest to an API. For tool decisions beyond the model itself, see Choosing AI Tools for Your Company.
What do you need in place before you choose?
You need three things: a clear description of what the agent does, a classification of your data by sensitivity, and a test set of real cases. Without a test set you are comparing impressions rather than models, and you will have no way to tell whether a later switch made things better.
Task description. Break the process into steps. A quotation agent might read the enquiry, find products in the catalogue, pull prices from the ERP system, write a draft and list open questions. Each step makes different demands on the model.
Data classes. Decide which data may go to an external vendor, which only to a specific region, and which only to your own servers. This often rules out whole groups of models before any testing starts.
Test set. Collect 30 to 100 real, anonymised cases with expected outcomes. Include hard ones on purpose: incomplete requests, contradictory details, cases where the agent should decline or ask a question. Anthropic's guide calls a good evaluation set the most important step in the whole process.
People. Agree who scores the results. Specialists from the process spot a wrong part number or an off tone faster than IT does.
How do you run the selection, step by step?
Work through six steps: weight the criteria, shortlist candidates, run the test set, repeat each case, calculate cost per case, and record the decision. With a test set ready, this takes a few days and can be rerun whenever a new model appears.
- Weight the criteria. Separate hard constraints (for instance, data must stay in the EU) from weighted factors such as cost and latency.
- Shortlist. Drop every model that fails a hard constraint. Use public leaderboards only to narrow the field to three to five candidates.
- Run the test set. Give each candidate every case with identical prompts and tool definitions. Score the tool calls as well as the final result.
- Repeat. Run every case several times. A model that solves a case in three runs out of five is not ready for an agent without human review, as pass^k shows.
- Calculate cost per case. Record tokens and the number of calls per completed case, then apply current vendor prices or the running cost of your own server.
- Record the decision. Note which model and version handles which task, and why. That makes future switches easier and answers questions from data protection or audit.
Two starting strategies are common, and both appear in Anthropic's documentation. Either begin with a small, fast model and move up only when tests reveal a gap. Or begin with a highly capable model, confirm the task can be solved at all, then work down towards smaller ones. For complex, highly autonomous agents the documentation favours the second route; you learn first whether the idea is viable at all.
What are the limits and risks?
The biggest risks are false confidence from benchmarks, vendor lock-in and quiet behaviour changes after a model update. No model choice replaces safeguards inside the agent itself: approvals, plausibility checks and logs remain mandatory.
Benchmarks are not your process. Public tests model airline bookings or online retail, not your bills of materials. The τ-bench maintainers themselves note in their repository that the original tasks are outdated and point to newer versions. Leaderboards age quickly.
Vendor lock-in. If prompts, tool definitions and output formats are tailored tightly to one model, you pay twice when you switch. Keep the integration swappable and the test set vendor-neutral.
Deprecations and updates. API vendors retire models on a regular basis. A newer model may score higher on your test set and still behave differently in individual cases. Without tests you will find out in production.
Running your own models. An open-weight model on your own hardware needs updates, monitoring and someone who understands the inference server. That is entirely doable, but it is not a side project.
Not legal advice. Questions about data processing agreements, international transfers or the EU AI Act belong with your data protection officer or legal counsel. The notes here are for orientation only.
When is each route the wrong one?
An API model is the wrong fit when contracts or regulation keep data inside your network. An open-weight model is the wrong fit when you do not want to run infrastructure. Fine-tuning is the wrong fit when the underlying knowledge changes often.
Sometimes the model is not the real problem at all. If master data is incomplete, interfaces are missing or the process itself has never been written down, every model will struggle. In our own operations, clear tool definitions with precise field descriptions have often made more difference than a model switch.
How do you get started?
Start with one well-defined process and a small test set. Try two or three models, including at least one that meets your data protection constraints. Within a few days you will know whether the task is solvable and which model handles it at an acceptable cost.
In practice:
- Pick a process with many similar cases, such as tickets in technical service.
- Classify the data involved and drop models that do not fit.
- Build 30 real test cases with expected outcomes.
- Run each candidate on each case five times and count the cases that came out right every time.
- Work out cost per completed case, then decide.
If sensitive data has to stay in-house, look at sovereign AI with open-weight models on your own infrastructure. For an overview of where agents fit, see AI agents; key terms are explained in the glossary.
Frequently asked questions
Is there one LLM that works for every business agent?
No. A model that handles customer support well can still stumble on technical inspection reports. Only tests with your own cases tell you which model fits.
Do agents always need the largest model?
No. Routine steps such as classifying, extracting fields or routing are often handled well by smaller models. Reserve the large model for planning and ambiguous decisions.
How often should we revisit the model choice?
Whenever your provider releases a relevant new model and whenever a model you rely on is deprecated. A ready-made test set turns that review into a routine task.
Can we switch models later?
Yes, as long as prompts, tool definitions and tests are not tied to one vendor. A thin abstraction layer and a vendor-neutral test set keep the switch manageable.
Sources
- Anthropic: Choosing the right model platform.claude.com
- Berkeley Function Calling Leaderboard (BFCL) V4 gorilla.cs.berkeley.edu
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains arxiv.org
- sierra-research/tau-bench (GitHub) github.com
- OpenAI API: Your data developers.openai.com
- Llama 3.1 Community License Agreement dev.meta.ai
- Qwen3-32B Model Card (Hugging Face) huggingface.co
- vLLM: Tool Calling docs.vllm.ai