Skip to content

Open-Weight Models Compared: Which LLM to Self-Host?

Self-hosting a language model means choosing from dozens of model families, each with a new release every few months. Public leaderboards are a weak guide: they average scores across generic tests, not across your support tickets, purchase orders or inspection reports. This guide sorts open-weight models by the two questions that actually decide a deployment: what job does the model need to do, and what hardware will it run on? Everything below reflects the vendors' own model cards as of November 2026.

What are open-weight models, and why do businesses use them?

Open-weight models are language models whose trained weights you may download and run on your own infrastructure. Organisations choose them when data must stay in-house, when per-request costs need to be predictable, or when a model has to remain available in a fixed version for years.

"Open weight" is not the same as open source. What is published is the outcome of training. Most vendors keep the training data and training code private. The licence matters too: some models ship under Apache 2.0 or MIT, others under vendor-specific licences with extra terms.

Common business uses:

  • Internal assistant: answering questions about manuals, policies and product data without sending text to a cloud provider.
  • Document processing: reading delivery notes, invoices or test reports and turning them into structured fields.
  • AI agents: a model calls tools, such as an ERP lookup or a ticket system, and works through multi-step processes.
  • Coding support: engineering teams keep source code on their own machines.

Whether self-hosting beats a cloud API depends on data protection needs, utilisation and the operational effort you can carry. The page on sovereign AI sets out that trade-off.

How do you choose: task, hardware, licence?

Think of model selection as three filters applied in order. The task defines the capabilities you need, such as image input or reliable tool calls. Your hardware caps model size. The licence decides whether a technically suitable model is usable in your organisation at all. Only models that pass all three filters go into testing.

Filter 1: model size and architecture

Two designs dominate in 2026. Dense models use every parameter for every token; Qwen3.8-27B, for instance, has 27 billion parameters according to its model card. Mixture-of-experts (MoE) models hold many parameters but activate only some per token. Per its model card, Mistral Small 4 has 119 billion parameters with 6.5 billion active per token.

The practical consequence: an MoE model can respond nearly as fast as a small model, yet it still needs memory for all its parameters. Total size drives GPU memory; active parameters mostly drive speed.

Filter 2: a rough memory estimate

At 16-bit precision each parameter takes two bytes, so a dense 27-billion-parameter model needs roughly 54 GB for its weights alone. Four-bit quantisation brings that down to about a quarter. On top comes the KV cache, which grows with context length and the number of concurrent users. How that translates into GPUs and servers is covered in the guide to hardware for local LLMs; that guide sizes the machine, this one chooses the model.

Filter 3: licence

Apache 2.0 and MIT permit commercial use with light obligations. Vendor licences can add conditions. One example: according to the Llama 4 Scout model card, the Llama 4 Community License requires a separate licence from Meta for anyone with more than 700 million monthly active users at release. That threshold rarely matters for mid-sized companies, but it shows why every licence needs reading.

Which models fit which task?

For each common workload there are several credible candidates in late 2026. What separates them is less a leaderboard rank than concrete capabilities: context length for long documents, built-in tool calling for agents, image input for forms. The table lists candidates whose capabilities are documented in the vendors' model cards.

ModelDesignContextInputLicence
Qwen3.8-27Bdense, 27B262,144 tokenstext, image, videoApache 2.0
Gemma 4 31Bdense, 31B256K tokenstext, imageApache 2.0
Mistral Small 4MoE, 119B / 6.5B active256K tokenstext, imageApache 2.0
gpt-oss-20b / -120bMoE, 21B / 117Bnot stated on cardtextApache 2.0
DeepSeek-V4-FlashMoE, 284B / 13B active1M tokenstextMIT

Figures from the model cards for Qwen3.8-27B, Gemma 4, Mistral Small 4, gpt-oss-120b and DeepSeek-V4-Flash, checked November 2026.

Chat and internal assistants

An internal assistant needs to follow instructions reliably, write well in your working languages and answer quickly. Google's Gemma 4 model card states pre-training on more than 140 languages; Mistral Small 4 lists dozens, including English, French, Spanish, German and Italian. Language lists say little about tone and terminology, so test with your own documents.

RAG over your own documents

Retrieval-augmented generation (RAG) retrieves relevant passages and hands them to the model as context. You want a long context window and a model that stays within the supplied material. Qwen3.8-27B offers 262,144 tokens natively per its card; the mid-sized Gemma 4 models offer 256K. The embedding model used for retrieval is a separate, much smaller model chosen on its own merits.

Agents and tool calling

Agents need a model that calls functions against a fixed schema (tool use, or function calling) and returns structured output without drifting. The gpt-oss model card lists native function calling and structured outputs, with the caveat that the model must be prompted in OpenAI's harmony format. Mistral Small 4 lists native function calling and JSON output; Gemma 4 lists native structured tool use. For examples of agents inside real business processes, see AI agents for quotation, service and sales.

Code

Coding work needs long context, so whole files and repositories fit, plus the ability to work step by step inside developer tools. The Qwen3.8-27B card specifically highlights stronger autonomous planning and compatibility with agent harnesses and development tools. Always judge coding models on tasks from your own codebase rather than on public scores.

Images, forms and video

Processing delivery notes, rating plates or photos of damaged parts requires image input. Per their model cards, Qwen3.8-27B accepts images and video, while Mistral Small 4 and Gemma 4 accept images. Google adds that the smaller Gemma 4 sizes (E2B, E4B and 12B) also take audio.

What hardware class does each model need?

Hardware narrows the field more than any ranking. Three classes cover most deployments: a workstation GPU for pilots, a single data-centre GPU for departmental use, and multi-GPU servers for large MoE models or many concurrent users.

Class 1: workstation GPU

Smaller or quantised models live here. According to its model card, gpt-oss-20b runs within 16 GB of memory thanks to MXFP4 quantisation and targets local, lower-latency use. Quantised builds of Qwen3.8-27B or Gemma 4 26B A4B are also candidates. This class suits pilots with real data and individual specialist users.

Class 2: one 80 GB data-centre GPU

The gpt-oss-120b card states that the model fits on a single 80 GB GPU such as an NVIDIA H100 or AMD MI300X. Dense models in the 27 to 31 billion range run here at 8-bit precision with headroom for context and several users. For many departmental workloads, this is the cost-effective middle ground.

Class 3: multi-GPU servers

Large MoE models need more than one GPU. The serving example on the Mistral Small 4 card spreads the model across two GPUs with vLLM. DeepSeek-V4-Flash, at 284 billion parameters in mixed FP4/FP8 precision, belongs firmly in this class. It pays off when many people work concurrently or when a task demonstrably only succeeds with a large model.

What do you need in place before you start?

A self-hosted model is only as useful as what surrounds it. You need a test set of real tasks, serving software, connections to business systems and a named owner for updates, monitoring and model changes. Projects rarely fail on the model itself; they fail on operations.

  • Test set (evals): 30 to 100 real tasks with expected answers, for example typical customer questions, excerpts from inspection reports or coding tasks. Every model decision rests on it.
  • Serving software: for production the Qwen card recommends SGLang or vLLM; for single machines Ollama is common and is mentioned on the gpt-oss card as a local option. The overview of LLM inference servers compares them.
  • Integrations: agents need access to ERP, CRM, ticketing or document storage, with role-based permissions.
  • Ownership: who applies updates, who watches load and error rates, who signs off a model change?
  • Data protection: a local model can be operated in line with the GDPR if logging, access rights and retention are handled properly. Running on-premises does not settle those questions by itself.

How do you run a model selection, step by step?

A six-step process works well: define the task, build a test set, fix the hardware class, shortlist two to four candidates, test them all the same way, and keep observing in production. The result is a decision you can repeat cheaply every time a new model appears.

  1. Define the task precisely. Not "a chatbot", but "answer technical follow-up questions from maintenance manuals, with a citation".
  2. Build the test set. Real cases, including hard and ambiguous ones. Agree up front what a correct answer looks like.
  3. Fix the hardware class. What do you own, what is budgeted? That sets the maximum model size.
  4. Shortlist candidates. Two to four models that meet task, hardware and licence. Comparing one dense and one MoE model is often revealing.
  5. Test like for like. Same prompts, same quantisation, same serving stack. Measure accuracy, latency and memory.
  6. Observe in production. Collect failures and add them to the test set. In our own operations, those collected failure cases have proved the most valuable input when the next model change comes round.

For guidance on building the team and governance around this, see fractional head of AI.

What are the limits and risks of open models?

Open-weight models are not a free shortcut. They move cost from a cloud invoice into hardware, operations and evaluation. There are also functional risks: hallucinations, gaps in domain vocabulary and release cycles short enough that today's choice ages quickly.

  • Capability gap: for the hardest tasks, large hosted models may still do better. Only your own test set tells you whether that matters for your workload.
  • Leaderboard trap: public benchmarks and vendor figures do not measure your processes. A model near the top of a ranking can still struggle with your forms.
  • Formats and quirks: per its card, gpt-oss works correctly only with the harmony format. Other models need their own tool-call parsers. These details cost integration time.
  • Reasoning mode and latency: many current models think before answering. On Qwen3.8-27B this is on by default and can be switched off per request, per its card. Test this for interactive chat.
  • Licence changes: terms can change between generations. Re-check the licence of the exact version at every upgrade.
  • Short shelf life: new versions appear every few months. Treat model changes as routine, not as a project.

Licensing and data protection are described here for orientation only; for binding answers, consult a lawyer.

How do you get started?

Start small and measurable: one task, one test set, one workstation GPU, two models. After two to four weeks you will know whether quality is sufficient and which hardware class production needs. Only then does a server investment make sense.

A pragmatic starting kit as of November 2026:

  • Candidate A, dense: Qwen3.8-27B or Gemma 4 31B, quantised, on a workstation GPU.
  • Candidate B, MoE: gpt-oss-20b for agent and tool-calling tests on the same machine.
  • Serving: Ollama for the first trial; vLLM or SGLang once several users connect.
  • Measurements: accuracy on your test set, latency, and memory use at realistic context lengths.

Deeper guides on VRAM sizing, licences, non-English language quality and tool calling will appear in this section and are listed here automatically. Terms such as quantisation, MoE and KV cache are explained in the AI glossary, and the guides hub gives an overview of every topic.

Frequently asked questions

Is open weight the same as open source?

No. Open weight means the trained weights are downloadable. Training data and code usually stay closed, and the licence may add conditions.

Can I run a useful model without a GPU?

Small quantised models run on CPUs for experiments. For a team of users you will want at least one dedicated GPU.

How often should we re-evaluate our model choice?

A quarterly check against your own test set is usually enough. Switch only when a new model measurably handles your cases better.

Which licences allow commercial use?

Apache 2.0 and MIT generally do; vendor-specific licences can add conditions. Read the licence of each exact version; this is not legal advice.

Sources

  1. Qwen3.8-27B – model card (Hugging Face) huggingface.co
  2. Gemma 4 – model card (Google AI for Developers) ai.google.dev
  3. Mistral Small 4 119B – model card (Hugging Face) huggingface.co
  4. gpt-oss-120b – model card (Hugging Face) huggingface.co
  5. gpt-oss-20b – model card (Hugging Face) huggingface.co
  6. DeepSeek-V4-Flash – model card (Hugging Face) huggingface.co
  7. Llama 4 Scout – model card and licence (Hugging Face) huggingface.co