Skip to content

LLM Inference Servers: Ollama, vLLM, llama.cpp and More

Running language models on your own infrastructure means choosing a piece of software that rarely gets attention: the inference server. It sits between the hardware and every application that needs a model. This guide explains what that layer does, which open-source servers matter today, how model formats and APIs fit together, and where each option stops being a good fit. Versions are checked as of 5 November 2026.

What is an LLM inference server?

An LLM inference server loads a model's weights into GPU or CPU memory, accepts requests over HTTP, runs text generation and streams the result back. It turns a model file into a shared service that chat interfaces, agents, scripts and internal tools can all call, instead of each one loading gigabytes of weights on its own.

The server is where your most expensive resource, GPU memory, gets shared between users. Get this layer wrong and even a good model feels slow or falls over under load.

Common reasons organisations run one:

  • An internal chat assistant. A front end such as Open WebUI talks to the server, and prompts stay inside your network.
  • Agents and automation. An agent that triages support tickets or drafts quotations calls the model through the API, not through a chat window.
  • Embeddings for search. A retrieval system turns documents into vectors, known as embeddings, using a model hosted on the same server.
  • Development. Engineers try out models on a workstation before anything reaches production.

Whether you should self-host at all is a strategic question, covered under sovereign AI and in the guide to local AI for business. This article assumes the answer is yes and asks which serving software to use.

How does an inference server work?

Every inference server combines four parts: a model format it can read, a compute engine for your hardware, a scheduler that interleaves concurrent requests, and an HTTP API. The projects differ mainly in which of these they optimise: ease of installation, or throughput across many users and GPUs.

Model formats: GGUF and the Hugging Face layout

Two formats dominate. The GGUF specification describes GGUF as "a file format for storing models for inference with GGML and executors based on GGML". A GGUF file packs weights and metadata into one file built for fast loading, and models are typically distributed in it already quantised, meaning stored at lower numerical precision so they fit into less memory.

The other is the Hugging Face layout: a directory with config files, the tokenizer and weights in Safetensors files. vLLM and SGLang load this by default. Ollama can import both, but it does not quantise GGUF models during import, so you quantise beforehand with llama.cpp's tools.

Memory management

Besides the weights, GPU memory fills up with the KV cache: the intermediate state for every conversation in flight. How a server manages that cache decides how many users it can handle at once. The vLLM README lists PagedAttention for this, plus prefix caching, which reuses work when many prompts start with the same system instructions.

Scheduling: continuous batching

Modern servers do not process one request at a time. With continuous batching, new requests join the running computation as soon as there is room. Both vLLM and llama.cpp's server (llama-server README) list it as a feature. Ollama handles concurrency through OLLAMA_NUM_PARALLEL, which the Ollama FAQ says defaults to one request per model.

The API: OpenAI compatibility as the common contract

Almost every server offers routes that mimic the OpenAI API, above all /v1/chat/completions. That shared contract is the most useful property of this layer: your applications and agent frameworks code against one interface, and the engine underneath can be replaced.

# The same request works against Ollama, vLLM, llama-server or SGLang;
# only the base URL and model name change
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [{"role": "user", "content": "Summarise this support ticket in three sentences: ..."}]
  }'

Compatible is not identical, though. Ollama's OpenAI compatibility page notes that /v1/responses is stateless only. vLLM needs a chat template for its chat endpoint, and per its documentation you must supply one if the model's tokenizer does not define it.

Which inference servers are worth considering?

Four open-source servers cover most self-hosting needs today. Ollama and llama.cpp prioritise simple setup and run on mixed hardware, including plain CPUs. vLLM and SGLang prioritise throughput on data-centre GPUs. Hugging Face's Text Generation Inference is now in maintenance mode and is not a sensible choice for new deployments.

ServerVersion (5 Nov 2026)LicenceModel formatStrength
Ollamav0.40.14MITGGUF, Safetensors importsimple operation, model management
llama.cpp (llama-server)build b12124MITGGUFlightweight, CPU and GPU
vLLMv0.32.0 (v0.33 in release candidate)Apache-2.0Hugging Face, GGUF and morethroughput, multi-GPU
SGLangv0.5.23Apache-2.0Hugging Facethroughput, wide hardware list

Version numbers come from each project's GitHub releases (Ollama, llama.cpp, vLLM). All four ship frequently, so check before you install.

Ollama: the easy on-ramp

Ollama wraps model execution in a tool with its own model library: ollama pull downloads a model, ollama run starts it, and a Modelfile defines custom variants. According to the FAQ, it binds to 127.0.0.1 on port 11434 by default and unloads idle models after five minutes. Several models can stay loaded at once if memory allows.

Alongside its native /api/ routes, Ollama provides OpenAI-compatible routes for chat, completions, models, embeddings and audio. One detail matters for anyone exposing it on a network: the local server ignores the API key that OpenAI clients send, according to its compatibility documentation. Access control has to come from something in front of it, such as a reverse proxy. A hands-on tutorial on running Ollama for a team is planned for this section.

llama.cpp: the lean engine

llama.cpp is a C/C++ inference library that ships an HTTP server, llama-server. Its README lists OpenAI-compatible chat completions, responses and embeddings routes, a reranking endpoint, parallel decoding for multiple users, continuous batching and schema-constrained JSON output.

# Serve a GGUF model, all layers on the GPU, four parallel slots
llama-server -m ./models/qwen3-8b-q4_k_m.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl all -np 4 --api-key "$LLM_API_KEY"

Unlike Ollama, llama-server checks API keys itself via --api-key. What it lacks is model management: you download GGUF files yourself and run one process per model. It fits teams that want direct control over every parameter, or that run on CPU-only servers or with only some layers offloaded to the GPU via -ngl.

vLLM: throughput for many users

vLLM is designed to keep GPUs busy under heavy concurrent load. Its README lists PagedAttention, continuous batching, prefix caching, speculative decoding, many quantisation formats (including FP8, AWQ, GPTQ and GGUF) and tensor and pipeline parallelism across GPUs. Beyond NVIDIA, it supports AMD and Intel GPUs as well as CPUs.

# Serve a Hugging Face model with an API key from the environment
vllm serve Qwen/Qwen3-8B --api-key "$LLM_API_KEY"

The OpenAI-compatible server documentation covers completions, chat completions, responses, embeddings and audio transcription. Be careful with authentication: --api-key only protects routes under /v1, /v2 and /inference, and the documentation explicitly warns against relying on it alone. A side-by-side comparison of Ollama and vLLM for testing versus production is planned.

SGLang: a second high-throughput option

SGLang targets the same high-throughput use case as vLLM. Its quickstart launches a model with sglang serve and queries it via /v1/chat/completions. The README lists NVIDIA data-centre GPUs from A100 to GB300, as well as DGX Spark and Jetson Orin, plus AMD Instinct, Google TPU, Intel and Apple Silicon.

Treat SGLang as a serious alternative to benchmark against vLLM. Which engine performs better depends on the model, the hardware and your request mix, so general claims are of limited use. Measure with your own prompts.

Text Generation Inference: legacy only

Hugging Face's Text Generation Inference (TGI) was once a common choice. Its README now states that the project is in maintenance mode, accepting only minor fixes, and recommends vLLM, SGLang, llama.cpp or MLX instead. The latest release, v3.3.7, dates from December 2025. Existing installations keep working; new ones should pick something else.

What about the user interface?

An inference server has no notion of company users, groups or chat history. That is the job of a front end such as Open WebUI, which, according to its README, connects to local models through Ollama or to local and cloud models through OpenAI-compatible APIs. It is a separate layer, not a replacement for the server; a dedicated guide on setting up sign-in and permissions is in preparation.

What do you need before you start?

You need three things: hardware with enough GPU memory for the model plus concurrent requests, a realistic picture of how the system will be used, and someone who can run Linux servers, containers and networking securely. Self-hosting projects rarely fail on model quality. They fail on operations.

Hardware. Memory demand depends on model size, quantisation and context length multiplied by the number of parallel requests. A small quantised model runs on a workstation GPU; large models for many users need data-centre cards or several GPUs with tensor parallelism. The guide to sizing GPU servers for local LLMs walks through the calculation, including the KV cache, which can outgrow the weights under long contexts and many users.

Load profile. How many people will use it at the same time? Short chat questions or long documents? Overnight batch jobs? These answers decide the server more reliably than any feature list.

Ownership. Someone has to apply updates, approve new models, manage access and notice outages. Some of these projects release several times a month. Without a plan, you end up with an outdated, unpatched install within a year. The guide on operating self-hosted AI covers updates, monitoring and responsibilities.

Data and licences. Keeping data in-house does not automatically make a system compliant; it can be operated in line with the GDPR or other privacy laws only if logging, retention and access rights are handled properly. Also check the model's own licence, which is separate from the server's licence. This is orientation, not legal advice.

How do you choose, step by step?

Work from the use case to the software, not the other way round. Describe usage and load first, pick a model, then choose the server that runs that model well on your hardware. A short load test with real prompts settles the decision better than any comparison table, including the one above.

  1. Define the use case. Company-wide chat, a specific agent, or embeddings for search? Each has different latency and concurrency needs.
  2. Pick a model. Choose one whose licence allows commercial use and that performs well in the languages you need. Check which formats it is published in.
  3. Shortlist a server. A small team with one GPU can start with Ollama or llama.cpp. Many concurrent users, several GPUs or large models point to vLLM or SGLang.
  4. Build a prototype. Install the server, bind it to localhost only and connect one application through the OpenAI-compatible API.
  5. Load test. Use realistic prompts and the expected number of concurrent users. Record time to first token, tokens per second and memory usage.
  6. Secure it. Put a reverse proxy with TLS and authentication in front, restrict network access and decide what gets logged.
  7. Hand over to operations. Agree update cadence, monitoring, a release process for new models, and who is responsible.

Because the servers share a similar API, switching later is manageable. A sensible path is to prototype on Ollama and move to vLLM or SGLang for production once usage grows. That works best if your applications avoid server-specific extensions.

What are the limits and risks?

The main risks sit in operations rather than in model quality: servers exposed without authentication, underestimated memory needs, gaps in API compatibility and fast-moving releases. On top of that, self-hosted open models can fall short of the largest hosted models on hard reasoning tasks.

Exposed endpoints. Ollama ignores API keys, and vLLM's key does not cover every route. Binding a server to 0.0.0.0 without an authenticating proxy in front makes the model, and depending on the routes some management functions, available to anyone on that network.

Memory and stability. A server that feels fast with one tester can reject requests or slow down sharply once the KV cache fills up under real load. The usable context window and the number of parallel requests have to match the hardware.

API differences. "OpenAI-compatible" covers the core, not every option. Tool calling, structured output, streaming details and the Responses API behave differently from server to server. Test exactly the features your application relies on.

Maintenance. These projects move quickly and configuration flags change between versions. Pin versions, test upgrades before rolling them out and read the release notes.

When self-hosting is the wrong call. If you need the capabilities of the largest models, have only occasional load, or lack staff to run servers, a hosted API under a suitable contract may serve you better. Mixed setups are common too, with sensitive workloads local and the rest in the cloud; see hybrid AI with local models and cloud.

How do you get started?

Start small and measurable: one model, one server, one use case, reachable only inside your network. In our own operations, the most useful early decision was to fix the API layer first and build every application against the OpenAI-compatible endpoint, so the engine underneath stays swappable.

A practical first step on a Linux workstation with a GPU:

# Install Ollama, pull a model, check the OpenAI-compatible route
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen3:8b
curl http://127.0.0.1:11434/v1/models

Next, connect a first application, for example an internal document search of the kind described under AI agents for knowledge management. When usage grows, load test vLLM or SGLang on the same hardware before committing.

For the wider questions of which data must stay on your premises, how to size infrastructure and who owns operations, see sovereign AI. If you want to build this capability inside your organisation, the page on a fractional head of AI explains how ownership and skills can be set up. Related terms such as on-premise LLM are explained in the glossary.

Frequently asked questions

Can I self-host an LLM without a GPU?

Yes. llama.cpp and Ollama run on CPUs, but generation is much slower. Once several people use the system at the same time, or the model is large, a GPU with enough memory becomes a practical requirement.

Will my existing OpenAI client code work against a local server?

Usually, for the core calls. All four servers covered here expose routes such as /v1/chat/completions, so changing the base URL, model name and key is often enough. Test tool calls, structured output and streaming separately, because those details differ.

Should I still deploy Hugging Face Text Generation Inference?

Not for new projects. Its README states that TGI is in maintenance mode and points users to vLLM, SGLang, llama.cpp or MLX instead.

What is the difference between GGUF and Safetensors?

GGUF is a single-file format from the ggml project, usually already quantised, and is what llama.cpp and Ollama run natively. Safetensors files sit inside a Hugging Face model directory and are the usual input for vLLM and SGLang.

Sources

  1. Ollama Docs: OpenAI compatibility docs.ollama.com
  2. Ollama Docs: FAQ docs.ollama.com
  3. Ollama Docs: Importing a model docs.ollama.com
  4. Ollama releases (GitHub) github.com
  5. vLLM Docs: OpenAI-Compatible Server docs.vllm.ai
  6. vLLM README (GitHub) github.com
  7. vLLM releases (GitHub) github.com
  8. llama.cpp: llama-server README github.com
  9. llama.cpp releases (GitHub) github.com
  10. GGUF specification (ggml) github.com
  11. SGLang Docs: Quickstart docs.sglang.io
  12. SGLang README (GitHub) github.com
  13. Text Generation Inference README (GitHub) github.com
  14. Open WebUI README (GitHub) github.com