# What Self-Hosted AI Actually Delivers: In-House Use Cases

> Transcription, document search with RAG, classification, data extraction and agents with a handful of well-defined tools run reliably on your own hardware today. For long multi-step reasoning and hard coding tasks, the strongest cloud models are still clearly ahead.

Canonical URL: https://www.ruebstudio.de/en/guides/local-ai/self-hosted-ai-use-cases  
Author: Dr.-Ing. Marcus Rüb, CEO  
Published: 2026-10-22

Plenty of organisations want the benefits of generative AI without sending design files, pricing, HR records or customer correspondence to an external provider. Self-hosted AI, meaning models that run on hardware you control inside your own network, has become a realistic way to do that. This guide sets out which use cases genuinely work in-house today, what components you need, and where cloud models still have the edge. Status: October 2026.

## What is self-hosted AI and why do companies use it?

Self-hosted AI means the model, the data and the processing all stay on infrastructure you operate, either on-premises or in a data centre you choose. Organisations use it when data must not leave their control, when per-request cloud billing is a problem, or when a system has to keep working without an internet connection.

It is possible because of open-weight models that anyone can download and run. A prominent example is gpt-oss, released by OpenAI under the Apache 2.0 licence. According to its [model card](https://huggingface.co/openai/gpt-oss-120b), gpt-oss-120b is designed to fit on a single 80 GB GPU, while gpt-oss-20b runs within 16 GB of memory. Alongside language models there are open models for speech recognition, text embeddings and document layout analysis.

The usual reasons for going in-house:

- **Confidentiality:** engineering documents, cost calculations and personnel data stay on your network.
- **Predictable costs:** you buy or lease hardware instead of paying per token.
- **Independence:** no disruption when a vendor changes prices, retires a model or rewrites its terms.
- **Disconnected sites:** factory floors, remote facilities or segregated production networks.

What self-hosted AI is not: a drop-in replacement for the most capable commercial models. Teams that expect that are disappointed. Teams that pick their tasks carefully get dependable tools.

## How does a self-hosted AI stack work?

A self-hosted AI system almost always has four layers: GPU hardware, an inference server that exposes the models, the models themselves, and an application layer that connects your data and feeds results back into existing workflows. Each layer can be swapped without rebuilding the others.

**Hardware.** For language models, GPU memory is the constraint that matters most. It determines which model sizes you can load at all and how many requests you can serve at once. Sizing is covered in detail in the guide to [hardware for local LLMs](https://www.ruebstudio.de/en/guides/local-ai/hardware-local-llm-gpu-server-sizing). A pilot often runs on one server with a single strong GPU; production use with many people needs more memory and headroom.

**Inference server.** This loads the model and provides an API. Two common choices:

- **Ollama** focuses on easy installation and suits workstations, prototypes and small teams. Its [documentation](https://docs.ollama.com/capabilities/tool-calling) describes tool calling via the chat endpoint at `localhost:11434/api/chat`, including parallel tool calls and multi-turn agent loops.
- **vLLM** is built for throughput on servers. The [vLLM documentation](https://docs.vllm.ai/en/latest/) lists PagedAttention for memory management, continuous batching of incoming requests and an OpenAI-compatible API server.

That OpenAI-compatible interface matters in practice: much tooling written for cloud APIs can be pointed at a local model by changing the base URL.

**Models.** Each job gets its own model: a language model for text and agents, an embedding model for search, a speech model for audio, and sometimes a layout model for tables and forms.

**Application layer.** This is where the value and most of the effort sit: connectors to file shares, ERP, ticketing or email; permission checks; logging; and an interface people will actually use.

## Which AI applications run reliably in-house today?

The applications that work well on your own hardware share a pattern: a clear input and an output someone can check. Turning speech into text, searching internal documents, pulling fields out of forms, sorting and summarising text, and running agents with a few tools all fall into that category. Each area below covers what works and where it stops working.

### Transcription and meeting minutes

Speech recognition is the most mature self-hosted use case. OpenAI published Whisper, code and weights, under the MIT licence. The [Whisper repository](https://github.com/openai/whisper) lists roughly 10 GB of GPU memory for the large model and about 6 GB for `turbo`, an optimised version of `large-v3`. That is enough to transcribe internal meetings, dictation or consented support calls on a server you own, then have a local language model draft the minutes, action items and decisions.

Limits: the repository itself notes that accuracy varies widely by language, specialist vocabulary and accents need correction, and identifying who said what is a separate processing step. A hands-on guide to local transcription with Whisper is in preparation in this section.

### Internal search and knowledge bases (RAG)

[Retrieval-augmented generation (RAG)](https://www.ruebstudio.de/en/glossary/retrieval-augmented-generation-rag) combines search over your own documents with a language model that writes an answer from the retrieved passages and cites them. The search side needs an embedding model. An open option is BGE-M3: its [model card](https://huggingface.co/BAAI/bge-m3) states support for more than 100 languages, inputs of up to 8,192 tokens, combined dense and lexical retrieval, and an MIT licence.

Local RAG fits manuals, work instructions, service documentation, standards excerpts and internal policies. The quality depends less on the language model than on how documents are prepared and how permissions are enforced, both covered in the guide to [enterprise RAG architecture](https://www.ruebstudio.de/en/guides/ai-agent-architecture/enterprise-rag-architecture): if someone cannot open a file on the share, the assistant must not show it to them either. The page on [AI agents for knowledge management](https://www.ruebstudio.de/en/ai-agents/knowledge-management) describes how such an assistant is scoped. A tutorial for a fully local RAG knowledge base is in preparation.

### Document processing and data extraction

Delivery notes, inspection reports, purchase orders and data sheets hold information that is still typed in by hand in many companies. In-house, this usually becomes a two-stage pipeline: a converter turns the document into structured text, then a language model extracts the required fields as JSON.

For stage one, [Docling](https://github.com/docling-project/docling) is an open tool that handles PDF, DOCX, PPTX, XLSX, HTML and more, exports to Markdown or JSON, is MIT-licensed and explicitly lists local execution for sensitive data and air-gapped environments. For stage two, a mid-sized language model is usually enough when the fields are fixed, because the task is narrow and easy to check against samples.

Limits: handwriting, poor scans and deeply nested tables remain error-prone. Nothing should be written to the ERP automatically without plausibility checks and a human sign-off for exceptions.

### Classification, summaries and drafts

Assigning emails to the right case, triaging tickets by urgency, condensing long reports and drafting replies from approved text blocks are tasks local models handle reasonably well. They are short and repetitive, and a person reviews the output before anything goes out. In service teams this is often the first step: incoming requests are pre-sorted before an [AI agent for technical service](https://www.ruebstudio.de/en/ai-agents/technical-service) assembles suggested fixes from the documentation.

### Agents with tool use

An [AI agent](https://www.ruebstudio.de/en/glossary/ai-agent) calls tools on its own, a pattern known as [tool use](https://www.ruebstudio.de/en/glossary/tool-use-function-calling): it reads a record from the ERP, checks stock levels, creates a draft. Open models now support this natively. The [gpt-oss model card](https://huggingface.co/openai/gpt-oss-120b) mentions function calling with defined schemas and structured outputs, and Ollama exposes tool calling through its chat API.

In our own operations, local agents are dependable when they have a small number of well-described tools and a short, predictable sequence of steps. As the tool list grows and chains get longer, errors pile up noticeably faster than with the strongest cloud models. A separate tutorial on building agents on a local model is in preparation.

## Where are local models enough, and where do cloud models win?

Local models are enough when the task is narrow, the output is verifiable and the knowledge comes from your documents rather than from the model itself. Cloud models stay ahead on long multi-step reasoning, difficult code, very long contexts and agents juggling many tools.

Epoch AI has measured the gap. Its [November 2024 report](https://epoch.ai/blog/open-models-report) found that the strongest open language models trailed the strongest closed ones by 5 to 22 months on benchmarks, concluding that frontier capabilities reach open models after about a year. That is a 2024 snapshot, but it is still a useful planning assumption: open models catch up while the frontier keeps moving.

A practical way to read this:

| Task | Self-hosted | Top cloud model |
|---|---|---|
| Transcription | usually sufficient | little advantage |
| Document search with citations (RAG) | sufficient with good preparation | better on hard questions across many documents |
| Fixed-field extraction | usually sufficient | edge on messy layouts |
| Summaries and drafts | mostly sufficient | often more polished |
| Agents with many tools | only within tight limits | clearly more robust |
| Complex code and analysis | limited | clearly stronger |

That is why many organisations settle on a hybrid setup: sensitive data and routine work stay in-house, while non-sensitive or particularly demanding requests go to a cloud model. How to design that split with digital sovereignty in mind is covered on the [sovereign AI](https://www.ruebstudio.de/en/sovereign-ai) page.

## What do you need before you start?

You need three things: hardware with enough GPU memory, data that is accessible and reasonably organised, and people who own the system after launch. Projects rarely fail on hardware. They fail on scattered, duplicated documents and on nobody being responsible once the pilot ends.

**Data.** Documents must be machine-readable and reachable, for example through SharePoint, a file share or a document management API. Outdated versions, duplicates and missing metadata hurt search quality, locally just as much as in the cloud.

**Systems.** Agents need APIs to the target systems such as ERP, CRM or ticketing, plus a service account with exactly the permissions the agent requires and nothing more.

**People and governance.** Someone has to update models, review failures and decide when to extend a use case. Where employee data is involved, many jurisdictions require consultation with staff representatives; in Germany, for instance, the works council usually has a say. This guide offers orientation only, not legal advice.

**Operations.** Self-hosted AI is infrastructure. It needs monitoring, backups, updates to the inference software and a plan for hardware failure.

## How do you roll it out step by step?

A staged approach works best: choose one narrow use case, build a test set from real examples, prototype with a mid-sized model, measure against the test set, and only then decide on larger hardware or further use cases.

1. **Pick the use case.** Frequent, rule-bound work involving confidential data with a checkable result, such as meeting minutes or fields from delivery notes.
2. **Build a test set.** 30 to 100 real, anonymised examples with the expected output. Without one, choosing a model is guesswork.
3. **Prototype with standard parts.** Ollama or vLLM, an open model, a short script. Skip the custom user interface for now.
4. **Measure and compare.** Run the same test set through a local model and, with non-sensitive data, through a cloud model. That shows the real gap for your specific task.
5. **Integrate with safeguards.** Connect target systems, enforce permissions, log everything, and require human approval for anything external or anything that triggers a transaction.
6. **Hand over to operations.** Clear ownership, monitoring and a recurring slot to test new model releases against your test set.

## What are the limits and risks?

The main risks are overestimating what the models can do, underestimating the operational load and assuming privacy is solved because the server is in your building. It is not: without permission checks, an internal search assistant can surface confidential files to every employee.

- **Hallucinations.** Local models invent things too. Citations in answers and spot checks are essential.
- **Fast-moving models.** New open models appear frequently. Design for swapping the model from day one instead of locking in a version.
- **Licences.** Open models come with very different terms. Apache 2.0 and MIT are straightforward; others include usage restrictions. Tools such as Docling may bundle models with their own licences.
- **Hardware ceilings.** GPU memory caps model size. A tight budget today can rule out a larger model next year.
- **Shadow IT.** Local models are easy to install on individual laptops, which creates unmanaged copies of sensitive data.
- **Adoption.** If the system is slower or less reliable than the cloud tool people already know, they will work around it.

A self-hosted system can be operated in line with the GDPR once access control, retention, logging and responsibilities are defined. Server location alone answers none of those questions.

## How do you get started?

Start with a task that is frequent, touches confidential data and produces output a person can check quickly. Transcription and internal document search are the obvious candidates: both are mature in self-hosted form and show value fast.

A realistic four-week start: week one, agree the use case and test set; week two, set up a prototype on existing or rented GPU capacity; week three, measure against the test set and compare with a cloud model; week four, decide on integration and ownership with the people who will use it.

If you first need to work out which tasks in your organisation are worth automating at all, a structured [AI workshop](https://www.ruebstudio.de/en/workshop) with the relevant teams is a good starting point. For more on planning, operating and choosing tools, browse the [guides](https://www.ruebstudio.de/en/guides), and look up terms such as RAG, embeddings or tool use in the [glossary](https://www.ruebstudio.de/en/glossary).

## Frequently asked questions

### Do I need my own data centre to run AI in-house?

No. Speech recognition and embedding models run on a single graphics card, and a mid-sized language model fits on one server with a capable GPU. Dedicated data-centre capacity only becomes necessary with many concurrent users or very large models.

### Are open models as good as the big commercial ones?

Not across the board. Epoch AI estimated in 2024 that open models reach frontier capabilities roughly a year after closed ones, which is often enough for narrow tasks such as extraction but frequently not for complex reasoning.

### Does running AI on-premises make it GDPR-safe by default?

No. Keeping data in-house helps, but access control, retention rules and documentation still have to be in place. A self-hosted system can be operated in line with the GDPR once those organisational points are settled.

### Can I mix self-hosted and cloud models?

Yes, and many teams do. A routing layer decides per request, based on data sensitivity and difficulty, whether the local model is sufficient or whether non-sensitive data may go to a cloud model.

## Sources

- [OpenAI: gpt-oss-120b model card (Hugging Face)](https://huggingface.co/openai/gpt-oss-120b)
- [OpenAI: Whisper repository and model overview](https://github.com/openai/whisper)
- [Docling Project: Docling document conversion](https://github.com/docling-project/docling)
- [BAAI: BGE-M3 embedding model (Hugging Face)](https://huggingface.co/BAAI/bge-m3)
- [vLLM documentation](https://docs.vllm.ai/en/latest/)
- [Ollama documentation: tool calling](https://docs.ollama.com/capabilities/tool-calling)
- [Epoch AI: How Far Behind Are Open Models? (November 2024)](https://epoch.ai/blog/open-models-report)
