Hybrid AI: Combining Local Models and the Cloud Sensibly
Most debates about enterprise AI start as a binary choice: run models on your own hardware, or use a cloud provider. Most organisations end up somewhere in between, with some traffic on internal servers and some in the cloud. The real design question is the rule that decides which request goes where. This guide describes one rule that holds up in practice: route by data class first, then by task difficulty.
What is hybrid AI and what is it used for?
Hybrid AI means running language models on your own infrastructure (an on-premise LLM) and models from a cloud provider side by side, with every request assigned to one of them by explicit rules. The goal is to keep sensitive data inside your network while still using large cloud models for the hardest work.
It helps to separate this from two patterns that often go by the same name:
- Uncoordinated mix. One team has a cloud subscription, another runs a local model on a spare machine. There is no shared control point, no shared logging, and no guarantee that confidential material stays out of the cloud.
- Local as a backup. A local model waits in case the cloud provider has an outage. That is a resilience measure, not an architecture built around data sensitivity.
A real hybrid setup has one entry point that every request passes through, and a routing policy you can read, test and audit. Typical workloads include internal search over manuals and contracts, summarising support tickets and inspection reports, drafting quotes and replies, coding assistance, and agents that act inside ERP or CRM systems. Some of these touch payroll, pricing or engineering drawings; others only touch public information. That mix is exactly why a single blanket decision rarely fits.
How does routing by data class and task difficulty work?
Each request is judged on two axes. Data class decides where a request is allowed to go at all. Task difficulty decides which model, among the allowed destinations, should handle it. Data class is a hard boundary; difficulty is an optimisation. Never let the second override the first.
Axis 1: data class
Three or four classes are usually enough, ideally mapped to an information security policy you already have:
- Public: website product copy, published standards, general technical questions. Any provider is fine.
- Internal: work instructions, internal slide decks, routine email. Allowed to providers with a data processing agreement and hosting in your required region.
- Confidential: cost calculations, design files, customer contracts, HR records. Stays on your own infrastructure.
- Restricted: health data, material under special secrecy obligations or export control. Often limited to dedicated systems, sometimes not cleared for AI at all.
In the EU, a cloud provider that handles personal data on your behalf is usually a processor. Article 28 GDPR requires, among other things, a written contract, processing only on documented instructions and rules for sub-processors. Other jurisdictions have their own equivalents. Whether a specific provider is acceptable for a specific class is a case-by-case assessment, and this guide is not legal advice. For the wider privacy picture, see GDPR and AI agents.
Retention terms belong in the same assessment. Anthropic's Privacy Center, for example, states that API inputs and outputs are deleted from its backend within 30 days by default, with exceptions such as usage policy violations or separately agreed arrangements. Statements like this are the evidence behind each "allowed" cell in your policy.
Axis 2: task difficulty
Within the allowed destinations, difficulty decides:
- Easy: classification, pulling fields from a delivery note, short summaries, translating boilerplate. Smaller open models on your own hardware often handle these well.
- Medium: answering from a knowledge base with citations, drafting emails and quotes. Mid-sized open models or smaller cloud models.
- Hard: multi-step agents with tool use, long contracts with cross-references, complex coding. Large cloud models are frequently much stronger here.
Research supports routing by difficulty. The authors of RouteLLM train routers that pick between a stronger and a weaker model at inference time and report cost reductions of more than half in certain cases without a drop in response quality. That is a research result, not a forecast for your workload.
The routing matrix
Combining both axes gives a small matrix:
| Data class | Easy | Medium | Hard |
|---|---|---|---|
| Public | local small | cloud | cloud large |
| Internal | local small | regional cloud | regional cloud large |
| Confidential | local small | local medium | local large or refuse |
| Restricted | dedicated | dedicated | refuse |
The bottom-right corner is the honest part. Some tasks cannot be done at the quality you want with confidential data on the hardware you have. Someone then has to choose: buy bigger hardware, reduce the data before processing, or leave the task manual.
What components does a hybrid AI architecture need?
You need four parts: local inference servers, one or more cloud endpoints, a gateway that receives and distributes every request, and logging plus policy. The gateway matters most, because it is the only place where routing rules can be enforced centrally rather than trusted to each application.
Local inference servers. Tools such as vLLM and Ollama serve open models behind an OpenAI-compatible API, including /v1/chat/completions and /v1/embeddings. That lets applications use the same client library for local and cloud models. Which one fits depends mainly on the number of concurrent users and how much operational effort you accept, so test both under your expected load.
Cloud endpoints. One or more providers under suitable contracts, ideally with a regional deployment for the "internal" class. Each endpoint is registered in the gateway as its own destination, tagged with the classes it may receive.
Gateway. A service that exposes one API and routes internally. The LiteLLM router, for instance, groups several deployments under one model name and offers strategies including simple-shuffle, latency-based-routing, least-busy and cost-based-routing, plus custom routing logic. It also handles retries, fallbacks and temporary cooldowns for failing deployments.
Logging and policy. Who called which model, with which data class, and when? Without that record you cannot show your privacy officer, works council or auditors that the rules are actually followed.
Example: model groups per data class
The configuration below uses LiteLLM proxy syntax (as of October 2026). Applications call group names that carry the data class, not raw model names:
model_list:
# Confidential: own vLLM server only
- model_name: confidential-default
litellm_params:
model: openai/local-model-medium
api_base: http://gpu01.internal:8000/v1
api_key: os.environ/LOCAL_API_KEY
# Second local server as backup
- model_name: confidential-backup
litellm_params:
model: openai/local-model-small
api_base: http://gpu02.internal:8000/v1
api_key: os.environ/LOCAL_API_KEY
# Internal: cloud deployment in an approved region
- model_name: internal-strong
litellm_params:
model: azure/regional-deployment
api_base: os.environ/AZURE_REGION_ENDPOINT
api_key: os.environ/AZURE_REGION_KEY
router_settings:
# Fallbacks only within the same data class
fallbacks: [{"confidential-default": ["confidential-backup"]}]
The last comment is the key line. In LiteLLM's fallback model, one model group points to another. If a confidential group falls back to a cloud model, a single outage of your own server quietly breaks the whole data class policy. Fallbacks must never cross class boundaries.
What variants of hybrid AI exist?
Hybrid setups differ mainly in who decides the data class and where routing happens. Three variants are common. They can be combined and introduced one after another as your setup matures.
Application-side routing. Each application knows where its data comes from and calls the matching group directly; an HR knowledge assistant always calls confidential-default. Simple and easy to audit, but it relies on discipline in every development team.
Gateway-side routing. The application passes the data class as metadata. The gateway picks the destination and double-checks the request, for instance by detecting IBANs, employee IDs or project codes. Rules live in one place, but the gateway now carries custom logic that someone must maintain.
Learned routers. A small model estimates difficulty and sends the request to a small or large model, as in the RouteLLM work. Use this only for the difficulty axis. Never let a learned model decide the data class: a wrong call cannot be reversed once data has left your network.
A special case is redaction before the cloud call: a local model strips names, amounts or identifiers so that a sanitised version can go to a large cloud model. It sounds neat but is fragile, because context often still allows re-identification. For confidential material it is rarely sufficient on its own.
What are the prerequisites?
Before routing the first request, you need three things: a written data classification that explicitly covers prompts, an inventory of systems that feed data into prompts, and a named owner for the gateway and its rules. The technology is usually the smaller part of the work.
Data. Many policies classify documents, not prompts. An agent that combines a bill of materials from the ERP with a customer email produces a request of mixed sensitivity. The rule has to be: the highest class present decides the destination.
Systems. The local side needs server- or workstation-class GPU hardware (sizing is covered in hardware for local LLMs), network access from applications and monitoring for load and latency. The cloud side needs contracts, key management and sign-off from whoever owns privacy.
Organisation. Someone decides which model sits in which group, when models are replaced and how exceptions are requested. These roles fit into an existing AI governance framework rather than a new committee. Once agents enter business processes – for example AI agents for quotes, service and sales – business owners join that decision, because only they can reliably judge how sensitive their records are.
How do you build it step by step?
Start with classification and one use case, not with hardware. Building in stages avoids ordering expensive servers before you know which model sizes your tasks actually need.
- List and classify use cases. For each planned use, write down the data involved, its class and the task difficulty. This fills in the first version of your matrix.
- Gateway first. Even with only one cloud model connected, every application talks to the gateway from day one. Moving a workload to a local model later becomes a configuration change rather than a rewrite.
- Name groups by class. Names like
confidential-defaultinstead of raw model names make the policy visible in code and simplify reviews. - Pilot a local model on your most common confidential task. Test with real examples whether an open model on available hardware meets the bar. Only then decide on more hardware.
- Define fallbacks and refusals. For every group, decide what happens on failure: another destination in the same class, or a clear error.
- Turn on logging and review it. After a few weeks, the data shows which groups are saturated, where latency is too high and whether applications use the correct classes.
- Refine by difficulty. Once data classes run reliably, start splitting traffic within a class between smaller and larger models.
In our own operations, step 2 paid off most. Because every tool went through one shared interface from the start, individual tasks could later move between local and cloud models without touching the tools themselves.
What are the limits and risks?
Hybrid AI is more complex than either pure option. You operate two environments whose models behave differently, plus a control layer that can fail or be misconfigured. If you only have a few low-risk use cases, a single provider is often the better choice.
- Different model behaviour. Prompts tuned for a large cloud model often perform worse on a smaller local one. Each group needs its own tests, and prompts may need per-group variants.
- Wrong data classes. The architecture is only as good as the classification. If an application labels HR data as "internal", no gateway will catch it reliably.
- Cross-class fallbacks. The most common technical mistake, described above. Check after every configuration change.
- The gateway as a single point of failure. If it goes down, every AI application goes down. Give it redundancy and monitoring like any other core service.
- Logs as a new data store. Full prompt and response logging creates a database of confidential content. Log metadata by default and content only with a reason and a retention limit.
- Fixed hardware cost. Local servers only pay off above a certain volume. With low usage they sit idle most of the time.
How do I get started?
The most useful first step is a one-page matrix of data classes against task difficulty for your five most important planned AI use cases. It shows immediately whether you need local models at all, for which tasks and at what size, and whether the cloud covers the rest.
Next comes a gateway with one or two destinations that every new application uses. If confidential tasks must run locally, follow with a pilot of one open model on a single GPU server. For the strategic side – provider choice, operating model, ownership – the overview of sovereign AI covers running your own models. To build the matrix together with business teams and IT, a workshop is a practical format.
Once the rules exist, every new AI application comes down to two questions: which data class, and how hard is the task? The answer is already in the matrix.
Frequently asked questions
When is a hybrid setup not worth it?
When you have only a handful of low-risk use cases and no data that must stay in-house. Running two environments plus a gateway adds operational work that a single cloud provider avoids.
What hardware do I need for the local side?
A single server- or workstation-class GPU machine running an inference server such as vLLM or Ollama is enough for a pilot. Size further hardware only after testing open models on your real tasks.
Can an automatic fallback send requests to the cloud?
Only for request classes that are already allowed in the cloud. Fallbacks for confidential traffic should point to another internal model or fail with a clear error.
Should a machine learning model decide the data class?
No. Learned routers are useful for estimating task difficulty, but data classification should come from the calling application and fixed rules, because a wrong decision cannot be undone once data has left your network.
Sources
- LiteLLM: Router – load balancing, routing strategies, fallbacks docs.litellm.ai
- LiteLLM Proxy: fallbacks, retries, cooldowns docs.litellm.ai
- vLLM: OpenAI-compatible server docs.vllm.ai
- Ollama: OpenAI compatibility docs.ollama.com
- Ong et al.: RouteLLM – Learning to Route LLMs with Preference Data arxiv.org
- Art. 28 GDPR – Processor gdpr-info.eu
- Anthropic Privacy Center: how long organisation data is stored privacy.claude.com