Skip to content

Local AI for Business: When to Run AI In-House

Whether a company calls a model through a cloud API or runs it itself should not be a matter of taste. It follows from the data the model will see. This guide sets out a simple decision path: classify the data, then pick the operating model. Hardware sizing, costs and mixed set-ups have their own articles, linked below.

What does local AI mean for a business?

Local AI means a language model runs on hardware your organisation controls: in your own server room, in a rented rack only you manage, or on an edge device on the shop floor. Prompts, documents and answers stay inside your infrastructure. The alternative is sending requests to a model provider's API.

Most real decisions sit somewhere between those two extremes. People often equate "local" with "secure" and "cloud" with "risky". That is too simple. A badly run in-house server with open ports is riskier than a well-contracted cloud service. On the other hand, no contract helps if your own agreements say certain data may not leave your premises at all.

"Sovereign AI" goes a step further. It is less about where processing happens and more about whether you can switch models, providers or infrastructure without a business process grinding to a halt. You are sovereign when you can choose, not when you build everything yourself. The service page on sovereign AI describes that idea from a provider's point of view; this article stays with the neutral decision logic.

How does data sensitivity drive the operating model?

Start with the data, not the technology. Once you know what information an AI system will touch and how sensitive it is, the operating model almost picks itself. Teams that start with models and servers tend to argue for months without a clear use for either.

Four data classes are enough for most mid-sized companies:

  1. Public: website copy, product sheets, published price lists. Nothing is lost if they leak.
  2. Internal: work instructions, internal slides, routine correspondence. Unwelcome outside the company, but not seriously damaging.
  3. Confidential: personal data about customers and staff, cost calculations, quotes, contracts. Data protection law applies, and usually non-disclosure agreements too.
  4. Strictly confidential: design files, formulations, process parameters, source code, and anything your own customers allow you to process only on your premises.

Four operating models map onto these classes:

Operating modelWhere the model runsTypical fit
Vendor APIProvider's data centres, often outside the EUpublic, internal
EU cloudThird-party data centre inside the EUinternal, confidential
Own GPU serverYour premises or your own rackconfidential, strictly confidential
Edge deviceNext to the machine or on sitestrictly confidential, no network

Treat the table as a rule of thumb. Confidential data can go to an EU cloud if contracts and audits are in order. Internal data may still run locally because the process must work without an internet connection. What matters is that the mapping is a conscious, documented decision.

What are the four operating models in practice?

The options differ mainly in who could access your data, technically or legally, how quickly you can start, and who is responsible for keeping things running. Raw model quality is less often the deciding factor, because open-weight models now handle many routine business tasks well enough.

Vendor API

You send text to an endpoint and get an answer back. There is nothing to operate, you get large models immediately, and you pay per use. If personal data is involved, the provider usually acts as a processor. Article 28 GDPR then requires a binding contract covering, among other things, the subject matter, duration, nature and purpose of processing, and obliging the provider to follow your documented instructions.

If the provider is outside the EU, international transfer rules apply as well. Under Article 44 GDPR, such transfers are only allowed if the conditions of Chapter V are met. For the United States, the European Commission adopted an adequacy decision for the EU-US Data Privacy Framework on 10 July 2023, which allows data to flow to US companies that participate in it, as the Commission's page on EU-US data transfers explains. That settles the data protection question for personal data. It does not answer whether you want to hand over your engineering know-how.

EU cloud

The model runs in an EU data centre operated by a hosting company or a large cloud provider. That avoids a third-country transfer in the narrow sense, but not every access risk. If the operator is a US company, the US CLOUD Act of March 2018 applies. It requires providers to disclose data in their possession, custody or control "regardless of whether such communication, record, or other information is located within or outside of the United States", according to the statutory text published by the US Department of Justice.

For European buyers, Germany's federal security agency publishes the C5 criteria catalogue, a set of minimum requirements for secure cloud computing. Auditors attest whether a provider meets them, and customers are advised to request the audit report and assess it themselves, since the agency does not review the reports. Whatever attestation a provider shows you, read the report, not just the logo.

Own GPU server

One or more servers with graphics processors sit in your building or in a rack only you control. An inference server on them makes an open-weight model available to your applications. Data never leaves your infrastructure, and there is no third party that could be compelled to hand it over. In exchange, procurement, operation, updates and resilience are your job.

Sizing such a server is covered in Hardware for Local LLMs: Sizing a GPU Server Properly. For the strategic decision, one reference point is enough: the gpt-oss-120b model card states that this 117-billion-parameter open-weight model fits on a single 80 GB GPU thanks to quantisation, while the smaller gpt-oss-20b runs within 16 GB of memory. Both are released under the Apache 2.0 licence, which permits commercial use.

Edge device

A compact computer with a Jetson-class or server-class GPU sits next to a machine, in a warehouse or on a remote site. This makes sense where connectivity is unreliable, response times must be very short, or machine data must never leave the plant network. Models are smaller and tasks narrower: classifying fault messages, for example, or retrieving instructions from the equipment manual.

Which use cases favour local AI?

Local AI is worth considering when at least one of three conditions holds: the data is strictly confidential, you want to avoid depending on a single provider, or the process must run without internet access. If none applies, a vendor API or EU cloud is usually the more pragmatic route.

Typical cases in mid-sized companies:

  • Quotes and costing: an agent that drafts quotes from enquiries and past offers sees prices, margins and customer details. That is confidential at the very least. How such agents work in general is described on the page about AI agents for quotation.
  • Technical service: maintenance reports, fault logs and engineering drawings contain know-how competitors should not see. See AI agents for technical service.
  • Knowledge management: making manuals, test reports and project folders searchable often means giving the model your entire document store. See AI agents for knowledge management.
  • Production: assistants close to the line at sites with weak or deliberately isolated networks.

For marketing copy, translating public content or general research, there is little case for owning hardware.

Why do trade secrets matter as much as privacy?

Protecting know-how is often a stronger reason for local AI than data protection. Under German law, for example, trade secret protection depends on the owner having taken appropriate steps to keep the information secret. Pasting design data into external tools without rules can undermine that protection.

Specifically, section 2 of the Trade Secrets Act (GeschGehG) only protects information that is subject to secrecy measures appropriate to the circumstances. What counts as appropriate depends on the case. A clear rule on which data classes may enter which systems, enforced technically, is an important part of it. Local processing makes that rule easy to demonstrate: the data verifiably stays on your own systems.

The article Trade Secrets and AI: Protecting Know-how When Using AI Tools goes into more depth. This overview is not legal advice; for specific contractual questions, please consult your own lawyer.

What do you need in place before going local?

Before ordering hardware you need three things: a clearly described process with real sample data, a classification of that data, and a named person responsible for operations. Without all three, local AI tends to become an expensive experiment.

Data: collect 20 to 50 real cases from the target process, such as enquiries paired with the quotes you actually sent. These become your test set. Without them, choosing a model is guesswork.

Systems: work out where the model will get its information (ERP, document management, ticketing) and whether those systems offer an interface. The model is rarely the bottleneck; integration usually takes longer.

People: since 2 February 2025, Article 4 of the EU AI Act has required organisations deploying AI systems to take measures to ensure sufficient AI literacy among their staff. That applies whether the model runs locally or in the cloud, so plan training and usage rules from the start.

Security: Article 32 GDPR requires technical and organisational measures appropriate to the risk, such as encryption, the ability to restore systems after an incident, and regular testing. Running your own server does not remove these duties. It puts all of them on your desk.

How should you make the decision, step by step?

Work from process to technology: pick a process, classify its data, derive the operating model, test with real cases, and only then commit to hardware or contracts. That order prevents buying a server before there is anything for it to do.

  1. Pick a process. Text-heavy, with a clear input and a clear output, such as "enquiry in, draft quote out".
  2. Classify the data. Which classes flow through the process? The highest one present sets the bar.
  3. Derive the operating model. Use the table above for a first mapping. If one process mixes several classes, look at hybrid AI that combines local models and the cloud: routine requests go to an API, sensitive ones stay in-house.
  4. Test a model. Open-weight models can be tried on a spare workstation. Ollama serves a local HTTP interface at http://localhost:11434, including an OpenAI-compatible variant. For production use with many users, vLLM offers an OpenAI-compatible server started with vllm serve <model>.
  5. Judge the results. Let subject experts rate the answers to your test cases. If a local model falls short, that is a finding, not a failure.
  6. Plan operations. Only now size hardware or review cloud contracts, name owners, and plan updates, monitoring and backups.

Because vLLM and Ollama speak the same interface as many cloud APIs, you can build applications so that changing the operating model later is a configuration change. That is sovereignty in practice: the decision stays reversible.

# Example: request to a local OpenAI-compatible endpoint (Ollama, model gpt-oss:20b)
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-oss:20b", "messages": [{"role": "user", "content": "Summarise this maintenance report in three bullet points: ..."}]}'

What are the limits and risks of local AI?

Local AI moves responsibility to you. You gain control over data and supplier choice, but you take on operations, security and keeping models current. Some providers also offer their models only through an API, not as open weights, so on very demanding tasks a local model may perform worse.

The most common pitfalls:

  • Underestimated operations. An unmonitored server will fail eventually, usually at a bad moment. You need an operations plan with an owner, monitoring and cover from day one.
  • False sense of security. Local does not mean safe. If anyone on the network can reach the inference server without authentication, confidential answers are visible to all. Access control and logging are part of the job.
  • Untested model quality. Picking a model without test cases means discovering in production that it confuses your technical terms.
  • Oversizing. Buying hardware for models the process does not need ties up capital. Whether own servers beat APIs depends heavily on usage volume.
  • Licences overlooked. Not every open-weight model may be used commercially without restrictions. Check the licence on the model card before deployment.

Local AI is usually the wrong choice if you use AI only occasionally, process only public data, or have nobody who can look after a server. In that case, a vendor API with a proper contract is the more honest answer.

How do you get started?

Start with a single process and a one-page list of data classes. Test an open-weight model on an existing machine with real cases before you decide on servers, contracts or budgets. What you learn from that will be more useful than any general market overview.

A pragmatic four-week start could look like this. Week one: choose the process and collect test cases. Week two: classify the data and derive the operating model. Week three: test one or two open-weight models against your cases. Week four: decide between local, EU cloud, vendor API or a mix. Treat that timeline as a planning assumption for a contained process, not a guarantee.

If the answer is your own hardware, the guides on sizing GPU servers and on combining local and cloud models cover the next step. More background is available in the guides section and the glossary.

Last reviewed: November 2026. Legal notes in this article are for orientation only and are not legal advice.

Frequently asked questions

Does running AI locally make it GDPR-safe by default?

No. Location only changes who could access the data. Access rights, logging, retention and purpose limitation still have to be handled on your own server.

Are open-weight models good enough for everyday business work?

For well-defined tasks such as summarising, classifying or answering from your own documents, they often are. Check with real cases from your process before you buy any hardware.

Do we need an in-house IT team to run local AI?

Someone has to run the server, apply updates and notice outages. That can be an employee or a service provider, but it cannot be nobody.

Can we move from a vendor API to local models later?

Yes, if your application talks to an OpenAI-compatible interface. Local servers such as vLLM and Ollama expose one, so switching becomes mostly a configuration change.

Sources

  1. Art. 28 GDPR – Processor gdpr-info.eu
  2. Art. 32 GDPR – Security of processing gdpr-info.eu
  3. Art. 44 GDPR – General principle for transfers gdpr-info.eu
  4. European Commission – EU-US data transfers commission.europa.eu
  5. U.S. Department of Justice – CLOUD Act Resources justice.gov
  6. CLOUD Act – statutory text (U.S. Department of Justice) justice.gov
  7. German Trade Secrets Act (GeschGehG), section 2 gesetze-im-internet.de
  8. BSI – C5 Criteria Catalogue bsi.bund.de
  9. EU AI Act – Article 4: AI literacy artificialintelligenceact.eu
  10. Hugging Face – gpt-oss-120b model card huggingface.co
  11. vLLM – Online serving docs.vllm.ai
  12. Ollama – API reference docs.ollama.com
  13. Ollama Library – gpt-oss ollama.com