AI Without Internet: What Really Works Offline and What Doesn't
For many organisations the question is no longer whether to use AI but where the data goes when they do. Engineering drawings, HR files and pricing models are things a lot of companies would rather not send to someone else's data centre. Offline AI is a real option today for a good range of tasks. It also comes with trade-offs, and running it becomes your job.
Who is offline AI for?
Offline AI suits organisations whose data should not, or may not, leave the building: manufacturers with confidential designs, healthcare and HR teams with personal data, defence suppliers and operators of critical infrastructure. This guide is written for decision-makers who are not infrastructure specialists and want a realistic picture.
If you need the engineering detail of a fully isolated deployment, including network separation, model verification and operations, read the guide on air-gapped AI in isolated networks. This article answers the question that comes first: which level of isolation do you actually need, and what do you give up for it?
For the wider business case of running AI in-house, see the overview of local AI for business.
What are the three levels of "no internet"?
People mean three different things by offline AI: a model on your own server that still has internet access, a setup with no cloud services at all, and a fully disconnected network. Each level gives more protection and demands more operational effort.
Level 1: In-house server. The model runs on a machine you control. Staff prompts never leave the corporate network, but the server can still reach the internet to fetch models and security patches. This is the simplest level and is enough for many companies.
Level 2: No cloud. On top of that, you switch off every cloud feature, including the ones that local tools quietly offer. The Ollama runtime states that "Ollama runs locally. We don't see your prompts or data when you run locally." It also offers cloud models and web search, and the same FAQ explains that setting OLLAMA_NO_CLOUD=1 disables them. At this level, every tool is checked for outbound calls.
Level 3: Fully offline. The network hosting the AI has no route to the internet at all. Models and updates enter only through a controlled path such as verified removable media. This is the level for classified work, critical infrastructure and production networks that are already isolated.
| Level | Data leaves the site? | Server online? | Operating effort |
|---|---|---|---|
| In-house server | No | Yes | low |
| No cloud | No | restricted | medium |
| Fully offline | No | No | high |
What do you lose offline?
Going offline costs you three main things: live web search, access to hosted frontier models such as ChatGPT or Claude, and the convenience of automatic updates. A disconnected model also knows nothing about recent events. For internal work that is often acceptable; for some roles it is not.
Web search. Many assistants now look things up before answering. Offline, they cannot. In Ollama, for example, web search is a hosted service on ollama.com that requires an account and an API key, according to the web search documentation. Without a connection, the model is limited to its training data and your internal documents.
Hosted frontier models. According to Anthropic's documentation, Claude is available through the Claude API and through cloud platforms on AWS, Google Cloud and Microsoft Azure. All of these need a network connection. A fully offline setup uses open-weight models instead, which you can download and operate yourself. Whether they are good enough for your work is something only a test on real tasks will show.
Freshness. A model only knows what was in its training data. New regulations, prices or standards reach it only if you supply the documents.
Automatic updates. Operating system, driver and AI software patches no longer arrive by themselves. Someone has to fetch them outside, check them and install them.
What still works without a connection?
Anything a model can do with your own data keeps working: drafting and summarising text, searching internal documents, transcribing meetings, explaining code and pulling fields out of forms. All you need is an open-weight model and suitable hardware.
Open-weight language models. One example is OpenAI's gpt-oss family. The gpt-oss-20b model card lists the smaller model under the Apache 2.0 licence and says it runs within 16 GB of memory, and the gpt-oss-120b model card says the larger one fits on a single 80 GB GPU such as an NVIDIA H100. Several other vendors publish open model families as well.
Search over your own knowledge. You can give the model manuals, inspection reports or service procedures. It retrieves the relevant passages and answers with a reference to the source document. Offline, this is often worth more than web search, because the knowledge is yours.
Speech to text. OpenAI's Whisper speech recognition model is released with its weights under the MIT licence, according to the GitHub project, and runs locally. Meetings and dictation can be transcribed without any cloud service. The project notes that accuracy varies widely by language.
Tools with an offline switch. Developer libraries often have an offline mode too. With HF_HUB_OFFLINE=1, the Hugging Face library makes no HTTP calls to the Hub and only uses cached files, as the environment variable reference describes.
How do you choose the right level?
Three questions decide it: how sensitive is the data, what will the AI actually do, and who will run it long term? Pick the lowest level that meets your protection requirements, because every step up adds operating overhead.
- Data sensitivity: If it is enough that prompts stay on site, level 1 will do. If contracts or regulators require network separation, level 3 is mandatory.
- Type of work: If staff need live information such as market prices or new standards, full isolation is a real loss.
- Quality bar: Test open models on real tasks from your daily work before committing. Summarising a quotation is not the same as reviewing a contract.
- Hardware: Larger models need server GPUs. Check early whether your server room has space, power and cooling.
- Operations: Name the person who patches, swaps models and handles incidents.
This guide is for orientation only and does not replace legal advice on your specific obligations.
What does a sensible rollout look like?
Start small: one task, one open model, one test machine. Only once the results hold up should you decide on the isolation level and permanent hardware. That way you avoid buying expensive servers for a use case the model cannot handle well enough.
- Pick a task. Choose a process with clear value, such as searching service manuals or summarising inspection reports.
- Set the level. Agree with your data protection and security teams which of the three levels applies.
- Test models. Try two or three open models on real but non-sensitive sample data.
- Audit the tools. Check every piece of software for telemetry, cloud features and update checks, and turn them off.
- Plan operations. Decide how new models and patches will enter the network. For level 3, the air-gapped AI guide covers the steps.
- Roll out. Add more users and more tasks only after that.
How such a project can be set up on your own infrastructure is described on the sovereign AI page.
Where do offline projects usually go wrong?
The usual problems are hidden outbound connections, inflated expectations of small models and no plan for updates. All three are avoidable if you design for them from the start instead of discovering them after launch.
- Hidden connections. Many programs check for updates or send usage statistics at start-up. At levels 2 and 3 these must be disabled or blocked at the firewall.
- Inflated expectations. A model running on a modest workstation will not match the largest hosted models. Test honestly.
- Stale knowledge. Offline, both the model and your documents age. Schedule regular refreshes of the internal knowledge base.
- No owner. A server nobody looks after becomes a security hole.
- Unchecked downloads. Models fetched from the internet should be verified for origin and integrity before they enter the internal network.
Is offline AI worth it?
Offline AI pays off when your data must stay on site and the work revolves around your own knowledge. If your teams depend on live information from the web or on the most capable hosted models, isolation takes a lot away.
For most organisations, level 1 is the sensible starting point: an in-house server, no data leaving the site, and updates over the internet. Levels 2 and 3 make sense when regulation or risk demands them. In our own operations, solving a single task well before deciding on hardware and network separation has proved the more reliable order.
Frequently asked questions
Do we need in-house IT staff to run offline AI?
Someone has to own the server, patching and model changes. That person can be internal or external, but the responsibility must be assigned before go-live.
Can ChatGPT or Claude run offline?
No. Both are delivered as hosted services that need a network connection. Offline setups rely on models whose weights you can download and run yourself.
Does an offline model know about recent events?
No. It only knows its training data plus whatever internal documents you give it. Fresh information has to come from your own data sources.
How does a new model get into a network with no internet?
It is downloaded on a separate connected machine, checked, and moved across on controlled media or through a transfer gateway.
Sources
- Ollama FAQ (local operation, disabling cloud features) docs.ollama.com
- Ollama docs: Web search docs.ollama.com
- Hugging Face Hub: environment variables (HF_HUB_OFFLINE) huggingface.co
- Hugging Face: openai/gpt-oss-20b model card huggingface.co
- Hugging Face: openai/gpt-oss-120b model card huggingface.co
- GitHub: openai/whisper github.com
- Claude API overview (access via API and cloud platforms) platform.claude.com