# Cost of Self-Hosted AI: When Own GPU Servers Beat APIs

> Self-hosting pays off when a GPU server stays busy for years and your team can run it. With small or spiky volume, the API stays cheaper. The costs people miss are staff time, power, redundancy and low utilisation.

Canonical URL: https://www.ruebstudio.de/en/guides/local-ai/self-hosted-ai-cost-vs-api  
Author: Dr.-Ing. Marcus Rüb, CEO  
Published: 2026-11-07

Most cost comparisons between self-hosted models and cloud APIs boil down to one line: the purchase price of a GPU server against a monthly invoice. That comparison misses the point. The two options have opposite cost structures. Self-hosting is mostly fixed cost; an API is almost entirely variable. This guide walks through both structures and gives you a model to populate with your own figures. It deliberately avoids quoting prices, because they change too quickly to be useful in an evergreen article.

## Why do self-hosted and API costs behave so differently?

With an API, you pay per token processed, so you only pay for actual use. With self-hosting, you pay for capacity: hardware, power, floor space and staff cost the same whether the server is busy or idle. That makes utilisation the single most important driver of your cost per request.

Picture two curves. API spend rises in a straight line with volume. Self-hosted spend looks like a staircase: roughly flat until one server is full, then a jump to the next step — another server, more GPUs, more operational effort. Where the line crosses the staircase is your break-even point, and it is different for every organisation.

Be clear about what is being compared. An open-weight model on your own hardware and a large frontier model behind an API do not automatically produce the same quality. A fair comparison assumes both options solve your task well enough. Otherwise you are pricing two different products.

## What goes into an API bill?

An API bill is input tokens times the input rate plus output tokens times the output rate, adjusted by discounts and surcharges. Prompt caching, batch processing, regional data residency and reasoning tokens all move the total. Looking only at the headline input price almost always gives a misleading estimate.

Using Anthropic's published pricing structure as a reference (checked November 2026):

- **Separate input and output rates.** Output tokens cost more than input tokens. For Claude Sonnet 5.5, [Anthropic's models overview](https://platform.claude.com/docs/en/about-claude/models/overview) lists an output rate five times the input rate. Workloads that generate long documents are therefore dominated by output.
- **Reasoning tokens.** Models that think before answering generate extra tokens. According to [Anthropic's thinking documentation](https://platform.claude.com/docs/en/build-with-claude/thinking), these are billed as output tokens even when the thinking text is not returned, so the billed count will not match what you see in the response.
- **Prompt caching.** Repeated prefixes such as system prompts, manuals or product catalogues can be cached. [Anthropic's pricing page](https://platform.claude.com/docs/en/about-claude/pricing) states that a cache hit costs 10% of the base input price by default, less on some models, while writing to the cache carries a premium.
- **Batch processing.** If an answer can wait, requests can be submitted in bulk. Anthropic's [batch documentation](https://platform.claude.com/docs/en/build-with-claude/batch-processing) describes a 50 percent discount on input and output, with most batches finishing within an hour and expiring after 24 hours. OpenAI's [Batch API guide](https://developers.openai.com/api/docs/guides/batch) also lists a 50% discount with a 24-hour completion window.
- **Data residency.** Pinning processing to a region can cost extra. Anthropic's pricing page lists a 1.1x multiplier on all token categories for US-only inference on Claude 4.6 and later models.

The practical lesson: measure your token profile, not just your token count. How long are typical inputs and outputs? How much of each prompt repeats and can be cached? Which jobs can tolerate a delay and run as batches? Only then do you get a realistic blended price per request.

## What does running your own GPU server actually cost?

A self-hosted inference server has four cost blocks: buying or leasing hardware; running costs for power, cooling and data centre space; staff to set it up and operate it; and spare capacity for failures and growth. Only the first block shows up on the vendor's invoice. The other three frequently decide whether the investment pays off.

**Hardware.** Spread the purchase price over the service life you realistically expect, convert it into a monthly figure and agree the depreciation schedule with your finance team. Besides the GPUs, include CPU, memory, fast storage, networking and chassis. For sizing a server to your model, see [Hardware for Local LLMs: Sizing a GPU Server Properly](https://www.ruebstudio.de/en/guides/local-ai/hardware-local-llm-gpu-server-sizing).

**Power and cooling.** Data centre GPUs draw a lot of power. [NVIDIA's specifications](https://www.nvidia.com/en-us/data-center/h100/) give a maximum thermal design power of up to 700 W for the H100 SXM and 350–400 W for the PCIe-based H100 NVL. A multi-GPU server quickly reaches several kilowatts, and all of that heat has to be removed. You need average draw, operating hours and your electricity rate including cooling overhead.

**Staff.** Someone has to maintain drivers, the inference stack, models and security patches, set up monitoring and respond to incidents. This is the most underestimated block, because it never arrives as an invoice — it arrives as engineering time that is missing elsewhere. The guide to [operating self-hosted AI](https://www.ruebstudio.de/en/guides/local-ai/operating-self-hosted-ai) lists what day-to-day operations involve.

**Headroom.** A single server is a single point of failure. If a production process depends on it, you need spare parts, a second node or a fallback, often the API itself. Peak load also requires capacity that sits idle most of the time.

## How much does utilisation matter?

Utilisation is the biggest lever in the whole calculation. A GPU server costs almost the same each month whether it handles one request or a million. The more useful work it does, the lower your cost per token. A server running at half capacity costs twice as much per token as one running flat out.

Modern inference servers extract much more from the hardware when many requests arrive at once. The [vLLM documentation](https://docs.vllm.ai/en/latest/) lists continuous batching of incoming requests, PagedAttention for efficient key-value memory management and prefix caching among its core features. For a comparison of the options, see the [overview of LLM inference servers](https://www.ruebstudio.de/en/guides/local-ai/llm-inference-servers-overview). In practice, throughput rises with the number of concurrent requests until memory or compute runs out.

Two planning consequences follow:

1. **Steady load is cheaper than peaky load.** A server classifying documents around the clock is used very differently from one answering chat questions during office hours and idling overnight and at weekends.
2. **Several workloads share the fixed cost.** If the same machine handles interactive requests by day and batch jobs such as summarisation or data extraction at night, the cost per token falls for both.

Don't guess utilisation. Run a pilot with realistic requests and realistic concurrency, and measure throughput in tokens per second.

## How do I calculate the break-even point myself?

Divide the total monthly cost of your own infrastructure by the tokens it actually processes each month, then compare the result with your blended API price per token. If the self-hosted figure stays below the API figure under realistic assumptions — and quality is comparable — self-hosting wins on cost.

Step by step:

1. **API side:** Measure monthly input, output and reasoning tokens. Account for the cached share and the batch share, and apply current list prices. The result is monthly API spend and a blended price per million tokens.
2. **Self-hosted fixed costs:** Monthly hardware depreciation plus hosting or room costs plus the relevant share of staff time.
3. **Self-hosted variable costs:** Power (average kilowatts × hours × rate), including cooling.
4. **Capacity:** Measured throughput × operating hours × realistic utilisation gives the tokens the server can actually deliver per month.
5. **Compare:** Self-hosted cost per million tokens against the blended API price.

A small Python sketch to plug your numbers into:

```python
# Fill in your own values - no sample prices here
api_price_per_m = 0.0          # blended, after caching and batch discounts
hardware_per_month = 0.0       # purchase price / depreciation months
hosting_per_month = 0.0        # space, maintenance, insurance
staff_per_month = 0.0          # share of engineering time
avg_kw = 0.0                   # measured draw incl. cooling
price_per_kwh = 0.0
tokens_per_second = 0.0        # measured throughput under your load
utilisation = 0.0              # share of time doing useful work (0-1)

hours = 730
power = avg_kw * hours * price_per_kwh
self_hosted_total = hardware_per_month + hosting_per_month + staff_per_month + power
m_tokens = tokens_per_second * 3600 * hours * utilisation / 1e6

if m_tokens > 0:
    self_hosted_per_m = self_hosted_total / m_tokens
    print(f"self-hosted: {self_hosted_per_m:.2f} per M tokens, API: {api_price_per_m:.2f}")
```

Then run a pessimistic and an optimistic scenario. If self-hosting only wins in the optimistic case, the cost argument is not robust enough to base a purchase on.

## Which costs get overlooked most often?

On the self-hosted side, people miss the costs that never appear on an invoice: staff time, resilience, model updates and evaluation. On the API side, they underestimate reasoning tokens, agent loops and tokenizer differences. Both options have blind spots, and an honest comparison has to look at each.

**Self-hosted blind spots:**

- **Model churn and evaluation.** New open-weight models appear constantly. Every switch means re-testing quality on your tasks and possibly rewriting prompts.
- **Security and patching.** The inference stack, GPU drivers and operating system need regular updates. An unpatched server with access to company data is a liability.
- **Idle time.** A server sized for peak demand runs below capacity most of the time.
- **Lead times and spares.** Data centre GPUs are not always available at short notice. A hardware fault can stall a process for longer than a typical API outage.

**API blind spots:**

- **Reasoning tokens.** Thinking models generate tokens you pay for but never see.
- **Agent loops.** An AI agent that calls tools resends the conversation so far as input at every step. Token volume compounds with each turn unless you cache or trim the history.
- **Tokenizer changes.** The same text does not produce the same token count on every model. Anthropic's [models overview](https://platform.claude.com/docs/en/about-claude/models/overview) notes that one million tokens equals roughly 555,000 words on its current tokenizer, compared with about 750,000 words on older models. Re-measure your volumes whenever you switch models.
- **Price changes.** API prices move between model generations, sometimes up, sometimes down. Plan with ranges rather than a single figure.

## When is self-hosting the wrong choice on cost grounds?

On pure cost, self-hosting is the wrong choice when volume is small or highly variable, when nobody can operate the stack, or when only the largest API models deliver the quality you need. In each case you either pay for capacity you don't use or accept weaker results.

Warning signs:

- You are early on and don't yet know your volume. The API is the cheaper place to experiment.
- Load arrives in rare bursts, such as month-end close or campaign launches.
- Your task needs very long context or complex reasoning that self-hostable models don't yet handle reliably.
- No one in-house can or wants to own operations.

Conversely, the strongest case for self-hosting is often not cost at all but data control, network isolation, latency or independence from a single vendor. Those are valid reasons, but name them for what they are instead of dressing them up as savings. The page on [sovereign AI](https://www.ruebstudio.de/en/sovereign-ai) covers how they fit together.

## Can you combine self-hosting and APIs?

Yes, and most organisations should at least consider it. A hybrid setup runs the steady base load on your own hardware and routes peaks, rare specialist tasks or the hardest requests to an API. Your server stays well utilised without being sized for the worst day of the year.

Common ways to split the work:

- **By data sensitivity:** Confidential documents stay on-premises; non-sensitive requests go to the API.
- **By difficulty:** A smaller local model handles routine work like classification or extraction; a large API model takes the hard cases.
- **By timing:** Interactive traffic through the API during the day, batch jobs locally overnight — or the reverse, depending on where capacity is cheaper.

The overview of [hybrid AI combining local and cloud](https://www.ruebstudio.de/en/guides/local-ai/hybrid-ai-local-and-cloud) explains how such architectures are built. For the calculation, split your token profile and cost each slice separately. Keep in mind that a hybrid setup needs a routing layer that decides where each request goes, and that layer takes engineering and maintenance time too.

## Where should I start?

Start by measuring, not buying. Log real token volumes for your use cases over a few weeks via an API, test an open-weight model on rented GPUs in parallel, and compare quality and throughput. Only with those figures does a break-even calculation mean anything.

A workable sequence:

1. **List your use cases** with volume, response-time needs and data sensitivity for each.
2. **Measure API usage** split by input, output, cached and batch tokens. Provider billing dashboards already show this.
3. **Check local quality:** run an open-weight model on rented GPUs against your real tasks before buying hardware.
4. **Measure throughput** with an inference server such as vLLM under realistic concurrency.
5. **Build the calculation with ranges:** pessimistic, expected, optimistic.
6. **Decide with an exit path:** keep an API fallback available even if you self-host.

Work through these steps and you end up with a calculation you can defend to finance and leadership rather than a gut call. You will also know which figure would have to change for the decision to flip — a significant API price cut, for example, or much higher utilisation from new workloads.

## Frequently asked questions

### Is there a token volume above which self-hosting is always cheaper?

No. The threshold depends on model size, hardware, utilisation, staff costs and current API prices. Only your own numbers can tell you where it sits.

### Are open-weight models free to run?

The weights are often free to use; running them is not. You pay for hardware, electricity, cooling, maintenance and the people who keep the stack current. Check each model's licence terms as well.

### Does self-hosting make sense for a single use case?

Rarely on cost alone. Data control, network isolation or latency are more common reasons. Running several workloads on the same hardware raises utilisation and improves the numbers.

### How often should the comparison be redone?

At least whenever API providers change prices, when a new model generation ships and before any hardware expansion. Prices and tokenizers shift noticeably between generations.

## Sources

- [Anthropic: Pricing (Claude API)](https://platform.claude.com/docs/en/about-claude/pricing)
- [Anthropic: Models overview](https://platform.claude.com/docs/en/about-claude/models/overview)
- [Anthropic: Batch processing](https://platform.claude.com/docs/en/build-with-claude/batch-processing)
- [Anthropic: Thinking](https://platform.claude.com/docs/en/build-with-claude/thinking)
- [OpenAI: Batch API](https://developers.openai.com/api/docs/guides/batch)
- [NVIDIA H100 Tensor Core GPU specifications](https://www.nvidia.com/en-us/data-center/h100/)
- [vLLM documentation](https://docs.vllm.ai/en/latest/)
