Skip to content

Operating Self-Hosted AI Long Term: Updates, Monitoring, Ownership

Getting an on-premise LLM to answer questions on a GPU server is a project of days. Keeping it useful for years is a different job. Drivers age, new model versions appear, staff start relying on the output, and at some point the server will fail on a busy morning. This guide covers what day-to-day operation involves, who should own it, and how to plan for it before the pilot becomes a production service.

What does operating self-hosted AI actually involve?

Operation covers everything after go-live: keeping the platform current, changing models in a controlled way, watching both system health and answer quality, handling incidents and making ownership explicit. Pilots usually skip exactly these tasks, because their cost only shows up after weeks or months.

A self-hosted AI service has more layers than a typical business application, and each layer changes at its own pace:

  • Hardware and OS: GPU servers, firmware, the Linux distribution. Rarely changes, but failures hurt. Sizing the servers up front is covered in the guide to hardware for local LLMs; this article picks up after that.
  • GPU drivers and runtime: NVIDIA drivers, CUDA, the container runtime. Frequent releases with tight version dependencies.
  • Inference engine: vLLM, Ollama, llama.cpp or similar. These projects move fast; flags and metrics change between releases.
  • Model: weights, tokenizer, quantisation. Every change alters the behaviour of the whole system.
  • Application: prompts, the retrieval pipeline, connectors to the ERP, document management or ticketing system.
  • Data: the indexed documents, embeddings and access rights.

Operating the service means deciding, for each layer, who maintains it, how often it changes and how you will notice when something is wrong.

Why isn't standard server operations enough?

Standard monitoring tells you whether a service responds, not whether the response is right. A language model can be perfectly healthy from an infrastructure point of view and still give worse answers after an update. Self-hosted AI therefore needs a quality owner alongside IT operations.

Three properties set LLM services apart from most enterprise software:

  1. Behaviour, not just function. A new model can pass every technical check and still change the tone of a service report, drop a required output field or cite documents less precisely, up to hallucinations that did not occur before.
  2. Resources at the edge. GPU memory is scarce. Longer inputs, more concurrent users or a larger model can tip a stable service into overload.
  3. Cross-layer dependencies. A new inference engine release often requires a newer driver, which in turn requires a particular CUDA version. One upgrade becomes three.

The practical consequence: an update is finished when the quality check passes, not when the service restarts.

Who should own it?

At minimum, two roles must be named: technical operations, responsible for the platform and availability, and a business owner who judges quality and signs off model changes. In a smaller organisation these can be two part-time people, but they can never be nobody.

A split that works in practice:

TaskTechnical operationsBusiness ownerData protection / security
Drivers, inference engine, OSresponsibleinformedinformed
Model changeexecutesapprovesconsulted on new data flows
Test setsupportsresponsible–
Monitoring and alertsresponsiblesets quality thresholds–
Incident with wrong answerssupportsdecides on shutdowninformed
Document access rightsimplementsdefinesreviews

The single most important question is who may switch off a running service that is visibly producing wrong results. Write that down before go-live. In our own operations, giving that authority to the business owner, with technical operations carrying out the decision, has proved the clearest arrangement.

The NIST AI Risk Management Framework, released on 26 January 2023, structures this kind of work into four functions: Govern, Map, Measure and Manage. Govern (roles and policies) and Manage (handling risks over time) are the ones that matter most once a system is live. In July 2024 NIST added a Generative AI Profile, NIST AI 600-1.

How should updates and model changes work?

Platform updates belong in fixed maintenance windows; model changes follow their own approval process. The pattern is always the same: pin the version, test it in a staging environment against your own test set, approve it, switch over, and keep the way back open.

Platform updates in a maintenance window

Drivers, OS packages and the inference engine should be updated in a regular window, for example monthly outside core hours, with urgent security fixes as the exception. Before each update, record in the change log which driver and CUDA versions the new engine release expects.

Pin model versions

Always load a model by an exact version, never by "latest". For models on Hugging Face, the revision parameter does this. According to the huggingface_hub documentation, it accepts a branch, a tag or a commit hash, and a commit hash must be the full-length hash rather than the seven-character short form:

from huggingface_hub import snapshot_download

# Pin the model to a fixed commit instead of "main"
path = snapshot_download(
    repo_id="organisation/model-name",
    revision="<full-40-character-commit-hash>",
)
print(path)

The weights then sit in the local cache or in your own model store, and a server restart loads exactly the same model as yesterday.

Approve against your own test set

Before any model change, run a fixed set of real-world requests against both the old and the new model: quotation requests, ticket summaries, questions against the document archive. The business owner compares the results and approves. Public leaderboards cannot replace this step, because they know nothing about your documents, formats or terminology.

Keep a way back

Keep the previous model available for a few weeks after a switch, so you can roll back with a configuration change. Without that, every model change is a gamble.

What needs to be monitored?

Monitor two levels: the system, using standard metrics, and answer quality, using sampled reviews. For the system side, the four golden signals from Google's SRE book (latency, traffic, errors, saturation) map well onto LLM services, extended with GPU metrics such as memory and temperature.

Google's SRE book defines the four golden signals as latency, traffic, errors and saturation. Applied to an LLM service:

  • Latency: time to first token and total request time. The vLLM documentation lists histograms such as vllm:time_to_first_token_seconds and vllm:e2e_request_latency_seconds on the /metrics endpoint.
  • Traffic: running and waiting requests (vllm:num_requests_running, vllm:num_requests_waiting).
  • Errors: API error responses, aborted generations, timeouts in the calling application.
  • Saturation: KV cache usage (vllm:kv_cache_usage_perc, where 1 means 100%) and GPU memory.

For the GPUs themselves, NVIDIA's DCGM Exporter publishes telemetry in Prometheus format. Its default counters include DCGM_FI_DEV_GPU_UTIL (utilisation), DCGM_FI_DEV_FB_USED (framebuffer memory used), DCGM_FI_DEV_GPU_TEMP (temperature) and DCGM_FI_DEV_XID_ERRORS (the last XID error, an early sign of driver or hardware trouble).

A starting point for Prometheus alert rules:

groups:
  - name: llm-operations
    rules:
      - alert: LLMSlowFirstToken
        # p95 time to first token over a 10-minute window
        expr: histogram_quantile(0.95, sum(rate(vllm:time_to_first_token_seconds_bucket[10m])) by (le)) > 5
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "First token takes longer than 5 seconds (p95)"
      - alert: LLMQueueBacklog
        expr: max(vllm:num_requests_waiting) > 20
        for: 10m
        labels:
          severity: critical

Treat these thresholds as placeholders for tuning, not recommendations; sensible limits only emerge after a few weeks of measurement on your own workload. Note also that vLLM retires metrics on a fixed schedule: a metric deprecated in release X.Y is hidden in X.Y+1 and removed in X.Y+2. Check your dashboards whenever you upgrade the engine.

Answer quality

System metrics cannot detect a wrong answer. Schedule a regular sample, for instance a handful of real requests and responses each week, reviewed by the business owner. A simple feedback button in the application, letting users flag bad answers, adds a second signal.

The SRE book's rule on alerting applies here as well: every page should be actionable. If nobody can act on an alert, it belongs in a report, not in the on-call rotation.

How do you handle incidents?

Self-hosted AI has two kinds of incident: the service is down, or it is up and producing wrong answers. Both need predefined steps, a named contact and a decision about what users do in the meantime.

Outage. A server, GPU or the inference service fails. Normal IT processes apply: ticket, contact, recovery. The LLM-specific question is whether there is a fallback, such as a second server, a smaller model on other hardware or simply the manual process that existed before. Decide this per use case: an internal assistant may be allowed to be down for a few hours, while an agent in order intake usually needs a fallback.

Quality incident. The service runs but answers are wrong, perhaps after a model change, a change in the document base or because an integration delivers stale data. The business owner decides whether to keep running, roll back to the previous model or switch off. Afterwards, the cause goes into the change log and the triggering case goes into the test set, so it is checked on every future change.

Which services need a fallback depends on what they do; the overview of self-hosted AI use cases describes typical ones. For both cases, record who is notified, where the rollback instructions live and what users are told.

For most internal use, operational duties come mainly from data protection and information security law. If you deploy a high-risk AI system under the EU AI Act, Article 26 adds duties such as human oversight, monitoring and log retention. This section is orientation, not legal advice.

Under Article 26 of the EU AI Act, deployers of high-risk AI systems must, among other things:

  • assign human oversight to people with the necessary competence, training and authority,
  • monitor operation according to the provider's instructions for use, inform the provider of risks and suspend use where needed,
  • keep the automatically generated logs under their control for at least six months, unless other law sets a different period,
  • inform workers and their representatives before using the system in the workplace.

According to the version published on that site, these obligations apply from 2 December 2027 for high-risk systems listed in Annex III (as of October 2026). Whether your use case is high-risk at all depends on its purpose and needs legal review. Either way, the list matches what sound operations require anyway: named owners, monitoring and traceable logs.

Outside the EU, frameworks such as the NIST AI RMF set out comparable expectations on a voluntary basis. Where logs could reveal how individual employees work, check whether local law requires consulting staff or their representatives; in Germany, for example, the works council has a say over technical systems designed to monitor employees.

What should be in place before go-live?

Before go-live, roles should be named, versions pinned, monitoring running, a test set built and a rollback path documented. If any of these is missing, you are running an extended pilot, not a production service.

A compact checklist:

  • Organisation: technical operations and business owner named, deputies agreed, shutdown authority written down.
  • Versions: driver, CUDA, inference engine and model commit documented and reproducibly installable, for example through container images with fixed tags.
  • Monitoring: latency, traffic, errors and GPU metrics collected; every alert has a recipient.
  • Quality: a versioned test set of real, anonymised cases exists.
  • Incidents: instructions for restart, model rollback and the manual fallback are written.
  • Data: it is clear which requests and responses are logged, and for how long.

For the broader question of how to set up your own AI platform and which deployment options exist, see the overview of sovereign AI on your own infrastructure.

How do you introduce operations step by step?

The easiest route is to build operations into the pilot rather than bolting it on afterwards: versions and roles first, then monitoring, then the approval process for model changes. After a few weeks of data, refine thresholds and maintenance windows.

  1. Inventory: list every component with its version, from firmware to prompt.
  2. Pin: fix the model by commit hash and software by container tag. No automatic updates in production.
  3. Name owners: technical operations, business owner, deputies, shutdown authority.
  4. Wire up monitoring: add the inference engine's metrics endpoint and the GPU exporter to your existing monitoring stack, starting with generous thresholds.
  5. Build a test set: collect typical and difficult real cases together with the expected answer.
  6. Set the rhythm: a schedule for platform updates and an approval process for model changes.
  7. Rehearse: roll back to the previous model once on purpose and walk through a server outage.
  8. Refine: after a few weeks, adjust thresholds, sample size and logging.

What are the limits and risks?

The biggest risk is underestimating the effort. Self-hosted AI needs ongoing time for updates, testing and quality review, and the knowledge often sits with one person. Fast-moving software and hardware that does not scale on demand add to that.

  • Single point of knowledge: if only one person knows how the server is built, every holiday is an operational risk. Documentation and a deputy are requirements, not extras.
  • Software pace: inference engines change flags and metrics between releases. Teams that rarely upgrade end up facing one large, risky jump.
  • Model ageing: a model that fits today may fall well behind newer ones within a year, and each switch costs testing effort.
  • Capacity: GPU memory is a hard limit. More users or longer documents may need extra hardware, and procurement takes time.
  • Silent quality decay: when documents, forms or integrations change, quality often drops without anyone noticing. Without sampling, you find out from a complaint.

When self-hosting is the wrong choice: if nobody in-house has time for these roles and no external partner takes them on, a hosted service with contractually defined operations is often the more honest option, provided your data protection and confidentiality requirements allow it.

Where do you start?

Start with two things that cost little and clarify a lot: a list of every component with its version, and a written split of responsibilities. Monitoring and a test set follow. Only once all four are in place should a self-hosted AI system count as production.

A pragmatic sequence for the first month:

  • Week 1: component inventory, pin model and software, name owners.
  • Week 2: connect engine and GPU metrics to existing monitoring.
  • Week 3: build the test set and run the first review with the business owner.
  • Week 4: test the model rollback, agree maintenance windows, complete the documentation.

If you are planning your own AI platform from scratch, the sovereign AI overview covers the foundations, and the guides section collects further articles on infrastructure, cost and regulation.

Frequently asked questions

Can our IT team simply run the LLM server like any other server?

For hardware, operating system and backups, yes. Whether answers are still correct after an update is something IT cannot judge alone; that needs a business owner with their own test set.

How often should we switch to a newer model?

When a candidate model does measurably better on your own test set, or when the current one stops receiving fixes. Fixed schedules make more sense for drivers and inference software than for models.

Which metrics matter most on day one?

Time to first token, error rate, queue length and GPU memory. Add a weekly human review of real answers, because system metrics cannot see a wrong answer.

Does the EU AI Act apply to every internal LLM tool?

The Article 26 duties apply to deployers of high-risk systems, and many internal assistants are not high-risk. Whether a specific use case is high-risk needs a legal assessment.

Sources

  1. Google SRE Book: Monitoring Distributed Systems sre.google
  2. vLLM Documentation: Production Metrics docs.vllm.ai
  3. NVIDIA DCGM Exporter: Default Counters github.com
  4. Hugging Face Hub: Download files from the Hub huggingface.co
  5. NIST AI Risk Management Framework nist.gov
  6. EU AI Act, Article 26: Obligations of Deployers of High-Risk AI Systems artificialintelligenceact.eu