The Complete CIO's Guide to On-Premise AI

On-premise AI means running AI infrastructure and models inside an organisation's own data centre, colocation cage, or private cloud, under its own physical and administrative control, rather than calling a public cloud AI API or a managed SaaS product. The core trade-off is straightforward: higher upfront capital and operational ownership in exchange for data control, predictable unit economics at scale, and independent verifiability. It is a spectrum rather than a binary choice, running from single-node evaluation rigs through to fully air-gapped, high-assurance clusters, and most organisations land somewhere in the middle.

This guide is written for the person who has to make that call and then defend it: a CIO, CTO, head of infrastructure, or IT director weighing owned infrastructure against continuing on cloud APIs, and a platform engineer who then has to build it. It covers the decision drivers, the hardware and networking architecture, security and identity, audit and regulatory obligations, deployment topologies, cost modelling, staffing, migration, and procurement. Each section is deliberately complete enough to stand alone and restrained enough to leave detail to dedicated deep dives.

Why do organisations choose on-premise AI?

Four drivers show up consistently in why organisations move AI workloads onto owned infrastructure. None of them is ideological. Each is a specific, evidenced reason tied to a specific kind of organisation and workload, and most organisations that make the move can point to at least two of the four applying to them at once, which is usually what tips the decision from "interesting" to "funded."

Data sensitivity and regulatory exposure

The most direct driver is that data never has to leave the organisation's boundary. For healthcare providers handling patient records, financial institutions handling account and transaction data, defence and government bodies handling classified or sensitive-but-unclassified material, law firms handling privileged material, and operators of critical national infrastructure, sending data to a third-party API means it crosses an administrative and often a jurisdictional boundary the organisation does not fully control. On-premise AI removes that boundary crossing entirely: the data, the model, and the inference all stay inside infrastructure the organisation owns or directly contracts for.

Cost economics at sustained scale

Cloud token pricing and owned infrastructure are not universally cheaper than each other. The general shape found in vendor and analyst TCO research is consistent: cloud API pricing wins at low or spiky usage, because the organisation pays only for what it consumes and carries no idle capacity. Owned infrastructure wins once usage is sustained and GPU utilisation is high, because the capital cost amortises across a large, continuous volume of inference.

Directionally, break-even is commonly discussed in the range of months rather than years for workloads that sustain roughly 70 to 80 percent GPU utilisation or higher. Below that utilisation threshold, cloud usually still wins, because idle GPU capacity is simply wasted capital. Token-based thresholds are also commonly cited as rules of thumb: workloads running below roughly 500,000 tokens a day on a 7B-class model, or below roughly 2 million tokens a day on a 70B-class model, tend to stay cheaper on cloud APIs; above those volumes, owned infrastructure starts to close the gap and then overtake it. These are commonly cited ranges from public vendor and industry TCO research, not a guarantee, and every organisation should verify the maths against its own workload mix before committing capital.

Latency, availability and independence from a vendor's control plane

Calling a third-party API means accepting that party's uptime, rate limits, pricing changes, and model deprecation schedule as constraints on the business. Owned infrastructure removes that outbound dependency. There is no risk of a vendor deprecating the exact model version a production system was validated against, no risk of a rate limit throttling a time-sensitive workflow at the worst moment, and no exposure to a pricing change made unilaterally by a supplier.

Sovereignty and auditability

The fourth driver is the ability to prove, not merely assert, what happened to a given piece of data or a given inference. When inference happens inside infrastructure the organisation controls end to end, the organisation can build a verifiable record of every prompt, every piece of context retrieved, and every output produced, independent of a vendor's own word about what its system did. That distinction between an assertion and a provable record is the thread that runs through the audit and governance section of this guide.

What hardware and GPUs does on-premise AI actually need?

GPU selection should be workload-driven, not vendor-driven. The starting point is a simple, reusable piece of arithmetic that every CIO evaluating on-premise AI should be able to do without a vendor in the room.

The VRAM rule of thumb

Model memory footprint scales with parameter count and numeric precision. As a rule of thumb: a 70-billion-parameter class model needs roughly 140GB of VRAM at FP16 precision, roughly 70GB at FP8 or INT8, and roughly 35 to 46GB at INT4, before accounting for KV cache and activation overhead, which grows with context length and concurrent requests. This single formula, parameter count times bytes per parameter at the chosen precision, plus a working overhead for cache and activations, is the most load-bearing sizing fact in the entire field, and it applies at any model size: halve it for a 35B-class model, double it for a 140B-class model, and always add headroom for the cache.

The sizing failure mode

Cloud capacity can be resized with a configuration change. On-premise hardware cannot. GPU procurement lead times commonly run 8 to 16 weeks from order to rack-ready, so a sizing mistake is not a quick fix, it is a quarter lost. Oversizing wastes capital that could have funded other priorities. Undersizing bottlenecks the business at the exact moment demand is growing, which is the worse of the two failure modes because it is visible to the whole organisation.

The commonly cited practical response is to size to sustained peak, not average, and to build in headroom: plan for roughly two to three times the initial model count as usage and use cases grow, rather than sizing tightly to the first production workload and hoping the next one fits.

Rack-scale reality

Modern AI hardware is dense. Contemporary AI racks commonly hold tens of GPUs connected by high-speed interconnect, and that density changes the nature of the procurement decision. It stops being "buy a server" and becomes "plan a room": power delivery, cooling capacity, floor loading, and interconnect topology all become first-order design decisions rather than afterthoughts bolted on once the hardware arrives. A CIO evaluating on-premise AI for the first time should expect the facilities conversation to be as substantial as the compute conversation.

Not every workload needs flagship hardware

CPU-only and smaller-GPU tiers exist and matter. Not every on-premise deployment needs top-tier datacentre GPUs. Three broad categories are worth naming without picking a vendor as the answer: datacentre-class GPUs for training and high-throughput inference, workstation-class GPUs for smaller models, development, and lower-concurrency inference, and CPU-only inference for smaller quantised models or workloads where latency tolerance is generous. Matching the tier to the actual workload, rather than defaulting to the largest available part, is one of the most common places a build overspends.

Quantisation is a design decision, not an afterthought

The VRAM formula above shows why quantisation, reducing numeric precision from FP16 down to FP8, INT8, or INT4, is one of the highest-leverage decisions in a hardware plan: it can roughly halve or quarter the memory footprint of the same model. It is not a free lunch. Lower precision trades some accuracy for that memory saving, and the acceptable trade-off point differs by workload: a customer-facing reasoning task usually tolerates less precision loss than a bulk classification or retrieval-ranking task. The practical approach is to benchmark the organisation's own evaluation set at each candidate precision level before committing hardware spend to a particular tier, rather than assuming a published benchmark from a different workload will transfer cleanly.

What does AI storage architecture actually require?

AI storage is a different discipline to general enterprise storage. Its job is to feed GPUs at line rate. If storage cannot keep up, the (very expensive) GPU sits idle waiting for data, which is the single most wasteful outcome in an on-premise AI build.

The disaggregated storage pattern

The common architecture pattern separates storage from compute: NVMe drives live in dedicated storage servers, and GPU servers access that storage over the network using NVMe-over-Fabrics rather than local disks. This decouples storage capacity and performance scaling from GPU server scaling, so an organisation can grow either independently as needs change.

GPUDirect Storage and RDMA to VRAM

Modern architectures push this further: storage traffic can bypass the server's CPU entirely and land directly in GPU memory using RDMA-based transfer paths, commonly referred to under the GPUDirect Storage pattern. This removes a CPU-mediated copy step that would otherwise sit in the critical path of every read. It requires a fast, lossless network fabric underneath it to work reliably, which is covered in the networking section below.

Tiering

A CIO needs three storage tiers, sized and retained differently:

Why storage bandwidth, not just capacity, is the binding constraint

Capacity planning gets most of the attention in a first pass, but bandwidth is usually the constraint that actually bites. Loading a large model's weights, and refreshing KV cache and retrieval context on every request under concurrent load, means the storage path has to sustain sequential and random read throughput measured in multiple gigabytes per second, sustained, not burst. A storage tier sized correctly on capacity but under-provisioned on throughput produces exactly the same symptom as undersized GPU compute: expensive accelerators sitting idle waiting on data, except the root cause is invisible from the GPU utilisation graph alone, which is why storage throughput needs its own monitoring, separate from GPU utilisation metrics.

Data lifecycle as a governance question

What gets kept, for how long, and why, is a governance decision as much as a technical one. Retention schedules for the hot and warm tiers are usually driven by operational need, but the cold tier's retention period is usually driven by regulatory and audit obligations, which means the storage architecture and the audit programme have to be designed together, not handed to two separate teams that never compare notes. This connects directly to the audit and governance section later in this guide.

InfiniBand or Ethernet: which networking fabric does an AI cluster need?

The GPU-to-GPU fabric decision is the single highest-leverage networking decision in an AI build, because it determines how efficiently a distributed job can share gradients and activations across many GPUs. Two options dominate, and the honest answer depends on scale.

InfiniBand

InfiniBand is purpose-built for HPC and AI workloads. It delivers lower and more consistent latency than general-purpose Ethernet, and it supports in-network compute acceleration, where the switch fabric itself performs in-switch aggregation for collective operations used heavily in distributed training. This materially speeds up large distributed training jobs. InfiniBand is mature at large scale, with proven deployments spanning hundreds to thousands of GPUs.

RoCE (RDMA over Converged Ethernet)

RoCE runs RDMA over standard Ethernet. It is generally lower cost than InfiniBand and integrates more easily into existing enterprise networking skills, tooling, and supply chains, since it builds on Ethernet expertise most infrastructure teams already have. It requires careful congestion control tuning, commonly priority flow control and explicit congestion notification tuned specifically for the AI traffic pattern, to remain lossless under sustained load, which is a real operational burden but a well-understood one for a team that already runs enterprise Ethernet. RoCE is commonly recommended today for enterprise clusters in the tens to low-hundreds of GPUs, where existing networking expertise and cost matter more than extracting the last 15 to 20 percent of throughput.

The honest trade-off

InfiniBand for maximum training performance at large scale. RoCE or standard Ethernet for most enterprise inference workloads and mid-size training workloads, where the existing networking team's expertise and the lower cost outweigh the performance ceiling InfiniBand offers. Neither is universally correct; the right choice is a function of scale and the workload's sensitivity to collective-communication latency.

The perimeter question

Separate from the GPU fabric decision is the network perimeter: inbound and outbound network posture. The recommended default for security-sensitive AI deployments is zero-egress by default, with explicit allow-lists for any external connection the system genuinely needs. For deployments handling sensitive data, this perimeter decision matters more than raw fabric throughput, because it is what actually prevents data leaving the boundary the organisation built the whole system to protect.

What software stack does compute orchestration for AI require?

This layer is the one most guides skip, but it is where an on-premise AI platform actually gets operated day to day. Three categories matter.

Container orchestration

Most production AI workloads run on Kubernetes-based orchestration, which handles scheduling, scaling, and failover for containerised services including model-serving endpoints. Training-heavy organisations, particularly those running large distributed training jobs, often layer an HPC scheduler such as Slurm alongside or instead of Kubernetes for the training workload specifically, since HPC schedulers are built around the batch-job, tightly-coupled-parallel-process model that training jobs need.

Model serving layers

A dedicated inference server sits between the orchestration layer and the raw model weights, handling request batching, quantisation, and multi-model routing so that GPU capacity is used efficiently across concurrent requests rather than one request monopolising a GPU. This layer is where most of the difference between a proof-of-concept and a production-grade inference service actually lives.

MLOps and LLMOps tooling categories

A mature on-premise AI programme needs, as categories rather than specific products: experiment tracking, a model registry that versions and governs which model artefacts are approved for production, pipeline orchestration for data and training workflows, and feature stores where structured feature data feeds multiple models consistently. Most of these categories have both credible open-source and commercial options, and the right choice depends on the organisation's existing tooling investment more than any category having a single obviously correct answer.

How should identity and access work for AI infrastructure?

The core shift AI infrastructure demands is that identity, not network location, becomes the control plane. A workload being inside the corporate network is not, on its own, a reason to trust it with model weights, training data, or inference endpoints.

RBAC is necessary but not sufficient

Role-based access control answers the question "what can this identity do." Zero trust adds a second question on top of it: "should it do it right now, from this device, in this context." An identity that legitimately holds a role can still be the wrong actor to trust in a given moment, for example if the request originates from an unexpected device or an unusual pattern of activity. Both questions need answering; RBAC alone answers only the first.

The AI-specific wrinkle

AI agents and service accounts behave differently from human users or static services. A human user's role changes infrequently. A static service account's behaviour is predictable and narrow. An AI agent can act across many different contexts within minutes, chaining tool calls and touching multiple systems in a single session, which means identity systems designed around infrequent role changes struggle to keep up. The emerging practice in response is to verify not just that an agent holds a valid credential, but that the underlying model or code executing under that credential has not been tampered with, an integrity attestation layered on top of ordinary authentication rather than a replacement for it.

Practical controls a CIO should expect

• Multi-factor authentication everywhere, with no standing exceptions for infrastructure or service accounts.
• Least-privilege access by default, granted for the narrowest scope and duration a task requires.
• Policy-as-code, using an approach such as Open Policy Agent, so that access rules are versioned, reviewed, and auditable rather than living as undocumented state in a console.
• Short-lived credentials in place of long-lived static keys, especially for anything an AI agent or automated pipeline uses, since a leaked short-lived credential has a naturally bounded blast radius.

What does the AI-specific threat model actually look like?

The threat model for AI infrastructure differs from conventional IT in specific, concrete ways: prompt injection, where malicious instructions embedded in input content attempt to hijack a model's behaviour; model or weight tampering, where the artefact itself is altered before or after deployment; data exfiltration via model outputs, where sensitive information leaks out through what the model says rather than through a conventional network breach; supply-chain risk in pretrained weights and their dependencies, since a model file and its surrounding software stack can carry the same kind of hidden risk as any other third-party software supply chain; and insider threat from staff with privileged infrastructure access, which is not new to AI but is sharpened by how much a single compromised credential can now touch.

The authoritative framework

The NCSC (UK) and CISA Guidelines for Secure AI System Development, published jointly and UK-led, are the primary authoritative framework for this threat model. They structure guidance across four lifecycle stages: secure design, secure development, secure deployment, and secure operation and maintenance. This is the single most citable UK-anchored standard for AI system security, and it deserves to be named accurately rather than paraphrased into something vaguer, because the four-stage structure is itself the useful part: it gives a CIO a checklist for where in the lifecycle a given control belongs.

Input and output guardrails

A baseline AI deployment, not an advanced one, should include: PII and PHI redaction on inputs and outputs, secrets detection to catch credentials or keys that end up in prompts or generated text, prompt-injection detection, and output filtering. Treating these as optional extras rather than baseline expectations is one of the more common gaps found in early-stage on-premise deployments.

Supply-chain integrity for models

Before any model weights or dependency bundle enter a trusted environment, they should be cryptographically signed and hash-verified, commonly using SHA-256, so that what actually loads onto GPU memory is provably the artefact that was validated, not a substituted or tampered file. A Software Bill of Materials should accompany every deployed bundle, so the organisation knows exactly what is inside it, down to the dependency level, rather than trusting a vendor's summary description.

Insider threat needs its own control, not a generic one

Privileged infrastructure access to an AI platform is a materially bigger blast radius than privileged access to a conventional application server, because a single administrator with unrestricted access can touch model weights, training data, and audit logs in the same session. The practical response is the same principle covered in the identity section above, applied specifically to infrastructure administrators: least-privilege scoping even for the platform team itself, separation of duties so that the person who can alter a model artefact is not the same person who can alter the log recording that change, and routine review of privileged-access activity rather than trusting that privilege alone is sufficient assurance.

What counts as a defensible AI audit trail?

CIOs commonly conflate two separate audit obligations. Separating them clearly is the single most useful thing this section can do.

Two distinct obligations

The first is proving what the AI system did: an evidentiary, per-action audit trail covering who asked what, what data was touched, what the model returned, and when. This is operational, granular, and specific to each interaction. The second is proving that the organisation manages AI responsibly as a programme: a management-system standard covering governance, risk processes, and organisational controls, assessed at the level of the whole programme rather than any single interaction.

The per-action audit trail

The practical requirement here is tamper-evident logging that survives a failover and cannot be silently edited after the fact. This is a genuinely different requirement to a normal application log, because a regulator or external auditor needs to be able to trust the log independent of trusting the vendor who built the system that produced it. A log an administrator can quietly edit is not evidence, it is a claim, and the distinction matters enormously the first time a regulator asks a hard question.

A defensible AI audit trail practically needs to record, for every relevant interaction: the prompt, the model version that served it, the retrieved context or data sources touched during the response, the output produced, a timestamp, the identity that initiated the request, and, where the output fed into a business process, the decision that resulted. "We have logs" is not the same claim as "we have a verifiable audit trail an outside party can check without trusting us," and a CIO defending an on-premise AI decision to the board or a regulator needs to be precise about which of the two claims is actually true of their system.

What "tamper-evident" means in practice

Tamper-evident is a specific, testable property, not a marketing description. A log that meets the bar is typically append-only at the storage layer, so individual entries cannot be edited or deleted after being written, and cryptographically chained, so that each new entry incorporates a hash of the previous one, meaning any retroactive edit to an earlier entry breaks the chain in a way that is detectable on inspection rather than something the log owner would have to disclose voluntarily. The test a CIO can apply to any vendor or in-house system claiming this property is simple: ask what happens, concretely, if an administrator with full database access tries to edit a historical entry, and whether the answer is "the system prevents it" or merely "our policy forbids it." Only the first answer is a genuine audit trail.

The management-system standard: ISO/IEC 42001

ISO/IEC 42001:2023 is the world's first AI management system standard. It specifies requirements across the standard high-level ISO clause structure, clauses 4 through 10, covering context, leadership, planning, support, operation, performance evaluation, and improvement, applied specifically to an organisation's AI risk management, system lifecycle management, impact assessment, and third-party and supplier oversight. As adoption grows, ISO/IEC 42001 is increasingly treated as an expected baseline credential for a serious AI programme rather than a differentiator that sets one organisation apart from its peers.

The voluntary framework: NIST AI RMF

The NIST AI Risk Management Framework, AI RMF 1.0, was published in January 2023. It is voluntary and US-originated, but it is globally referenced well beyond US borders. It is organised around four core functions, Govern, Map, Measure, and Manage, mapped against a set of trustworthiness characteristics: validity, reliability, safety, security and resilience, accountability, transparency, explainability, privacy, and fairness. A companion NIST AI RMF Playbook exists alongside the framework itself, offering practical implementation guidance for organisations working through each of the four functions.

How does the AI regulatory landscape affect on-premise decisions?

Regulation here is best treated as a compliance calendar to plan against, not as news to react to. The dates below are the corrected, current position; treat any source stating otherwise as stale.

The EU AI Act

The original headline deadline for high-risk obligations under the EU AI Act was 2 August 2026. That date has been deferred and should never be presented as a live deadline; it is now the original, superseded date only. Following the EU's Digital Omnibus, the corrected timeline is: stand-alone Annex III high-risk obligations are now due 2 December 2027, and high-risk AI embedded in regulated products under Annex I is deferred to 2 August 2028. Article 50 transparency obligations largely remain on their original schedule and were not deferred. The deferral is best read as a build window rather than a reprieve: the underlying proof and governance requirements survive the delay intact, so an organisation that waits loses runway, not obligation.

United Kingdom

The UK's primary anchor is the NCSC and CISA Guidelines for Secure AI System Development covered above, alongside UK GDPR Article 32, which requires technical and organisational security measures and has been explicitly extended by NCSC's newer agentic-AI guidance to raise that "state of the art" bar for systems where an AI agent acts with more autonomy. As of writing, there is no standalone UK AI act; the UK's approach remains principles-based and sector regulator-led, in contrast to the EU's binding statute. Neither approach should be overstated relative to the other: the UK's is more flexible and slower to bind, the EU's is more prescriptive and binding by default.

United States public sector

FedRAMP is the baseline cloud-security authorisation for US federal use. DoD Impact Levels are a separate, additional layer on top of FedRAMP, not a FedRAMP tier themselves, and this distinction is commonly stated wrong. IL5 builds on FedRAMP High plus DoD-specific NIST SP 800-53-derived controls, mandates US data residency, and requires US-person-only personnel access; it covers Controlled Unclassified Information and sensitive unclassified National Security Systems. IL6 covers classified information up to SECRET. An organisation targeting DoD workloads needs to plan for both layers, not assume FedRAMP authorisation alone is sufficient.

General enterprise baseline

SOC 2 Type II and ISO/IEC 27001 are widely-held table-stakes certifications for any serious AI infrastructure vendor or in-house programme today. Neither functions as a differentiator on its own; both are closer to a minimum bar a serious counterparty is expected to clear.

Sector regulators still apply on top of general AI rules

None of the horizontal AI frameworks above replace an organisation's existing sector regulator. A UK bank running on-premise AI still answers to its financial services regulators on model risk and operational resilience; a healthcare provider still answers to its clinical and data-protection regulators; a defence contractor still works within its existing security accreditation regime. The practical implication for a CIO is that an on-premise AI programme needs mapping against the sector-specific regime the organisation already lives under, not just against the general AI frameworks named above, since the general frameworks are additive to existing sector obligations rather than a replacement for them.

The honest close on regulation

Certifications and frameworks are necessary evidence, not the substance itself. The substance is whether the architecture can actually produce the proof each framework asks for, on demand, in a form an outside party can verify without simply taking the organisation's word for it.

What are the deployment topology options for on-premise AI?

Deployment topology is a spectrum, and most real deployments blend more than one point on it rather than picking a single option outright.

Single-node

A single-node deployment is appropriate for development and evaluation only. It has no high availability and no redundancy, and running production workloads on a single node is a decision that eventually produces an outage with no failover path.

Multi-node clustered

Multi-node clustered deployment is the production standard: distributed compute with built-in redundancy, so that a single node or GPU failure does not take the whole service down. This is the topology most of the architecture discussed earlier in this guide, storage tiering, networking fabric choice, orchestration, assumes as the baseline.

Hybrid

Hybrid deployment keeps sensitive workloads on-premise and bursts non-sensitive or spiky workloads to cloud. This is the most common real-world enterprise pattern, and it deserves to be presented neutrally as often the pragmatic middle path, rather than framed as a compromise or a failure to commit. An organisation that keeps its most sensitive, highest-volume workloads on owned infrastructure while leaving exploratory or unpredictable workloads on cloud APIs is usually making a sound engineering trade-off, not hedging out of indecision.

Edge

Edge deployment runs smaller, quantised models on lower-power hardware close to where data is generated: branch offices, factory floors, vehicles. Edge is the right answer where latency or connectivity, not just data sensitivity, is the driving constraint, since it removes the round trip to a central cluster entirely.

Disaster recovery for AI specifically

A common and costly failure mode is bolting AI onto an existing general-IT disaster recovery plan that was never designed for what an AI platform actually holds. An AI platform's DR-relevant assets are not the same as a conventional application's: model weights, vector indices built from a document corpus, agent session state, and audit trails all need their own recovery point and recovery time thinking. Inheriting RPO and RTO assumptions wholesale from database or application DR planning, without accounting for these AI-specific assets, is where real incidents happen: an organisation restores its databases cleanly and discovers its vector index or its audit trail did not come back with it.

What does air-gapped and high-assurance AI deployment involve?

True air-gap has a precise definition: no live network connection to any external network, full physical separation. It is worth being precise about this because the term gets used loosely.

The realistic middle ground: restricted networks with a data diode

Most organisations that describe their deployment as air-gapped are actually running a restricted network with a hardware-enforced one-way data diode. This allows controlled inbound updates while making outbound exfiltration physically impossible, not merely policy-blocked, because the diode enforces the one-way constraint in hardware rather than in software that could in principle be misconfigured or bypassed. This is the practical, buildable version of "air-gapped" for most enterprises. A literal disconnected vault, with no physical media path in or out at all, is reserved for the most sensitive tier of deployment and carries a correspondingly higher operational cost.

Getting models and updates into an air-gapped environment

Three practices consistently show up in credible air-gapped update processes: physical media transfer with integrity verification, meaning cryptographic hash and signature checks performed on arrival before anything is trusted; pre-packaged, self-contained dependency bundles, so nothing is pulled live from the internet at runtime; and a Software Bill of Materials accompanying every bundle, so the receiving environment knows precisely what it is accepting.

Observability without phoning home

Telemetry inside an air-gapped environment should never phone home to a vendor's control plane. If any metrics leave the boundary at all, they should go outward only, through a one-way diode, to a separately accredited monitoring environment, never as a two-way connection back to the vendor.

The cross-domain update pattern

Classified and other high-assurance environments commonly use a specific, citable pattern for reviewing updates: verify the update on the lower side, transfer it through a cross-domain solution, then re-verify it on the high side with independent checksums before it is trusted. This three-step verify-transfer-reverify pattern is a useful concrete process a CIO can map onto their own environment even outside a classified context, anywhere the requirement is to trust nothing that crosses a security boundary without checking it twice, independently, on both sides.

How should a CIO model the total cost of on-premise AI?

Cost is a genuine trade-off, not a foregone conclusion in either direction. Cloud AI APIs generally win at low, spiky, or unpredictable usage. Owned infrastructure generally wins once usage is sustained and utilisation is high, commonly cited as roughly 70 to 80 percent or higher sustained GPU utilisation over a multi-year horizon.

The full TCO component list

A naive comparison typically misses several of these:

• Hardware depreciation across the useful life of GPUs, storage, and networking equipment.
• Power and cooling, which scale directly with the rack density discussed in the hardware section.
• Physical space or colocation costs.
• Networking equipment and the fabric choice covered above.
• Storage across all three tiers.
• Software licensing for orchestration, serving, and MLOps tooling.
• Operations labour, the most commonly underweighted line in a naive comparison and the subject of the staffing section below.

A further hidden-cost category is worth naming explicitly: rate-limit engineering and prompt rework incurred when migrating away from a cloud vendor, since prompts and integration patterns tuned against one API often need real rework against a different serving stack, plus the opportunity cost of any downtime incurred during that transition.

Reading the workload-threshold table

The token-based thresholds cited earlier are easiest to use as a simple lookup, always with the caveat that these are commonly cited public ranges, not a guarantee for any specific organisation's mix:

These figures move as GPU pricing, cloud API pricing, and model efficiency all move independently, which is exactly why the guide frames them as directional lookups rather than a fixed number to plan a budget against. Re-run the comparison against current pricing and the organisation's own measured token volume before committing capital, not against a number from an article, however recent.

A usable mental model, not a single number

Because actual dollar figures move too fast to stay evergreen and vary hugely by workload mix, the more useful mental model frames break-even in workload terms, tokens per day at a given model size, rather than a universal price point. Any unit-economics claim in vendor marketing, including a vendor's own published figures, should be treated as directional and workload-dependent, never as a guarantee. A CIO's own workload, measured over a representative period, is the only number worth building a business case on.

What staffing does an on-premise AI programme actually need?

The roles a CIO needs to plan for, stated honestly rather than optimistically: ML and AI engineers, data engineers, MLOps and platform engineers, and increasingly a dedicated AI product owner function that connects the platform to business use cases. Smaller programmes commonly run with 5 to 8 people covering these roles; larger programmes scale to 15 to 20 or more as the number of production use cases grows.

The actual skill blend MLOps roles need

MLOps and platform engineering roles are not a purely data-science skillset. They combine software engineering fundamentals, DevOps practice covering containers, CI/CD, and infrastructure-as-code, and ML-specific tooling literacy across orchestration and pipeline tools and model or feature registries. Hiring purely for data-science credentials and expecting those hires to also run the platform is a common and avoidable mismatch.

The organisational failure mode

On-premise AI programmes are usually under-staffed on the operations side relative to the model and data-science side. The operational burden, patching, scaling, monitoring, incident response, disaster recovery, and producing audit evidence on demand, is easy to underestimate until the platform is actually live and someone has to be paged at two in the morning. This is, honestly, the single biggest reason build-versus-buy-versus-rent decisions fail in practice: not the hardware bill, which is visible and gets budgeted for, but the standing operational team the platform requires, which is far easier to underestimate at the planning stage.

How do organisations migrate from cloud AI to on-premise?

Migration is best framed as a staged discovery-to-cutover process, not a single cutover event.

The staged process

• Estate discovery. Map current AI usage, data flows, and integration points across the organisation, including shadow usage that never went through a formal procurement process.
• Needs assessment. Determine which workloads actually justify moving, using the decision drivers and cost model covered earlier in this guide, rather than moving everything by default.
• Architecture and topology selection. Choose from the deployment topologies covered above, matched to the specific workloads identified in the needs assessment.
• Phased migration plan. Move workloads in stages rather than as a single big-bang cutover, so problems surface against a smaller blast radius.
• Security and identity baseline, established and tested before go-live. This has to happen before cutover, not during it.

The specific pitfalls

Skipping the security-baseline step leaves newly migrated workloads exposed mid-transition, at exactly the point when they are least understood by the team operating them. Skipping proper cost modelling produces on-prem bills that exceed the old cloud bill, because nothing was right-sized against the actual workload, which is the single fastest way to lose executive confidence in an on-premise programme after it has already consumed capital.

Hybrid as the actual path, not a final destination

Hybrid is frequently where a migration ends up staying, not a waypoint on the way to fully on-premise. Most organisations move their sensitive or high-volume workloads first and keep exploratory or spiky workloads on cloud APIs indefinitely. That is sound engineering judgement given the cost and complexity trade-offs covered above, not a failure to fully commit to the on-premise decision.

Should a CIO build, buy, or rent on-premise AI infrastructure?

This is a genuine three-way comparison, and each option is legitimate for a different organisation and risk appetite.

Build

Self-integrating hardware and open tooling gives maximum control and maximum operational burden. It requires the full staffing stack covered in the section above, in-house, and it is the option best suited to organisations with an existing platform engineering capability and a genuine reason to want every layer of the stack under direct control.

Buy

A packaged on-premise or private AI platform gives faster time to value, because the vendor carries much of the integration and support burden. This requires genuine due diligence on the vendor's own security posture, audit capability, and exit posture: can the organisation get its data and models back out cleanly if the relationship ends. A vendor that cannot answer that question clearly is a red flag regardless of how capable the platform looks in a demonstration.

Rent

Managed private cloud or dedicated GPU hosting is a middle path: physical or logical isolation without full capital ownership. This is a legitimate option, worth naming honestly, for organisations that are not yet ready to own hardware outright but still need stronger isolation guarantees than a shared multi-tenant cloud API provides.

Where a fully owned platform fits

Fully owned, offline-capable sovereign AI platforms exist as a category for organisations whose requirement is complete ownership plus a per-action verifiable audit trail of the kind described in the audit and governance section above. Mickai's SIOS is one option in that category. It is one credible answer among the legitimate build, buy, and rent options covered in this section, not a reason to skip evaluating the others on their own merits.

The fuller architecture behind the patterns in this guide is set out at /sovereign-ai, and the film at /film shows the interface in operation.

Frequently asked questions

What is on-premise AI?

On-premise AI is AI infrastructure and models that run inside an organisation's own data centre, colocation cage, or private cloud, under its own physical and administrative control, rather than through a public cloud AI API or managed SaaS product. It trades higher upfront capital and operational ownership for data control, more predictable unit economics at scale, and independently verifiable proof of what happened to data and inferences.

Is on-premise AI cheaper than cloud AI?

It depends on usage volume and sustained GPU utilisation, not on the architecture alone. Cloud APIs generally win at low or spiky usage, while owned infrastructure generally wins once usage is sustained and utilisation clears roughly 70 to 80 percent over a multi-year horizon. Below that utilisation, cloud usually still wins, so the honest answer is to model against actual workload volume rather than assume either option is universally cheaper.

What GPU do I need to run a 70B parameter model?

As a rule of thumb, a 70-billion-parameter class model needs roughly 140GB of VRAM at FP16 precision, roughly 70GB at FP8 or INT8, and roughly 35 to 46GB at INT4, before adding KV cache and activation overhead. This scales linearly with parameter count, so the same formula, parameters times bytes per parameter at the chosen precision, plus headroom, applies at any model size.

What is the difference between air-gapped and on-premise AI?

On-premise describes where infrastructure is owned and controlled, not whether it has a network connection to the outside world. Air-gapped specifically means no live network connection to any external network at all. Most on-premise deployments are connected to the internet under a controlled perimeter; true air-gap, or its practical equivalent using a hardware-enforced one-way data diode, is a stricter, additional constraint layered on top of an on-premise deployment, not a synonym for it.

Is the EU AI Act deadline still 2 August 2026?

No. The original 2 August 2026 date for high-risk obligations has been deferred following the EU's Digital Omnibus. Stand-alone Annex III high-risk obligations are now due 2 December 2027, and high-risk AI embedded in regulated products under Annex I is deferred to 2 August 2028. Article 50 transparency obligations largely remain on their original schedule.

What is ISO/IEC 42001?

ISO/IEC 42001:2023 is the world's first AI management system standard. It specifies requirements across the standard ISO clause structure, covering context, leadership, planning, support, operation, performance evaluation, and improvement, applied to an organisation's AI risk management, system lifecycle management, impact assessment, and third-party oversight. It is increasingly treated as an expected baseline for a serious AI programme rather than a differentiator.

Do I need InfiniBand or is Ethernet enough for an AI cluster?

It depends on scale and workload. InfiniBand delivers lower, more consistent latency and in-network compute acceleration, and is proven at scales of hundreds to thousands of GPUs, making it the right choice for maximum training performance at large scale. RoCE over standard Ethernet is generally lower cost, easier to integrate with existing networking skills, and commonly recommended for enterprise clusters in the tens to low-hundreds of GPUs, particularly for inference and mid-size training workloads.

How many people does it take to run an on-premise AI platform?

Smaller programmes commonly run with 5 to 8 people covering ML and AI engineering, data engineering, and MLOps or platform engineering; larger programmes scale to 15 to 20 or more as production use cases grow. Programmes are commonly under-staffed on the operations side relative to the model and data-science side, and that operational gap, not the hardware bill, is the most common reason build-versus-buy-versus-rent decisions fail in practice.

Can on-premise AI be audited independently?

Yes, but two separate obligations need to be distinguished. Proving what the AI system did on a given interaction requires a per-action, tamper-evident audit trail that survives a failover and cannot be silently edited after the fact. Proving the organisation manages AI responsibly as a programme is a separate management-system claim, typically evidenced against a standard such as ISO/IEC 42001. Having application logs is not the same claim as having a verifiable audit trail an outside party can check without trusting the organisation's word.