Run Capable AI Without the Cloud: A Guide for Regulated Sectors

Yes, you can run capable AI without the cloud, on hardware you own, fully offline, and for a great deal of regulated work that is the sounder choice. The reason it now works is that capable open-weight models run well on workstation-class machines, the kind an organisation already buys. In Mickai the component that does this is Poros, our inference engine: it loads a large model and serves it on the loopback interface only, so requests never leave the machine. Around that engine a regulated deployment needs four things (a tested egress boundary, hardware-bound licensing, retrieval grounded in private data, and a verifiable audit record). Get those right and you have capable AI with custody, at the cost of cloud elasticity.

• Capable models now run locally on workstation-class hardware, not only in a hyperscale data centre.
• Poros serves a large model on loopback only: fully offline, with nothing sent to a third party.
• Require four things of any local deployment: a tested egress boundary, hardware-bound licensing, private-data grounding, and a verifiable audit record.
• Grounding keeps answers anchored to your own documents, which never leave your premises.
• The trade-off is real: you gain custody and predictable cost, you give up on-demand elasticity.

Can you actually run capable AI without the cloud?

Yes, and the reason is a shift in what the hardware can hold. A few years ago, running a genuinely capable model meant renting time on someone else's accelerators, because the model would not fit anywhere else. That is no longer the constraint. Open-weight models have improved sharply, quantisation lets a large model run in far less memory with little loss of quality, and workstation-class GPUs now carry enough memory to serve one locally at useful speed. The class of task that used to demand a data centre now runs on a machine you can put under a desk or in a rack you control.

'Without the cloud' should be taken literally here. It does not mean a private tenancy in someone else's data centre, and it does not mean a model that phones home for telemetry. It means inference that executes on hardware you own, in a room you control, with no dependency on an external service to function. For regulated work (where data residency, sovereignty, and the ability to operate through a network outage are not niceties but requirements) that literal reading is the entire point.

How local inference works with Poros

Poros is the inference engine inside Mickai. In practice it loads a large open-weight model into memory once and exposes it as a local service, listening only on the loopback interface (127.0.0.1). Loopback-only is a deliberate boundary: the model is reachable by applications on the same machine and by nothing else. There is no external endpoint to secure because there is no external endpoint at all. Everything above it (the studios, the assistant, the retrieval layer) talks to Poros over that local socket.

Because the engine and the weights sit on your hardware, the model works with the network cable unplugged. That is the test I hold it to. Sovereign inference is not a mode you switch on when a regulator visits, it is the default and only behaviour: prompts, documents, and generated output stay on the machine, and there is no code path that ships them elsewhere. I have written this approach up as a technical preprint on sovereign inference, published on Zenodo under my ORCID (0009-0000-5511-5858), for buyers who want the method rather than the marketing.

What a regulated buyer should require

If you are procuring AI for a regulated environment, do not accept 'it runs locally' as a claim. Ask for evidence of four specific properties, each of which can be tested.

• A tested egress boundary: the guarantee that data does not leave the machine, checked as an automated release gate, not asserted in a datasheet.
• Hardware-bound licensing: the software runs on the machines you are entitled to run it on, and cannot be silently copied elsewhere.
• Retrieval grounded in private data: the model answers from your documents, which stay on your premises, rather than from an external index.
• A verifiable audit record: a log of what the system did that you can check yourself, offline, without trusting the vendor.

The first of these is the one buyers most often forget to test. In Mickai the no-egress property is exercised as a release gate: a build does not ship unless the boundary holds under test. An egress boundary that is merely designed and never verified is a hope. One that fails the build when it regresses is a control.

Grounding: retrieval that never leaves the building

A capable model on its own is a generalist. To be useful for your work it needs your context: your policies, your contracts, your case files, your codebase. The wrong way to provide that is to upload it all to a third-party service. The right way is retrieval-augmented grounding that runs where the data already is. The documents are indexed locally, the relevant passages are retrieved locally, and they are handed to the local model as context. Nothing is sent out to be embedded or searched.

This is what lets a sovereign system be both private and current. The model does not need to have memorised your corporate knowledge, and you do not need to fine-tune it on sensitive material to get grounded answers. You keep your knowledge in your own store, the retrieval layer reads from it under your access controls, and the model reasons over what it is given. Your private data stays private precisely because the grounding never leaves the building.

Licensing bound to the hardware

Custody of the software matters as much as custody of the data. Mickai licensing is hardware-bound: entitlement is tied to specific machines rather than to an account in someone's cloud console. For a regulated buyer this closes two gaps at once. It prevents the software (and the capability it carries) from being copied onto machines outside the controlled estate, and it means your right to run does not depend on a licence server you have to reach over the internet. The system you paid for keeps working inside your perimeter, and only inside it.

Proving what the system did

The last requirement, a verifiable audit record, is what makes a local deployment defensible after the fact. Running offline is necessary but not sufficient: you still need to prove what the system did. Mickai writes every consequential action to the Open Audit Record, an append-only, hash-chained log whose entries are signed with a post-quantum signature (ML-DSA-65 under FIPS 204) and verifiable offline against a checkpoint you hold yourself. It is tamper-evident: a silent edit breaks the chain and fails verification. That gives you the paper trail a regulator or an internal auditor will ask for, without a line of it leaving your control.

The honest trade-off

I will not pretend local inference is free of compromise. The thing you give up, compared to a hyperscaler, is elasticity: you cannot conjure a thousand extra accelerators for an hour and hand them back. You size for your real workload, and a genuine spike beyond that capacity waits or is scheduled. If your demand is wildly unpredictable and occasionally enormous, the cloud's ability to absorb that is a real advantage, and it is fair to weigh it.

What you get in return is custody and predictable cost. Your data stays on your hardware, your capability keeps working through an outage, and your bill is the hardware and the licence rather than a metered charge that scales with every token and surprises you at month end. For a great deal of regulated, steady-state work (which is most of the actual work) that is the better trade, and it is the one Mickai is built to make. If that is the trade you want to make, our closed beta is open at mickai.co.uk/beta, where we are onboarding a first cohort of design partners.

Frequently asked questions

Can you run capable AI without the cloud?

Yes. Capable open-weight models now run on workstation-class hardware, and with an engine like Poros they serve locally on the loopback interface only, fully offline. For regulated work that keeps data on your premises and lets the system operate through a network outage.

What hardware do you need to run a large model locally?

Workstation-class hardware with a modern GPU carrying enough memory to hold the model. Quantisation reduces the memory a large model needs with little loss of quality, so the requirement is well within what an organisation already procures for demanding work, rather than a data-centre build.

Does offline AI mean a weaker model?

Not necessarily. The gap between open-weight models you can run yourself and the largest hosted models has narrowed considerably. For most regulated, task-specific work, a capable local model grounded in your own data outperforms a bigger remote model that has no access to your context.

How do you keep the model grounded in private data without sending it away?

Through retrieval that runs where the data lives. Your documents are indexed locally, relevant passages are retrieved locally under your access controls, and they are passed to the local model as context. Nothing is uploaded to an external service to be embedded or searched, so the data never leaves the building.

What is the main downside of running AI locally?

Elasticity. You size for your real workload and cannot instantly summon a hyperscaler's worth of extra capacity for a short spike. In exchange you get custody of your data and predictable cost. For steady-state regulated work that is usually the better trade.

How do I try Mickai?

Our closed beta is open at mickai.co.uk/beta, where we are onboarding a first cohort of design partners. Mickai is built by Mickai LTD (Companies House 17166618), and I am glad to talk to organisations with genuine sovereignty requirements.