How do you acceptance test an on-premise AI before go-live?

Run the go-live gate: eight checks against pass criteria fixed before testing starts, covering capability, retrieval accuracy, refusal behaviour, hardware performance, audit-record verification, access control, backup and restore, and operational handover. The rule that makes this a real test rather than theatre is simple: if the pass criteria were not written down before testing began, what happened was a demo, not an acceptance test, however impressive it looked.

The question matters because on-premise AI removes the excuse a cloud pilot has of blaming the vendor's infrastructure. Once the system runs on hardware the organisation owns, everything that goes wrong in production was, in principle, checkable before go-live, which makes a rigorous gate genuinely worth the time it costs.

What does capability testing on your own documents actually mean?

Scoring the system against a frozen golden set built by the people who do the work, not a generic benchmark. A golden set is a fixed collection of real tasks and correct answers drawn from your own documents, agreed and locked before testing, so that a result can be compared against a known standard rather than judged by impression. Testing against someone else's benchmark tells you how the system performs on someone else's problems.

How do you measure retrieval accuracy rather than just vibing it?

By checking, for a sample of realistic queries, whether the system cites the correct internal source and whether that source actually supports the answer given. This should be a scored exercise with a pass threshold agreed in advance, not a handful of impressive-looking answers demonstrated live. A system that retrieves confidently from the wrong document is more dangerous than one that visibly struggles, because the error is harder to spot.

What is refusal and boundary testing?

Confirming the system declines what policy says it must, tested deliberately rather than assumed. This means constructing prompts specifically designed to probe boundaries, requests it should refuse, data it should not surface to a given user, and confirming the refusal actually happens rather than trusting a policy document that describes intended behaviour.

Why test performance on your own hardware at realistic concurrency?

Because a system that performs well with one user in a demo can behave very differently under the concurrent load a live deployment actually produces. Testing should cover realistic simultaneous usage on the actual hardware that will run in production, including both CPU and GPU paths if both will be used operationally, since performance characteristics differ meaningfully between them.

What does audit-record verification actually involve?

Sealing a full day's worth of actions to the ledger, then independently verifying that ledger offline using the independent verifier, confirming the record checks out without needing to trust the system that produced it. Critically, this step must also include a deliberate tamper test: attempt to alter a sealed record and confirm that verification then fails. A ledger that has never been tested against tampering has not actually been proven tamper-evident, it has only been assumed to be.

How do you prove access control rather than review the configuration?

By testing it directly with a live account, not by reading a policy setting. The check that matters is whether a sales team user can actually reach the HR data store, attempted for real with that user's credentials, not confirmed by inspecting an access control list and assuming it works as configured. Configuration review finds intended access; live testing finds actual access, and the two are not always the same.

What rounds out the gate before go-live?

A full backup and restore drill, with ledger continuity confirmed intact after restoration, since a backup that cannot be restored without breaking the audit chain is not a real backup; and an operational handover check, confirming the organisation's own named operator can start, stop, monitor and read system health without the vendor on the call. A deployment that only its vendor knows how to operate has not actually been handed over, whatever the contract says. Once all eight checks pass, a bounded period of parallel running, humans checking outputs before reliance, closes the gap between passing a test and trusting a system in production.

“If the pass criteria were not written down before the test started, it was a demo, not an acceptance test.”

How the go-live gate fits into the wider architecture of offline verification and hardware-attested identity is set out at /sovereign-ai, and the film at /film shows the interface in operation.

Frequently asked questions

How long should an on-premise AI acceptance test take?

Long enough to complete all eight checks properly against pre-agreed pass criteria, which varies by deployment scale and complexity; no fixed duration is asserted here, but a rushed test that skips the tamper test or the live access-control check has not actually completed the gate regardless of the calendar time spent.

What is the single most commonly skipped check in a real acceptance test?

The deliberate tamper test on the audit ledger. Teams routinely verify that the ledger works when everything is normal, but skip the step of trying to alter a sealed record and confirming verification fails, which is the check that actually proves tamper evidence rather than assuming it.

Should the acceptance test use real production data or synthetic data?

The golden set should be built from real internal documents and tasks wherever possible, since testing against synthetic or generic data tells you how the system performs on problems it was not actually bought to solve, and the whole point of on-premise testing is checking it against your own reality.

Is a successful vendor demo the same as passing acceptance testing?

No. A vendor demo is typically curated and single-user; acceptance testing requires the organisation's own golden set, its own concurrency profile, its own access control accounts, and pass criteria the organisation itself fixed in advance, none of which a vendor-run demo satisfies.

Who should own the pass criteria before testing starts?

The people who do the work the system is meant to support, alongside whoever is accountable for the deployment's compliance and security posture, so that the golden set and the thresholds reflect real operational and regulatory requirements rather than a vendor's suggested benchmark.