Validating a system that never returns the same answer twice

In a GxP environment you qualify a system in three stages, installation, operational, and performance qualification, and all three rest on an assumption nobody writes down, because until recently it was free: the same input produces the same output, so a test that passes today passes tomorrow. A language model breaks that on the first call, since temperature aside, the vendor can replace the weights behind a stable name, the retrieval corpus shifts under you, and someone improves a prompt on a Tuesday, which together mean the suite you passed in March is not describing the system running in July.

Before any of the machinery, one boundary decides whether this is even a legal question to ask, and I didn’t draw it early enough the first time. Draft EU GMP Annex 22 keeps probabilistic and generative models out of critical GMP decisions entirely, so everything below lives in the non-critical space it does allow. That constraint shapes every choice that follows: an audit record detailed enough to tell you when an output stopped being reproducible, a promotion gate that can prove it refuses to ship an unqualified configuration, an OQ criterion that bounds a distribution instead of checking a value, and a way to size human review that does not scale with volume. The standards are real, and read carefully they tell you where to stop.

Qualifying a distribution instead of a value

The mental move is small and everything follows from it. For deterministic software, OQ asks whether a function returns the expected value, whereas for a probabilistic system that question has no answer, so it becomes whether the output distribution stays inside a qualified region. You stop qualifying an output and start qualifying a bound.

Intended use is the other word that has to be re-earned. A calculator has a range, and outside that range it errors, so the software itself enforces the boundary of its intended use, whereas a language model has no such edge. Ask it something the qualification never covered and it answers, fluently, with nothing in the output marking that it left the qualified region. So the specification carries an explicit out-of-scope list, and the boundary becomes a control in front of the model deciding what it may be asked. That control is deterministic, so you qualify it once, and it is the most load-bearing component most of these systems have.

A word on the scaffold, because a validation lead will catch it otherwise. GAMP 5 Second Edition and ASTM E2500 moved the field away from prescriptive IQ/OQ/PQ toward risk-based specification and verification, and I keep the IQ/OQ/PQ vocabulary because it is still what most quality systems are written against, and because the work lives in the mapping while the label does nothing. The category language needs the same care. A hosted LLM is a supplied product, Category 3 or 4 shaped, because the vendor built it and you configure it, whereas the prompt templates, retrieval, and orchestration you write around it are Category 5, custom and bespoke. Those are two kinds of thing wearing one system, and the rigor attaches to the bespoke parts you actually control.

One more piece of scaffolding, for whoever insists on scripted expected results. FDA’s computer software assurance guidance, reissued in February 2026 under a new title, names unscripted testing as assurance evidence in its own right, with scenario testing, error guessing, and exploratory testing beneath it, and says the record need not carry more evidence than the identified risk requires. It is written for device production and quality system software, and it is no pharma predicate rule, but the register transfers: for a system whose correct output you cannot write down in advance, unscripted evidence is the difference between a strategy and an impasse.

Rewriting the three stages themselves gives you something a validation lead can sign:

The rest of this is those three rows in code.

Where the regulator draws the line

I put this second because it decides the shape of everything after it. Draft EU GMP Annex 22, published for consultation on 7 July 2025, is the first GMP text written directly at AI, and its scope section is unusually blunt about model type. It covers static models, frozen after training, with deterministic output, while models that keep learning in production and models with probabilistic output are named as things that should not be used in critical GMP applications, with generative AI and large language models called out by name in the same category.

What the draft does allow is non-critical use, the applications with no direct impact on patient safety, product quality, or data integrity, with a qualified person responsible for the outputs. Its operations section goes further, since depending on how critical the process is and how much the model was tested, conformance may require a consistent review or test of every output, following a procedure. Read that next to the economics later and the reconciliation is the whole design. You do not get to replace review of critical decisions with a sample, because the draft forecloses putting the model in that path at all. What you get is the model where a wrong answer is recoverable, a person accountable for it, and statistics to size the monitoring of that stream.

The record that tells you when replay broke

Start here, because everything else gates on it. The first time an auditor asked me to reproduce a specific output, I couldn’t. The row showed the generated text, a timestamp, a user, and an approval flag, but it did not show which model version produced it, which prompt template was live that day, or which passages retrieval had returned, and months later the corpus had moved and the retriever returned different passages. The record was legible and attributable and still not reproducible, which under ALCOA+ means it was not complete. An output you cannot regenerate is an assertion wearing a record’s clothes.

So the record captures the full generating configuration, and the fields that let you diagnose a failed replay:

from dataclasses import dataclass

@dataclass(frozen=True)
class AuditRecord:
    record_id: str
    created_at: str            # ISO 8601, UTC, contemporaneous
    # --- what generated it (so a replay can be attempted and diagnosed) ---
    model_version: str         # resolved dated id: "gpt-4o-2024-11-20", not "gpt-4o"
    system_fingerprint: str    # backend config; seeded replay holds only while this is stable
    prompt_hash: str           # hash of the exact template + system prompt
    context_hash: str          # hash of the retrieved passages, in order
    temperature: float
    top_p: float
    seed: int | None
    max_tokens: int
    input_hash: str            # hash of the user input (no raw PII in the record)
    output_hash: str           # hash of the produced output
    # --- the signature manifestation (21 CFR 11.50): name, time, meaning ---
    reviewer_name: str | None  # printed name of the signer
    reviewer_id: str | None    # unique to one individual (Part 11 Subpart C)
    decision: str | None       # the meaning of the signature: approved | rejected | amended
    decision_at: str | None
    # --- integrity ---
    package_version: str       # the validation package this ran under

Four fields carry disproportionate weight, and I learned each one by not having it. The model_version has to be the resolved dated identifier, since a moving alias logs the name of a thing that changed. The context_hash covers the retrieved passages in the order they were fed in, which is the field whose absence made that first output unrecoverable. The seed is necessary for a replay attempt, but nowhere near sufficient, because hosted inference is best-effort, GPU nondeterminism breaks bit-exact reproduction on its own, and a seeded call only reproduces while the backend is unchanged. That is what system_fingerprint is for. It does not make replay work. It tells you the day replay stopped being possible, which in a program whose data-integrity argument rests on regenerability is the difference you need to name.

Which leaves what you do on the day replay stops working. GxP has always validated instruments that never return the same number twice, and nobody replays a chromatography run. The batch is defensible because system suitability was confirmed at the time of use, on that instrument, against criteria fixed in advance, applied alongside the samples instead of reconstructed from them later. The same instrument works here: a small pinned probe set, run against the live deployment on a schedule, its result written into the same store as the outputs it covers. The validity argument then rests on the system having been in a qualified state when it produced the output, which is how a system can be irreproducible and still validated, and it demotes replay from load-bearing evidence to a diagnostic.

There is a reason I flatten the parameters into typed fields instead of keeping a params dict. A frozen dataclass is only hashable if its fields are, and a dict field makes any hash() on the record raise, which is a trap that surfaces the first time you put records in a set.

Build the record inline with generation so the two cannot drift apart:

import uuid, hashlib, json
from datetime import datetime, timezone
from openai import OpenAI

llm = OpenAI()

def _hash(obj) -> str:
    return hashlib.sha256(json.dumps(obj, sort_keys=True, default=str).encode()).hexdigest()

def record_output(user_input: str, retrieved: list) -> tuple[AuditRecord, str]:
    prompt = render_prompt(user_input, retrieved)
    # Chat Completions on purpose: it is the surface that exposes seed and
    # system_fingerprint, the two fields that make a replay attempt diagnosable.
    # The Responses API is the newer default and surfaces neither.
    resp = llm.chat.completions.create(
        model="gpt-4o-2024-11-20",                     # dated id, never the alias
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2, top_p=1.0, seed=7, max_tokens=800,
    )
    text = resp.choices[0].message.content
    rec = AuditRecord(
        record_id=str(uuid.uuid4()),
        created_at=datetime.now(timezone.utc).isoformat(),
        model_version=resp.model,                      # what actually served it
        system_fingerprint=resp.system_fingerprint,    # None if the backend won't say
        prompt_hash=_hash(prompt),
        context_hash=_hash([(p.id, p.text) for p in retrieved]),
        temperature=0.2, top_p=1.0, seed=7, max_tokens=800,
        input_hash=_hash(user_input),
        output_hash=_hash(text),
        reviewer_name=None, reviewer_id=None, decision=None, decision_at=None,
        package_version=ACTIVE_PACKAGE,
    )
    return rec, text

Two properties make this hold up instead of merely look tidy. The store is append-only and write-once, so a record can be superseded by a linked version but never edited in place, which is close to a literal reading of 21 CFR Part 11 §11.10(e) on secure, computer-generated, time-stamped audit trails that independently record actions creating, modifying, or deleting records. And the human decision lives in the same record: when a reviewer signs you write a new linked version carrying their printed name, the time, and the meaning of the signature, which is what §11.50 requires of a signature manifestation and what §11.70 and Subpart C ask you to bind to the record. The signature covers the output hash, so an approval is tied to the exact bytes reviewed and cannot slide onto a later edit.

A hash does not discharge the whole obligation, though. §11.10(b) asks for the ability to generate accurate and complete copies of records in both human readable and electronic form, suitable for inspection, and a context hash proves the passages have not changed while telling an investigator nothing about what the model read. So the record has to resolve back to the passages themselves, which is where the audit trail becomes larger than anything anyone will read. The context runs far longer than the output, and it is the part of the record with the shortest expected readership and the longest retention. Do not copy the passages into every record to solve it. Keep the corpus under version control with immutable snapshots and let the record carry the snapshot id, passage ids, and offsets, so a copy is producible on demand without storing the corpus once per output. What you cannot end up with is a record that is perfectly verifiable and unreadable, where at inspection you prove integrity and still cannot produce the document.

Write-once storage has one collision worth naming, because it passes every technical review and fails a legal one. Put a seven-year Object Lock retention in COMPLIANCE mode over records containing user input, then have a data subject exercise a GDPR erasure right, and you have built two controls that cannot both be honored. COMPLIANCE mode means not even the root account can delete the object before its retention date, which is exactly why it satisfies Part 11 and exactly why it cannot satisfy an erasure request. The way out is to keep personal data out of the immutable object: hash or tokenize it, store the raw input under a separate, erasable key, and let the locked record hold references. Object Lock also has preconditions people miss, that it is enabled at bucket creation and needs versioning on, and that a delete against a locked version writes a delete marker rather than removing anything.

Landing this exposed how poor the existing trail had been. The share of outputs that replayed to the same result from their logged inputs started at 61%, mostly missing retrieved context and seeds, and climbed as the record filled in.

A 99% floor with an alarm below it is a defensible criterion for the non-critical stream. The remaining fraction is almost always a non-replayable call, a tool timeout or a truncated context or a backend that changed its fingerprint mid-week, and those are worth looking at one at a time.

A promotion gate that fails closed, and proves it

The record makes an output defensible. The gate makes a release defensible, and the rule is blunt: nothing promotes unless a complete, current, signed validation package exists. As policy-as-code with Open Policy Agent it lives in one file CI evaluates, instead of a wiki page people intend to follow.

The first version of this gate I wrote failed open, and it took a deliberate negative test to catch it, which is the whole lesson. Two bugs compounded. The Python wrapper built {"input": {...}} and piped it to opa eval -I, but -I already treats stdin as input, so the policy read input.input.manifest and every real field came back undefined. And allow if count(deny) == 0 is true whenever deny is empty, which on an empty or malformed request it always is, because every deny body references fields that are not there and never fires. A default allow := false does nothing against a defined allow rule that evaluates to true. The gate returned allow: true on an empty document. A control that reports success by receiving nothing is the failure mode you least want here.

The fix is a rule that denies on a structurally incomplete request, so allow depends on the request being complete as well as clean:

package release.gate

default allow := false

allow if {
	required_present
	count(deny) == 0
}

required_present if {
	input.manifest.model_version
	input.manifest.prompt_hash
	input.manifest.package_version
	input.validation.iq.model_version
	input.validation.iq.prompt_hash
	input.validation.oq.lower_bound
	input.spec.min_pass_rate
	input.signoff.package_version
}

deny contains "validation package is incomplete: a required field is missing" if {
	not required_present
}

# IQ: the manifest under review must match what was qualified, exactly.
deny contains msg if {
	input.manifest.model_version != input.validation.iq.model_version
	msg := sprintf("IQ mismatch: running %v, qualified %v",
		[input.manifest.model_version, input.validation.iq.model_version])
}

deny contains msg if {
	input.manifest.prompt_hash != input.validation.iq.prompt_hash
	msg := "IQ mismatch: prompt hash differs from the validated package"
}

# OQ: the qualification run must have cleared its statistical acceptance.
deny contains msg if {
	input.validation.oq.lower_bound < input.spec.min_pass_rate
	msg := sprintf("OQ fail: 95%% lower bound %.3f < spec %.3f",
		[input.validation.oq.lower_bound, input.spec.min_pass_rate])
}

# Change control: an open change must carry a signed validation record.
deny contains msg if {
	some c in input.change_control.open
	not c.signed_off
	msg := sprintf("change %v is not signed off", [c.id])
}

# Human sign-off: the QA release signature must be bound to this package.
deny contains msg if {
	input.signoff.package_version != input.manifest.package_version
	msg := "release signature does not match the package being promoted"
}

The negative test is three lines and belongs in CI next to the policy, because it is the only thing that proves the control is a control:

$ echo '{}' | opa eval -I -d release.rego 'data.release.gate' --format=values
[{"allow": false, "deny": ["validation package is incomplete: a required field is missing"]}]

An empty document denies. A package that fails all five checks denies with all five reasons. A complete, matching, signed package allows. CI calls it as one step, payload passed straight through, no wrapper:

import json, subprocess

def gate(manifest, validation, change_control, spec, signoff) -> None:
    payload = {"manifest": manifest, "validation": validation,
               "change_control": change_control, "spec": spec, "signoff": signoff}
    out = subprocess.run(
        ["opa", "eval", "-I", "-d", "release.rego",
         "data.release.gate", "--format=values"],
        input=json.dumps(payload), capture_output=True, text=True, check=True,
    )
    result = json.loads(out.stdout)[0]                 # -I: stdin IS input, do not wrap
    if not result["allow"]:
        raise SystemExit("release blocked:\n  - " + "\n  - ".join(sorted(result["deny"])))

Which changes re-open qualification

Two failures convinced me the classification has to be code instead of judgment, and both were provable only after the fact, which is their point. In the first, an engineer moved a constraint from the end of a prompt to the top, which was an improvement, but it shipped as a config change, and that path did not touch the model change-control process. Three weeks later a reviewer noticed a whole class of outputs had started carrying a caveat they used to omit, and nobody could say when it changed or whether it had been validated. In the second, a team fine-tuned the base model to fix a recurring failure, and although a weight update is the largest change you can make to a probabilistic system, it went out with a model card and no change record, surfacing months later when someone compared the deployed checkpoint hash to the last validated one. Nothing harmful reached anyone, but it was still a finding, because for a quarter the running system was not the validated system and we could only prove it by accident.

Both are classification failures. Nobody decided those changes needed re-validation because nobody was asked. So the decision becomes a function.

from enum import IntEnum

class Action(IntEnum):
    LOG_ONLY = 0
    OQ_RERUN = 1
    IQ_RECHECK_OQ = 2
    FULL_REVALIDATION = 3

def classify_change(change) -> Action:
    if change.kind == "prompt_edit":
        # a semantic diff, not a character diff: whitespace and typos are cosmetic
        return Action.LOG_ONLY if change.semantically_equivalent else Action.OQ_RERUN
    if change.kind == "decoding_params":
        return Action.OQ_RERUN
    if change.kind == "corpus_update":
        return Action.IQ_RECHECK_OQ            # corpus hash moves; behavior can shift
    if change.kind == "model_version":
        return (Action.IQ_RECHECK_OQ if change.same_family
                else Action.FULL_REVALIDATION)
    if change.kind in ("fine_tune", "weight_update"):
        return Action.FULL_REVALIDATION        # the largest change there is
    return Action.FULL_REVALIDATION            # unknown change: fail safe, escalate

The prompt branch is the subtle one. A character diff is the wrong test, because reformatting whitespace should not trigger an OQ while a one-word change to a constraint should, so the classifier asks whether the edit touches instruction content or only formatting, escalates when it cannot tell, and routes anything unrecognized to full re-validation.

The corpus row is the one teams argue about, because a corpus update feels like data and not code, and it is the coupling that makes this hard. Retrieval output is a function of the corpus, so changing the corpus changes the function, and the data team that edits the corpus rarely knows it just moved a validated behavior owned by QA. It re-opens OQ for the same reason a prompt edit does.

Pre-authorizing the changes you already know are coming

Read that matrix and the objection writes itself. This is unlivable. If a model version bump forces a full qualification cycle, you will be qualifying continuously and shipping nothing, and teams respond by freezing the model for a year, which trades one risk for a slowly worsening one. The instrument that resolves it comes from the device side and is worth knowing about even if you never file anything. The FDA’s predetermined change control plan, whose guidance for AI-enabled device software functions was finalized in December 2024 and rests on authority FDORA gave the agency under section 515C, lets a manufacturer describe planned modifications in advance, with the method used to validate each one and an assessment of its impact. Where the plan is authorized, those changes can be made without a new submission, provided the plan is followed.

The shift is from qualifying an artifact to qualifying a procedure for changing it, so you stop asking whether this exact model is validated on every bump and start asking whether the change sits inside an envelope you already validated a process for. Bounded is the operative word: a PCCP is permission to change specified things by a specified method. That maps onto the classifier directly, since each change kind carries a pre-agreed method and acceptance criterion, so a change inside its declared bounds routes to that method instead of a qualification cycle, and anything outside the plan falls through to the escalation branch already there.

Even with no submission and no agency reading your plan, writing the envelope down before you need it keeps change control from becoming a standing committee, and forces the argument about what counts as a bounded change at a moment when nobody is under pressure to ship. For the pharmaceutical case the material to hand a validation lead is no longer a single appendix. GAMP 5 Second Edition carries the AI/ML thinking in Appendix D11, and in July 2025 ISPE published the standalone GAMP Guide: Artificial Intelligence, around 290 pages built on that base, which is now the reference rather than the appendix.

The vendor’s clock runs faster than yours

All of that assumes you decide when the model changes. On a hosted deployment you usually do not, and the setting that decides it sits outside the validation package. Azure, which documents this most explicitly, gives a deployment three version upgrade policies: opt out of automatic upgrades, upgrade once a new default becomes available, or upgrade once the current version expires. Two of them move the weights under a manifest QA signed, one on the two weeks of notice Azure commits to before a new default lands, the other on the retirement date, while the third holds your qualified version and stops accepting requests once that version retires. So the only policy that preserves the validated state is the one that guarantees an outage on a date the vendor picks, and the two that keep the service answering are the two that end your qualification without raising anything. The deployment keeps working, it just stops being the system in the package.

Two things follow. The upgrade policy becomes an IQ field, read back from the deployments API and compared against the manifest, because a deployment whose weights can move on someone else’s schedule is not the artifact you qualified. And the validation package carries the model’s published retirement date as its own expiry, so the package goes stale before the model does, which makes periodic review on an annual cycle against a snapshot with a shorter life than that a calendar the model wins every time.

The supplier file has the matching hole. Annex 11 requires formal agreements with third parties carrying clear statements of their responsibilities, and asks for periodic evaluation that a system remains in a valid state. The clause a quality unit expects in that agreement, that the supplier notifies you of changes affecting validated status, is one no model provider will sign for weights behind an alias, so what you can get is a dated snapshot, a published retirement date, and a notice floor measured in months against a validated state measured in years. Supplier qualification here is no assessment of the vendor’s change control. It is a compensating control on your side, and the IQ hash comparison you already run stands in for the notification that is never coming.

OQ acceptance as a confidence bound

Now the part that has to survive a statistician on the audit team, who in regulated work is a real person. “The outputs looked good” is not an acceptance criterion. A confidence bound on a pass rate is.

Fix an automated check that returns pass or fail per output, and a frozen qualification set. Run the system over it and count passes. The point estimate alone says little, because a small set can look excellent by luck, so what you want is a lower bound: with 95% confidence, at least what fraction of outputs pass? The Wilson score interval behaves far better near the boundary than the normal approximation, and requiring its one-sided lower bound to clear spec gives you a criterion that is a specific integer. The one detail that silently changes the answer is the multiplier: this is a one-sided 95% bound, so z = 1.645 where a reader fills in 1.96 by reflex, and that single substitution moves the headline from three allowed failures to four.

$\text{LB}_{95} = \frac{\hat{p} + \frac{z^2}{2n} - z\sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}, \quad z = 1.645 \ \text{(one-sided 95\%)}$

from statsmodels.stats.proportion import proportion_confint

def oq_lower_bound(passes: int, n: int, confidence: float = 0.95) -> float:
    """One-sided 95% lower bound: the lower end of a two-sided interval at
    alpha = 2*(1-confidence). At confidence 0.95 that is alpha 0.10, whose
    per-tail z is 1.645, which is the one-sided 95% multiplier."""
    lo, _ = proportion_confint(passes, n, alpha=2 * (1 - confidence), method="wilson")
    return lo

def oq_accept(passes: int, n: int, spec: float = 0.95) -> bool:
    return oq_lower_bound(passes, n) >= spec

# the acceptance rule falls out of the arithmetic, not out of taste:
#   oq_accept(196, 200) -> True    (lower bound 0.9562)
#   oq_accept(195, 200) -> False   (lower bound 0.9495)

Put numbers on it. To qualify that at least 95% of outputs pass, on a set of 200, the lower bound reaches 0.95 at 196 passes and falls to 0.9495 at 195, so the rule is no more than four failures in 200, derived from the confidence you claim and never chosen because it sounded strict. Change the confidence, or the set size, and the integer moves with it. A reference build clearing the check on 98% of the set passes, whereas a drifted build at 82% fails, and what fails it is the shape of its whole distribution. No single output does it.

If the check is graded instead of binary, a 1-to-5 quality score say, do not reach for Wilks’ tolerance limits. They assume continuous order statistics, but an ordinal score has neither. Dichotomize at a threshold, pass when the score is 4 or above, and reuse the same Wilson bound. The useful fact that falls out is a sample size: to defend 95% conforming at 95% confidence with zero failures, you need $\lceil \ln 0.05 / \ln 0.95 \rceil = 59$ labeled outputs, because $0.95^{59}$ is the first power to drop below 0.05.

Sizing review as process monitoring

PQ is where regulated AI programs tend to give up and declare that a human reviews everything. For a system producing thousands of outputs a week that is a hiring requirement more than a plan, and it decays into a rubber stamp, which is worse than a smaller review done attentively. So people reach for acceptance sampling, and the analogy that gets you there also misleads.

Acceptance sampling disposes of a lot: you inspect a sample, and you accept or reject the batch before it ships. There is no batch to reject here, because the week’s outputs already reached users the moment they were generated, so what you are doing is estimating a running defect rate on delivered traffic, which is process monitoring, closer to a control chart than to lot release. The operating-characteristic curve still tells you the probability that a sample of size n passes without flagging a defect rate that has in fact drifted, which is your detection power. It just does not license the word accept.

$P_a(p) = \sum_{k=0}^{c} \binom{n}{k}\, p^k\, (1-p)^{n-k}$

Two quality levels anchor a plan: the defect rate you are willing to pass almost always, the acceptable quality level, and the rate you want to catch almost always, the rejectable quality level. Producer’s risk is flagging a stream that is good, consumer’s risk is missing one that is bad, and choosing n and c places the curve between them.

from scipy.stats import binom

def plan_risks(n: int, c: int, aql: float, rql: float) -> dict:
    """P_a(p) is the binomial CDF evaluated at the acceptance number c."""
    return {
        "producer_risk": 1 - binom.cdf(c, n, aql),   # flag a stream that is good
        "consumer_risk": binom.cdf(c, n, rql),        # miss a stream that is bad
    }

plan_risks(50,  1, 0.01, 0.05)   # -> producer 0.089, consumer 0.279
plan_risks(200, 4, 0.01, 0.05)   # -> producer 0.052, consumer 0.026

The gap between those plans is the whole argument. Both pass a clean stream almost always. Against a stream that is genuinely 5% defective, the small sample misses it 28% of the time while the larger one misses only 3%.

A zero-acceptance sample is worth carrying in your head. Its curve is $P_a(p) = (1-p)^n$, and if you review n outputs and find no defects the 95% upper bound on the true rate is about $3/n$, the rule of three, so reviewing 100 and finding zero lets you defend a rate under 3%. That turns the review-budget conversation from “how many can we afford” into “how tight a bound do we need”, which a risk assessment can answer.

It also changes the economics, because once monitoring is a sampling problem its cost stops tracking volume. A stream of 4,000 outputs reviewed in full at four minutes each is about 267 reviewer-hours a week, close to seven full-time people, whereas a 200-output sample is about 13 hours and stays there whether the week is 4,000 outputs or 40,000, since the bound you can defend depends on sample size and never on traffic.

Three honesty checks keep this from being a way to review less and feel safe. A sample bounds a rate; it does not un-ship the outputs already sent, and at 40,000 a week a 3% defect rate is 1,200 wrong answers in users’ hands before any sample speaks. It assumes the reviewer is right, and a reviewer who misses defects inflates your consumer risk beyond what the curve shows. And the sample has to be drawn at random by construction, from a seeded selection over the week’s records, because a convenience sample of the outputs someone happened to notice measures nothing. None of this touches the outputs where a single wrong answer is not recoverable. Those follow the Annex 22 reading: a qualified person reviews every one, or a probabilistic model does not serve them at all, and the FDA, Health Canada, and MHRA Good Machine Learning Practice principles say the same thing from the other direction, putting the human-AI team at the center in Principle 7.

A fourth check is about the arithmetic instead of the practice, and it is the one that eventually breaks the instrument. A binomial bound assumes defects arrive independently. Model defects do not, because they cluster on one bad corpus document, one prompt template, one topic the retriever handles badly, so a draw that lands several outputs on the same broken template is not that many independent trials, and the interval it produces is tighter than the evidence supports. Stratify by template or use case, take the bound within stratum, and when a sample does hit a cluster treat it as a reason to stop counting and go find the cause rather than as defects toward a rate.

One procedural question outranks the statistics, and I got it wrong the first time. When a reviewer finds a bad output, is that a deviation? The acceptance criterion already states that a fraction of outputs will fail, so a failure inside the qualified bound is the process performing as specified, and the deviation is the rate breaching the bound, which is awkward, because the event that constitutes the deviation is an aggregate no reviewer can observe. A reviewer sees one wrong answer. The person with standing to open a deviation sees a weekly rate.

Leave that unwritten and you get one of two failures, and I’ve watched both. Reviewers open a deviation per bad output, the quality system fills with records that all close as expected model behaviour, and people stop flagging. Or nobody opens anything, because every case was, correctly, within expected behaviour, and the rate walks past its bound with a complete audit trail behind it. So the procedure names the defect as the unit a reviewer logs, the statistic as the trigger, and one owner for the statistic, and the training says out loud that the acceptance criterion pre-authorizes a failure rate, because a reviewer who has not been told that will either escalate everything or stop reading.

Where a model does not belong

Everything above validates the system as it is today, but it will move. Traffic wanders off the qualification set, a corpus refresh shifts retrieval, the vendor moves the weights. Drift is the normal condition, and the defense is a schedule: re-run OQ on a cadence and on every classified change.

The cheaper move is to have less to validate. If an output is checkable, check it and skip the model there. A dose calculation should be arithmetic with a unit test. A required field should be schema validation. Reserve the probabilistic machinery for the open-ended parts, summarizing a document or drafting a narrative, where no cheaper oracle exists and where, not by coincidence, a wrong answer is recoverable. Annex 22 arrives at this from the regulatory side and I arrived at it from the maintenance side, and we meet in the same place: every deterministic component you carve out is one you qualify once and never sample again.

Nothing here makes a language model deterministic. It makes the system around it auditable, which is what GxP is actually asking for. You can reproduce an output or name the day you no longer could, you can name who is accountable for it, and you can show that what shipped is what was validated. The model will still surprise you. The record, the gate, and the sample are what let you answer for it.