mavai-R

This page is the front door to the mavai statistical model. It is intended for engineers, reviewers, product owners, and auditors who need to understand what the model is for, what kind of evidence it produces, and what it deliberately does not claim — without working through the full statistical companion.

The statistical companion remains the canonical reference for formulae, derivations, and edge cases. This document summarises the conceptual frame around it.

1. Motivation: deterministic testing breaks at uncertainty's boundary

Traditional software testing assumes that the same input should produce the same output. For deterministic software, that assumption is usually sound: one failing test is meaningful evidence of a defect. But modern AI-backed services, especially LLM-based services, often do not behave that way. Their behaviour is distributional. The same prompt may produce a correct response most of the time, an invalid response occasionally, and a borderline response rarely.

In such systems, uncertainty is not a nuisance to be engineered away. It is an intrinsic property of the system under test. The distribution of outputs is the behaviour.

The practical consequence is that the question "did this invocation pass?" is no longer the right question. The better question is: how often does this service satisfy its contract, under stated conditions, and with what evidential strength?

2. Scope: what the model is for

The mavai model focuses on two quality dimensions:

• Functional stochasticity — whether the service produces an acceptable result. Correctness is modelled as repeated pass/fail observations.
• Temporal stochasticity — how long successful invocations take. Latency is modelled through empirical percentiles rather than averages.

A contract may exercise either dimension on its own or both together; when both are in scope, the overall verdict requires each to pass, and the model does not assume the two are statistically independent. Every assertion in either dimension counts towards the verdict by default; a run-time switch can make one dimension, or both, advisory, so that its verdict is still reached and reported but cannot fail the test.

What the model is not for:

The model does not claim to prove that an AI system is safe, truthful, or correct in any absolute sense. It provides disciplined, repeatable evidence about observed behaviour under stated sampling conditions. It is a testing and evidence framework, not a philosophical theory of AI correctness.

3. The core model, in plain language

Each invocation of the service is treated as a trialⓘA single invocation of the service, evaluated against a criterion, yielding a pass or a fail. The basic unit of evidence in the functional model.Called a sample in the punit and feotest implementations. The two terms are synonymous in mavai's vocabulary; this document uses trial in keeping with the statistical literature.. For a given criterion — for example, "the response parses as JSON" or "the answer satisfies a rubric" — the trial either passes or fails. Repeating the invocation over a defined set of inputs produces a count: how many trials passed out of how many attempted. Under stated assumptions of approximate independenceⓘThe assumption that one trial's outcome does not influence another's. The model asks for approximate independence and records the assumption rather than assuming it away. and stationarityⓘThe assumption that the underlying success rate does not drift over the period during which trials are collected. Assumed and documented, not proven., that count can be treated as binomialⓘThe distribution of the count of passes in a fixed number of independent trials with the same success probability. evidence about the service's success rate.

The model does not decide on the raw observed percentage. An observed rate is not the same thing as evidence: if a service passes 93 out of 100 trials, the model does not pretend the true success rate is exactly 93%. Each verdict is instead decided by an exact statistical test, which turns the question into a bar on the pass count: the smallest number of passes that counts as sufficient evidence. The bar is set so that the chance of a wrong verdict stays within the configured confidence levelⓘThe long-run reliability of a statistical procedure over repeated experiments. For a decision, 99% means the test reaches the wrong verdict it guards against at most 1% of the time; for an interval, that it covers the true value 99% of the time. A property of the procedure, not of any single verdict.. Against a requirement of 90% with 100 trials at the 99% level, for example, the bar is 97 passes, so 93 passes FAILS even though 93% exceeds 90%. The requirement is the claim; the count is the evidence the claim needs. Which test sets the bar depends on the question being asked (section 4).

Reports also show a Wilson score intervalⓘA confidence interval on a binomial proportion that behaves well at extreme rates and small sample sizes, where the naive normal approximation fails. The model reports it to describe the uncertainty around an observed rate; it decides no verdict. around each observed rate. It describes how uncertain the rate is; it plays no part in the verdict.

4. Compliance vs regression

The model distinguishes two testing questions that are often confused.

• A compliance test asks whether the service has shown enough evidence to satisfy an external requirement, such as an SLA, SLO, or policy threshold.
• A regression test asks whether current behaviour has degraded relative to a measured baseline.

Each question has its own test. Compliance of a success rate is decided by the exact binomial testⓘA test on a pass count that uses the binomial distribution itself, so its error rate holds at every sample size. The model uses it for compliance: PASS when the count reaches the bar the requirement demands., which asks whether the pass count reaches the bar the requirement demands. Regression of a success rate is decided by the one-sided Fisher exact testⓘA test that compares two pass counts, the baseline's and the test's, without assuming the baseline's rate is known exactly. The model uses its one-sided form for regression., which compares the test's pass count with the baseline's and allows for the fact that the baseline is itself only a sample. Latency follows the same split: a latency requirement is judged by the exact binomial test applied to the count of responses within the time limit, and latency regression by a precedence testⓘA test that compares two samples of measurements by the order in which their values fall, assuming nothing about the shape of their distribution. The model uses it for latency regression., which compares the test's percentile against the ordered baseline latencies without assuming any shape for their distribution.

Both compliance and regression test kinds use repeated observations and one-sided decision rules, but the interpretation of PASS and FAIL differs. A compliance PASS means evidence supports compliance at the configured level. A regression PASS means no degradation signal was observed at the configured cutoff; it does not prove equivalence to the baseline.

Keeping these two questions separate is the single most important discipline for avoiding overclaiming.

5. Criteria: why one "overall correctness" number is not enough

Real service contracts rarely contain one kind of failure. A malformed JSON response, a slightly unsuitable tone, a missing required field, and a safety violation are all "failures," but they are not failures of the same kind. Aggregating them into one pass rate can hide exactly the failure mode that matters most.

The mavai model therefore allows a contract to be decomposed into criteria, each with its own threshold, confidence level, input set, and verdict. The combined contract verdict is structural: all required criteria must pass. The model does not assume that criteria are statistically independent merely because they are reported separately.

6. Empirical vs categorical clauses

A natural objection to any rate-based model is: "surely some failures are unacceptable at any rate." The model takes this seriously and distinguishes two kinds of clause.

• An empirical clause says, in effect, "this behaviour must succeed at least this often." It is bounded by a rate.
• A categorical clause says, "this must not happen." It is not bounded by a rate at all.

The model does not pretend that setting a threshold to 99.999% converts a categorical obligation into a statistical one. Intolerable failures are handled architecturally — by guardrails, filters, schemas, refusal mechanisms — and those architectural controls can (and indeed should!) themselves be tested statistically.

Tolerable failures are bounded statistically; intolerable failures are bounded architecturally; and the architecture itself is bounded statistically.

7. What the model offers that ordinary tests cannot

Four claims:

1. It turns flaky-looking behaviour into measurable behaviour.
2. It separates observed rates from statistically supported claims.
3. It records the assumptions under which a verdict is meaningful: factor configuration (e.g. model version, temperature), covariates (e.g. model version, deployment region, time of day), baseline age, sampling design, and population claim.
4. It produces audit-friendly evidence: counts, thresholds, confidence bounds, cutoffs, caveats, and provenance.

The model's transparent-statistics mode is designed for exactly this audience: auditors, stakeholders, educators, and regulators who need to see the reasoning behind a verdict, not just the verdict.

8. What the model does not offer

The model does not remove uncertainty. It measures it.

It does not guarantee that future behaviour will match past behaviour. It assumes, and then documents, the conditions under which repeated observations can be interpreted: approximate independence, stationarity, and a defined input population. Where those assumptions are doubtful, the model does not magically repair them; it surfaces the doubt as part of the evidence trail.

A reported confidence level is therefore not a claim that an individual verdict has a certain probability of being correct. It is a statement about the long-run behaviour of the decision procedure under the stated assumptions. The model is frequentist by design, for exactly this reason: it is a discipline for producing defensible evidence, not a machine for producing certainty.

Glossary

Short definitions of the key statistical terms used above. The full glossary lives alongside this overview; only the terms that appear in the body are repeated here.

• Binomial — the distribution of the count of passes in a fixed number of independent trials with the same success probability. The companion uses binomial reasoning to relate observed pass counts to the underlying success rate.
• Confidence level — the long-run reliability of a statistical procedure if the experiment were repeated under the same conditions. For a decision, a 99% level means the test reaches the wrong verdict in the direction it guards against (a false PASS for compliance, a false alarm for regression) at most 1% of the time; for an interval, that the interval covers the true value 99% of the time. It is a property of the procedure, not a probability attached to any single verdict.
• Exact binomial test — a test on a pass count that uses the binomial distribution itself rather than an approximation to it, so its error rate holds at every sample size. The model uses it for compliance: PASS when the count reaches the bar the requirement demands.
• Fisher exact test — a test that compares two pass counts, here the baseline's and the test's, without assuming the baseline's rate is known exactly. The model uses its one-sided form for regression: FAIL when the test's count falls too far below what the baseline makes plausible.
• Frequentist — the school of statistical inference in which probabilities describe the long-run behaviour of procedures over repeated sampling, not degrees of belief about individual events. The mavai model is frequentist by design (the companion sets out why it prefers this to a Bayesian alternative); this is why a confidence level is a property of the procedure rather than a probability of the verdict.
• Independence — the assumption that the outcome of one trial does not influence the outcome of another. Required for binomial reasoning to apply; the model asks for approximate independence and records the assumption rather than assuming it away.
• One-sided decision rule — a test that asks whether the evidence exceeds (or fails to exceed) a single threshold in one direction only — for example, "did at least 97 of 100 trials pass?", where 97 is the bar that a 90% requirement demands — rather than testing equality against a two-sided interval.
• Percentile — the value below which a stated fraction of observations fall. Latency is summarised by empirical percentiles (e.g. p95) rather than means, because the upper tail is what users and SLAs care about.
• Population — the set of inputs and conditions about which a verdict is meant to generalise. The model's claims are only as strong as the population definition recorded alongside them.
• Precedence test — a test that compares two samples of measurements by the order in which their values fall, assuming nothing about the shape of their distribution. The model uses it for latency regression: FAIL when the test's percentile lies beyond a baseline latency it should rarely exceed.
• Stationarity — the assumption that the underlying success rate does not drift over the period during which trials are collected. Like independence, this is assumed and documented, not proven.
• Trial — a single invocation of the service, evaluated against a criterion, yielding a pass or a fail. The basic unit of evidence in the functional model.
• Wilson score interval — a confidence interval on a binomial proportion that behaves well at extreme rates (near 0 or 1) and at small sample sizes, where the naive normal approximation fails. The model reports it to describe the uncertainty around an observed rate; it decides no verdict. (Up to companion 1.4.1 it did; the exact tests above replaced it.)

For the full derivations, formulae, and the rules each implementation must conform to, see the statistical companion. For the language-agnostic conformance fixtures derived from the companion, see ../inst/cases/.