Acceptance Criteria for AI-Based Systems
In traditional software, acceptance criteria are binary: a requirement is either met or it isn't. Because AI systems are probabilistic, that rigid approach breaks down. This topic covers how to redefine acceptance criteria using statistical thresholds, tolerance ranges, and context-dependent metrics - while explicitly covering AI-specific quality characteristics like fairness and robustness.
~12 min read
The core ideas, explained
Shifting from deterministic to probabilistic criteria
You cannot write an acceptance criterion for an AI system that demands 100% perfection - if you do, the test will always fail. Instead, AI acceptance criteria define an acceptable range of performance based on statistical metrics, not an exact match.
| Feature | Conventional acceptance criteria | AI-based acceptance criteria |
|---|---|---|
| Nature of output | Deterministic - exact matches. | Probabilistic - ranges and thresholds. |
| Pass/fail logic | Binary (true/false). | Statistical (meets a minimum threshold). |
| Example (functional) | "When the user clicks 'Checkout,' the cart total must equal Item A + Item B exactly." | "The model must correctly classify images of stop signs with an accuracy of ≥ 98%." |
| Example (performance) | "The API must return the user profile data in under 2.0 seconds." | "The AI must process real-time voice translation with a confidence score of ≥ 85%." |
Context-dependent thresholds
The statistical thresholds in an acceptance criterion can't be arbitrary - they must be heavily dictated by the context of use and the associated risk. A lower threshold is fine where a wrong answer is harmless; a near-perfect threshold is required where a wrong answer could be catastrophic.
Low-risk context - a music recommendation engine: if the AI recommends a song the user dislikes, the impact is negligible, so a modest acceptance criterion is fine - "the system must recommend a song the user listens to for at least 30 seconds with a precision of ≥ 60%."
High-risk context - an autonomous braking system: if the AI fails to detect a pedestrian, a fatal accident can occur, so the threshold must be astronomically high - "the computer vision system must detect pedestrians in the vehicle's path with a recall rate of ≥ 99.999%, across diverse weather conditions."
Writing AC for AI-specific quality characteristics
Acceptance criteria must explicitly cover the non-functional AI characteristics from the quality-characteristics topic - bias/fairness, robustness, explainability, and the rest - translating each abstract concept into a measurable target rather than a vague aspiration.
-
Bias / fairness criteria:
A criterion like "the AI must not be biased" is unmeasurable and shouldn't be accepted - it gives no number to test against.
Example: Strong AC: "the facial recognition system's false rejection rate must not vary by more than 2% across any tested demographic groups (age, gender, ethnicity)."
-
Robustness criteria:
Likewise, "the AI must handle bad data" gives testers nothing concrete to verify.
Example: Strong AC: "the model's classification accuracy must remain ≥ 90% even when up to 15% digital noise (blur/static) is introduced into the input images."
-
Explainability criteria:
And "the AI must explain its choices" doesn't specify what an acceptable explanation actually looks like.
Example: Strong AC: "for every denied loan application, the system must output the top three weighted variables that negatively influenced the decision, in human-readable text."
Evolving acceptance criteria (adaptability)
If an AI system is Adaptable - it learns continuously in production - its acceptance criteria might need to evolve over time too. A system that passes staging criteria on Monday can adapt itself to new user behavior and fail those same criteria on Friday, without a single code change.
The mitigation is to write AC as ongoing monitoring thresholds rather than one-time gates - e.g. "the model's accuracy must not drift below 92% over any 7-day period in production." This ties directly back to test monitoring and maintenance testing after deployment.
Key points to remember
- Perfection is impossible: don't write AC demanding 100% accuracy - set realistic statistical thresholds instead.
- Risk dictates the threshold: high-risk applications need strict thresholds; low-risk applications can tolerate looser ones.
- Non-functional AC are mandatory: write measurable criteria for bias, robustness, safety, and explainability - not just functional accuracy.
- Continuous monitoring: adaptable AI needs acceptance criteria that act as ongoing production guardrails, watching for concept drift rather than being checked once.
Terminology
A few terms from this topic worth knowing precisely.
The conditions an AI-based system must satisfy to be considered acceptable - typically statistical thresholds or tolerance ranges (e.g. "accuracy \u{2265} 95%") rather than an exact, binary pass/fail condition.
Bias is when an AI system's outputs systematically disadvantage certain individuals or groups; fairness is the degree to which it treats all demographic groups equitably.
An AI system's ability to maintain its performance when faced with invalid inputs, noisy data, or malicious changes such as adversarial attacks.
The degree to which a system can give a human-understandable reason for one single, specific output - a black-box property of that output.
An AI system's ability to change its own behavior over time as it encounters new data or environmental changes, even without a developer releasing new code.
A designed safeguard where a low-confidence system falls back to a safe state, or hands control to a human, rather than continuing to act on an uncertain decision.
A false positive wrongly flags a negative case as positive (e.g. blocking an innocent post); a false negative misses a real positive case (e.g. letting offensive content through) - which one matters more depends on context.
A gradual decline in an AI model's real-world performance over time, as the patterns in live production data diverge from what the model was originally trained on.
Summary
Acceptance criteria for AI-based systems must abandon the binary, deterministic mindset of traditional software. Because AI relies on probabilities, testers and business stakeholders must collaborate to define acceptable statistical thresholds (e.g. "≥ 95% accuracy"). These thresholds are entirely context-dependent, driven heavily by the risk of the application. Abstract AI quality characteristics - like fairness, robustness, and explainability - must also be translated into strict, measurable criteria to ensure the system is safe and compliant before release.
| Concept | One-line memory hook |
|---|---|
| Deterministic vs. probabilistic AC | Exact match vs. statistical threshold |
| Context-dependent thresholds | Higher risk demands a stricter threshold |
| Non-functional AC | Bias, robustness, and explainability need numbers too |
| Vague vs. measurable AC | "Must be fair" fails; "≤ 2% variance" passes |
| Adaptability and AC | Adaptable systems need ongoing monitoring thresholds |
| FAR vs. FRR | Stricter security often means weaker usability, and vice versa |
Check your understanding
15 quick questions - click an option to see if you got it right.