Quality Characteristics for AI-Based Systems
Established quality models like ISO/IEC 25010 measure attributes such as performance, usability, and security - but AI introduces entirely new dimensions of risk. This topic covers the AI-specific quality characteristics you need to evaluate: can it explain its decisions, adapt without breaking, operate safely without supervision, and make fair, unbiased choices?
~16 min read
The core ideas, explained
Autonomy
Autonomy is the degree to which an AI system can perform its tasks and make decisions without human intervention. The higher the autonomy, the harder the system is to test, because it may encounter situations the developers never anticipated.
Low autonomy: a medical AI highlights a potential tumor on an X-ray, but a human doctor must still make the final diagnosis and prescribe treatment (human-in-the-loop).
High autonomy: an automated algorithmic trading system executes millions of dollars in stock trades per second based on market fluctuations, with no human approval at all.
Adaptability (continuous learning)
Adaptability refers to an AI system's ability to change its own behavior over time as it encounters new data or environmental changes. If a system adapts dynamically in production, a test that passed on Monday might fail on Friday - not because anything was deployed, but because the underlying model has changed itself.
A spam filter continuously learns new vocabulary used by hackers - if hackers start using the word "Fr33," the system adapts to block it without a developer writing a single line of new code.
A dynamic pricing algorithm for a ride-sharing app raises prices in response to a sudden snowstorm, even if it has never encountered a snowstorm in that specific city before.
Bias and fairness
Bias occurs when an AI system produces outputs that systematically disadvantage certain individuals or groups; fairness is the metric by which we ensure the AI treats all demographic groups equitably. Bias usually stems from historical prejudice hidden inside the training data, rather than from malicious code written by developers.
Selection bias: a resume-screening AI is trained on 10 years of past hiring data at a firm that historically hired mostly men. It "learns" that male candidates are statistically more successful and begins unfairly downgrading female applicants.
Representation bias: a facial recognition system trained mostly on images of lighter-skinned individuals shows a significantly higher failure rate when recognizing darker-skinned users.
Robustness
Robustness is a system's ability to maintain performance when faced with invalid inputs, noisy data, or malicious environmental changes such as adversarial attacks. AI can be surprisingly fragile when given data that falls just slightly outside its normal training distribution.
Noisy data: a self-driving car's computer vision system keeps correctly identifying lane markings even when the camera lens is partly obscured by rain or mud.
Adversarial attack: a hacker places a few small, strategically placed stickers on a "Stop" sign. A human still reads it as a Stop sign, but a non-robust AI might suddenly classify it as a "Speed Limit 65" sign.
Transparency, Interpretability, and Explainability
These three concepts are closely related but distinctly different - together they deal with penetrating AI's "black box" nature, each from its own angle: the process, the model, and a single output.
| Characteristic | Definition | The perspective | Example |
|---|---|---|---|
| Transparency | The overarching concept that a system's design, data sources, and limitations are openly documented and accessible. | The process | Publishing a document stating exactly what datasets were used to train the AI and what its known limitations are. |
| Interpretability | The degree to which a human can comprehend the entire underlying mechanics of the model. | The model (white-box) | A simple decision tree is highly interpretable, since a tester can trace every branch; a deep neural network is not, since it uses millions of hidden weights. |
| Explainability | The degree to which the system can give a human-understandable reason for one single, specific output. | The output (black-box) | An AI denies a loan application. The user asks "Why?" and the system answers: "Your debt-to-income ratio is 5% too high." |
AI and safety
Safety is the absence of unreasonable risk of harm to human life, health, property, or the environment. Because AI systems are probabilistic and often highly autonomous, they introduce novel safety risks that conventional software rarely has to deal with.
-
Unpredictability in novel situations:
Conventional software fails predictably. AI software, placed in an environment it wasn't trained for, can fail unpredictably instead.
Example: An autonomous drone trained to fly in clear skies might behave erratically and crash into a crowd if it encounters heavy snowfall for the first time.
-
Failsafe mechanisms and human overrides:
Safety-critical AI systems must be designed with strict boundaries - if the AI becomes confused (a low confidence score), it must fail safely rather than keep guessing.
Example: An autonomous vehicle's AI loses confidence in its camera feed due to blinding sun glare. Instead of guessing where the road is, the failsafe instantly triggers the hazard lights, pulls the car to the shoulder, and hands control back to the driver.
- Safety vs. performance trade-off: Sometimes, maximizing an AI's raw performance reduces its safety margins - testers must weigh these two conflicting characteristics against each other rather than optimizing for just one.
Key points to remember
- ISO/IEC 25059 is the international standard that extends traditional software quality models to include AI-specific characteristics.
- Interpretability vs. Explainability: interpretability is understanding how the whole machine works; explainability is understanding why it gave one specific answer.
- Autonomy levels dictate testing rigor - a fully autonomous system needs exponentially more rigorous safety testing than one that only recommends actions to a human.
- Bias is a data problem, not a code problem: you can't test for bias by reading the source code - you test for it by auditing the data and analyzing the statistical output.
- Robustness protects against the unexpected: testing it means intentionally feeding the AI noisy, corrupted, or maliciously altered data (adversarial testing) to see if it breaks.
Terminology
A few terms from this topic worth knowing precisely.
The degree to which an AI system can perform its tasks and make decisions without human intervention - the higher the autonomy, the harder the system is to test.
An AI system's ability to change its own behavior over time as it encounters new data or environmental changes, even without a developer releasing new code.
Bias is when an AI system's outputs systematically disadvantage certain individuals or groups; fairness is the degree to which it treats all demographic groups equitably.
An AI system's ability to maintain its performance when faced with invalid inputs, noisy data, or malicious changes such as adversarial attacks.
The overarching principle that a system's design, data sources, and known limitations are openly documented and accessible - a process-level property.
The degree to which a human can comprehend the entire underlying mechanics of a model - a white-box property of the model itself.
The degree to which a system can give a human-understandable reason for one single, specific output - a black-box property of that output.
The absence of unreasonable risk of harm to human life, health, property, or the environment - AI's probabilistic, autonomous nature introduces novel safety risks.
A designed safeguard where a low-confidence system falls back to a safe state, or hands control to a human, rather than continuing to act on an uncertain decision.
Summary
Testing AI requires expanding the traditional view of software quality to include AI-specific characteristics. Testers must evaluate an AI's Autonomy (can it act alone?), its Adaptability (does it change over time?), and its Robustness (can it handle bad data?). Because AI often acts as a black box, testers must also evaluate its Interpretability (model transparency) and Explainability (output justification). Crucially, testers must guard against historical Bias and ensure the probabilistic, unpredictable nature of the AI does not compromise human Safety.
| Characteristic | One-line memory hook |
|---|---|
| Autonomy | How much can it do without a human stepping in? |
| Adaptability | Does it learn and change its behavior in production? |
| Bias / Fairness | Does it treat all demographic groups equally? |
| Robustness | Does it still work if the input data is messy or malicious? |
| Interpretability | Can a human understand the underlying math/structure? |
| Explainability | Can it tell me why it made this specific decision? |
| Safety | Will a failure cause physical or catastrophic harm? |
Check your understanding
20 quick questions - click an option to see if you got it right.