Archiv 2026

From the Butterfly Effect to a practical assurance concept for multi-agent AI systems

Abstract

As AI evolves from individual models toward interacting autonomous agents, assurance must evolve with it. Testing individual agents is insufficient when outputs become inputs for other agents, decisions depend on previous agent decisions, and deviations can propagate through an interaction network.

This article proposes Agent System Stability as an assurance property and presents Agent System Stability Testing (ASST) v1.1: a structured method for establishing the system’s inherent stochastic variability, introducing controlled perturbations, tracing their propagation, selecting perturbation-appropriate measures, observing detection and recovery, and assessing outcomes against acceptance criteria defined before testing.

ASST does not claim that LLM agent systems are mathematically chaotic, nor that its individual measurement ideas are unprecedented. Controlled fault injection, propagation analysis and amplification factors already appear in emerging multi-agent research. ASST is proposed as an assurance-oriented integration and generalisation across perturbation classes, linking measurements to acceptance decisions and to the placement of verification and human-control gates.

Don’t just test the agents. Test the dynamics — against a baseline and an acceptance envelope.

1. The System Is More Than Its Agents

Artificial intelligence is moving from individual language models toward systems of interacting agents. One agent may analyse requirements. Another may retrieve evidence. A third may challenge the analysis. A fourth may evaluate risks. An orchestrator may combine their conclusions and decide which agent acts next.

Each component can be tested individually. But this leaves an important question unanswered: What happens when these agents interact?

Consider a simplified architecture:

Requirements Analyst → Technical Analyst → Independent Reviewer → Risk Analyst → Validator

The output of one agent becomes part of the input of another. Consequently, the behaviour of the overall system depends not only on the quality of its individual agents but also on the way information travels through the network. The system therefore possesses an additional characteristic: interaction dynamics.

Recent research into networks of LLM-based cognitive agents supports the importance of this distinction. Zomer and De Domenico demonstrated experimentally that communication topology can affect collective behaviour. In their experiments, LLM agents exhibited consensus tendencies and, in some configurations, premature convergence; changing the network topology could alter these effects.

This means that the communication architecture itself can influence system behaviour. That is where Agentic AI becomes a systems-engineering problem.

2. From the Butterfly Effect to Agentic AI

In 1963, meteorologist Edward Lorenz published Deterministic Nonperiodic Flow. Lorenz studied a small deterministic system of nonlinear differential equations representing atmospheric convection. His results became foundational to the study of sensitivity to initial conditions.

The important lesson is often oversimplified. Chaos does not simply mean randomness. A deterministic system can follow defined rules while nevertheless being highly sensitive to small changes in its state.

This does not mean that modern LLM agent systems are therefore chaotic systems. No such general conclusion follows. But the underlying question is highly relevant:

What happens when a small perturbation enters a complex interacting system?

This question has already been investigated in classical agent-based modelling. Bertolotti, Locoro and Mari examined two deterministic agent-based models in 2020. They changed a single initial condition associated with one agent and analysed the resulting macro-level behaviour. In both models examined, the impact of that change was substantial. They also found qualitative evidence that factors such as the number of agents and their connectivity were related to sensitivity.

The experiment therefore demonstrated something important: A small local change can materially affect global behaviour in at least some agent-based systems.

3. The Agent Butterfly Effect

Consider a technical requirement:

Maximum permitted response time: 500 ms.

Now imagine an agent transforms this during an early processing step into:

Expected response time: approximately 500 ms.

The textual difference appears small. The engineering meaning is not. In the first case, 500 ms represents a constraint. In the second, it has become an expectation.

A reviewer might return to the original requirement, detect the deviation and correct it. The perturbation disappears. But another system might behave differently. The next agent accepts the statement. An architecture agent uses it as an assumption. A risk agent evaluates the resulting architecture. A decision agent receives conclusions that are internally consistent but derived from an altered premise.

The relevant question is therefore no longer only: Did Agent A make an error?

What did the system do with that error?

4. Propagation Is a Real Multi-Agent Problem

This question is increasingly appearing in current research. A 2026 preprint, PropUQ-MAS, represents multi-agent execution as a communication-structured graph and explicitly models uncertainty inherited from upstream agent messages. Its premise is that intermediate errors can be inherited by downstream agents, and isolated uncertainty measurements therefore fail to capture all reliability risks in a multi-agent system.

Another 2026 preprint, Hallucination Cascade, investigated factual inconsistencies across sequential LLM-agent interactions. Its findings caution against a simplistic assumption that errors must always grow. The reported experiments found that hallucination scores could decrease across the investigated agent cascade, even while factual accuracy declined slightly.

Agent interaction may therefore correct one type of deviation while simultaneously degrading another quality dimension. We should neither assume amplification nor correction. We should measure what the system actually does.

5. Three Fundamental Responses to a Perturbation

Attenuation

The deviation decreases. Another agent, an evidence check or an independent verification mechanism corrects the problem. The system moves back toward an acceptable state.

Propagation

The deviation survives. Other agents inherit the changed information without materially increasing or decreasing its impact.

Amplification

The deviation becomes more consequential. Downstream agents derive additional conclusions from it, distribute it to additional agents or incorporate it into decisions.

Agent-system stability should not be assumed from agent reliability. It should be tested experimentally.

6. Introducing Agent System Stability

This article proposes using the term Agent System Stability to describe how a multi-agent AI system reacts to controlled perturbations in its inputs, agents, tools or communication structure.

A stable system should not necessarily produce identical outputs. That would be an unrealistic requirement for many probabilistic AI systems. Instead, stability concerns whether small disturbances remain within an acceptable behavioural envelope.

The key questions become: Does the deviation propagate? Does it grow? Does another component detect it? Does the system recover? Does the final outcome remain acceptable?

This shifts assurance from purely component-level evaluation toward system-level behaviour.

7. The Agent System Stability Test – ASST v1.1

For probabilistic agent systems, a single nominal execution is not a sufficient baseline. Two unperturbed runs may already diverge because of model sampling, retrieval variation, tool timing or other stochastic effects. ASST must therefore begin by characterising inherent system variability.

Step 0 — Establish the null baseline

Execute the unperturbed system n times under the same controlled conditions:

S₀¹, S₀², …, S₀ⁿ

Record the relevant trajectories and outcomes. Pairwise or reference-based distances between these nominal runs form an empirical baseline distribution, D_null. The number of runs is context-dependent and should be justified by the intended confidence and variability of the system; ASST does not prescribe a universal n.

A perturbation effect should only be interpreted as system sensitivity when it can be distinguished from this inherent run-to-run variability. Depending on the metric and sample size, that distinction may use confidence intervals, quantiles, effect sizes or a justified statistical test.

Without D_null, ASST risks measuring stochastic variability and calling it sensitivity.

Define the acceptance envelope before perturbation

The Acceptable Behavioural Envelope (ABE) must also be defined before the perturbation results are examined. It should be derived from the use case, hazard and risk analysis, system requirements, assurance objectives and the consequence of an incorrect outcome. In regulated or safety-related contexts, the chain should be explicit:

Hazard / Risk → Assurance Requirement → Acceptance Criterion → Test Metric → Evidence → Decision

The envelope may contain quantitative limits, categorical constraints and veto conditions. It must not be adjusted after observing a poor result merely to make that result acceptable.

Perturbation classes

• Information perturbation: modify a factual input or numerical parameter.
• Semantic perturbation: alter wording while retaining substantially the same task intent.
• Agent perturbation: modify an intermediate agent response.
• Tool perturbation: corrupt, delay, remove or replace a tool result.
• Topology perturbation: change a communication path, dependency or interaction sequence.
• Confidence perturbation: modify uncertainty or confidence information communicated between agents.

These classes are deliberately heterogeneous. ASST v1.1 therefore does not force all perturbations into one universal scalar measure.

8. The ASST v1.1 Process

1. Baseline: execute repeated unperturbed runs and establish D_null.
2. Specify: derive the ABE, acceptance criteria, veto conditions and selected metrics from the assurance context before testing.
3. Perturb: introduce one defined and reproducible disturbance.
4. Trace: observe how the deviation travels through agents, evidence, tools and system states.
5. Measure: use a metric appropriate to the perturbation class.
6. Verify & Recover: determine whether controls detect the disturbance and whether the system returns to an acceptable state.
7. Assess: compare the evidence with D_null and the predefined ABE, then apply the decision rule.
8. Improve & Retest: place or strengthen controls where propagation, non-detection or failed recovery is observed, then repeat the test.

BASELINE → SPECIFY → PERTURB → TRACE → MEASURE → VERIFY/RECOVER → ASSESS → IMPROVE/RETEST

9. Measuring System Response

For a state representation S(t), an application-specific distance function d may still be useful. But in a stochastic system the comparison is no longer simply one S₀(t) against one Sε(t). The perturbed trajectory must be interpreted against the distribution of nominal behaviour.

For quantitative artefacts, distance may be numerical. For requirements it may concern constraint violations; for hazard analysis, changed hazards, classifications or mitigations; for structured artefacts, graph or schema differences. The metric must preserve the engineering meaning of the artefact being assessed.

ASST v1.1 distinguishes between ordinal/metric perturbations and categorical/structural perturbations.

10. Amplification Is One Measure — Not the Universal Measure

Where the perturbation has a meaningful ordinal magnitude ε and outcome distance is defined on a compatible, normalised scale, an amplification-style measure can be useful:

A = ΔOutcome / ε

Recent work on hallucination cascades has already used an amplification-factor interpretation in which values below one represent net attenuation. ASST therefore treats amplification measurement as prior art to be adapted and generalised, not as a novel universal metric.

A tolerance band must be defined in advance. Rather than treating “A ≈ 1” as self-explanatory, the test specification should define a context-specific band [1−τ, 1+τ]. Only within that declared band should behaviour be described as approximately proportional propagation.

A should not be used where the perturbation magnitude is undefined or approaches zero. This includes meaning-preserving semantic reformulations, removal of a tool, or deletion of a communication edge unless a justified ordinal scale has independently been defined. In such cases the ratio may be meaningless, infinite or dimensionally incoherent.

Alternative measures for categorical and structural perturbations

For these perturbations ASST v1.1 uses outcome-oriented measures such as:

• Detection Rate (DR): proportion of injected disturbances detected before an unacceptable outcome.
• Recovery Rate (RR): proportion of detected or injected disturbances after which the system returns to the predefined ABE.
• Outcome Acceptability Rate (OAR): proportion of perturbed runs whose final outcome remains within the ABE.
• Propagation Depth (PD): number or proportion of downstream boundaries crossed before detection, containment or termination.

The purpose is not to produce a single impressive number. It is to select a measure that is well-defined for the disturbance being tested.

11. Why This Is Not a Lyapunov Exponent

The analogy with dynamical-systems theory is nevertheless useful. Lyapunov methods analyse stability by examining how system trajectories behave under perturbations.

Xu and colleagues published a 2026 study in npj Artificial Intelligence that formulates constraint fusion in an LLM-based multi-agent autonomous system as a stability-convergence problem. Their framework uses a Lyapunov function as a system-level state metric and combines it with cooperative differential-game mechanisms.

This provides evidence that concepts from stability and control theory can be meaningfully explored in LLM multi-agent architectures. But scientific precision matters. The proposed ASST amplification measure should not simply be called a Lyapunov exponent.

For engineering assurance, the operational question is simpler: Does a small disturbance remain small?

12. Stability Is Multidimensional — and Requires a Decision Rule

Agent System Stability should not be compressed into one universal score. A system may tolerate semantic variation while reacting strongly to incorrect tool data, or recover from one agent failure while remaining sensitive to communication topology.

The proposed Agent System Stability Profile therefore retains separate dimensions: Semantic, Information, Agent, Tool, Topology and Recovery Stability. For each dimension, the test specification should identify the applicable metric, D_null where relevant, the ABE, and whether the criterion is veto-capable or informative.

Proposed assurance decision logic

• GO: all veto criteria are satisfied and the tested behaviour remains within the predefined ABE.
• CONDITIONAL GO: no veto criterion is violated, but one or more informative criteria require a documented mitigation, limitation or follow-up test.
• NO-GO: at least one veto criterion is violated — for example an unacceptable safety-related outcome, failure to detect a defined critical tool fault, or recovery below a predefined minimum.

Which dimensions are veto-capable is not universal. It must be derived from the system’s risk and assurance requirements. In many critical applications, recovery failure or undetected corruption of authoritative tool/evidence inputs would be plausible candidates, but ASST does not prescribe those choices independently of the hazard analysis.

13. Consensus Is Not Verification

Multi-agent architectures frequently use several agents to review or challenge one another. This can improve robustness, but additional agents do not automatically create independent verification.

Three agents can still share the same erroneous evidence, inherit the same incorrect assumption or influence one another. Consensus therefore cannot be equated with correctness.

The work of Zomer and De Domenico provides an interesting example: their experiments showed consensus tendencies among cognitive agents and demonstrated that communication topology can influence collective behaviour and premature convergence.

For safety-relevant systems, independent verification should therefore ideally return to the underlying evidence.

Verification should challenge evidence, not merely repeat conclusions.

14. Observability Becomes an Assurance Property

Suppose an agent system produces an incorrect final result. Can we identify where the decisive deviation occurred?

At ICML 2025, Zhang and colleagues introduced a dataset containing failure logs from 127 LLM multi-agent systems. Their best evaluated method identified the failure-responsible agent with 53.5% accuracy, but the decisive failure step with only 14.2% accuracy. The authors concluded that automated failure attribution remains difficult.

If we cannot reliably reconstruct a failure after the event, sufficient observability needs to be designed into the system beforehand. For critical applications, traceability should therefore include original evidence, agent inputs and outputs, tool interactions, state changes, verification decisions and escalation points.

Observability is not merely useful for debugging. It becomes part of Agent System Assurance.

15. Assurance Gates Should Follow Measured Propagation Risk

Placing a human reviewer only at the end of a long agent workflow may be insufficient. By then, an incorrect assumption may have crossed several boundaries and been transformed into apparently coherent downstream artefacts.

The Hallucination Snowball provides particularly relevant evidence: in its investigated sequential pipeline, verification at agent boundaries reduced hallucination survival substantially compared with end-of-pipeline checking. This does not establish a universal gate location for every agent architecture, but it supports the principle that when verification occurs can matter as much as whether verification occurs.

ASST v1.1 therefore links measurement to architecture. An assurance gate should be considered where one or more of the following are observed:

• amplification exceeds the predefined tolerance for a metric perturbation;
• detection probability deteriorates across a boundary;
• recovery falls below the predefined acceptance criterion;
• propagation crosses into a higher-consequence subsystem or decision;
• shared evidence creates correlated-agent agreement without independent verification;
• an irreversible or safety-relevant action would otherwise follow.

The resulting engineering loop is:

TEST → MEASURE → IDENTIFY PROPAGATION BOUNDARY → PLACE/IMPROVE CONTROL → RETEST

Human intervention is one possible gate, but not the only one. Depending on the risk, the control may be independent evidence retrieval, deterministic validation, schema or constraint checking, tool-result verification, an isolated reviewer, or a human decision gate with explicit authority to reject or stop the workflow.

16. Related Work: ASST as Integration, Not Isolation

ASST sits within a rapidly developing research field and should be interpreted against that prior work.

The Hallucination Snowball models error propagation as state transitions through a sequential multi-agent pipeline and quantifies escape probabilities at agent boundaries. Its results are directly relevant to ASST’s trace-and-measure logic and to the placement of verification gates.

AgentChaos provides programmatic, runtime and non-intrusive fault injection at the LLM API layer across a systematic fault taxonomy. This is closer to ASST’s perturbation concept than general position papers on chaos engineering and demonstrates that controlled fault injection can be operationalised reproducibly.

HalluProp / Before Agents Speak models systemic hallucination risk before inter-agent interaction by combining intrinsic agent risk with semantic influence and communication topology. It reinforces the need to treat topology as part of system-level assurance.

Collective Hallucination in Multi-Agent LLMs likewise treats hallucination as a system-level, time-evolving phenomenon shaped by communication topology, recursive information flow and control mechanisms.

These works reduce the need for broad novelty claims and strengthen the core ASST proposition: system-level perturbation behaviour is measurable, architecture-dependent and relevant to assurance. ASST’s proposed contribution is to connect that emerging evidence to a risk-derived acceptance envelope, perturbation-appropriate metrics, explicit assurance decisions and architecture improvement.

17. From AI Assurance to Agent System Assurance

A safety-relevant agent system should potentially be assessed at several levels:

• Agent Reliability – can individual agents perform their assigned functions within defined limits?
• Interaction Integrity – is information exchanged correctly and traceably between agents?
• Evidence Integrity – can critical conclusions be traced to authoritative evidence?
• Propagation Behaviour – how do errors, uncertainty and changed assumptions travel through the system?
• Perturbation Stability – how strongly does the system react to controlled disturbances?
• Recovery Capability – can verification mechanisms detect and correct deviations before unacceptable consequences occur?
• Human Control – are human intervention and escalation points positioned where system risk requires them?

Together these characteristics describe something broader than AI-model assurance: Agent System Assurance.

18. Implications for Safety-Critical AI

This becomes particularly relevant when Agentic AI enters safety-related engineering processes. Traditional verification frequently asks: Does the component produce an acceptable result for the specified input?

Agent systems require an additional question:

Does the system remain within an acceptable behavioural envelope when one component, input or interaction deviates from its nominal state?

A robust safety architecture should therefore not merely attempt to eliminate every possible agent error. Instead, the architecture should also ensure that errors are observable, critical evidence remains traceable, propagation paths are understood, independent verification can interrupt error chains, recovery mechanisms exist, and human intervention occurs before an unacceptable state is reached.

The goal is not perfect agents. The goal is a controlled system despite imperfect agents.

19. The ASST v1.1 Assurance Loop

BASELINE → SPECIFY → PERTURB → TRACE → MEASURE → VERIFY/RECOVER → ASSESS → IMPROVE → RETEST

The process is repeated across perturbation classes, magnitudes where meaningful, operating scenarios and relevant architecture variants. Results are retained as an Agent System Stability Profile with explicit evidence for each acceptance decision.

From proposed framework to applied method

A necessary next step is empirical application. One useful demonstrator is an assurance-oriented multi-agent chain in which several reviewing agents share an upstream evidence base. Such a system directly tests the principle “Consensus Is Not Verification”: if the common evidence source is corrupted, multiple downstream reviewers may agree for the same wrong reason.

An ASST experiment can compare an architecture with shared evidence flowing directly downstream against a variant that introduces an independent evidence gate before analysis and review. Controlled evidence perturbations can then measure detection, propagation depth, recovery and final outcome acceptability. This would move ASST from a proposed framework toward a method that has been applied and empirically characterised at least once.

20. Conclusion – The Butterfly Is the Test Case

Chaos theory does not prove that LLM agent systems are chaotic. Emergent behaviour does not prove deterministic chaos either. Those claims would go beyond the available evidence.

But more than sixty years after Lorenz demonstrated the significance of sensitivity to initial conditions, the underlying engineering question has become surprisingly relevant again:

What happens to a small deviation after it enters a complex interacting system?

Research on classical agent-based models has demonstrated sensitivity to initial conditions in specific deterministic models. Current research on LLM-agent networks demonstrates that communication structure can influence collective behaviour. Failure-attribution research demonstrates how difficult it can be to identify the decisive source of failure after a multi-agent execution has gone wrong. Recent research is beginning to apply explicit stability concepts and Lyapunov-guided methods to LLM multi-agent systems. Chaos Engineering research is exploring deliberate fault injection as a method for evaluating LLM multi-agent robustness.

Together, these developments suggest an important direction for AI Assurance:

The reliability of an Agentic AI system cannot be established solely from the reliability of its individual agents. The stability of their interaction must also be examined.

Agent System Stability Testing is proposed here as one possible assurance-oriented approach: first establish the system’s nominal variability and acceptance envelope; then perturb the system, trace the deviation, apply a metric appropriate to the perturbation class, verify detection and recovery, assess the outcome and use the evidence to improve the architecture.

Don’t just test the agents. Test the dynamics.

And perhaps the most useful lesson from chaos theory is therefore not that the agent system is chaotic.

The butterfly is not the conclusion. It is the test case.

Scientific Basis and References

Lorenz, E. N. (1963). Deterministic Nonperiodic Flow. Journal of the Atmospheric Sciences, 20(2), 130–141. DOI: 10.1175/1520-0469(1963)020<0130:DNF>2.0.CO;2.

Bertolotti, F., Locoro, A., & Mari, L. P. (2020). Sensitivity to Initial Conditions in Agent-Based Models. Lecture Notes in Artificial Intelligence 12520, 501–508. DOI: 10.1007/978-3-030-66412-1_32.

Zhang, S. et al. (2025). Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. Proceedings of ICML 2025, PMLR 267, 76583–76599.

Owotogbe, J. (2025). Assessing and Enhancing the Robustness of LLM-Based Multi-Agent Systems Through Chaos Engineering. IEEE/ACM CAIN 2025, 250–252. DOI: 10.1109/CAIN66642.2025.00039.

Zomer, N., & De Domenico, M. (2026). Unraveling the emergence of collective behavior in networks of cognitive agents. npj Artificial Intelligence 2, 36. DOI: 10.1038/s44387-026-00091-5.

Xu, Z. et al. (2026). Lyapunov-guided cooperative games enable stable constraint fusion in LLM-based multi-agent autonomous systems. npj Artificial Intelligence 2, 60. DOI: 10.1038/s44387-026-00110-5.

Emerging research / preprints and closely related work

Liu, Y. et al. (2026). PropUQ-MAS: Propagation-Aware Uncertainty Quantification for LLM Multi-Agent Systems. arXiv:2608.22130.

Jamshidi, S. et al. (2026). Hallucination Cascade: Analyzing Error Propagation in Multi-Agent LLM Systems. arXiv:2606.07937. This work is treated as prior art for amplification-factor interpretation.

Singh, P., & Pawar, B. (2026). The Hallucination Snowball: Modeling Error Propagation as State Transitions in Multi-Agent LLM Pipelines. arXiv:2608.14588.

Tan, G. et al. (2026). AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection. arXiv:2608.06790; DOI: 10.1145/3832783.3837437.

Lin, S. et al. (2026). Before Agents Speak: Pre-hoc Failure Risk Inference in Multi-Agent Systems. arXiv:2607.26836. DOI: 10.48550/arXiv.2607.26836.

Jamshidi, S. (2026). Collective Hallucination in Multi-Agent LLMs: Modeling and Defense. arXiv:2606.07941.

---

Version 1.1 methodological status: Agent System Stability Testing (ASST) remains a proposed engineering framework. Version 1.1 explicitly incorporates prior art, stochastic baseline characterisation, perturbation-specific metrics, ex-ante acceptance criteria and decision logic. Experimental validation, formalisation, replication and domain-specific adaptation remain necessary before ASST can be considered a validated assurance method.

Human oversight is often presented as the safety net for generative and agentic AI. In high-stakes environments, however, simply placing a person at the end of an automated workflow does not create assurance. The human must have the evidence, authority, competence and time to challenge what the AI has produced.

As AI systems move from single-model assistance towards agentic workflows, the phrase Human-in-the-Loop can become dangerously reassuring. A nominal approval step may exist while the real reasoning, evidence selection and intermediate decisions have already been delegated to AI agents.

The relevant assurance question is therefore not “Is a human involved?” It is “Can the human meaningfully understand, challenge and stop the AI-supported process?”

Human presence is not the same as human control

A reviewer can only exercise meaningful oversight if the information needed for a decision remains visible. If an agentic workflow compresses dozens of sources, assumptions and intermediate decisions into one polished recommendation, the reviewer may see the conclusion without seeing the basis for it.

This creates a risk of automation bias: the apparent coherence and speed of the system can make human approval procedural rather than substantive. Effective assurance therefore requires more than a final “approve” button.

NIST’s AI Risk Management Framework explicitly calls for roles and responsibilities in human-AI configurations to be defined and differentiated, and for human oversight processes to be defined, assessed and documented. The NIST Generative AI Profile also notes that generative AI may require different levels of oversight, additional human review, tracking and documentation.

From Human-in-the-Loop to Human Decision Authority

For high-stakes use cases, the human role should be designed around decision authority, not merely participation. AI may search, structure, compare, analyse and propose. The accountable person must retain the authority to accept, reject, escalate or demand additional evidence.

At Topp Consulting we describe this as Human Epistemic Authority: the human retains authority over what is accepted as sufficiently supported knowledge for the decision at hand.

That authority requires at least four conditions:

• Visibility: evidence, assumptions, uncertainty and unresolved contradictions are available to the reviewer.
• Challengeability: the reviewer can inspect and question intermediate AI reasoning products and supporting sources.
• Intervention: the reviewer can stop, reject or redirect the workflow without being forced into an automated path.
• Accountability: the final decision and its evidential basis remain attributable to a defined human role.

Agentic AI changes the assurance problem

In an agentic system, one model may research while another analyses, a third reviews and additional tools retrieve documents or execute actions. This can improve capability, but it also distributes error across a chain of components.

A human reviewer at the end cannot realistically reconstruct that chain unless traceability has been engineered into the workflow. The system therefore needs to preserve a path such as:

Source → Evidence → Agent Action → Claim → Independent Challenge → Verification → Human Decision

This is why agentic AI requires assurance by architecture. Controls cannot be added only at the user interface.

Independent agents are useful only when their roles are genuinely different

Adding a second AI agent does not automatically provide independent verification. Two agents using the same model family, the same evidence and similar prompts can reproduce the same assumptions and failure modes.

A stronger design separates responsibilities. A research agent should be optimised for evidence retrieval. An analysis or engineering agent should transform evidence into structured claims. A reviewer or red-team agent should actively search for unsupported claims, contradictions, missing evidence and boundary violations.

Where practical, verification should also include deterministic checks, authoritative-source comparison or other mechanisms that do not merely ask another language model whether the first language model was correct.

Human oversight must be risk-proportionate

Not every AI-assisted activity needs the same human control. The assurance depth should follow the potential consequence of error, the autonomy of the system and the context in which its output will be used.

The EU AI Act reflects this principle for high-risk AI systems: Article 14 requires effective human oversight and states that oversight measures should be proportionate to the risks, level of autonomy and context of use.

In engineering practice, this supports a graded approach. Low-impact drafting may permit broad delegation. Requirements interpretation, compliance evidence or safety-related analysis may require explicit verification and approval gates. Decisions that carry formal accountability may remain human-exclusive.

The reviewer needs an assurance package, not just an answer

For consequential decisions, the human should receive a compact assurance package alongside the AI result. Depending on the use case, this can contain:

• authoritative sources and provenance;
• claims linked to supporting evidence;
• assumptions and known limitations;
• conflicting or missing evidence;
• results of independent review and deterministic checks;
• model, prompt, tool and configuration identifiers where relevant;
• the defined acceptance, escalation and stop criteria.

This changes the human task from “read the AI answer and decide whether it looks plausible” to “evaluate whether the evidence and verification are sufficient for the intended decision.”

Fail Visible is essential for meaningful oversight

Human oversight fails when the system hides uncertainty behind fluent language. An assurance-oriented workflow should instead make unresolved conditions explicit.

If evidence is missing, the system should say so. If agents disagree, the disagreement should remain visible. If verification fails, the workflow should stop or escalate. If the requested action exceeds the agent’s authority, execution should be blocked.

This is the principle of Fail Visible: uncertainty and failure become reviewable engineering information rather than being converted into plausible output.

Human-in-the-Loop becomes a system property

Effective human oversight is not a single step. It is a property of the complete socio-technical system: architecture, roles, evidence, interfaces, permissions, verification, escalation paths and governance must all preserve meaningful human authority.

This is particularly important in railway and critical infrastructure, where AI-supported work may interact with established engineering lifecycles, formal verification and validation, safety evidence and accountable decision structures.

Our earlier article, AI Assurance for Safety-Critical AI: From AI Output to Engineering Evidence, describes the broader assurance chain. Human decision authority is one of the controls that makes that chain defensible.

A practical assurance pattern for agentic AI

A robust pattern for high-stakes agentic workflows can therefore be expressed as:

Controlled Delegation + Evidence Traceability + Independent Challenge + Fail Visible + Human Decision Authority

None of these controls is sufficient alone. Together, they create a structure in which AI can contribute substantial analytical capability without silently becoming the final authority.

Conclusion: the human must be able to say NO

The strongest test of Human-in-the-Loop is simple: Can the accountable person understand why the system reached its recommendation and meaningfully refuse it?

If the answer is no, human oversight may exist on paper while effective control has already been delegated to the AI system.

For safety-critical and regulated environments, the objective should therefore be stronger than Human-in-the-Loop. It should be Human-in-Control through evidence, verification and explicit decision authority.

References and further reading

• NIST AI RMF — AI Risk Management and Human-AI Interaction
• NIST — Generative Artificial Intelligence Profile (NIST AI 600-1)
• European Union — AI Act, Regulation (EU) 2024/1689, Article 14

---

About Topp Consulting

Topp Consulting supports organisations introducing AI into regulated, safety-critical and critical-infrastructure environments. Our approach combines AI governance with evidence traceability, independent verification, controlled delegation and accountable human decision authority.

Explore our AI Assurance approach
Read: AI Assurance for Safety-Critical AI
Related expertise: Railway & Critical Infrastructure

Generative and agentic AI can accelerate engineering, research and compliance work. In safety-critical environments, however, producing a plausible answer is not enough. Organisations need to demonstrate why an AI-supported result can be trusted, what evidence supports it, where uncertainty remains and who is accountable for the final decision.

Artificial intelligence is rapidly moving beyond simple productivity tools. Large language models can analyse requirements, evaluate documents, identify inconsistencies, propose test cases, support risk analysis and coordinate increasingly complex workflows through AI agents.

For organisations operating in regulated or safety-critical environments, this creates significant opportunities. It also creates a fundamental assurance problem.

An AI-generated result may look convincing while being incomplete, based on weak evidence or influenced by incorrect assumptions. In engineering, critical infrastructure or safety-related processes, the consequences can be considerably more serious.

The relevant question therefore changes from “Can AI produce a useful result?” to “Can we demonstrate why this result may be trusted?” That distinction is at the heart of AI Assurance.

AI Governance is necessary — but it is not AI Assurance

AI governance establishes important organisational boundaries. It can define which models may be used, how sensitive information is handled, who owns an AI system and which use cases require approval. These controls are necessary, but they do not automatically establish the reliability of an individual AI-supported engineering result.

For that, additional questions must be answered: Where did the information come from? Which evidence supports the generated claims? Which assumptions were made? Has the result been independently challenged or verified? What uncertainty remains? Who has the authority to accept the result?

AI Assurance therefore needs to connect organisational governance with the actual evidence and decision-making process. The objective is to create a defensible chain from evidence to decision.

From AI Output to Engineering Evidence

A language model can generate an output within seconds. That output should not automatically become engineering evidence. A controlled assurance process introduces additional stages:

Source → Evidence → AI Analysis → Independent Verification → Human Decision → Auditable Result

The source establishes where information originated. Evidence establishes whether that information can support a relevant claim. AI analysis can structure, compare, interpret or transform the available evidence. Independent verification challenges the resulting claims and identifies contradictions, unsupported assumptions or missing information. Finally, an accountable human decides whether the available evidence is sufficient for the intended use.

Agentic AI requires stronger assurance

The assurance challenge becomes more complex with agentic AI. Instead of a person asking a single model a question, an AI system may contain several specialised agents performing different tasks.

More agents do not automatically create more trustworthy results. If several agents rely on the same source, assumptions or similar model behaviour, they can reinforce the same error. A robust agentic architecture therefore requires functional separation of responsibilities.

Evidence Sources → Research Agent → Analysis / Engineering Agent → Independent Reviewer or Red-Team Agent → Verification → Human Decision Gate → Auditable Result

The reviewing agent should challenge assumptions, identify missing evidence, detect contradictions and determine whether claims are actually supported. AI systems should be designed to expose unresolved problems rather than optimise them away.

Human-in-the-Loop is necessary — but not sufficient

Simply placing a human at the end of an automated workflow does not create effective assurance. The decision-maker must be able to see the relevant evidence and its origin, assumptions, unresolved contradictions, uncertainty, verification results and the limits of the AI system.

The human must retain genuine authority to reject, escalate or request further evidence. This is what we describe as Human Epistemic Authority: AI may search, structure, analyse and propose, while the accountable human retains authority over what is accepted as sufficiently supported knowledge for the decision at hand.

Fail Visible instead of Fail Plausible

Generative AI is very good at producing plausible language. That becomes problematic when missing information is transformed into a confident-looking answer. AI Assurance should follow the opposite principle: Fail Visible.

If evidence is insufficient, the system should state that evidence is insufficient. If sources contradict each other, the contradiction should remain visible. If verification fails, the workflow should stop or escalate. In safety-critical AI, an explicit “insufficient evidence” can be considerably more valuable than an impressive answer.

Traceability must be designed into the workflow

AI-supported engineering should extend rather than break established engineering traceability. For important AI-supported claims, it should be possible to reconstruct:

Source → Claim → Analysis → Verification → Decision → Result

Sources should be retained or referenced, claims distinguishable from evidence, AI-generated interpretations attributable, verification results recorded and human approvals traceable. Changes to models, prompts, tools or source information should be treated as potentially relevant changes to the assurance context.

Verification and Validation remain essential

AI does not eliminate the distinction between verification and validation. An AI system can behave as designed and still produce a result inappropriate for a safety-related engineering decision. Assurance therefore needs both perspectives.

Depending on the use case, this may involve independent review, deterministic checks, comparison against authoritative sources, requirements-based testing, consistency analysis, controlled test datasets or human expert validation. NIST provides resources for testing, evaluation, verification and validation (TEVV) within trustworthy AI risk management.

The required assurance depth should depend on the potential impact of the AI-supported activity. An AI system drafting meeting minutes does not require the same assurance architecture as an AI agent identifying safety requirements or proposing changes to an engineering baseline.

Controlled Delegation

Not every activity should be delegated to AI in the same way. A practical AI Assurance framework should distinguish between delegable activities, controlled automation and human-exclusive decisions.

These boundaries should be defined before deploying autonomous agents. Automation capability should not determine delegation authority. Risk, evidence requirements and accountability should.

What an AI Assurance framework should contain

A practical assurance framework for safety-critical or regulated AI should establish governance and accountable roles, classify AI use cases according to risk and impact, define permitted levels of delegation and specify requirements for evidence and source traceability.

It should also establish independent verification and validation mechanisms, effective human decision gates, audit trails, model and configuration change control, and explicit escalation or stop conditions. For agentic systems, additional controls are required around agent responsibilities, tool permissions, evidence access and interaction between agents.

Most importantly, the framework must distinguish between generating information and accepting information as evidence.

AI Assurance in Critical Infrastructure

These principles become particularly relevant in railway and critical infrastructure environments.

Railway, energy, transport and other regulated environments already rely on structured engineering lifecycles, risk management, verification processes and formal evidence. AI should not create a parallel universe beside those established processes. The stronger approach is to integrate AI Assurance into the existing engineering and governance framework.

The assurance question is not simply whether the AI model itself is safe. It is whether the complete socio-technical process in which AI participates remains controlled, traceable and accountable.

AI Assurance is ultimately about accountability

AI Assurance should not attempt to prove that an AI system will always be correct. Instead, assurance should create a controlled environment in which organisations can determine when AI-generated results are sufficiently supported for their intended use — and when they are not.

That requires evidence, traceability, independent verification, visible uncertainty and accountable human authority.

The defining question for safety-critical AI should therefore not be “Did the AI produce an answer?” It should be: “Can we demonstrate why this answer may be trusted, where its limits are, and who remains accountable for acting on it?”

References and further reading

• NIST — AI Risk Management Framework (AI RMF)
• NIST — Generative Artificial Intelligence Profile (NIST AI 600-1)
• UK Department for Science, Innovation and Technology — Introduction to AI Assurance

---

About Topp Consulting

Topp Consulting supports organisations introducing AI into regulated, safety-critical and critical-infrastructure environments. Our approach combines AI governance and assurance with engineering traceability, independent verification, human decision authority and experience in complex IT/OT transformation.

Explore our AI Assurance approach
Related expertise: Railway & Critical Infrastructure