Agent System Stability – A Missing Dimension of AI Assurance?

From the Butterfly Effect to a practical assurance concept for multi-agent AI systems

Abstract

As AI evolves from individual models toward interacting autonomous agents, assurance must evolve with it. Testing individual agents is insufficient when outputs become inputs for other agents, decisions depend on previous agent decisions, and deviations can propagate through an interaction network.

This article proposes Agent System Stability as an assurance property and presents Agent System Stability Testing (ASST) v1.1: a structured method for establishing the system’s inherent stochastic variability, introducing controlled perturbations, tracing their propagation, selecting perturbation-appropriate measures, observing detection and recovery, and assessing outcomes against acceptance criteria defined before testing.

ASST does not claim that LLM agent systems are mathematically chaotic, nor that its individual measurement ideas are unprecedented. Controlled fault injection, propagation analysis and amplification factors already appear in emerging multi-agent research. ASST is proposed as an assurance-oriented integration and generalisation across perturbation classes, linking measurements to acceptance decisions and to the placement of verification and human-control gates.

Don’t just test the agents. Test the dynamics — against a baseline and an acceptance envelope.

1. The System Is More Than Its Agents

Artificial intelligence is moving from individual language models toward systems of interacting agents. One agent may analyse requirements. Another may retrieve evidence. A third may challenge the analysis. A fourth may evaluate risks. An orchestrator may combine their conclusions and decide which agent acts next.

Each component can be tested individually. But this leaves an important question unanswered: What happens when these agents interact?

Consider a simplified architecture:

Requirements Analyst → Technical Analyst → Independent Reviewer → Risk Analyst → Validator

The output of one agent becomes part of the input of another. Consequently, the behaviour of the overall system depends not only on the quality of its individual agents but also on the way information travels through the network. The system therefore possesses an additional characteristic: interaction dynamics.

Recent research into networks of LLM-based cognitive agents supports the importance of this distinction. Zomer and De Domenico demonstrated experimentally that communication topology can affect collective behaviour. In their experiments, LLM agents exhibited consensus tendencies and, in some configurations, premature convergence; changing the network topology could alter these effects.

This means that the communication architecture itself can influence system behaviour. That is where Agentic AI becomes a systems-engineering problem.

2. From the Butterfly Effect to Agentic AI

In 1963, meteorologist Edward Lorenz published Deterministic Nonperiodic Flow. Lorenz studied a small deterministic system of nonlinear differential equations representing atmospheric convection. His results became foundational to the study of sensitivity to initial conditions.

The important lesson is often oversimplified. Chaos does not simply mean randomness. A deterministic system can follow defined rules while nevertheless being highly sensitive to small changes in its state.

This does not mean that modern LLM agent systems are therefore chaotic systems. No such general conclusion follows. But the underlying question is highly relevant:

What happens when a small perturbation enters a complex interacting system?

This question has already been investigated in classical agent-based modelling. Bertolotti, Locoro and Mari examined two deterministic agent-based models in 2020. They changed a single initial condition associated with one agent and analysed the resulting macro-level behaviour. In both models examined, the impact of that change was substantial. They also found qualitative evidence that factors such as the number of agents and their connectivity were related to sensitivity.

The experiment therefore demonstrated something important: A small local change can materially affect global behaviour in at least some agent-based systems.

3. The Agent Butterfly Effect

Consider a technical requirement:

Maximum permitted response time: 500 ms.

Now imagine an agent transforms this during an early processing step into:

Expected response time: approximately 500 ms.

The textual difference appears small. The engineering meaning is not. In the first case, 500 ms represents a constraint. In the second, it has become an expectation.

A reviewer might return to the original requirement, detect the deviation and correct it. The perturbation disappears. But another system might behave differently. The next agent accepts the statement. An architecture agent uses it as an assumption. A risk agent evaluates the resulting architecture. A decision agent receives conclusions that are internally consistent but derived from an altered premise.

The relevant question is therefore no longer only: Did Agent A make an error?

What did the system do with that error?

4. Propagation Is a Real Multi-Agent Problem

This question is increasingly appearing in current research. A 2026 preprint, PropUQ-MAS, represents multi-agent execution as a communication-structured graph and explicitly models uncertainty inherited from upstream agent messages. Its premise is that intermediate errors can be inherited by downstream agents, and isolated uncertainty measurements therefore fail to capture all reliability risks in a multi-agent system.

Another 2026 preprint, Hallucination Cascade, investigated factual inconsistencies across sequential LLM-agent interactions. Its findings caution against a simplistic assumption that errors must always grow. The reported experiments found that hallucination scores could decrease across the investigated agent cascade, even while factual accuracy declined slightly.

Agent interaction may therefore correct one type of deviation while simultaneously degrading another quality dimension. We should neither assume amplification nor correction. We should measure what the system actually does.

5. Three Fundamental Responses to a Perturbation

Attenuation

The deviation decreases. Another agent, an evidence check or an independent verification mechanism corrects the problem. The system moves back toward an acceptable state.

Propagation

The deviation survives. Other agents inherit the changed information without materially increasing or decreasing its impact.

Amplification

The deviation becomes more consequential. Downstream agents derive additional conclusions from it, distribute it to additional agents or incorporate it into decisions.

Agent-system stability should not be assumed from agent reliability. It should be tested experimentally.

6. Introducing Agent System Stability

This article proposes using the term Agent System Stability to describe how a multi-agent AI system reacts to controlled perturbations in its inputs, agents, tools or communication structure.

A stable system should not necessarily produce identical outputs. That would be an unrealistic requirement for many probabilistic AI systems. Instead, stability concerns whether small disturbances remain within an acceptable behavioural envelope.

The key questions become: Does the deviation propagate? Does it grow? Does another component detect it? Does the system recover? Does the final outcome remain acceptable?

This shifts assurance from purely component-level evaluation toward system-level behaviour.

7. The Agent System Stability Test – ASST v1.1

For probabilistic agent systems, a single nominal execution is not a sufficient baseline. Two unperturbed runs may already diverge because of model sampling, retrieval variation, tool timing or other stochastic effects. ASST must therefore begin by characterising inherent system variability.

Step 0 — Establish the null baseline

Execute the unperturbed system n times under the same controlled conditions:

S₀¹, S₀², …, S₀ⁿ

Record the relevant trajectories and outcomes. Pairwise or reference-based distances between these nominal runs form an empirical baseline distribution, D_null. The number of runs is context-dependent and should be justified by the intended confidence and variability of the system; ASST does not prescribe a universal n.

A perturbation effect should only be interpreted as system sensitivity when it can be distinguished from this inherent run-to-run variability. Depending on the metric and sample size, that distinction may use confidence intervals, quantiles, effect sizes or a justified statistical test.

Without D_null, ASST risks measuring stochastic variability and calling it sensitivity.

Define the acceptance envelope before perturbation

The Acceptable Behavioural Envelope (ABE) must also be defined before the perturbation results are examined. It should be derived from the use case, hazard and risk analysis, system requirements, assurance objectives and the consequence of an incorrect outcome. In regulated or safety-related contexts, the chain should be explicit:

Hazard / Risk → Assurance Requirement → Acceptance Criterion → Test Metric → Evidence → Decision

The envelope may contain quantitative limits, categorical constraints and veto conditions. It must not be adjusted after observing a poor result merely to make that result acceptable.

Perturbation classes

• Information perturbation: modify a factual input or numerical parameter.
• Semantic perturbation: alter wording while retaining substantially the same task intent.
• Agent perturbation: modify an intermediate agent response.
• Tool perturbation: corrupt, delay, remove or replace a tool result.
• Topology perturbation: change a communication path, dependency or interaction sequence.
• Confidence perturbation: modify uncertainty or confidence information communicated between agents.

These classes are deliberately heterogeneous. ASST v1.1 therefore does not force all perturbations into one universal scalar measure.

8. The ASST v1.1 Process

1. Baseline: execute repeated unperturbed runs and establish D_null.
2. Specify: derive the ABE, acceptance criteria, veto conditions and selected metrics from the assurance context before testing.
3. Perturb: introduce one defined and reproducible disturbance.
4. Trace: observe how the deviation travels through agents, evidence, tools and system states.
5. Measure: use a metric appropriate to the perturbation class.
6. Verify & Recover: determine whether controls detect the disturbance and whether the system returns to an acceptable state.
7. Assess: compare the evidence with D_null and the predefined ABE, then apply the decision rule.
8. Improve & Retest: place or strengthen controls where propagation, non-detection or failed recovery is observed, then repeat the test.

BASELINE → SPECIFY → PERTURB → TRACE → MEASURE → VERIFY/RECOVER → ASSESS → IMPROVE/RETEST

9. Measuring System Response

For a state representation S(t), an application-specific distance function d may still be useful. But in a stochastic system the comparison is no longer simply one S₀(t) against one Sε(t). The perturbed trajectory must be interpreted against the distribution of nominal behaviour.

For quantitative artefacts, distance may be numerical. For requirements it may concern constraint violations; for hazard analysis, changed hazards, classifications or mitigations; for structured artefacts, graph or schema differences. The metric must preserve the engineering meaning of the artefact being assessed.

ASST v1.1 distinguishes between ordinal/metric perturbations and categorical/structural perturbations.

10. Amplification Is One Measure — Not the Universal Measure

Where the perturbation has a meaningful ordinal magnitude ε and outcome distance is defined on a compatible, normalised scale, an amplification-style measure can be useful:

A = ΔOutcome / ε

Recent work on hallucination cascades has already used an amplification-factor interpretation in which values below one represent net attenuation. ASST therefore treats amplification measurement as prior art to be adapted and generalised, not as a novel universal metric.

A tolerance band must be defined in advance. Rather than treating “A ≈ 1” as self-explanatory, the test specification should define a context-specific band [1−τ, 1+τ]. Only within that declared band should behaviour be described as approximately proportional propagation.

A should not be used where the perturbation magnitude is undefined or approaches zero. This includes meaning-preserving semantic reformulations, removal of a tool, or deletion of a communication edge unless a justified ordinal scale has independently been defined. In such cases the ratio may be meaningless, infinite or dimensionally incoherent.

Alternative measures for categorical and structural perturbations

For these perturbations ASST v1.1 uses outcome-oriented measures such as:

• Detection Rate (DR): proportion of injected disturbances detected before an unacceptable outcome.
• Recovery Rate (RR): proportion of detected or injected disturbances after which the system returns to the predefined ABE.
• Outcome Acceptability Rate (OAR): proportion of perturbed runs whose final outcome remains within the ABE.
• Propagation Depth (PD): number or proportion of downstream boundaries crossed before detection, containment or termination.

The purpose is not to produce a single impressive number. It is to select a measure that is well-defined for the disturbance being tested.

11. Why This Is Not a Lyapunov Exponent

The analogy with dynamical-systems theory is nevertheless useful. Lyapunov methods analyse stability by examining how system trajectories behave under perturbations.

Xu and colleagues published a 2026 study in npj Artificial Intelligence that formulates constraint fusion in an LLM-based multi-agent autonomous system as a stability-convergence problem. Their framework uses a Lyapunov function as a system-level state metric and combines it with cooperative differential-game mechanisms.

This provides evidence that concepts from stability and control theory can be meaningfully explored in LLM multi-agent architectures. But scientific precision matters. The proposed ASST amplification measure should not simply be called a Lyapunov exponent.

For engineering assurance, the operational question is simpler: Does a small disturbance remain small?

12. Stability Is Multidimensional — and Requires a Decision Rule

Agent System Stability should not be compressed into one universal score. A system may tolerate semantic variation while reacting strongly to incorrect tool data, or recover from one agent failure while remaining sensitive to communication topology.

The proposed Agent System Stability Profile therefore retains separate dimensions: Semantic, Information, Agent, Tool, Topology and Recovery Stability. For each dimension, the test specification should identify the applicable metric, D_null where relevant, the ABE, and whether the criterion is veto-capable or informative.

Proposed assurance decision logic

• GO: all veto criteria are satisfied and the tested behaviour remains within the predefined ABE.
• CONDITIONAL GO: no veto criterion is violated, but one or more informative criteria require a documented mitigation, limitation or follow-up test.
• NO-GO: at least one veto criterion is violated — for example an unacceptable safety-related outcome, failure to detect a defined critical tool fault, or recovery below a predefined minimum.

Which dimensions are veto-capable is not universal. It must be derived from the system’s risk and assurance requirements. In many critical applications, recovery failure or undetected corruption of authoritative tool/evidence inputs would be plausible candidates, but ASST does not prescribe those choices independently of the hazard analysis.

13. Consensus Is Not Verification

Multi-agent architectures frequently use several agents to review or challenge one another. This can improve robustness, but additional agents do not automatically create independent verification.

Three agents can still share the same erroneous evidence, inherit the same incorrect assumption or influence one another. Consensus therefore cannot be equated with correctness.

The work of Zomer and De Domenico provides an interesting example: their experiments showed consensus tendencies among cognitive agents and demonstrated that communication topology can influence collective behaviour and premature convergence.

For safety-relevant systems, independent verification should therefore ideally return to the underlying evidence.

Verification should challenge evidence, not merely repeat conclusions.

14. Observability Becomes an Assurance Property

Suppose an agent system produces an incorrect final result. Can we identify where the decisive deviation occurred?

At ICML 2025, Zhang and colleagues introduced a dataset containing failure logs from 127 LLM multi-agent systems. Their best evaluated method identified the failure-responsible agent with 53.5% accuracy, but the decisive failure step with only 14.2% accuracy. The authors concluded that automated failure attribution remains difficult.

If we cannot reliably reconstruct a failure after the event, sufficient observability needs to be designed into the system beforehand. For critical applications, traceability should therefore include original evidence, agent inputs and outputs, tool interactions, state changes, verification decisions and escalation points.

Observability is not merely useful for debugging. It becomes part of Agent System Assurance.

15. Assurance Gates Should Follow Measured Propagation Risk

Placing a human reviewer only at the end of a long agent workflow may be insufficient. By then, an incorrect assumption may have crossed several boundaries and been transformed into apparently coherent downstream artefacts.

The Hallucination Snowball provides particularly relevant evidence: in its investigated sequential pipeline, verification at agent boundaries reduced hallucination survival substantially compared with end-of-pipeline checking. This does not establish a universal gate location for every agent architecture, but it supports the principle that when verification occurs can matter as much as whether verification occurs.

ASST v1.1 therefore links measurement to architecture. An assurance gate should be considered where one or more of the following are observed:

• amplification exceeds the predefined tolerance for a metric perturbation;
• detection probability deteriorates across a boundary;
• recovery falls below the predefined acceptance criterion;
• propagation crosses into a higher-consequence subsystem or decision;
• shared evidence creates correlated-agent agreement without independent verification;
• an irreversible or safety-relevant action would otherwise follow.

The resulting engineering loop is:

TEST → MEASURE → IDENTIFY PROPAGATION BOUNDARY → PLACE/IMPROVE CONTROL → RETEST

Human intervention is one possible gate, but not the only one. Depending on the risk, the control may be independent evidence retrieval, deterministic validation, schema or constraint checking, tool-result verification, an isolated reviewer, or a human decision gate with explicit authority to reject or stop the workflow.

16. Related Work: ASST as Integration, Not Isolation

ASST sits within a rapidly developing research field and should be interpreted against that prior work.

The Hallucination Snowball models error propagation as state transitions through a sequential multi-agent pipeline and quantifies escape probabilities at agent boundaries. Its results are directly relevant to ASST’s trace-and-measure logic and to the placement of verification gates.

AgentChaos provides programmatic, runtime and non-intrusive fault injection at the LLM API layer across a systematic fault taxonomy. This is closer to ASST’s perturbation concept than general position papers on chaos engineering and demonstrates that controlled fault injection can be operationalised reproducibly.

HalluProp / Before Agents Speak models systemic hallucination risk before inter-agent interaction by combining intrinsic agent risk with semantic influence and communication topology. It reinforces the need to treat topology as part of system-level assurance.

Collective Hallucination in Multi-Agent LLMs likewise treats hallucination as a system-level, time-evolving phenomenon shaped by communication topology, recursive information flow and control mechanisms.

These works reduce the need for broad novelty claims and strengthen the core ASST proposition: system-level perturbation behaviour is measurable, architecture-dependent and relevant to assurance. ASST’s proposed contribution is to connect that emerging evidence to a risk-derived acceptance envelope, perturbation-appropriate metrics, explicit assurance decisions and architecture improvement.

17. From AI Assurance to Agent System Assurance

A safety-relevant agent system should potentially be assessed at several levels:

• Agent Reliability – can individual agents perform their assigned functions within defined limits?
• Interaction Integrity – is information exchanged correctly and traceably between agents?
• Evidence Integrity – can critical conclusions be traced to authoritative evidence?
• Propagation Behaviour – how do errors, uncertainty and changed assumptions travel through the system?
• Perturbation Stability – how strongly does the system react to controlled disturbances?
• Recovery Capability – can verification mechanisms detect and correct deviations before unacceptable consequences occur?
• Human Control – are human intervention and escalation points positioned where system risk requires them?

Together these characteristics describe something broader than AI-model assurance: Agent System Assurance.

18. Implications for Safety-Critical AI

This becomes particularly relevant when Agentic AI enters safety-related engineering processes. Traditional verification frequently asks: Does the component produce an acceptable result for the specified input?

Agent systems require an additional question:

Does the system remain within an acceptable behavioural envelope when one component, input or interaction deviates from its nominal state?

A robust safety architecture should therefore not merely attempt to eliminate every possible agent error. Instead, the architecture should also ensure that errors are observable, critical evidence remains traceable, propagation paths are understood, independent verification can interrupt error chains, recovery mechanisms exist, and human intervention occurs before an unacceptable state is reached.

The goal is not perfect agents. The goal is a controlled system despite imperfect agents.

19. The ASST v1.1 Assurance Loop

BASELINE → SPECIFY → PERTURB → TRACE → MEASURE → VERIFY/RECOVER → ASSESS → IMPROVE → RETEST

The process is repeated across perturbation classes, magnitudes where meaningful, operating scenarios and relevant architecture variants. Results are retained as an Agent System Stability Profile with explicit evidence for each acceptance decision.

From proposed framework to applied method

A necessary next step is empirical application. One useful demonstrator is an assurance-oriented multi-agent chain in which several reviewing agents share an upstream evidence base. Such a system directly tests the principle “Consensus Is Not Verification”: if the common evidence source is corrupted, multiple downstream reviewers may agree for the same wrong reason.

An ASST experiment can compare an architecture with shared evidence flowing directly downstream against a variant that introduces an independent evidence gate before analysis and review. Controlled evidence perturbations can then measure detection, propagation depth, recovery and final outcome acceptability. This would move ASST from a proposed framework toward a method that has been applied and empirically characterised at least once.

20. Conclusion – The Butterfly Is the Test Case

Chaos theory does not prove that LLM agent systems are chaotic. Emergent behaviour does not prove deterministic chaos either. Those claims would go beyond the available evidence.

But more than sixty years after Lorenz demonstrated the significance of sensitivity to initial conditions, the underlying engineering question has become surprisingly relevant again:

What happens to a small deviation after it enters a complex interacting system?

Research on classical agent-based models has demonstrated sensitivity to initial conditions in specific deterministic models. Current research on LLM-agent networks demonstrates that communication structure can influence collective behaviour. Failure-attribution research demonstrates how difficult it can be to identify the decisive source of failure after a multi-agent execution has gone wrong. Recent research is beginning to apply explicit stability concepts and Lyapunov-guided methods to LLM multi-agent systems. Chaos Engineering research is exploring deliberate fault injection as a method for evaluating LLM multi-agent robustness.

Together, these developments suggest an important direction for AI Assurance:

The reliability of an Agentic AI system cannot be established solely from the reliability of its individual agents. The stability of their interaction must also be examined.

Agent System Stability Testing is proposed here as one possible assurance-oriented approach: first establish the system’s nominal variability and acceptance envelope; then perturb the system, trace the deviation, apply a metric appropriate to the perturbation class, verify detection and recovery, assess the outcome and use the evidence to improve the architecture.

Don’t just test the agents. Test the dynamics.

And perhaps the most useful lesson from chaos theory is therefore not that the agent system is chaotic.

The butterfly is not the conclusion. It is the test case.

Scientific Basis and References

Lorenz, E. N. (1963). Deterministic Nonperiodic Flow. Journal of the Atmospheric Sciences, 20(2), 130–141. DOI: 10.1175/1520-0469(1963)020<0130:DNF>2.0.CO;2.

Bertolotti, F., Locoro, A., & Mari, L. P. (2020). Sensitivity to Initial Conditions in Agent-Based Models. Lecture Notes in Artificial Intelligence 12520, 501–508. DOI: 10.1007/978-3-030-66412-1_32.

Zhang, S. et al. (2025). Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. Proceedings of ICML 2025, PMLR 267, 76583–76599.

Owotogbe, J. (2025). Assessing and Enhancing the Robustness of LLM-Based Multi-Agent Systems Through Chaos Engineering. IEEE/ACM CAIN 2025, 250–252. DOI: 10.1109/CAIN66642.2025.00039.

Zomer, N., & De Domenico, M. (2026). Unraveling the emergence of collective behavior in networks of cognitive agents. npj Artificial Intelligence 2, 36. DOI: 10.1038/s44387-026-00091-5.

Xu, Z. et al. (2026). Lyapunov-guided cooperative games enable stable constraint fusion in LLM-based multi-agent autonomous systems. npj Artificial Intelligence 2, 60. DOI: 10.1038/s44387-026-00110-5.

Emerging research / preprints and closely related work

Liu, Y. et al. (2026). PropUQ-MAS: Propagation-Aware Uncertainty Quantification for LLM Multi-Agent Systems. arXiv:2608.22130.

Jamshidi, S. et al. (2026). Hallucination Cascade: Analyzing Error Propagation in Multi-Agent LLM Systems. arXiv:2606.07937. This work is treated as prior art for amplification-factor interpretation.

Singh, P., & Pawar, B. (2026). The Hallucination Snowball: Modeling Error Propagation as State Transitions in Multi-Agent LLM Pipelines. arXiv:2608.14588.

Tan, G. et al. (2026). AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection. arXiv:2608.06790; DOI: 10.1145/3832783.3837437.

Lin, S. et al. (2026). Before Agents Speak: Pre-hoc Failure Risk Inference in Multi-Agent Systems. arXiv:2607.26836. DOI: 10.48550/arXiv.2607.26836.

Jamshidi, S. (2026). Collective Hallucination in Multi-Agent LLMs: Modeling and Defense. arXiv:2606.07941.

---

Version 1.1 methodological status: Agent System Stability Testing (ASST) remains a proposed engineering framework. Version 1.1 explicitly incorporates prior art, stochastic baseline characterisation, perturbation-specific metrics, ex-ante acceptance criteria and decision logic. Experimental validation, formalisation, replication and domain-specific adaptation remain necessary before ASST can be considered a validated assurance method.