Kategorie Uncategorized

Generative and agentic AI can accelerate engineering, research and compliance work. In safety-critical environments, however, producing a plausible answer is not enough. Organisations need to demonstrate why an AI-supported result can be trusted, what evidence supports it, where uncertainty remains and who is accountable for the final decision.

Artificial intelligence is rapidly moving beyond simple productivity tools. Large language models can analyse requirements, evaluate documents, identify inconsistencies, propose test cases, support risk analysis and coordinate increasingly complex workflows through AI agents.

For organisations operating in regulated or safety-critical environments, this creates significant opportunities. It also creates a fundamental assurance problem.

An AI-generated result may look convincing while being incomplete, based on weak evidence or influenced by incorrect assumptions. In engineering, critical infrastructure or safety-related processes, the consequences can be considerably more serious.

The relevant question therefore changes from “Can AI produce a useful result?” to “Can we demonstrate why this result may be trusted?” That distinction is at the heart of AI Assurance.

AI Governance is necessary — but it is not AI Assurance

AI governance establishes important organisational boundaries. It can define which models may be used, how sensitive information is handled, who owns an AI system and which use cases require approval. These controls are necessary, but they do not automatically establish the reliability of an individual AI-supported engineering result.

For that, additional questions must be answered: Where did the information come from? Which evidence supports the generated claims? Which assumptions were made? Has the result been independently challenged or verified? What uncertainty remains? Who has the authority to accept the result?

AI Assurance therefore needs to connect organisational governance with the actual evidence and decision-making process. The objective is to create a defensible chain from evidence to decision.

From AI Output to Engineering Evidence

A language model can generate an output within seconds. That output should not automatically become engineering evidence. A controlled assurance process introduces additional stages:

Source → Evidence → AI Analysis → Independent Verification → Human Decision → Auditable Result

The source establishes where information originated. Evidence establishes whether that information can support a relevant claim. AI analysis can structure, compare, interpret or transform the available evidence. Independent verification challenges the resulting claims and identifies contradictions, unsupported assumptions or missing information. Finally, an accountable human decides whether the available evidence is sufficient for the intended use.

Agentic AI requires stronger assurance

The assurance challenge becomes more complex with agentic AI. Instead of a person asking a single model a question, an AI system may contain several specialised agents performing different tasks.

More agents do not automatically create more trustworthy results. If several agents rely on the same source, assumptions or similar model behaviour, they can reinforce the same error. A robust agentic architecture therefore requires functional separation of responsibilities.

Evidence Sources → Research Agent → Analysis / Engineering Agent → Independent Reviewer or Red-Team Agent → Verification → Human Decision Gate → Auditable Result

The reviewing agent should challenge assumptions, identify missing evidence, detect contradictions and determine whether claims are actually supported. AI systems should be designed to expose unresolved problems rather than optimise them away.

Human-in-the-Loop is necessary — but not sufficient

Simply placing a human at the end of an automated workflow does not create effective assurance. The decision-maker must be able to see the relevant evidence and its origin, assumptions, unresolved contradictions, uncertainty, verification results and the limits of the AI system.

The human must retain genuine authority to reject, escalate or request further evidence. This is what we describe as Human Epistemic Authority: AI may search, structure, analyse and propose, while the accountable human retains authority over what is accepted as sufficiently supported knowledge for the decision at hand.

Fail Visible instead of Fail Plausible

Generative AI is very good at producing plausible language. That becomes problematic when missing information is transformed into a confident-looking answer. AI Assurance should follow the opposite principle: Fail Visible.

If evidence is insufficient, the system should state that evidence is insufficient. If sources contradict each other, the contradiction should remain visible. If verification fails, the workflow should stop or escalate. In safety-critical AI, an explicit “insufficient evidence” can be considerably more valuable than an impressive answer.

Traceability must be designed into the workflow

AI-supported engineering should extend rather than break established engineering traceability. For important AI-supported claims, it should be possible to reconstruct:

Source → Claim → Analysis → Verification → Decision → Result

Sources should be retained or referenced, claims distinguishable from evidence, AI-generated interpretations attributable, verification results recorded and human approvals traceable. Changes to models, prompts, tools or source information should be treated as potentially relevant changes to the assurance context.

Verification and Validation remain essential

AI does not eliminate the distinction between verification and validation. An AI system can behave as designed and still produce a result inappropriate for a safety-related engineering decision. Assurance therefore needs both perspectives.

Depending on the use case, this may involve independent review, deterministic checks, comparison against authoritative sources, requirements-based testing, consistency analysis, controlled test datasets or human expert validation. NIST provides resources for testing, evaluation, verification and validation (TEVV) within trustworthy AI risk management.

The required assurance depth should depend on the potential impact of the AI-supported activity. An AI system drafting meeting minutes does not require the same assurance architecture as an AI agent identifying safety requirements or proposing changes to an engineering baseline.

Controlled Delegation

Not every activity should be delegated to AI in the same way. A practical AI Assurance framework should distinguish between delegable activities, controlled automation and human-exclusive decisions.

These boundaries should be defined before deploying autonomous agents. Automation capability should not determine delegation authority. Risk, evidence requirements and accountability should.

What an AI Assurance framework should contain

A practical assurance framework for safety-critical or regulated AI should establish governance and accountable roles, classify AI use cases according to risk and impact, define permitted levels of delegation and specify requirements for evidence and source traceability.

It should also establish independent verification and validation mechanisms, effective human decision gates, audit trails, model and configuration change control, and explicit escalation or stop conditions. For agentic systems, additional controls are required around agent responsibilities, tool permissions, evidence access and interaction between agents.

Most importantly, the framework must distinguish between generating information and accepting information as evidence.

AI Assurance in Critical Infrastructure

These principles become particularly relevant in railway and critical infrastructure environments.

Railway, energy, transport and other regulated environments already rely on structured engineering lifecycles, risk management, verification processes and formal evidence. AI should not create a parallel universe beside those established processes. The stronger approach is to integrate AI Assurance into the existing engineering and governance framework.

The assurance question is not simply whether the AI model itself is safe. It is whether the complete socio-technical process in which AI participates remains controlled, traceable and accountable.

AI Assurance is ultimately about accountability

AI Assurance should not attempt to prove that an AI system will always be correct. Instead, assurance should create a controlled environment in which organisations can determine when AI-generated results are sufficiently supported for their intended use — and when they are not.

That requires evidence, traceability, independent verification, visible uncertainty and accountable human authority.

The defining question for safety-critical AI should therefore not be “Did the AI produce an answer?” It should be: “Can we demonstrate why this answer may be trusted, where its limits are, and who remains accountable for acting on it?”

References and further reading

• NIST — AI Risk Management Framework (AI RMF)
• NIST — Generative Artificial Intelligence Profile (NIST AI 600-1)
• UK Department for Science, Innovation and Technology — Introduction to AI Assurance

About Topp Consulting

Topp Consulting supports organisations introducing AI into regulated, safety-critical and critical-infrastructure environments. Our approach combines AI governance and assurance with engineering traceability, independent verification, human decision authority and experience in complex IT/OT transformation.

Explore our AI Assurance approach
Related expertise: Railway & Critical Infrastructure