AI Incident Management: What Should Happen When AI Goes Wrong?
Most AI governance focuses on what happens before an AI system is deployed.
Assess the risk.
Approve the use case.
Define human oversight.
Set permissions.
Test the system.
Document ownership.
Then the AI goes live.
And something unexpected happens.
An AI system produces materially incorrect information.
Sensitive data appears somewhere it should not.
An AI agent takes an unintended action.
A customer receives an inappropriate response.
A model behaves differently after an update.
Employees begin using a capability outside its approved purpose.
An automated workflow starts producing errors at scale.
Now the organisation faces a different question:
What happens when AI goes wrong?
At that moment, having an AI policy is not enough.
Leadership needs an AI incident-management capability.
Because effective AI governance is not simply about preventing problems.
It is also about ensuring the organisation can detect, contain, investigate, resolve and learn from them when they occur.
What is an AI incident?
An AI incident is an event involving an AI system, use case or agent that causes, or has the potential to cause, material harm, loss, disruption, policy breach or unintended business impact.
Examples might include:
incorrect or misleading outputs;
inappropriate automated decisions;
sensitive-data exposure;
security events;
unexpected agent actions;
unauthorised system access;
financial errors;
customer harm;
employee impact;
regulatory concerns;
systematic bias;
failure of required human oversight;
AI operating outside its approved purpose.
Not every poor AI output is necessarily an incident.
The organisation needs criteria for distinguishing:
normal error
from
material exception
from
AI incident.
For more on classifying AI according to potential consequence and exposure, read AI Risk Classification: How to Classify AI Use Cases by Risk.
Why AI incidents are different
Many organisations already have:
cybersecurity incident management;
data-breach procedures;
operational incident management;
complaints processes;
business continuity.
AI incidents may interact with all of these.
But AI can introduce additional questions.
Was the output generated or retrieved?
Was the AI operating within its approved purpose?
What model/version was involved?
What information did it use?
Did a human approve the action?
Was an agent acting autonomously?
Which tools did the agent access?
Can the action be reversed?
Did the same output affect other customers?
Has the underlying behaviour changed?
An AI incident process should therefore connect to existing incident-management systems rather than unnecessarily duplicate them, while capturing the AI-specific information required to understand what happened.
The first challenge is knowing an incident happened
Some AI failures are obvious.
A system stops working.
A customer complains.
A financial transaction fails.
Others are harder to detect.
An AI system may continue operating while:
quality deteriorates;
incorrect information increases;
employees override outputs repeatedly;
agent exceptions rise;
business outcomes worsen.
That means incident management begins with observability.
If the organisation cannot see how material AI is behaving, it may not know when intervention is required.
The Stratify AI Incident Management Framework
Structure AI incident management through seven stages:
DETECT → CONTAIN → ESCALATE → INVESTIGATE → DECIDE → RECOVER → LEARN
The objective is not merely to resolve the immediate problem.
It is to ensure the organisation understands:
what happened;
why it happened;
who was affected;
what should change;
whether the AI should continue operating.
1. DETECT — How do we know something has gone wrong?
AI incidents can be detected through:
system monitoring;
employee reports;
customer complaints;
human overrides;
exception rates;
security alerts;
performance measures;
audit activity;
agent logs;
business-outcome monitoring.
Detection requirements should reflect the consequence of the AI.
A low-risk internal assistant may need relatively light monitoring.
A customer-facing agent with financial authority may require much stronger observability.
The Detection Question
For every material AI use case, ask:
If this AI began behaving materially differently tomorrow, how would we know?
If the answer is:
“Someone would probably notice,”
the monitoring model may not be sufficient.
2. CONTAIN — How do we prevent further impact?
Once a material issue is detected, the immediate priority may be to limit further exposure.
Containment might include:
pausing the AI;
removing system access;
reducing agent permissions;
switching to human processing;
isolating affected data;
reverting to a previous version;
stopping external communication;
suspending transactions.
The appropriate response depends on the incident.
But containment authority needs to exist before the incident occurs.
Someone needs to be able to say:
Stop.
For more on defining who should have authority to intervene, override or stop AI, read AI Decision Rights: Who Should Be Allowed to Approve, Override or Stop AI?
Stop authority needs to be practical
It is not enough for a policy to say:
“AI may be suspended if required.”
Leadership should know:
Who can suspend it?
Can they technically do so?
How quickly?
Does suspension affect other systems?
What happens operationally after shutdown?
What fallback process exists?
A theoretical stop button is not the same as an operational stop capability.
3. ESCALATE — Who needs to become involved?
Not every incident requires executive attention.
But material incidents need a clear escalation route.
Depending on the event, this might involve:
business owner;
Technology;
Security;
Risk;
Legal;
Data Protection;
Compliance;
executive leadership;
Board.
Escalation should reflect:
impact;
urgency;
authority;
regulatory implications;
financial exposure;
customer/employee consequence.
For more on defining when an AI issue should move to a higher level of authority, read AI Escalation Framework: When Should an AI Decision Move Up the Organisation?
The Stratify AI Incident Severity Model
A useful first-pass model could classify incidents into four levels.
LEVEL 1 — MINOR
Limited impact.
Contained locally.
No material external consequence.
LEVEL 2 — SIGNIFICANT
Meaningful operational impact or repeated failure requiring formal investigation.
LEVEL 3 — MAJOR
Material customer, employee, financial, security, legal or operational impact.
Senior leadership involvement may be required.
LEVEL 4 — CRITICAL
Severe or widespread impact potentially requiring immediate executive action, external notification or Board oversight.
The exact definitions need to be tailored to the organisation.
The important point is that severity should determine response, rather than every incident following the same process.
4. INVESTIGATE — What actually happened?
Once immediate exposure is controlled, investigate.
The organisation may need to determine:
Which AI system was involved?
Which version?
What was the approved purpose?
What data was used?
What input triggered the event?
What output or action occurred?
Which systems were accessed?
Was a human involved?
Did required controls operate?
Was the action within permission?
How many people or transactions were affected?
Can the outcome be reproduced?
This creates the factual basis for deciding what happens next.
Don't assume the AI caused the incident
This is important.
An incident involving AI does not automatically mean the model itself failed.
The underlying cause could be:
poor data;
incorrect configuration;
integration failure;
human misuse;
weak workflow design;
inappropriate permissions;
vendor change;
missing control;
unclear policy;
unexpected model behaviour.
Incident investigation should identify the system cause, not simply label the event:
“AI error.”
5. DECIDE — What should happen to the AI now?
Once the organisation understands enough about the incident, leadership needs a decision.
Possible outcomes include:
CONTINUE
No material change required.
CHANGE
Modify data, prompts, controls, workflow, permissions or technology.
RESTRICT
Reduce scope or authority.
PAUSE
Suspend while additional investigation or remediation occurs.
STOP
End the use case.
This connects incident management to the wider AI investment and governance system.
For more on making explicit continuation decisions rather than allowing AI initiatives to continue automatically, read When Should You Stop an AI Project? A Framework for Scale, Change or Stop Decisions.
6. RECOVER — How do we return to safe operation?
Recovery is more than switching the AI back on.
The organisation may need to:
correct affected records;
communicate with customers;
restore data;
change permissions;
update workflows;
retrain employees;
retest the system;
validate controls;
monitor closely after restart.
Before resuming normal operation, ask:
What evidence tells us the problem has been sufficiently addressed?
That evidence should reflect the severity of the incident.
Recovery should include affected people
AI incidents may affect:
customers;
employees;
suppliers;
partners.
Technical recovery does not automatically resolve the wider impact.
The organisation may need:
communication;
remediation;
human review;
correction;
support.
Incident management should therefore consider the business outcome, not simply system availability.
7. LEARN — What should change because this happened?
This is where incident management becomes organisational learning.
Ask:
Why did the incident occur?
Why was it not prevented?
How was it detected?
Did escalation work?
Did stop authority work?
Was ownership clear?
Did monitoring provide sufficient evidence?
Could the same issue exist elsewhere?
What needs changing?
Potential changes may involve:
policy;
controls;
risk classification;
permissions;
training;
monitoring;
technology;
data;
governance;
responsibilities.
The incident should improve the system.
The Stratify AI Incident Lifecycle
The full lifecycle becomes:
DETECT
What happened?
↓
CONTAIN
How do we limit further impact?
↓
ESCALATE
Who needs to know or decide?
↓
INVESTIGATE
Why did it happen?
↓
DECIDE
Continue / Change / Restrict / Pause / Stop?
RECOVER
How do we restore appropriate operation?
↓
LEARN
What changes now?
So:
DETECT → CONTAIN → ESCALATE → INVESTIGATE → DECIDE → RECOVER → LEARN
AI agents make incident management more complex
An AI assistant may produce an incorrect answer.
An AI agent may:
produce the answer;
update a system;
send a communication;
initiate a transaction;
trigger another agent.
That means the incident can propagate beyond the original AI output.
Leadership may need to reconstruct:
what the agent did;
which tools it used;
which permissions were exercised;
what other systems were triggered;
whether actions can be reversed.
For more on defining the authority boundaries within which agents operate, read AI Agent Permissions: What Should an AI Agent Be Allowed to Access and Do?
The Agent Action Trail
For material agents, the organisation should ideally be able to reconstruct:
TRIGGER
What initiated the agent?
↓
CONTEXT
What information did it receive?
↓
DECISION
What did it determine?
↓
ACTION
What did it do?
↓
SYSTEM
Where did it act?
↓
OUTCOME
What happened?
↓
ESCALATION
Did human intervention occur?
This will become increasingly important as agentic AI expands.
Human oversight needs an incident role
Human oversight should not exist only during normal operation.
The oversight model should define what happens when:
AI confidence is low;
exceptions occur;
behaviour changes;
impact becomes material.
Humans need:
authority;
information;
time;
skills
to intervene meaningfully.
For more on designing effective human intervention rather than symbolic oversight, read Human Oversight in AI: When Does AI Need a Human in the Loop?
Every incident needs an owner
During an incident, multiple specialist teams may become involved.
That can create another accountability problem.
Who owns resolution?
The answer may vary according to incident type.
But there should be a clearly identified incident owner coordinating:
containment;
investigation;
decisions;
actions;
closure.
Specialists contribute expertise.
The incident owner ensures the overall response progresses.
The Stratify AI Incident Record
For every material AI incident, record:
INCIDENT ID
Unique reference.
AI USE CASE / AGENT
What was involved?
BUSINESS OWNER
Who owns the affected outcome?
INCIDENT OWNER
Who coordinates resolution?
DETECTION
How was it discovered?
SEVERITY
How material is it?
IMPACT
Who or what was affected?
CONTAINMENT
What immediate action occurred?
ESCALATION
Who became involved?
ROOT CAUSE
What contributed to the incident?
DECISION
Continue / Change / Restrict / Pause / Stop.
REMEDIATION
What needs changing?
EVIDENCE
What supports closure?
LEARNING
What should change elsewhere?
STATUS
Open / Contained / Remediation / Closed.
Incident management should connect to risk classification
Suppose a supposedly low-risk use case produces repeated material incidents.
That is evidence.
Its original classification may need revisiting.
Likewise, an agent operating safely for an extended period may provide evidence relevant to future controls.
Governance should therefore be dynamic.
Incidents should feed back into:
risk;
controls;
permissions;
monitoring;
investment;
scale decisions.
Incident management should connect to accountability
A material incident can expose accountability weaknesses.
Leadership may discover:
nobody owned the use case;
stop authority was unclear;
the business assumed Technology was monitoring;
Risk assumed the business was monitoring;
an agent had permissions nobody realised existed.
For more on identifying these organisational ownership failures before an incident exposes them, read The AI Accountability Gap: 7 Signs Nobody Really Owns AI.
Near misses matter too
Not every useful incident produces harm.
An employee may catch an AI error before it reaches a customer.
An agent may attempt an action that permissions correctly prevent.
A control may stop sensitive information leaving the organisation.
These are valuable signals.
They can reveal:
weaknesses;
control effectiveness;
emerging risks.
Organisations should consider capturing material near misses, not only incidents that caused actual harm.
The AI Incident Review
Leadership should periodically review:
VOLUME
How many material incidents occurred?
SEVERITY
How serious were they?
SOURCE
Which systems/use cases are involved?
CAUSE
What patterns are emerging?
RESPONSE
How quickly were incidents contained?
ACCOUNTABILITY
Was ownership clear?
CONTROLS
Did safeguards operate?
LEARNING
What has changed?
The objective is not simply to minimise the incident count.
A low incident count could mean:
AI is safe.
Or:
nobody is detecting/reporting problems.
The quality of the incident-management system matters.
The Fractional CAIO and AI incidents
A Fractional CAIO should not replace specialist:
Security;
Legal;
Risk;
Compliance;
Technology
incident-response functions.
The role can help ensure AI specific incidents are connected into the wider management system.
For example:
Does the incident affect the AI portfolio?
Does risk classification need changing?
Does ownership need changing?
Should investment continue?
Does another use case have the same weakness?
What does executive leadership need to know?
What action needs tracking?
This is another example of the Fractional CAIO acting as orchestrator rather than substitute specialist.
10 Questions to Test AI Incident Readiness
Can employees recognise what constitutes a material AI incident?
Can material AI incidents be detected through monitoring, reporting or exception processes?
Is stop or containment authority clearly assigned?
Are incidents classified according to severity?
Are escalation routes defined for material AI incidents?
Can the organisation reconstruct what a material AI system or agent did?
Is there a named owner responsible for coordinating each material incident?
Are Continue / Change / Restrict / Pause / Stop decisions explicitly recorded?
Do incidents and near misses feed back into risk, controls, permissions and monitoring?
Does leadership periodically review AI incident patterns and organisational learning?
If several answers are unclear, the organisation may have preventative AI governance without having a sufficiently developed response capability.
Good AI governance assumes something will eventually go wrong
No organisation can eliminate every AI failure.
Models can behave unexpectedly.
People make mistakes.
Data changes.
Systems fail.
Agents encounter unusual situations.
The objective of governance should therefore not be:
Nothing will ever go wrong.
It should be:
When something goes wrong, we will know, we can limit the impact, the right people will act, and the organisation will learn.
That is a much more realistic standard.
Because mature AI governance is not defined only by how organisations approve AI.
It is also defined by how effectively they respond when reality differs from what they expected.
Frequently Asked Questions About AI Incident Management
What is an AI incident?
An AI incident is an event involving an AI system, use case or agent that causes or could cause material harm, loss, disruption, policy breach or unintended business impact.
What are examples of AI incidents?
Examples can include sensitive-data exposure, inappropriate automated decisions, significant incorrect outputs, unauthorised agent actions, financial errors, security compromise or failures of required human oversight.
Who should own an AI incident?
A named incident owner should coordinate the response, while the underlying business owner remains accountable for the affected business outcome. Specialist teams may contribute depending on the incident.
Should every AI error be treated as an incident?
No. Organisations should define criteria distinguishing normal errors, operational exceptions and material incidents according to impact, risk and context.
How should an AI incident be escalated?
Escalation should reflect severity, potential impact, urgency, authority and relevant specialist requirements. Material incidents may require involvement from Business, Technology, Risk, Security, Legal, executives or the Board.
Should an AI system be stopped after an incident?
Not automatically. The appropriate response may be Continue, Change, Restrict, Pause or Stop depending on severity, root cause, controls and available evidence.
How are AI agent incidents different?
Agents can take actions across systems, meaning an incident may involve permissions, transactions, communications or delegated actions in addition to AI-generated outputs.
What should be recorded during an AI incident?
Record the affected AI, owners, severity, impact, detection method, containment, escalation, evidence, root cause, decisions, remediation and learning.
What is an AI near miss?
A near miss is an event where an AI issue could have caused material impact but was detected or prevented before that impact occurred. Near misses can provide useful governance evidence.
How can organisations learn from AI incidents?
Review patterns across incidents and near misses and use the evidence to improve risk classification, controls, monitoring, permissions, training, ownership and governance.
How Prepared Is Your Organisation for an AI Incident?
Effective AI governance needs to work when AI performs as expected — and when it does not.
Leadership needs confidence that material AI issues can be detected, contained, escalated, investigated and resolved, with clear ownership and evidence throughout the process.
Take our free 5-minute Executive AI Governance & Risk Assessment →
Discover where your organisation is strong, where governance gaps exist and what to prioritise next.
Comments