Artificial intelligence and automated systems are moving from the back office into the operational core: detecting anomalies on a plant, scheduling maintenance, screening permits, flagging unsafe behaviour on camera, even drafting the risk analyses themselves. Used well, they catch things people miss and free experts to do the work only people can do. Used carelessly, they don't remove risk — they relocate it, often somewhere harder to see.
This guide is a practical way to think about that shift. It looks at where AI and automation genuinely help, the failure modes they introduce, the quieter problem of dependence on systems we no longer fully understand, and how to keep all of it inside a risk picture you can actually manage — using the same bowtie method you'd apply to any other hazard.
The opportunity
Automation and machine learning are good at exactly the things humans are weak at, and that is the source of their value:
- Vigilance at scale. A model watching thousands of sensor streams, transactions, or camera feeds doesn't get bored at 3 a.m. It can surface weak signals — a slow drift in a bearing temperature, an unusual access pattern — long before they'd reach a person.
- Pattern recognition. Predictive maintenance, early fault detection, and anomaly detection find correlations across more variables than any analyst could hold in their head.
- Speed and structure. AI can turn a messy description into a first-draft analysis in seconds. SolidBowtie's own AI drafting does this for bowties — it gives a workshop a structured starting point so the experts spend their time challenging the model, not staring at a blank canvas.
- Consistency. A well-specified automated check applies the same rule the same way every time, where a tired human reviewer might not.
The pattern worth noticing: AI is strongest as an amplifier of human judgment, not a replacement for it. The opportunities are real, but every one of them creates a new thing you now depend on.
A new kind of dependence
In 1983 the researcher Lisanne Bainbridge described the "ironies of automation": the more you automate, the more you ask of the humans left in charge. Automate the routine and you leave people to handle only the rare, hard cases — the ones they now get little practice at — while also expecting them to monitor a system that is reliable enough to lull them into not watching.
That irony is the heart of automation risk. The better the system, the deeper the dependence, and the more its failure or absence costs:
- An operator who hasn't manually controlled a process in two years is not well placed to take over in the ten seconds after the automation hands them a problem.
- A team that trusts an AI screening tool will, over time, stop independently checking what it approves.
- A decision pipeline built on a single model or vendor inherits that system's outages and blind spots as common-cause failures across everything downstream.
Dependence isn't inherently bad — we depend on lots of engineered systems. The danger is unexamined dependence: leaning on something whose failure modes you haven't mapped, and whose absence you haven't planned for.
The risk landscape
AI and automation introduce failure modes that differ from mechanical ones — they tend to be quiet, correlated, and tied to human behaviour. The main ones to know:
- Automation bias and complacency. People over-trust automated output, defer to it even when it's wrong, and stop looking for disconfirming evidence. The more reliable the system usually is, the stronger the bias.
- Deskilling and lost situational awareness. Skills that aren't practised decay. Operators "out of the loop" lose the feel for the process they're supposed to supervise and recover.
- Opacity. Many models can't explain why they produced an output. When a decision can't be interrogated, it can't be properly challenged, audited, or trusted in a safety case.
- Data drift and out-of-distribution inputs. A model is only as good as the data it learned from. When the world shifts — new equipment, new conditions, a situation it never saw in training — confidence stays high while accuracy quietly collapses.
- Brittleness and silent failure. Automated systems often fail without failing loudly. They keep producing plausible outputs that are wrong, with no obvious alarm, until something downstream breaks.
- Common-cause and cascading failure. Shared models, shared data feeds, or a shared cloud service mean one fault can disable many "independent" barriers at once — the opposite of the defence-in-depth you were aiming for.
- Security and adversarial manipulation. Inputs can be poisoned, models stolen or evaded, and language-model systems steered by prompt injection or fed manipulated context. Automation expands the attack surface.
- Accountability gaps. When an AI-assisted decision goes wrong, who owns it — the operator who accepted it, the team that deployed it, the vendor that built it? Unclear ownership is itself a risk.
Putting AI on the bowtie
The useful move is to stop treating "AI" as a special category and start placing it on a bowtie like anything else. An automated system can sit in three different positions, and confusing them is where analyses go wrong:
| Role on the bowtie | What it means | What to watch |
|---|---|---|
| Threat / cause | The AI or automation failing is a pathway to the top event (e.g. a bad model output triggers an unsafe action). | Treat it as a credible threat with its own preventive barriers. |
| Barrier | The system is a control — anomaly detection, an automated trip, an AI screen catching unsafe permits. | Like any barrier it has effectiveness, availability, and reliability — and it degrades. |
| Escalation factor | Automation bias or deskilling that weakens a human barrier without being the direct cause. | The classic hidden one: the human check still exists on paper but no longer really functions. |
Two consequences follow directly from this.
First, an AI barrier is still a barrier, so it needs an owner, a health status, and assurance. "The model handles it" is not a control you can rely on unless someone is responsible for it, you can tell whether it's currently working, and there are tasks that keep it healthy — monitoring for drift, revalidation, checking the alerting actually fires. This is exactly the discipline covered in barrier management, applied to software instead of hardware.
Second, automation bias belongs on the bowtie as an escalation factor, because it's the most commonly missed risk. The diagram shows a human verification barrier; in reality, the human rubber-stamps whatever the machine says. Naming that escalation factor explicitly — and giving it its own control, such as forcing an independent check — is often the single highest-value thing an AI-aware bowtie adds.
If you run these conversations in a bowtie workshop, put the question directly to the room: for each automated control, what tells us it's still working, and what happens to the humans around it when it usually is?
Design for graceful degradation
Because automated systems fail quietly and dependence builds invisibly, the controls that matter most are the ones that handle the system being wrong or absent:
- Meaningful human oversight. Not a human who clicks "approve," but one with the time, information, authority, and competence to override. Oversight that can't realistically say no isn't a barrier.
- Keep the manual path alive. If people may have to take over, they have to practise. Drills, periodic manual operation, and rotation keep the fallback real.
- Fail loud, fail safe. Prefer systems that signal uncertainty and degrade to a safe state over ones that confidently carry on. Out-of-distribution detection and confidence thresholds turn silent failure into a visible one.
- Preserve independence. Don't let your "layers of protection" secretly share a model, a data feed, or a cloud region. Independence is what makes defence-in-depth work; AI quietly erodes it.
Governance and assurance considerations
Managing AI risk well is mostly ordinary risk discipline applied to a new kind of component, and it slots into frameworks you may already use — ISO 31000 for risk management overall, and functional-safety standards such as IEC 61508 / 61511 where automation performs a safety function. Newer, AI-specific guidance is worth knowing:
- ISO/IEC 42001 provides a management-system framework specifically for AI — the "how do we govern this responsibly" layer.
- The EU AI Act takes a risk-tiered approach, placing heavier obligations on high-risk uses; if you operate in or sell into the EU, where your use lands on that scale shapes what you must do.
Beyond standards, a few practices carry most of the weight:
- Clear ownership and accountability for every automated decision — a named human or role, not "the system."
- Validation before trust, monitoring after. A model that passed acceptance once is not validated forever; drift monitoring and periodic revalidation are the assurance tasks that keep the barrier healthy.
- Change management for models. A retrained or updated model is a modification to a control and deserves the same scrutiny as a hardware change — including re-checking the bowties that depend on it.
- Proportionality. Match the rigour to the stakes. A model recommending maintenance scheduling needs less ceremony than one that can trip a process or gate a safety-critical permit.
(Standards and regulations evolve and vary by jurisdiction; treat the references above as orientation and confirm the current requirements for your sector and region.)
The balance
None of this is an argument against AI. The opportunities are genuine, and refusing good automation has its own risks — slower detection, human error on tedious tasks, missed signals. The argument is for clear-eyed dependence: adopt the capability, and in the same breath map what you now rely on, how it can fail, and what holds when it does.
That is precisely what barrier-based thinking is for. AI doesn't need a separate rulebook; it needs to be put on the bowtie — as a threat where it can fail, as a barrier where it protects (with an owner and a health status), and as an escalation factor where it quietly weakens the people around it. Do that, and automation becomes what it should be: a way to make your risk picture sharper and more responsive, rather than a confident black box you've quietly come to depend on.
A living bowtie — barriers with owners, health that responds, and incidents that feed back — is the natural home for keeping those AI controls honest as they, and the systems around them, change.