Trust Is Not the Absence of Failure

A thought I had this morning while reading the newspapers moved me to write something about the concept of trust, governance and human reliability.

It's about the inner trust in processes, governance, controls we built along the past decades, and yet we have read about cases of organizations failing these same controls because the people who should have enforced them were measured on the same numbers as the people breaking them.

We read news about humans misbehaving, breaking rules, working around policies and processes, exploiting loopholes, or simply making bad decisions.

So I asked myself a question: If humans can only be trusted within limits, and even then fail, why are we scared about AI and not the organizations and the people that are working with it? I believe this question gets asked less often.

Think about how organizations operate today. Governance, auditing, separation of duties, access controls, compliance, monitoring, incident investigations, all of these things exist precisely because organizations do not simply assume that humans will always do the right thing.

I'm not saying human and AI misbehaving have the same effects and therefore controls. The scale and impacts of AI misbehavior can be a thousand time stronger, and faster! But the reason why an AI system is misbehaving could be linked to a human wrong-doing, either intentionally or unintentionally.

So my point is that we already know how to build systems around something we cannot assume will always behave correctly. Why should the fundamental principle be different with AI?

The answer was never to pretend incidents wouldn't happen. The answer was to build systems capable of detecting them, understanding them, containing them, and learning from them.

But these controls themselves are still designed, implemented, monitored, and sometimes overridden by humans. And humans have incentives. Companies want to move faster, teams have deadlines, investors expect returns and governments don't want to fall behind.

So good AI governance cannot only ask:

What happens if the AI behaves in a way we didn't expect?

The answer becomes good architecture, and sometimes also good architectures can fail.

The question now should become:

What happens when the people around the AI have an incentive to accept more risk, weaken a control, ignore a warning, or move faster than the system is ready for?

Which brings us back, strangely enough, to exactly the same problem: trust. Not blind trust in the AI. But not blind trust in the humans controlling it either.

Governance has to account for both. So perhaps we need to think differently about what we mean when we say we want to “trust AI.”

Trust is not the absence of failure.

Trust is having enough controls, visibility, accountability, and learning mechanisms around a system that, when failure happens, we can detect it, understand what happened, respond effectively, and improve.

But those controls cannot apply only to the AI. They also have to apply to the humans and organizations deciding where to use it, how much autonomy to give it, how much risk to accept, and when a warning can be ignored.

Because in the end, AI governance isn't only about controlling technology. It's about governing a system made of technology, people, incentives, and consequences.

We never made complex human systems reliable simply by trusting humans more. We built governance around human unreliability.

Now we need to do the same with AI, without forgetting that the humans are still there.