There is a particular kind of catastrophe that never comes with a countdown. It does not announce itself with a press release, a product launch or a dramatic demonstration. It simply happens â and then, afterwards, everyone agrees it was obvious all along.
That is the warning at the heart of a set of letters published by the Guardian, whose authors push back against the way public debate about artificial intelligence tends to be framed. Much of the conversation, they suggest, is organised around visible thresholds: the arrival of a system smarter than humans, a spectacular failure, a moment when regulators finally decide the technology has gone too far. The implication is that we will know when the danger has arrived, and that there will be time to react.
The correspondents argue this is a comforting fiction. The real harms of automated systems, they contend, are far more likely to accumulate quietly â inside bureaucracies, benefits systems, hiring processes, credit decisions, medical triage and military logistics â where the people affected have little visibility into how a decision was reached, and no obvious mechanism to challenge it. By the time a pattern of harm becomes statistically undeniable, it has usually been running for years.
The problem of invisible failure
One reason for this is structural. Traditional disasters have a shape: a collapse, a crash, a leak. They generate images, casualty figures and inquiries. Failures of algorithmic systems often generate nothing but slightly worse outcomes, distributed unevenly across a large population. A marginally higher rejection rate here, a marginally longer wait there. No single case looks like a scandal; the aggregate might be one.
Another reason is commercial. Systems are frequently deployed by organisations with strong incentives not to look too closely at their own performance, and evaluated using metrics chosen by the same organisations. Where independent auditing exists at all, it tends to arrive late and rely on data that the operators control.
The letters also touch on a subtler point: the danger of dependency. As institutions hand over more of their decision-making and their drafting, summarising and reasoning to automated tools, the human expertise required to notice a malfunction may itself erode. A system that is wrong in a plausible-sounding way is far more dangerous than one that is wrong in an obvious way â particularly if the people checking it have lost the habit of checking.
What follows from the argument
If the premise is accepted, the policy conclusions shift. Attention moves away from speculative scenarios about superintelligence and towards unglamorous infrastructure: mandatory logging of automated decisions, meaningful rights of appeal, independent access to training data and model behaviour, incident reporting regimes comparable to those in aviation or pharmaceuticals, and clear legal liability when a system causes harm.
None of that is as compelling as a debate about machine consciousness. It is also, the letter writers imply, the only kind of preparation that works against a threat with no launch date.
The broader message is a caution against waiting for permission to worry. Public alarm tends to require a triggering event, and regulation tends to require public alarm. If the most serious harms of artificial intelligence really do arrive unannounced â diffuse, deniable and difficult to attribute â then the familiar sequence breaks down. The alternative is to build the monitoring capacity now, before anyone can point to the wreckage and say it should have been foreseen. Read More
Leave a Reply