What will the world be like the day after AI doomsday

 Let me tell you about my mate Aaron. Apart from being an Arsenal fan there is nothing I don't like about the guy. I used to work with him in IT and he was always a good laugh. He was also extremely good at his job and very savvy with technology. I can recall talking to him about the opening of the Channel Tunnel. He said "I won't use it until someone has died on it in an accident". This seemed to me to be a very strange thing to say, but he explained. "When new technology is deployed, I never use it until someone has died and then they iron out all the bugs that they left in due to cost cutting". He wouldn't fly any new make of aircraft, use any new train or do anything, until there had been an accident and all of the problems caused by cost cutting ironed out.

I met Aaron for a beer last week and I was thinking about this in relation to AI. We see AI being used for management of all manner of IT systems. We even saw a report in the Guardian last week stating that AI was designing new viruses. 

What could possibly go wrong pic.twitter.com/ihy9lyvlcK

— A Better Mill Hill 🔸🌈🇬🇧🇪🇺 (@ABetterMillHill) August 7, 2026

What could possibly go wrong? Well I worked in IT for 37 years, mostly in support of systems. My job was to fix things and get things running when they'd gone wrong. Generally when things went wrong, it was when new systems were introduced. You soon learn the limits of what resilience you can design into systems, when you have such a job. About 90% of the problems we dealt with were not caused by really difficult and obscure issues, they were caused by absolutely basic mistakes and misunderstandings. Generally they were fixed in hours. As I worked in Payments (card transactions, ATM technology, etc) that operate 24 x 7 and have to be available I saw the effects first hand of the most basic errors. Let me give you a couple of examples

In one, a basic operator error resulted in all electronic cash tills at a service station on the M1 being unable to take card payments for 1 hour. People were pulling off to fill up, and being told at the till that they couldn't pay by card. Within 30 minutes, there was a 5 mile tailback on the M1 as people were trying to pull over to fill up. After that, I never let my fuel light come on again on a long journey. It took about 20 minutes to code a fix to prevent that. The second was when a system that used to dial into company card systems to retreive card payments was being tested. Some test data had been input and all of the test telephone numbers were set to all 9's. When the test was performed, at 3am, the West Midlands emergency services 999 service was swamped with calls from the system. Genuine emergency callers could not get through for over 2 hours.

 I was on a contract at one of the UK's largest banks at the time. The chairman was woken up by the chief of police demanding an explanation. I wasn't working on the test, but it was hugely embarrassing. 

Both of these made me realise just how vulnerable our society is to the effects of IT failures. These incidents happened in the 1990's. Think how much more embedded in our lives IT is now, and how much more vulnerable we are. Now lets consider AI. AI offers huge opportunities for good. Advances in medical science, all manner of goods and services that we never dreamed we needed. It will remove many jobs, make all manner of services affordable. It sounds great. However technology doesn't always work and the more we trust it and let it do the things we need to do, the bigger the shock and the problems when it goes wrong. 

I've given a lot of thought to how it could go wrong. The first thing to bear in mind is the old IT acronym GIGO. That means Garbage in Garbage out. When an AI system is fed bad information, it will produce bad results. Whereas humans generally tend to realise when they've got stuff wrong and things are going badly, if things are outside of the AI models parameters, who knows what it will decide. SO much of the information on the net is wrong, so I would find it hard to trust something AI tells me without doing my own homework. This doesn't matter when you write blogs about obscure punk rock bands, but if an AI system is managing a nuclear power station and the data it is fed is spurious, I do worry. Human systems tend to have things like checks by other people built in to spot calamatous decisions. Who vets AI?

The second thing is coding, programming and hardware are never perfect. The unexpected happens. Now of course the vendors of AI systems will claim their systems are resilient, but I had a job for the best part of 37 years dealing with the problems caused when systems don't do what they are supposed to. I recall going to the opening of a automated warehouse, for one of the UK's leading retailers,, where a pallet full of tellies was squashed into the roof because a sensor wasn't working. I also know someone who had a car with an automatic parking feature that wrote his car off the first time he used it. A sensor malfunctioned. These sort of things make me wary.

The third thing is that there are teams of malicious hackers. These fall into three main camps. Criminals, hostile governments and malicious ne'er do wells who love mayhem. All of these are hugely interested in the potential of AI for making mayhem for us. One of my last big IT jobs was looking at the resilience of a major UK company in the face of a serious, malicious IT attack. My boss had to present a paper at a conference. He asked me to write it for him. I looked at all the ways a hostile intruder may try and get in compromise the company's IT systems. I realised that all sorts of assumptions had been made that were completely ridiculous. The power of an AI system to exploit these weaknesses was simply mind boggling. As a matter of urgency the weakest vulnerabilities were fixed. My boss presented the paper. He said that the room was aghast with the conclusions. Many massive companies, particularly in the US had almost no protections in place beyong basic password controls. To keep a malicious party with an AI running on a super computer at bay, they had no chance.

It lead me to conclude that the West is walking a tightrope blindfolded when it comes to IT security. We can't see the crowd throwing rocks at us. If they all miss, we'll be fine, but we are unprepared for where it hits.

But that is not what interests me most. I believe that we will see a major catastrophie, with loss of life caused by one of these scenarios. And when this happens, we will see a massive re-assessment of our relationship with AI. The thing is, that may happen after the poinbt where we can put the genie in the box. If all the people who AI replaced no longer are around or have the skill sets to clean up the mess, we are in an apocolyptic situation, where we have to keep software running that we cannot trust, to do the basic things our society needs, like keeping the lights on in winter and the water flowing. It will be bad enough if the problems are just down to glitches or hardware problems. If they are down to a malicious actor gaining control of the systems that run our lives, using AI systems that we cannot circumvent, who knows what will happen?