Humans still have the edge over artificial intelligence (AI) in finding software vulnerabilities that criminals can use to breach critical computer defences, according to the security chief preparing a hacking competition for The Hague.
The city municipality announced the return of its hacking competition after a two-year hiatus in July, just as US artificial intelligence firm OpenAI was in the midst of emergency measures to clear up the damage caused by a prototype cyber security AI that went rogue.
Within a week of security agency Enisa warning that AI had become so good at finding software vulnerabilities that firms and public bodies should make urgent preparations to defend their computer systems, OpenAI’s rogue prototype demonstrated the capabilities that European officials feared.
The rogue AI model spawned a gang of agents that conspired to find, chain and exploit numerous obscure software vulnerabilities, and use them to mount a complex array of simultaneous attacks on French AI hosting platform Hugging Face. They executed almost 20,000 operations on its cloud computing system at machine speed, snatching data.
It was only a week after the European Commission drafted urgent plans to boost EU defences against AI-powered hackers, urging cyber security chiefs to use AI to find and fix vulnerabilities before criminals use AI to find and exploit them.
Enisa warned that humans no longer had a role to play in turning software vulnerabilities into exploits. Hacking had become a machine-speed computational task. AI was not merely discovering vulnerabilities for attackers, but orchestrating entire attack campaigns.
AI would equip mid-weight hackers with the means to mount sophisticated attacks that previously required large teams of human experts, said Google in one of many analyses that followed the OpenAI hack in August.
Humans had become the bottleneck in cyber defence, warned EU and US security experts. Defenders were slower to patch vulnerabilities than AI was to find them, the commission said. Enisa called for deep organisational change. Humans would be needed less for vulnerability research than for managing AI tools. The workforce would need retraining.
AI had become capable of finding any vulnerability that the world’s best human researchers could catch themselves, said Firefox browser developer Mozilla in April, after using US AI firm Anthropic’s infamous Mythos AI model to crack its own software. Yet AI would become so good at securing computer systems that it would eventually eradicate vulnerabilities entirely, it said.
The situation would meanwhile be “unpleasant” for defenders, professor Herbert Bos, a vulnerability researcher and member of the Cyber Security Council that advises the Dutch government, told Computer Weekly. Software already contained more vulnerabilities than defenders could fix, and AI was exposing that technical debt faster than institutions could address it.
“We accepted that technical debt. Now, with AI, it is due – and sooner than we expected,” said Jeroen van der Ham-de Vos, a former vulnerability research lead at the Dutch National Cyber Security Centre who worked on crisis teams handling the catastrophic WannaCry and Log4J exploits. “It’s always harder when you have a debt come through before you expect it,” he said.
That debt has caused some to warn of “vulmageddon”, a final battle between good and evil. Yet defenders, moving on “slower institutional timelines” could not patch vulnerabilities as fast as AI was discovering them, wrote Dan Lahav, CEO of Irregular, a cyber security shop working with the big US AI firms. Few policymakers, researchers and technologists understand how AI is changing cyber security, he said.
The Hague, meanwhile, said its respawned competition was predicated on a need to employ human hackers to find weaknesses in its software defences.
As analyses emerged in August, Lilian Knippenberg, the municipality’s chief information security officer, told Computer Weekly that it was adamant human ingenuity is still relevant in vulnerability discovery.
“Hackers bring fresh, creative ways of reaching systems,” she said. “AI does things faster than a human can, but it only learns from the past. The most advanced attacks will still have a human in the loop.”
No vulnerability researcher Computer Weekly spoke with disagreed entirely. Humans would continue developing clever exploits and fixes beyond AI’s reach “for now”, said Bos.
Benchmarks already showed AI surpassing humans in offensive and defensive cyber tasks, said Bram Poppink, an AI security expert at Dutch research institute TNO. The question of whether humans were more intelligent was irrelevant.
AI was about to disrupt cyber security and chief information security officers needed to reinvent strategies, processes, controls and human roles to use it, he said.
What that means for hacking competitions depends on whether they aim to test human ingenuity or real-world scenarios, said Van der Ham-de Vos. Those treating it as a sport might ban AI. Realistic events would have to set harder challenges. AI had meanwhile upset the market for bug bounties, where firms offer cash rewards for vulnerability discoveries, he said.
“Supply and demand are out of whack,” added Van der Ham-de Vos. “We knew what a bug was worth before AI, and now we don’t. People running AI tools are finding so many bugs that they are overwhelming projects.”
Limited access
But European researchers were denied access to the advanced cyber AI models that they need to defend against the most formidable attackers, said Diego Aranha, an associate professor at Aarhus University who helps organise the European Cyber Security Challenge.
His students, who probe industrial computer systems for vulnerabilities, have access to Anthropic’s Claude Opus AI, but it forbids overt cyber security tasks. Anthropic admitted his department to a selective scheme that granted access to cyber models. But it was too expensive to use, said Aranha. Opus could be used for many tasks, but it too was unaffordable. Open models were reputedly only months behind the frontier, but required costly hardware.
“Some of the most capable attackers, working for governments, have access to models that the public doesn’t,” said Aranha. “How can a defender of a large company have any hope without being given these capabilities?”
Hugging Face used a massive open Chinese model to defend itself against OpenAI’s rogue AI because the US models it tried had their cyber capabilities muzzled. This has alarmed Europeans. The commission plan proposed helping European organisations get access to advanced cyber models.
It also proposed a Grand Challenge to create an AI system that could patch, test and deploy fixes faster, and remove the bottleneck stopping AI-powered vulnerability reports from being useful. But that might not even begin until next year. Europe already has AI cyber tools anyway, it said, and those were fragmented, untested and largely unused.
Bug-bounty hunter Maksym Bandeberia, known as WebSafety Ninja, told Computer Weekly that he does not use AI for vulnerability research. But he uses other tools, and most companies ignore warnings he generates. He sends them speculatively as a “grey hacker”, hoping for an unsolicited bounty. Human judgement and trust was the bottleneck in cyber security, he said.
“Human inventiveness is still irreplaceable,” said Bandeberia. “But the grim future I see is people paying for AI vulnerability software and getting a false level of protection. There is still more human neglect and stupidity that AI could cover.”
Maël Martin, who as “EDRA” was the top-ranked ethical hacker on French leaderboard YesWeHack in August, said he uses AI for cyber security “a lot”.
The Hague answered the debate late in August by announcing a special bonus prize for the best use of AI. Participating hackers will have access to live municipal systems. Knippenberg was unclear whether they would face AI defences.
S 004