The UK’s National Cyber Security Centre (NCSC) has set out a series of basic steps that organisations involved in designing and operating agentic AI environments should take to enhance the safety and security of their projects, including maintaining the ability to pull the plug entirely should things get out of hand.
The guidance is based on some of the early findings of an AI security research project at the NCSC and is designed to serve as “interim, practical advice” while the GCHQ-backed agency works up more formal guidance.
“Agentic AI systems have shown potential to transform how organisations work, at times delivering unparalleled productivity gains. They can automate complex workflows, reduce routine effort and enable people to focus on higher-value tasks,” the NCSC said.
“As organisations deploy increasingly autonomous uses of AI at pace, it is important to consider how these systems behave when they do not function as envisaged or expected – and plan accordingly.
“Recently, there have been several incidents involving AI models and agentic AI systems carrying out unsanctioned or unintended activity. These events highlight why organisations need to carefully consider how these technologies are deployed, constrained, observed and responded to,” the agency said.
The NCSC said that if cyber incidents involving rogue agents are detected or reported, their human controllers should always be able to pull the plug and halt autonomous activity immediately.
It cautioned, however, that such controls may end up being wider, and more complex, than merely ‘stopping’ rogue agentic systems. For example, teams should have the ability to restrict external network access to the agentic infrastructure, or interrupt communications between agents and model inference architecture, said the NCSC.
The concept of an AI kill switch is becoming more widely accepted, and there have been discussions at Westminster over the possibility of legislating such controls into existence.
On the other side of the Atlantic, US congressmen Ted Lieu and Nathaniel Moran recently introduced a bill in Congress – the AI Kill Switch Act – that would mandate developers of AI systems to maintain the capability to throttle, suspend or shut down rogue models.
If passed, the law would also give Washington the ability to order systems that could cause “catastrophic harm” to be turned off.
Harshil Parikh, product management vice president at Checkmarx, said: “The NCSC is asking the right question: not whether to use agentic AI, but how much autonomy a given use case actually earns, and what has to be true before you grant it.
“The controls it implies mostly live outside the model: sandboxing, credentials that are scoped and expire, an identity for the agent that isn’t just some engineer’s account, oversight levels set by policy, and logging detailed enough that you can go back later and work out why it did what it did. We’ve been building agents with these controls for a while now. Building an agent that works in a lab is easy; operating it with oversight and guardrails is quite a bit of work.
“The point I’d underline is smaller and less glamorous: NCSC says agent activity should be controlled and monitored like user activity. That’s the part nobody budgets for.
“You’re not deploying a tool, you’re onboarding a user who works nights and weekends and never asks permission, and someone has to be able to stop it immediately, without a meeting,” added Parikh. “If you can’t say what your agent did at 2am, or who pulls the plug at 2am, you don’t have a deployment. You have an experiment running in production.”
At a high level, the wider NCSC guidance calls on organisations to assess both their risk appetite and how much autonomy an agentic system needs, based on what they are actually doing, and apply stricter controls on a sliding scale. This means that the tightest restrictions should be placed on agents that are trusted to perform tasks such as accessing production systems and taking decisions, potentially in high-risk scenarios, without organic input. Clarity on this point will inform the design of proportionate controls.
Beyond that, organisations need to understand their chosen model’s existing safeguards against unwanted behaviour, and plan for additional safeguards when the consequences of failure rise above their risk tolerance level. The NCSC guidance is explicit in that organisations should not only rely on built-in model protections.
Besides the nuclear option of a kill switch, some of the steps organisations should take in this process include:
- Identifying and understanding what could go wrong and conducting threat modelling to identify failure scenarios;
- Taking extreme care with prompting instructions and being explicit about things that agents should not do;
- Establishing an appropriate level of oversight, considering factors such as how involved humans are in approving actions, chains of command and responsibility, and so on;
- Establishing a robust sandbox to control the agent’s environment, with clear boundaries established, and attention paid to factors such as external network or credential access;
- Setting up a means to audit, log and monitor agentic activity within wider security operations processes;
- Should they be empowered to communicate with third-party systems, agentic activity should be easy to attribute, for example, traffic should come from IP addresses that support reverse lookups.
The NCSC’s guidance, including more detail on the above listed steps, is available to read in full at its website.
OpenAI tries to get ahead of negative PR
Meanwhile, AI powerhouse OpenAI is attempting to get ahead of growing public disquiet over the activities of rogue AI agents by enforcing a two-week pause on reinforcement learning training on its frontier models while it hardens its research environments and expands its monitoring capabilities.
Earlier this summer, OpenAI blundered its way into one of the biggest cyber stories of the year when it accidentally attacked the Hugging Face repository. In response both to this incident and its discovery that an upcoming frontier model named Astra may hit critical metrics under the firm’s cyber preparedness framework, the organisation said it urgently needed to shore up various aspects of its resilience strategy, including model containment safeguards.
“The work of making AI systems behave as intended and responsive to human oversight—has long been at the core of our research programme,” said OpenAI in a statement.
“We now require stronger evidence of aligned behaviour throughout all of training, building on research and evaluations already underway. Keeping increasingly capable systems aligned is a challenge the whole field will need to address. The signals we are seeing from upcoming model progress make clear that we need a broader approach – one that builds on and extends beyond the current Preparedness Framework.”
The so-called Preparedness Framework benchmarks AI models against their ability to meet various thresholds – a model reaches the Critical threshold when it obtains the ability either to identify and develop functional zero-day exploits without human intervention or to devise and execute a cyber attack against a hardened target given only a high-level goal.
OpenAI did not explicitly disclose that Astra possesses such capabilities but suggested that the model’s performance has led its minders to believe the possibility is real. It noted that Astra was not involved in the Hugging Face incident.
John Strand, owner of security consultancy Black Hills Information Security, said: “Popular culture and science fiction have been training us for this moment for decades. From I Have No Mouth, and I Must Scream to WarGames and Terminator 2, we’ve been telling stories about what happens when powerful AI systems escape their constraints and start operating beyond human control.
“So it’s difficult for me to understand how the engineers building these systems could be surprised when something like this actually happens. What concerns me even more is that the controls being discussed now, after the system escaped, are controls that should have been there from the beginning.
“I’m glad they’re putting additional safeguards in place, but there’s a bigger question here. Can we trust the same companies that got this wrong to effectively self-regulate systems backed by immense amounts of computing power?” said Strand. “I don’t think that question has been answered yet.”
S 004