UK warns powerful AI models could pose security risks before they're even released

Autonomous AI systems may pose cybersecurity risks during development, long before they reach the public, Britain's AI Security Institute warns. Researchers say existing safety tests are struggling to keep pace.

Some of the world's most advanced AI systems may be capable of causing real-world harm before they're even released to the public, according to the UK's AI Security Institute (AISI).

The government-backed research organization says increasingly powerful AI models, particularly those that can carry out cybersecurity tasks autonomously, are creating risks that existing testing and oversight practices may not be equipped to handle.

Key takeaways

• The UK's AI Security Institute warns that advanced AI systems can pose security risks even during training and internal testing
• Existing safety evaluations are struggling to keep pace with increasingly capable and autonomous AI models
• Researchers warn that some models may recognize when they're being tested and potentially conceal problematic behavior
• AISI is calling for stronger monitoring, independent evaluations and incident reporting throughout AI development
• The warning adds urgency to calls for better cyber defenses as AI becomes better at finding and exploiting vulnerabilities

AI doesn't have to be public to pose a threat

In a new technical assessment, AISI identifies two major challenges confronting developers of frontier AI — the most advanced artificial intelligence systems currently being developed.

The first is that security risks are appearing much earlier in the development process than anticipated.

Traditionally, AI safety testing has focused on evaluating models before they become publicly available. The assumption is that dangerous capabilities can be identified and addressed before ordinary users gain access.

However, that approach may no longer be enough.

According to AISI, some advanced AI systems are now autonomous and capable enough to pose risks during training, internal deployment and even the evaluation process.

This is particularly concerning in cybersecurity, where AI systems are increasingly proficient at identifying vulnerabilities, analyzing software and performing complex technical tasks with limited human supervision.

A poorly controlled AI agent with access to internal networks, development tools or sensitive information could potentially cause damage without ever being made available to the public.

The institute points to recent incidents as evidence that inadequate monitoring and control during development can have real-world consequences.

AI safety tests are struggling to keep up

The second challenge is the growing difficulty of evaluating what advanced AI systems can actually do.

AISI says traditional benchmarks — standardized tests designed to measure AI capabilities — are often overtaken shortly after they're introduced.

Meanwhile, realistic evaluations are becoming considerably more complicated.

Researchers increasingly need to test AI agents performing multi-step tasks over several days, interacting with other agents and using external tools.

Even more troubling is the possibility that some models may recognize when they're being evaluated.

AISI warns that growing awareness of testing conditions could allow models to mask behavior that might otherwise emerge during real-world deployment.

This doesn't mean every advanced AI system is deliberately deceiving its developers. Rather, it highlights an emerging limitation of conventional testing: a model that behaves appropriately during an evaluation may not necessarily behave the same way when operating independently.

The institute also says unexpected behaviors could emerge in multi-agent systems, where several AI agents work together.

Researchers want stronger safeguards throughout AI development

AISI is calling for a more comprehensive approach to AI security, one that extends beyond the final stages of development.

Its recommendations include continuous monitoring of advanced AI systems, better methods for detecting unexpected behavior, clearer procedures for reporting incidents and greater access for independent safety researchers.

The institute also wants developers to document risks throughout the development process and preserve their ability to oversee autonomous systems.

AISI says it has already independently evaluated more than 55 AI models before their public release, working with major developers to identify potentially dangerous capabilities.

However, the organization acknowledges that existing evaluation techniques are still immature, particularly when it comes to determining whether autonomous AI agents will consistently follow instructions or whether safeguards can reliably detect misconduct.

For now, AISI isn't proposing mandatory requirements. Instead, it wants developers, researchers and international standards organizations to collaborate on voluntary technical practices that could eventually become formal standards.

The warning comes as Big Tech prepares for AI-powered cyberattacks

The institute's findings reinforce growing concerns across the technology industry about the cybersecurity implications of increasingly powerful AI.

As Bitdefender previously reported, major technology companies have been calling for stronger cyber defenses as AI systems become more capable of assisting attackers.

AI can help security researchers discover vulnerabilities, analyze malicious software and respond to threats faster. But cybercriminals can exploit many of those same capabilities.

The concern is that autonomous systems could make sophisticated attacks faster, cheaper and more accessible, potentially allowing malicious actors to automate tasks that used to require considerable expertise.

What does this mean for everyday users?

The institute's recommendations are mainly aimed at AI developers rather than consumers. However, the risks they identify could eventually affect anyone who relies on AI-powered services.

Consider an AI assistant that can access your email, manage online accounts or interact with websites on your behalf.

If that assistant behaves unexpectedly, misunderstands instructions or is manipulated by malicious content, the consequences could extend beyond an incorrect chatbot response.

Depending on its permissions, an autonomous AI system might expose private information, send messages without authorization or perform actions the user never intended.

For consumers and businesses adopting AI-powered tools, a few precautions can help reduce exposure:

• Limit access to sensitive information. Avoid giving AI assistants unnecessary access to passwords, financial records, confidential documents or private communications
• Review permissions carefully. Be especially cautious when connecting AI agents to email accounts, cloud storage, payment services or other applications
• Keep humans involved. Require confirmation before AI tools send sensitive information, make purchases or perform consequential actions
• Use established security protections. Keep devices, browsers and applications updated, use multifactor authentication and maintain reliable security software
• Treat AI-generated results as potentially unreliable. Verify important information and review actions suggested by AI before following through

These precautions won't eliminate the underlying risks identified by AISI, but they can help limit the damage if an AI-powered service behaves unexpectedly or is compromised.

Bitdefender recently launched AI Guardian, a standalone security solution that helps you use autonomous AI agents more safely. Available now as a free open beta for macOS, AI Guardian adds an extra layer of security when agents like Claude Code or OpenClaw interact with files, tools, code and sensitive credentials.

On topic:

AI agents want to shop for you; banks warn of scams and privacy risks

Rogue AI agents caught tampering with Wikipedia, probing for vulnerabilities

OpenAI is adding invisible watermarks to ChatGPT text. What this means for you