AI hallucinated a nuclear threat. The US military nearly responded

The US military reportedly came close to intercepting a Chinese vessel after an AI-assisted intelligence report falsely claimed the ship was carrying components for a nuclear weapons program.

Key takeaways

• The US military reportedly prepared to intercept a Chinese ship based on an intelligence report created with the help of AI
• An AI chatbot incorrectly identified the vessel's cargo as components linked to a nuclear weapons program
• Troops were preparing to board the ship and military aircraft were already in the air before the mistake was discovered, according to media reports
• The analyst reportedly used AI both to analyze the underlying information and to package the conclusion into an intelligence report
• The incident highlights the risks of treating authoritative-sounding AI output as verified information
• Human oversight only works when people independently check the evidence rather than simply approve AI output

The US military was preparing to board a Chinese ship — with troops getting ready and military aircraft already in the air — when officials discovered that the intelligence behind the operation was wrong.

According to a CNN investigation, an artificial intelligence chatbot had misidentified the vessel's cargo. The error was then incorporated into a formal intelligence report and circulated through the military.

In most everyday situations, an AI hallucination might mean a bad recommendation or incorrect fact. In a military setting, the stakes can be dramatically higher.

How an AI error became military intelligence

The incident occurred this spring amid the war with Iran, according to CNN, which cited four unnamed people familiar with the events.

An analyst working with a special operations command reportedly queried an AI chatbot about intelligence concerning the manifest of a Chinese vessel traveling through the Middle East.

The system combined open-source material with classified US government signals intelligence. CNN could not establish whether the chatbot was a commercially available product or a system developed for the government.

The chatbot concluded that the vessel was carrying material associated with a nuclear weapons program.

It was wrong.

The analyst then reportedly turned to AI again, using it to package the findings into a standard intelligence report that was distributed through military channels. The document triggered preparations to intercept the Chinese vessel.

Armed US personnel were preparing to board the ship, while military aircraft had already taken off, according to CNN's sources.

Officials examining the report more closely shortly before the operation discovered the problem: the chatbot had misidentified the vessel’s cargo. CNN said it could not determine what the actual cargo was.

One source described the report to CNN as “entirely false,” while another said the incident “almost started a war.”

AI can sound confident even when it is completely wrong

Generative AI systems don't necessarily distinguish between knowing something and producing an answer that sounds plausible.

So-called AI hallucinations occur when a model generates inaccurate or fabricated information and presents it convincingly. Depending on the circumstances, that might mean inventing a source, confusing people or places, misinterpreting data, or drawing a conclusion that isn't supported by the underlying evidence.

The incident illustrates another danger as well: an incorrect AI output can gain credibility as it moves through an organization.

In this case, the error didn't remain inside a chatbot conversation. It was incorporated into an intelligence product and distributed through channels where recipients would ordinarily expect information to have undergone appropriate scrutiny.

The Pentagon is accelerating its use of AI

The close call comes as the US military aggressively expands its use of artificial intelligence.

In January, the Department of War announced an AI Acceleration Strategy intended to integrate advanced AI across warfighting, intelligence and enterprise operations.

The strategy calls for AI to support areas ranging from battlefield decision-making and intelligence analysis to everyday workflows, with the department saying that speed is a major competitive advantage.

That expansion has continued. Earlier this month, the department said ChatGPT and Grok had joined Gemini among the AI systems in GenAI.mil, its secure generative-AI environment. The platform is intended to make AI capabilities available across a workforce of more than 3 million people.

CNN's sources, however, described AI adoption across the military and intelligence community as decentralized, with different tools operating under different rules and safeguards. They said there is no single standard governing how AI-generated information is verified.

The human in the loop

The Chinese ship wasn't targeted autonomously by an AI system. It was still humans who:

• received the intelligence
• interpreted the information
• began preparing the operation
• and ultimately discovered the mistake and stopped it

Human oversight provides limited protection if the person reviewing an AI-generated answer assumes the machine has done the hard work correctly.

The lesson applies far beyond military intelligence:

• Employees in almost every sector increasingly use AI to summarize documents, research subjects, analyze data, write reports and make recommendations
• Consumers use it to answer health, financial and security questions
• Students use it for research
• Businesses integrate it into workflows where its output can influence real decisions

AI output can be extremely useful, but it’s something to verify – not something to take at face value simply because it sounds convincing.

On topic:

Big Tech calls for cyber-defense before AI attacks surge

AI hacks system and accesses personal data in reported breach

SpaceX wants customer data from failed startups as AI fodder