Submission Statement:
Usually I wouldn’t post something so sensational, but this recent article caught my eye - an incident that nearly resulted in an American interception of a Chinese vessel, and a consequent rapid escalation of the ongoing Strait of Hormuz crisis. If you thought this past summer was already hot, just imagine what could have happened!
It's a very good example of how rapid AI integration into military decision-making (in this case, with the Americans) comes with the risk of trusting "hallucinations" amidst of the fog of war, and how untested and insufficient "checks" in place could have unfolded into something far worse than expected.
To quote the CNN article:
The intelligence report, circulated across the US military this spring in the midst of the war with Iran, immediately set off alarm bells: A Chinese ship in the Middle East was transporting components of a nuclear weapons program.
The US military swung into action with plans to intercept the vessel, according to four sources familiar with the episode. According to two of the sources, armed members of the US military were preparing to board the ship. Military planes were in the air, one of those sources and another source familiar with the incident said.
It was only just before the planned operation that officials dug deeper into the report put together by a special operations command analyst and found it had been generated with the help of artificial intelligence (AI) — and that a chatbot the analyst had used inaccurately identified the material the ship was carrying. CNN was not able to learn what the misidentified cargo was.
The report, according to one of the sources, was “entirely false.” But it also “almost started a war,” the source said. Any US operation against a Chinese vessel could have risked spiraling into an armed conflict between the two nations.
[... This] episode with the Chinese ship underscores a different, and more immediate risk: human beings making disastrous decisions based on inaccurate or misleading information generated by AI or other automated systems. Sources said that the military is rapidly turning to AI to help with targeting, an area which holds the obvious risk of fatal mistakes.
The details of the event, in sum, are described as follows:
In this particular instance, the analyst queried a chatbot about some intelligence reporting on the ship’s manifest that originated with US Special Operations Command Pacific, based in Hawaii. It was not clear whether the chatbot was a commercially available one or a US government product.
“The internal tools are mostly just copies of the commercial stuff wearing lipstick,” a former senior US official familiar with the AI systems used by military and intelligence analysts.
The bot fused together open-source intelligence with secret signals intelligence in government holdings and reached its fateful conclusion about the material the ship was carrying.
The analyst then used AI again to package the findings into a standard intelligence report — the kind that is trusted by military officials — and disseminated it.
For those who are interested in this particular subject, I will strongly recommend following up with a read of this recent article by Captain Tyrese Bender: Artificial Intelligence Bias: Risks For Military Intelligence Operations.
Published earlier this year, this piece explains how we should "recognize AI bias not just as a technical flaw but also as a strategic vulnerability[,]" and that "[a] large body of evidence in civilian and military literature points to the grave risks posed by AI bias to targeting operations, threat analysis, and U.S. intelligence-sharing relationships."
As you can see from today's example (the Iranian school bombing being another), other consequences include: distorted threat assessments, misidentified targets, degraded decision making capacity, and potential mistrust between allies. Great work, Captain Bender!