Chapter 33. Who Is Responsible for an AI Mistaken Strike?

Who is responsible for AI errors? February 25, 1991, at the Dhahran base in Saudi Arabia, where the Gulf War was in full swing. Iraqi Scud missiles have arrived. The Patriot air defense system had to work. But it didn't work. It was a software bug. As the system's clock ran continuously for more than 100 hours, errors accumulated and it was unable to track the missile. The results were disastrous. 28 American soldiers were killed. Who was responsible for this accident? Raytheon, the company that created the Patriot system? Programmers who didn't find the bug? An operational unit that hasn't rebooted its system for over 100 hours?

Or the Department of Defense for not distributing software updates on time? Decades later, this question still has no clear answer. But this wasn't even an autonomous weapon yet. Humans were operating the system. What if autonomous lethal weapons (LAWS) decide to attack civilians? Who should be held responsible for that? This is the “Accountability Gap”. Current international law is designed based on human actions and intentions. Article 25 of the Rome Statute of the International Criminal Court (ICC) provides for individual criminal liability. Article 30 states that “intent and knowledge” are necessary to establish a crime.

But machines cannot have intentions. Algorithms can't go to jail. In December 2024, the ICC issued arrest warrants for Israeli Prime Minister Netanyahu and former Defense Minister Gallant. Among the charges are war crimes: intentional attacks on civilians. Israel has used AI targeting systems extensively in the Gaza Strip. This could be the first time the ICC directly addresses AI-related war crimes. But there are many legal obstacles. The “intent and knowledge” requirements of Article 30 of the Rome Statute are strict. In the 2014 Katanga case, the ICC ruled that dolus eventualis, that is, assuming a risk while foreseeing a harmful outcome,

It was confirmed that criminal liability cannot be held. This creates a critical loophole in the operation of autonomous weapons. When someone says “AI did that”, how can you prove their “intention”? Researcher Marta Bo took a deep dive into this issue. She warns that even if the unlawful consequences of deploying autonomous weapons are foreseeable, prosecution may become impossible if there are no identifiable actors with specific intent. The law is not keeping up with technology. So who is responsible? The first suspect is the commander.

In international law, there is a principle called “Command Responsibility.” If a commander knew or should have known of a subordinate's crime but did not prevent it, he or she is held liable. However, applying this principle to AI systems is difficult. Can we say that a commander “should have known” every decision made by a black box algorithm? The second suspect is the developer. Should the programmer who wrote the code be held responsible? However, deep learning-based AI is a black box. Even the developers themselves can't 100% explain why the AI ​​came to that conclusion.

A system that was perfect in millions of simulations can malfunction due to the unpredictable variables of the real battlefield. Is this intentional? Is it negligence? The third suspect is a defense company. Should the companies that manufactured and delivered the weapons be held liable? However, most defense contracts contain indemnification clauses. And defense companies are protected by the state. Current international law makes it very difficult to prosecute companies for war crimes. The fourth suspect... could it be the machine itself? This is impossible. You can't take AI to court.

There is no point in convicting the algorithm “guilty.” Ultimately, a situation arises where everyone is involved but no one is directly responsible. The 2024 UN General Assembly Resolution 79/239 is an important development. The resolution confirms that when AI is used in the military domain, international humanitarian law applies throughout its entire life cycle. It also added international criminal law to the applicable legal system. This highlights that personal criminal liability may apply in appropriate cases. There are efforts toward solutions.

First, the “Digital Audit Trail”. Like a black box on an airplane, it records every decision made by AI. Why did you identify that target as the enemy at that point? What data was it based on? What alternatives have you considered? This record is required so that accountability can be held after the fact. Second, strengthening national responsibility. The idea is to ensure that the country that operates the AI ​​weapon itself, rather than the individual, is responsible for compensation under international law. If countries are held accountable, governments will be more cautious about introducing AI weapons.

Third, mandatory weapons legal review. Article 36 of Additional Protocol I to the Geneva Conventions requires legal review of new weapons. For AI weapons, this review is even more important. It should be assessed whether the system can comply with the principles of distinction and proportionality of international humanitarian law. 2025, the International Committee of the Red Cross (ICRC) highlighted in its report to the Secretary-General of the United Nations. “It is not the AI ​​systems themselves that must comply with the law, but the humans who use them.” AI must remain a tool to aid human decision-making.

It should support humans, not replace them. As a pilot, I am prepared to take responsibility for my actions. If I accidentally attack a civilian, I will carry that burden with me for the rest of my life. You will be judged by a military court. That is the honor of a warrior. But what if I have to go to jail for something my AI wingman did? That's not fair. Conversely, what if no one is responsible for AI's mistakes? That is not justice either. Law is always slower than technology. But without law, only a jungle remains. Laws must exist even in war. This is so that we can remain human and not beasts.

No matter how advanced AI is, it is humans that bleed. The law must be the minimum courtesy to that blood.