Chapter 15. Explainable AI (XAI)

Explainable AI (XAI) July 3, 1988, Persian Gulf. The tragedy began in the Combat Information Center (CIC) of the U.S. Navy's Aegis cruiser USS Vincennes. An unidentified track appeared on the radar screen. The captain and crew were nervous. This is because military tensions with Iran were extremely high at the time. The computer system classified the track as an Iranian Air Force F-14 fighter jet. The captain ordered the missile launch. But it wasn't an F-14. Iran Air Flight 655 was a commercial airliner. All 290 passengers and crew died.

Later investigation revealed that the computer system displayed incorrect information and, in a state of extreme tension, the crew believed it. Why did this tragedy happen? The system at the time just said, “That’s an F-14.” They did not explain why they decided it was the F-14, how confident they were, or whether there were any other possibilities. The crew had no way to verify what the machine was saying. This is the ‘black box’ problem. A black box is a black box that cannot be seen inside. When you put in an input, you get an output, but you don't know what happens in between.

The same goes for modern deep learning artificial intelligence. Millions of parameters are complexly intertwined, and even developers are often unable to explain exactly why a conclusion was reached. For combat pilots, trust means life. If my wingman suddenly breaks out of formation and makes a sharp turn, I will grab the radio and yell. “Why are you doing this!” At that time, if the wingman responds, “We have confirmed a SAM (Surface-to-Air Missile) launch at 3 o’clock! We are evading!”, I immediately understand and I also enter evasive maneuvers. Because I explained why. But what if the wingman is an artificial intelligence drone?

The guy suddenly rushes into the middle of the enemy lines. I don't see anything on my radar. I don't know if this is a genius tactic, a system error, or if I've been hacked. At that moment I panic. Incomprehensible behavior is indistinguishable from treason. In 2020, in DARPA's Alpha Dog Fight Trial, Heron Systems' artificial intelligence 'Falco' completely defeated the human pilot 'Banger' 5 to 0. The results were shocking. But it was also strange. Falco performed maneuvers that no human pilot would ever perform. He used the 'head-on' tactic of charging head-on toward the enemy plane and firing the machine cannon dozens of times per second.

I shook the control stick slightly. Banger said after the fight: “That guy’s shooting skills are unbelievably accurate, but you can’t predict why he’s doing that maneuver or what he’s going to do next.” In a simulation, even if the artificial intelligence does something strange and crashes, you can just press the reset button. But on the actual battlefield? What if an artificial intelligence drone flying next to me suddenly launches a missile toward a civilian village? How do you know if it's hitting an enemy's camouflage point or if it's malfunctioning? XAI (eXplainable AI, explainable artificial intelligence) emerged to solve this problem.

This is a technology that explains “why” artificial intelligence made that decision in words that humans can understand. DARPA has had an XAI program in operation since 2016. Their goals are clear. “Create explainable models while maintaining high performance, and enable humans to understand, appropriately trust, and effectively manage AI partners.” Easier said than done, but difficult to realize. If the existing AI simply said, “This is an enemy tank. 97% probability,” XAI would say: “This is a T-90 tank. The basis for judgment is as follows. First, the shape of the turret is 95% consistent with the T-90’s database.

Second, the engine heat distribution captured by the infrared sensor shows the characteristics of a diesel engine. Third, the formation of the escort vehicles deployed around matches the Russian armored forces doctrine.” Only with this explanation can pilots verify the artificial intelligence's judgment. If the AI ​​identifies a target for the wrong reason (e.g. because of the shape of a tree's shadow), the pilot may cancel the attack, saying "That stupid machine was wrong again." In April 2025, the U.S. Air Force released 'Doctrinal Document on Artificial Intelligence (AFDN 25-1)'.

The document states that military AI development should use “transparent and explainable algorithms” and conduct “regular audits and evaluations.” Military applications can only be built if data is accessible and understandable. The concept of trust calibration is also important. This means that you should not trust artificial intelligence too much or too little. If you believe too much, the Vincennes tragedy will repeat itself. If you don't trust it too much, you can turn off or ignore artificial intelligence and not use it. Artificial intelligence must be honest about its level of certainty.

The human response should be different when you say, “This target has a 99% chance of being an enemy,” and when you say, “This target has a 60% chance of being an enemy. Identification is uncertain.”

French defense company Thales has launched the slogan ‘TrUE AI’. The goal is to create artificial intelligence that is transparent, understandable, and ethical. By making the decision-making process of artificial intelligence traceable, they are developing technology that can backtrack the thinking process of artificial intelligence, like analyzing an airplane's black box when an accident occurs. The MQ-28 Ghost Bat unmanned aerial vehicle, which the Australian Air Force is developing with Boeing, also focuses on this problem.

When a manned fighter pilot entrusts a mission to his wingman, the Ghost Bat, the pilot must be able to see at a glance the status of the unmanned aircraft and whether the commands have been properly understood. We need an interface that turns complex data into intuitive pictures. In May 2024, U.S. Air Force Secretary Frank Kendall personally boarded the X-62A VISTA fighter jet piloted by artificial intelligence. Secretary Kendall was in the backseat while the plane performed dogfighting maneuvers at speeds exceeding 550 miles per hour (approximately 880 kilometers per hour).

After the flight, he said, “I think we can leave the decision to fire weapons to an artificial intelligence.” But this is a trust that only became possible after thousands of hours of simulations and testing, and after artificial intelligence was able to explain why it makes the maneuvers it does. XAI also has limitations. Additional computations are required to generate the description. When fighters have to make millisecond decisions, they can't afford to waste time creating explanations. Also, explanations that are too complicated will confuse the pilot.

“The curvature of the turret shape is 0.73 and the standard deviation of the heat distribution is 2.4” makes sense only to engineers. The pilot needs to say "That's a tank, shoot!" There is also the threat of adversarial attacks. What if the enemy knows the weaknesses of our artificial intelligence? For example, if we notice that our AI is highly dependent on the turret shape, we can cover the tank with camouflage netting to distort the turret outline. You can also fool artificial intelligence by adding subtle patterns to images that are invisible to the human eye.

If XAI discloses the basis for artificial intelligence's judgments, it could mean informing the enemy of our weaknesses. Ethical issues also remain. Just because artificial intelligence provides explanations does not mean it is exempt from liability. If the artificial intelligence explains that “the target’s heat signature pattern matches 94% of an enemy fighter jet,” but in reality it was a civilian airliner, who is responsible? Are you a programmer who wrote code? Was it the pilot who pressed the fire button? Or is it the commander who decided to deploy artificial intelligence?

Both the International Committee of the Red Cross (ICRC) and the U.S. Department of Defense identify explainability as a key element in the development and use of autonomous weapons systems. The principle of “Meaningful Human Control” is emphasized. But will humans have time to intervene in the split-second of hypersonic missiles and hundreds of drones? Probably not.

At that time, we may have to delegate authority to artificial intelligence, saying, “You take care of it.” So XAI is not just about answering real-time questions like “Explain why now?” Before deploying artificial intelligence, we test it in thousands of scenarios and verify in advance how it makes decisions in certain situations. The goal is to enable artificial intelligence to honestly say, “This situation is beyond the scope of what I have learned. You be the judge, human.” As a fighter pilot, I love machines.

When the F-16's engines roar behind me and the fly-by-wire system accurately translates the slightest manipulation at my fingertips, I feel one with the machine. But that's because I understand the machine. Even if the engine sound is slightly strange, you will notice it. If the control's response is different from usual, you will feel it immediately. The same should be true for artificial intelligence. When we understand the thoughts of artificial intelligence and when it can explain its thoughts to us, we become a true 'team'.

At that time, I will be willing to let go of the control stick and take command while looking at the tactical screen displayed by artificial intelligence. Intelligence without explanation is indistinguishable from madness on the battlefield. You need a comrade in arms who can have your back, not a ghost in a black box. XAI is our effort to create those comrades.

Part 4. The Era of Loyal Wingmen