A recent policy discussion at the Center for Air and Space Law examined a scenario that captures one of the most consequential questions emerging in modern warfare.
A state deploys an AI-enabled drone system designed to identify and strike military targets using real-time data and machine learning. The system has been extensively tested and consistently outperforms human operators. It produces fewer targeting errors and results in lower civilian casualty rates.
Despite that, the military requires a human operator to approve every strike.
During an operation, the AI system flags a vehicle as likely civilian. The human operator overrides that assessment and authorizes the strike. Civilians are killed.
That is the scenario. And it is doing a lot of work.
From a legal standpoint, there was broad agreement on where to begin. “International humanitarian law regulates outcomes, not who makes the decision.” It regulates whether the decision complies with distinction, proportionality, and the obligation to take feasible precautions. On those terms, the failure here was not technological. It was human. Or, as one participant put it, “The failure here was not technological. It was human.”
But that clarity quickly gave way to a deeper divide.
Some participants emphasized the limits of automation. Targeting decisions are not static or binary. Civilian objects can become lawful military objectives depending on use or purpose. Proportionality requires weighing competing values that resist quantification. “Targeting decisions are not binary. Context and judgment still matter.” From this perspective, human judgment is not just relevant. It is essential to applying IHL in real-world conditions.
Others challenged that instinct. If an AI system is demonstrably more accurate and consistently reduces civilian harm, then insisting on human control begins to look less like a safeguard and more like a source of risk. Several participants pointed to the obligation to take “all feasible precautions” and asked a harder question. “If a more reliable system exists, can a state justify not relying on it?” At some point, “refusing to use better tools may itself raise legal and ethical concerns.”
As the discussion made clear, responsibility does not disappear when AI is introduced. It expands.” Participants consistently identified the operator as directly responsible for the override decision, commanders as responsible for training and oversight, and the state as ultimately responsible under international law. More pointedly, some questioned whether system design itself, particularly allowing unrestricted human override, creates structural risk before any individual mistake occurs.
One of the most compelling threads in the discussion was the recognition of a deeper problem. Even when AI improves targeting accuracy, human interaction with the system can undermine its benefits. In this case, “the human in the loop was not a safeguard. It was the point of failure.”
That leads to the underlying tension the discussion kept circling. What is IHL actually trying to prioritize?
Human control supports accountability and moral agency. Minimizing civilian harm is the law’s central humanitarian objective. For a long time, those goals have been treated as aligned. This scenario suggests they may not always be. “Human control and minimizing civilian harm are not always aligned.”
Most participants resisted treating this as a binary choice. Instead, they pointed toward a more structured middle ground. Systems where AI plays a decisive role, but human oversight remains disciplined rather than absolute. Not a reflexive ability to override, but a requirement to justify doing so.
Because once a system is capable of reducing harm more effectively than human judgment, “choosing not to rely on it is no longer neutral.” It becomes part of the legal and ethical analysis.
Steve E. Bartz, Jared Blackburn, Brian S. Brewer, Julian Butler, Sunshine H. Eversull, Chase C. Harris, Christian C. Kobres, Mark M. Majors, Kerry A. Mawn
Leave a Reply