About the author(s):
Yiokasti Mouratidi is a Doctoral Candidate in the Department of International and Operational Law at the Swedish Defence University, jointly with the Faculty of Law, Economics and Governance at Utrecht University. Her PhD focuses on compliance with the law of armed conflict, particularly through the lens of technological developments. Previously, Yiokasti obtained her LLM in Public International Law from Utrecht University (2022) and her LLB in Law with European Law from the University of Nottingham (2019). Yiokasti has a strong interest in international humanitarian law and international security law.
This post forms part of phase two of the Beyond Compliance Symposium: How to Prevent Harm and Need in Conflict, hosted by the Armed Groups and International Law blog. The introductory post can be found here. The symposium invites reflection on the conceptualisation of negative everyday lived experiences of armed conflict, and legal and extra-legal strategies that can effectively address both civilian harm and humanitarian need.
Introduction
It has been said that increasing integration of AI capabilities in the military domain, particularly machine-learning based AI, has led to discourse built upon a binary conception of “better” or “worse” compliance with international humanitarian law (IHL), particularly in the context of the conduct of hostilities. On the one hand, military AI enables data-processing at a higher speed and scale, arguably leading to greater situational awareness and exceeding human cognition in hypothesising different potential decisions and respective outcomes. On the other hand, risks of bias can lead to misidentification of targets, while increased speed in target identification can enable “mass production targeting” without due consideration to legal obligations. Beyond IHL, these technologies raise broader concerns, including the harm of subjecting a population to constant monitoring and mass surveillance, and ethically grounded arguments against the use of technology to harm humans and risk leading to an apocalyptic future.
Ordinarily, IHL compliance is determined on a pendulum between deference to military personnel decision-making and ex-post facto scrutiny by state and non-state “external influences”, including international courts and tribunals, non-governmental organisations and UN entities. Within this context, the challenge posed by different potential interpretations of IHL by different actors is part of the BCC’s rationale for going beyond compliance. Now, military AI capabilities add a new dimension to this challenge, given that they entail critical decisions prior to their actual use, during their design and development by engineers (see also here and here). This corresponds to the rationale of legal informatics, explained further below, at the core of which is ex-ante problem identification and solution.
Drawing from legal informatics perspectives on the relationship between law and technology, this blog post outlines the ex-ante design decisions at play when it comes to military AI capabilities, before the question of IHL compliance by end users even arises. In turn, it considers how these decisions intersect to shape the IHL targeting framework. This analysis reflects a common thread throughout, namely the oftentimes circular relationship between lawmaking and discourse on compliance: “[m]uch of the arguments about interpretation is aimed at compliance, but the line between compliance with existing law and creating new law is blurry” (p 8). With this in mind, this post presents a framework against which to conduct further research into the role of the private technology sector in designing and providing these capabilities, particularly as an actor and process that can be viewed through the lens of lawmaking (see here and here for recent publications on IHL and lawmaking/shaping, with the notable gap of this perspective). For present purposes, it thus comes within the BCC’s conceptualisation of private actors whose practices are not neatly captured by IHL, but are nonetheless part of the ecosystem contributing to harm and need in conflict settings.
Compliance in the age of algorithmic warfare: AI “tools” and “rules”
Legal informatics pertains to the bidirectional relationship between law and technology: technology itself can constitute a “tool” for the purposes of enforcing the law, while at the same time technology can be regulated through legal “rules”. The crucial aspect of this bidirectional relationship is the way in which it intersects, such that “only a combination of the two can lead to a full understanding [of the interplay of law and IT]” (p 35). As a result, there are three perspectives to consider: a. AI capabilities as regulatory tools themselves; b. as systems to be regulated; c. the intersection between the two. To elaborate on this further, I apply to the first two perspectives Prifti et al’s taxonomy of practices on “regulation by design” (RBD), based upon their review of academic literature on “by design” governance approaches, before turning to consider the third.
A. Military AI as a “tool” for IHL compliance
Under Prifti et al’s taxonomy, the notion of “compliance by design” is the first perspective to consider. Accordingly, software developers must adopt technical measures that implement legal requirements, thereby (automatically) enforcing the law. To achieve the goal of legality, hard or soft-coded requirements are the regulatory method, with designers as regulators and users as those being regulated.
When applied in the context of military AI, “compliance by design” would aim to embed existing IHL rules into code. Such efforts are apparent, for example, in how some of these products are being defended and advertised: a blog post by US tech company Palantir outlines how its AI-Decision Support Systems (AI-DSS) reinforce “critical military ethics and Law of War considerations” and “can help users understand the likely impact of any action across various categories. For instance, are there protected sites nearby or unexpected clustering of non-combatants? Are there alternative courses of action that would achieve comparably valuable military objectives but with reduced risk of civilian harm?” This can be understood through the lens of IHL targeting rules, particularly Arts 57(2)(a)(i) and 57(3) of Additional Protocol I 1977. In this sense, AI-DSS can be said to aim to “regulate” the “user” (military personnel) in relation to the IHL targeting framework, with IHL compliance as the “goal”.
To this end, some scholars have argued that the duty to respect and ensure respect with IHL under Common Article 1 of the Geneva Conventions entails ensuring that IHL standards are designed into algorithms, including by private sector developers, reflecting the state’s international obligations. Failing to do so, or procuring tools that are inherently incapable of entailing legal norms, would arguably incur state responsibility even before any physical harm materialises. Similarly, the “lawful by design” initiative by Article 36 Legal advocates for early-stage incorporation of IHL into the design and development of new weapons and systems by private industry.
Machine-learning, however, is shifting the paradigm from “law as code” (i.e. embedding black letter law into code), to “code as law” (i.e. self-learning and adaptive code responding to situations it was not explicitly programmed for, thereby dynamically evolving rules). While holding the promise of handling more complex tasks, machine-learning models generate outputs on the basis of assumptions which may not be available to humans and can thus “diverge from the wishes of those developing or deploying them” (p 5), even entailing “hallucinations” by identifying patterns or objects that do not exist. As a solution, some suggest an “iterative” approach to Article 36 legal reviews, by monitoring and auditing changes in performance on an ongoing basis (see here and here). The risks are nonetheless compounded by the “vagueness by design” of key IHL targeting provisions, which are ill-fitting to the outcome-based design of mathematical algorithms. This is why, for example, the ICRC considers it inappropriate to rely on AI for “open-ended” issues, such as whether there is a “belligerent nexus” for the purposes of determining whether someone is directly participating in hostilities.
Importantly, while such upstream efforts may, in principle, reduce the downstream risks of non-compliance, it is not “the system itself that must comply with the law, but the humans using it” (p 2). To this end, human operators will need to understand how such systems work, including their limitations, in order to comply with their legal obligations under IHL. Yet, while decisions remain the responsibility of humans, there is a risk of cognitive impacts including “automation bias”, the unquestioning trust towards a machine’s accuracy and reliability. Such risks bring to the fore the importance of military AI “rules”.
B. Military AI “rules” to be complied with
The perspective of AI “rules” falls under Prifti et al’s “optimisation by design” category, pertaining to the attainment of a particular standard by the technological system. Under their taxonomy of RBD practices, it is the system itself that is being regulated, rather than the end user, though there may be an impact on the user as a result. This remains compliance-oriented, but with what is “responsible” or “ethical” military AI, in IHL terms.
Indicatively, there are a number of terms that come under this umbrella concept within ongoing governance discourse by State and non-state actors, the exact parameters of which are subject to interchangeable language. AI should be “predictable” and “understandable”: it should carry out what is expected from it, and for intelligible/transparent reasons; concerns about bias across the lifecycle of design, development and deployment calls for “fairness” in AI, through the adoption of mitigating measures and inclusive processes to counter these risks of bias; the human-machine interaction which entails “distributed forms” of human and machine agency, and the resulting need to ensure “meaningful” human control and judgment, in light of the aforementioned risk of automation bias.
At present, factors falling under “responsible” military AI are not part of an international, binding regulatory framework, leaning instead into a soft law approach, and often elaborated domestically with further guidelines. Despite these elaborations by States and companies themselves (see here and here, for example), these principles continue to be indeterminate and subjective, which in turn makes “compliance” with them difficult to assess, as the “metrics chosen for measuring compliance with ethical principles will influence the character of that compliance” (p 189).
Moreover, a core criticism of focusing on “responsible” AI is that it enables actors “to be vague in their proposals for global governance and regulation. It potentially risks undermining efforts for the setting of binding legal norms…”. This emphasis further risks overshadowing power dynamics, as well as diverting attention from the actual end use and broader business models involved in this service provision, including venture capital investment which shapes markets based on “aggressive marketing and hyperbolic narratives” (p 8), and the “platformisation” of defence technologies which leads to dependencies and monopolies. In this regard, Lee et al outline how private corporations favour self-regulation in the form of “ethical AI”, as non-binding and unenforceable standards, placing them in a position of having “unique obligations and capacities to craft and enforce regulation”, while at the same time leading to “unclear obligations, limited options for accountability, and varied perceptions of AI regulation among both key stakeholders and the public” (p 416). This fragmentation leads to uncertainty about “who makes the rules and for how long” (p 190).
Against this background, there is a risk that the translation of responsible AI principles by private sector engineers into technical requirements becomes entrenched, binding future generations and raising legitimacy questions. For example, Bode and Huelss outline how in the context of air defence systems, which have entailed autonomous features since the 1960s, the design choices by European manufacturers over decades have had a “cumulative and incremental effect” on what counts as “appropriate” human control in the deployment of defensive air systems. Now, the same situation risks repeating in the context of offensive capabilities, thereby shaping the legal framework on the conduct of hostilities given the intersection of “rules” and “tools”.
C. At the intersection of military AI tools and rules: shaping IHL
The above perspectives of “rules” and “tools” are closely interrelated within discourse, since “lawfulness”, including vis-à-vis IHL, forms part of the factors for responsible AI within some conceptualisations. This approach, however, is arguably too simplistic, failing to account for how these two perspectives interact, shaping human cognition and behaviour and, by extension, stand to shape the legal framework on the conduct of hostilities.
As noted above, while AI “rules” target the system, there is nonetheless a resulting effect on the end user and thereby the implementation of IHL. Machine-learning based AI is trained and operates through the aggregation of data and pattern collation to generate an outcome based on probabilities. Within this, design choices as to the organising, filtering and presenting information to users may entail risks of distorted information and automation bias. Against the background of such risks, and coupled with increased speed in target identification, Dorsey and Bo analyse what this means for the duty to take “feasible” precautions in attack. In particular, the use of AI-DSS can shorten the target selection process, “shift[ing] key interpretative metrics” onto machines; in light of known error rates and “quantification logics”, this can lead to the normalisation of “higher thresholds of acceptable civilian harm, effectively leading to interpretation of a lower standard of care required under the principle of precautions” (p 27-28).
Such quantification logics of numeric thresholds and statistical models mark a shift from the traditionally qualitative assessments required in targeting. For example, in relation to proportionality assessments, which are inherently contextual and value-based judgments, there is a risk of normalising the displacement of discretionary legal reasoning and human agency, in favour of predictions based on probabilities calculated by algorithms. Similarly, Klonowska argues that the production of knowledge and recommendations resulting from algorithmic calculations are reshaping the standard of the “reasonable commander” in warfare, given reliance upon them.
Permeating the above is the challenge that data used to train AI systems may not be representative of dynamic and unpredictable contexts of battlefields, potentially leading to unreliable outputs and unknown risks. For example, during the 2021 conflict between Israel and Palestine, one Israeli military official reported that their platforms “go crazy with a lot of edge cases during wartime… We managed to update our systems 150 times in 10 days”. More recently, in reports on the use of Lavender in spring 2024, one Israeli operative noted that “the numbers changed all the time, because it depends on where you set the bar of what a Hamas operative is. There were times when a Hamas operative was defined more broadly…”. In light of this, Downey poses the question, “then who, or what, is ultimately defining the fateful distinction between a “combatant” and a “non-combatant?”” (p 11).
Concluding thoughts
The promises of RBD practices are a far cry from the paradigm of international law, which “provides a baseline but indeterminate structure, ‘deferring substantial resolution elsewhere’ and leaving room for interpretations in varied ways” (p 61). Not only do military AI capabilities not resolve long-standing challenges arising from this paradigm, but they give rise to further uncertainties and complexities, reflecting also how “disputes about the substance of the law are not infrequently also disputes over the making of the law” (p 3). This post has highlighted the importance of looking at the period of time before compliance, to outline the required design decisions which are invariably shaping legal norms and, by extension, the shaping of what compliance with the same means and entails.
Importantly, behind this “governance-by-design dystopia” is the role of the private technology sector, making design decisions that are framed as inevitable and apolitical. In practice, this is aptly leading to increasing attention to the private technology sector’s role in contributing to harm and need in armed conflict. In this context, going beyond compliance requires going beyond public spaces of negotiation, particularly to the sites and actors involved in designing these technologies.