Constraining Malicious AGI
Exploring the Capacities of States and Non-State Actors to Develop and Use Artificial General Intelligence for Harm, and the Legal Barriers That May Limit Them
Author’s Note: This article introduces a artificial intelligence policy research question I intend to develop over the coming months: to what extent do states and non-state actors differ in their capacities to create and use artificial general intelligence for malicious purposes, and how effectively can current legal regimes constrain them? This is an increasingly important question to consider, but the current discourse unfortunately inadequately addresses it. My coming publications will partially fill this lacuna.
Introduction
As the capabilities of artificial intelligence (AI) systems continue to advance, the prospect of artificial general intelligence (AGI) has transitioned from speculative possibility to plausible medium-term outcome. This transition invites a reconsideration of how the global system might constrain the malicious use of AGI—not merely in the abstract, but across the concrete capacities of actors who may seek to develop and deploy it. While international discourse often emphasizes state behavior, non-state actors—from research collectives to organized illicit networks—may also possess the capabilities necessary for AGI creation and use. Understanding these actors’ comparative capacities, the legal structures constraining them, and the gaps therein, is critical to effective governance in the years ahead.
Background: Artificial General Intelligence
Artificial intelligence refers to the field of creating machines capable of solving problems that require reasoning. Since the mid-twentieth century, this field has undergone several paradigm shifts, with the multilayered perceptron marking an early milestone. The modern era of AI began in earnest with the 2012 demonstration by Krizhevsky et al. that increasing model size yields large performance gains, inaugurating the era of scaled deep learning.
AGI refers to systems capable of solving nearly all problems that humans can solve using reasoning. In contrast to narrow AI—designed for specialized tasks such as image recognition or translation—AGI is intended to perform across domains and adapt to new problems. Recent progress toward AGI is largely driven by large language models (LLMs), which integrate symbolic reasoning, memory, and latent concept formation in a single architecture. These models are trained on massive corpora using transformer architectures and have shown generalization capacity beyond initial expectations.
Developing AGI entails challenges across four domains: scientific (developing new architectures and optimization techniques), engineering (operationalizing and scaling frontier techniques), data (curating, processing, and aligning high-quality training sets), and compute (securing the hardware and energy necessary for training large models). Scientific progress is largely confined to elite research communities. Engineering implementation is more broadly accessible but still requires deep technical skill. Data and compute are limited by institutional capacity.
In the medium-term future—defined here as the next three to seven years—AGI systems are likely to first solve all closed-form expert-level problems (e.g., those in the Humanity’s Last Exam benchmark), then extend to highly structured intellectual labor (e.g., evaluating weapons design feasibility), and finally encroach upon unstructured problem-solving domains (e.g., developing operational plans for unconventional biological attacks). These capacities will not emerge uniformly, but rather across a distributed ecosystem of actors with differing capabilities and constraints.
Background: Political Science and International Law
What is a state, and what is its capacity?
Under the Montevideo Convention (1933), a state is defined as a political entity with a defined territory, permanent population, governing structure, and the capacity to enter into relations with other states. Jurisprudential debate remains on whether this capacity requires institutional potential or actual functional engagement. Although the Convention is binding only on its American signatories, it is often treated as a codification of customary international law.
Alternative conceptions include the Weberian view, where the state holds a monopoly on legitimate violence, and the recognition-based positivist view, where statehood derives from international recognition. The concept of the state must also be distinguished from that of a polity—a broader term encompassing any internally governed political community. The administrative, fiscal, and military capacity associated with modern states has expanded considerably since the eighteenth and nineteenth centuries, when centralization and institutional consolidation became core to statecraft.
What is a non-state actor, and what is its capacity?
Non-state actors include all entities operating outside formal state institutions. These range from multinational corporations to criminal networks, religious movements, philanthropic foundations, and research collectives. Importantly, non-state actors are not inherently benevolent or malicious; Médecins Sans Frontières, for example, is widely regarded as a highly benevolent non-state actor.
Some non-state actors operate with substantial structural coherence and strategic sophistication. While they typically lack sovereign legal status, treaty access, or diplomatic protections (e.g., immunity), they may possess state-like capacities such as territorial control, coercion, and infrastructure. Their ability to develop AGI depends on access to compute, data, engineering talent, scientific insight, and their ability to evade regulatory oversight.
Comparative capacity analysis
States and non-state actors share several core capacities: the ability to organize labor, allocate resources, and pursue high-risk technical projects. However, states derive these from sovereignty, taxation, recognition, and institutional permanence; non-state actors rely on market position, ideological coherence, or network dynamics. States enjoy superior access to scale-dependent assets—e.g., national labs, classified datasets, and strategic infrastructure—but are often hindered by bureaucratic inertia. Non-state actors can iterate more rapidly and evade scrutiny, particularly if operating transnationally.
Legal exposure also differs: states may be shielded by sovereign immunity or enforcement asymmetries, while non-state actors face more direct coercive constraints—but also benefit from attributional ambiguity. In practice, state initiatives often depend on non-state contractors, blurring distinctions and creating hybrid dependencies.
How are their respective actions externally regulated?
International law lacks a centralized enforcement mechanism. Instead, compliance is encouraged through reputational, reciprocal, and institutional means—treaties, soft law, judicial forums, and multilateral sanctions. The Westphalian system used today, centered on the principle of anarchy—the absence of any supranational authority above states, and the presumption that no state holds formal legal authority over another—is similar in this way to other systems of international relations, such as the Sinocentric tributary system and the Islamic jurisprudential siyar, in that all three systems lacked centralized enforcement.
States are bound by treaties and customary norms, but enforcement is politically contingent. Non-state actors are regulated indirectly—primarily through state actions (e.g., sanctions, criminalization). AGI-related governance will likely exacerbate these structural enforcement gaps, especially given problems of jurisdiction, attribution, and multi-actor diffusion.
Literature Review: International Constraints on Malicious AGI Development and Use
International legal and normative frameworks aimed at constraining the malicious development and use of artificial general intelligence (AGI) are presently limited in both scope and enforceability. While efforts to govern artificial intelligence more broadly have accelerated in recent years, few existing instruments directly address the risks posed by general-purpose reasoning systems. This section reviews the primary international treaties, non-binding norms, and emerging institutional mechanisms intended to regulate harmful AGI use by states and non-state actors. Emphasis is placed on both the formal architecture of these regimes and their practical limitations in a world where AGI development is rapidly diffusing.
Binding Instruments and Legal Principles
The most consequential legally binding instrument to date is the Council of Europe’s Framework Convention on Artificial Intelligence, Human Rights, Democracy and the Rule of Law (2024). This treaty obliges signatories to ensure that AI development and deployment, including across the full system lifecycle, conforms to human rights obligations and democratic norms. Its jurisdictional reach remains regionally anchored but open to global accession. Crucially, it exempts national security and defense domains—precisely those where malicious AGI use may be most likely to arise.
Other binding legal sources derive from existing international humanitarian and criminal law. The Geneva Conventions and their Additional Protocols already govern state conduct in armed conflict, including the deployment of new technologies. Article 36 of Additional Protocol I requires legal review of novel weapons systems, a mandate applicable to AGI-enabled military tools. Nevertheless, compliance is self-administered, and enforcement is rare. Proposals for an AGI-specific arms control treaty remain speculative, though the UN General Assembly’s 2024 resolution calling for negotiations on lethal autonomous weapons (LAWS) marks a notable development. While not binding, this resolution signals growing consensus on the need for preemptive restriction of AI systems capable of causing mass harm.
In the realm of international criminal law, no specific offense of “malicious AGI development” exists. However, state or non-state actors who deploy AGI in ways that result in mass atrocity or systemic rights violations could, in principle, be held liable under existing war crimes or crimes against humanity statutes. Yet such avenues are reactive, depend on attribution, and lack preventive force.
Soft Law and Normative Frameworks
Non-binding norms have proliferated in the absence of comprehensive treaty law. UNESCO’s Recommendation on the Ethics of Artificial Intelligence (2021), adopted unanimously by 193 member states, articulates global principles around transparency, accountability, and non-maleficence. Similarly, the OECD AI Principles (2019), now endorsed by over 40 countries, emphasize human-centered design, safety, and robustness. These frameworks implicitly apply to AGI, although neither uses the term directly. Their persuasive authority lies in norm diffusion, peer review, and reputational pressure, rather than coercive sanction.
Multilateral declarations—such as the 2023 Bletchley Park Declaration on AI safety—extend these efforts into the frontier-AI domain. While voluntary, such documents increasingly reflect consensus that advanced AI systems, including potential AGI, require international coordination and risk mitigation strategies. Proposals from the UN Secretary-General’s High-Level Advisory Body on AI suggest the eventual emergence of a distributed global governance framework, though institutional design remains underdefined.
Efforts to regulate lethal autonomous weapons within the Convention on Certain Conventional Weapons (CCW) process have similarly produced guiding principles affirming the necessity of human control over use-of-force decisions. However, treaty negotiations have stalled amid geopolitical disagreement, and the potential for AGI-enhanced LAWS development remains unconstrained.
Enforceability and Structural Gaps
International legal enforceability remains weak across these regimes. Binding treaties are implemented domestically, with uneven fidelity. Non-binding norms rely on reputational incentives and voluntary compliance. There is no centralized AI oversight body equivalent to the International Atomic Energy Agency, and existing mechanisms—such as export control regimes or domestic regulatory frameworks—function in a fragmented and uncoordinated fashion.
Attribution poses a critical barrier to enforcement. As AGI systems become more autonomous and development more transnational, assigning responsibility for harms becomes increasingly complex. This is particularly true for non-state actors operating with plausible deniability or in permissive jurisdictions. Although international law holds states responsible for activities emanating from their territory, enforcement depends on proof of knowledge and effective control—both difficult to establish in distributed AGI projects.
Moreover, jurisdictional diffusion undermines regulatory coherence. An AGI trained across multiple cloud providers in different countries, developed by a decentralized research collective, and deployed via anonymized infrastructure, may elude any single legal regime. Existing doctrines of universal jurisdiction or state responsibility offer limited recourse in such scenarios.
Prospects for Future Constraint
While existing instruments fall short of comprehensive AGI governance, they lay normative groundwork for future regimes. A precautionary principle—requiring risk assessment and mitigation prior to deployment—has begun to take shape across ethical guidelines and summit declarations. Discussions around international safety standards, model evaluation protocols, and cross-border incident response mechanisms suggest emerging pathways for formalization.
Nonetheless, the current international legal order lacks the institutional density, technical capacity, and political alignment necessary to effectively constrain malicious AGI development and use. Any future regime will need to address enforcement asymmetries, enhance attribution mechanisms, and develop institutional architectures capable of monitoring high-risk AGI trajectories. Without such evolution, the gap between technological capability and legal constraint will only widen.
This literature review thus identifies both the partial scaffolding and profound insufficiency of current international frameworks for constraining malicious AGI. The burden of anticipatory governance remains unmet. Recognizing this shortfall is a necessary precondition for building effective future constraints.
Conclusion
As artificial general intelligence transitions from theoretical construct to foreseeable reality, the question of who may develop and wield such systems—and under what constraints—gains strategic urgency. This article has examined the comparative capacities of states and non-state actors to maliciously create and use AGI, as well as the external legal frameworks that currently exist to restrain such actions. States possess clear advantages in scale, infrastructure, and regulatory power, while non-state actors benefit from agility, transnational reach, and opacity. Both actor types face significant, though asymmetrical, constraints imposed by international law, norms, and treaty regimes.
Yet these constraints remain uneven, fragmented, and frequently reactive. The existing patchwork of international instruments—while foundational—does not yet rise to the level of a comprehensive AGI governance regime. Enforcement asymmetries, attribution difficulties, and normative ambiguities present systemic vulnerabilities that could be exploited by actors pursuing harmful ends. Bridging these gaps will require deliberate institutional innovation: the development of actor-specific safeguards, the refinement of legal attribution doctrines, and potentially the establishment of new treaty frameworks or monitoring institutions.
A clear understanding of which actors are most capable of—and most likely to—pursue malicious AGI trajectories is a prerequisite to meaningful intervention. Future work must build on this foundation, identifying specific technical thresholds, institutional failure modes, and enforcement architectures that can mitigate cross-actor risk. The regulatory space around AGI remains under-structured, but it need not remain so. By anticipating the shape of its challenges and the actors involved, policy can begin to move at the speed of the technologies it seeks to govern.