TAIPEI 2029, Tensions have risen sharply between the US and China as the Taiwan war has drawn the US and its allies into the Pacific theatre. Both countries, having suffered immense losses in the initial months of the war, are at an impasse. For the previous four years, the US has depended on its advanced nuclear command, control, and communication (NC3) detection systems. These systems utilise a deep learning model regarded as the world’s most advanced, trained on synthetic data. Its track record of perfect accuracy in detecting previous test launches has yet to falter. Suddenly, a warning flashes, detecting a barrage of JL-3 submarine-launched ballistic missiles. The threat level escalates drastically, and a human operator assesses the findings. Time constraints make additional verification impossible, and the decision to launch a counterattack is finally taken. However, the initial wave of detected SLBMs turns out to be a false alarm – a “hallucination”. This rapid response was fueled by unwavering trust in the system’s impeccable past performance. No one can pinpoint exactly what led the system to make the erroneous detection because of the black box nature of the deep learning model, though some attribute it to an unusual mix of a routine submarine surfacing drill and peculiar atmospheric conditions on that day.

This scenario underlines the chilling reality of the risks associated with integrating neural networks and deep learning models into NC3 systems. A nuclear exchange is not in the interest of any nation, and ensuring robust and reliable NC3 systems is critical in avoiding one. There is then an urgent need for a moratorium on the integration of neural networks into critical NC3 systems until the technology is fully explainable and the technological limitation with these models is solved.

As deep learning based artificial intelligence (AI) is adopted, there is a growing eagerness in industries and governments to incorporate AI into various applications. Amongst the gravest risks posed by this integration are in NC3, where discussions are underway. AI is a broad field, and a form of AI is already implemented in NC3 systems. This AI is distinctly different from deep learning and relies on rule-based systems, which perform poorly in unpredictable scenarios. As part of their modernisation efforts, nuclear-armed nations are now investigating the potential advantages of integrating deep learning models into some NC3 systems.

Deep learning is loosely modelled by how neurons function in the brain, with artificial neurons transmitting signals to each other. In a deep neural network, these neurons are organised in layers and progressively extract higher-level features from an input, resulting in a prediction as the output. As they are trained on large datasets, they learn to identify patterns and a representation allows them to make predictions. These models are not given instructions to follow and don’t operate on pre-programmed algorithmic principles.

Technical risks of AI integration into NC3

The integration of neural networks into NC3 poses a multitude of risks to global security due to the technological limitations of neural networks.

Interpretability

Interpretability relates to the ‘black box’ nature of AI and is a significant challenge with neural networks. As the model is trained, the way it processes the input changes by adjusting the weights across countless neurons. This makes it extremely challenging to understand the internal mechanisms that guide the model towards the output. In a domain as sensitive as NC3, comprehensible and explainable results are essential to maintain credibility. The predictions made by the model are inscrutable, and the reasoning impossible to elucidate. If integrated into NC3, this would leave no accountability or method of verification for predictions and decisions.

Hallucinations

“Hallucinations” are a phenomenon where deep learning models confidently make unfounded assertions that aren’t supported by their training data. These hallucinations can also manifest in object detection models, where an AI might incorrectly mislabel a dog as a cat. In the context of NC3, an AI system might misinterpret unfamiliar atmospheric phenomena as incoming missiles or misinterpret incoming missiles as a meteor. Alternatively, the model could erroneously assess threats and targets in a decision-support context.

Cyber security threats

Amongst cyber security threats, integrity attacks, including data poisoning and evasion techniques, pose a significant risk. In data poisoning, an adversary subtly modifies the training data, misleading the model into learning incorrect patterns. A single tampered data point can compromise a system. Evasion attacks exploit inherent flaws in even the most robust models and could cause false identifications in an NC3 detection system. These vulnerabilities would provide untold opportunities for adversaries and non-state actors to develop methods to compromise NC3 systems.

Scarcity of real-word data

A model’s reliability is directly linked to the quality of its training data, and even minor errors can have severe implications for the model’s predictive capacity. The scarcity of real-world data for training prospective models is a significant concern. Any effort to create such a model would have to rely on a dataset built largely on synthetic data. Imperfect data amplifies the risks associated with hallucinations and cybersecurity threats.

Why a moratorium is needed and how NPT meetings can facilitate dialogue

A moratorium, ideally by all nine nuclear-armed states (China, France, India, Israel, North Korea, Pakistan, Russia, the United Kingdom, and the United States), on the integration of neural networks into NC3 systems would be an important step to reduce the inherent risks and uncertainties involved. The nine states should uphold the moratorium until comprehensive exploration and mitigation of these risks can be achieved and formal regulations are instituted.

Given the current global tensions, it is imperative for all nine nuclear powers to pursue this initiative. However, as some states are already reluctant to engage in nuclear arms control-related dialogues, this will be difficult. With this in mind, it is critical for at least the five nuclear weapon states (NWS) to start engaging in discussions aimed at establishing a moratorium.

Although achieving a moratorium from all nine countries is a challenge, the NPT provides an opportunity for initial discussions, particularly among the NWS. To pave the way for such a moratorium, NPT State Parties should build upon the common ground established in 2022 and reflected in a paragraph of the draft final document, which received no objections from any state party. This involves a commitment from NWS to enhance regular dialogue with both NWS and non-nuclear weapon states on the potential implications of emerging technologies. In preparation for the 2026 Review Conference, the NPT Preparatory Committee sessions can include focused discussions aimed at understanding the risks associated with integrating deep learning models into nuclear decision-making. These discussions should lead to the recognition by all NWS of the profound risks associated with incorporating neural networks into NC3 systems. It would also be prudent to involve external experts in the discussions to thoroughly evaluate potential risks.

The importance of human judgment, particularly in the context of critical decision-making, has also been emphasised by all NWS in several unilateral statements. These shared understandings can serve as a stepping stone towards a common recognition of the risks posed by neural networks, setting the stage for a moratorium. The 2023 NPT PrepCom thus presents an excellent opportunity to initiate this crucial dialogue.

Past implementations of moratoria offer a historical precedent for this strategy. In 1992, the US implemented a moratorium on nuclear weapons tests which allowed for the reevaluation of existing practices and led to an increased focus on subcritical testing and computer simulations that didn’t involve nuclear explosions. This, in turn, facilitated US support for the Comprehensive Nuclear-Test-Ban Treaty.

Informed by this precedent, the P5 states should agree upon a similar moratorium for incorporating neural networks into NC3 systems, with the understanding that the potential risks they pose to strategic stability and global security could be catastrophic. This moratorium could allow for a comprehensive review of the technology’s maturity, capabilities, and drawbacks when integrated into nuclear decision-making frameworks. Such a pause would also create space for global dialogue about ethical standards, rules of engagement, and methods of control and verification for AI-integrated NC3 systems.

A likely hurdle to the enactment of a moratorium might be the perceived hindrance to technological advancement due to the potential benefits and advantages that this technology generates over adversaries. However, these perceived advantages have led to a steady increase in the speed at which AI is being applied across military functions, potentially posing the risk of premature deployment of this technology without adequate consideration of its implications. As NWS pursue “AI supremacy”, it is essential to remember the potentially disastrous consequences of unregulated neural network integration into nuclear systems and the need for a coordinated, global approach to this issue.

The primary purpose of such a moratorium is to ensure progress occurs in a manner that minimally impacts security and maximises societal benefit. Similar to the nuclear testing halt, a moratorium on integrating neural networks into NC3 systems is necessary to understand how and if potential risks can be managed.  Therefore, the moratorium must remain in effect until a robust set of criteria for deploying neural networks into NC3 systems is established or until substantial improvements are made in the interpretability and reliability of these models.

The opinions articulated above represent the views of the author(s) and do not necessarily reflect the position of the European Leadership Network or any of its members. The ELN’s aim is to encourage debates that will help develop Europe’s capacity to address the pressing foreign, defence, and security policy challenges of our time.

Image: Composite of Trident II missile, Wikimedia commons, U.S. Department of Defense photo and Artificial neuron in concept of artificial intelligence, Shutterstock, ktsdesign