Superintelligent AI: Should its development be stopped?

Approximate read time: 20 minutes

The House of Lords is scheduled to consider the following question for short debate on 29 January 2026:

Lord Hunt of Kings Heath (Labour) to ask His Majesty’s Government what plans they have to bring forward proposals for an international moratorium on the development of superintelligent AI.

1. Definitions and levels of AI

AI can be broadly defined as the ability of computer systems to perform tasks normally requiring human intelligence.^[1] AI has the potential to bring a range of benefits, but it also poses risks, such as those related to data privacy, biases, misinformation and cyber security. More details can be found in the House of Lords Library briefing ‘Artificial intelligence: Development, risks and regulation’ (18 July 2023).

Within this definition there are also distinctions between types of AI. It can range from so-called ‘narrow’ AI, designed to perform specific tasks, to what is known as ‘strong’ or ‘general’ AI with the capacity to learn and reason. The House of Commons Library has provided an overview of these different forms and other related definitions drawing upon research from Stanford University and other sources:

• Narrow AI is designed to perform a specific task (such as speech recognition), using information from specific datasets, and cannot adapt to perform another task. These are often tools that aim to assist, rather than replace, the work of humans.
• Artificial general intelligence (AGI—also referred to as ‘strong’ AI) is an AI system that can undertake any intellectual task/problem that a human can. AGI is a system that can reason, analyse and achieve a level of understanding that is on a par with humans; something that has yet to be achieved by AI. The US computer scientist Nils John Nilsson, for example, proposed that one way to test if a system had achieved AGI was if it could successfully learn the skills to perform the different jobs “ordinarily performed by humans”, from “knowledge work” (such as a library assistant) to “manual labour” (such as a roofer).
• Machine learning is a method that can be used to achieve narrow AI; it allows a system to learn and improve from examples, without all its instructions being explicitly programmed. It does this by finding patterns in large amounts of data, which it can then use to make predictions (for example what film or TV programme you might like to watch next on a streaming platform). The AI can then independently amend its algorithm based on the accuracy of its predictions.
• Deep learning is a type of machine learning whose design has been informed by the structure and function of the human brain and the way it transmits information. The application of deep learning can be seen in ‘foundation models’, of which ‘large language models’ (LLMs) such as ChatGPT, are one example. The term refers to those models that are trained on very large, unlabelled datasets and which can be adapted to do a wide range of tasks, despite not having been trained explicitly to do those tasks. In other words, the model can take information it has learnt about in one situation and apply it to another, different situation. Sometimes LLMs are refined or ‘fine-tuned’ (trained using additional data) to achieve a specific goal. ChatGPT, for example, has been fine-tuned to allow users to ask it a question, or make a request, and for it to generate “human-like text” in response.^[2]

In addition to the concepts of narrow and general AI, the US computer scientist Meredith Ringel Morris and her colleagues at Google also incorporated different levels of AI expertise into their definitions. Six levels of AI performance were identified, ranging from no AI to superhuman AI:^[3]

• no AI
• emerging AI: equal to or somewhat better than an unskilled human
• competent AI: at least 50th percentile of skilled adults
• expert AI: at least 90th percentile of skilled adults
• virtuoso AI: at least 99th percentile of skilled adults
• superhuman AI: outperforms 100% of humans

In this definition, examples of expert level narrow AI are spelling and grammar checkers, while emerging level general AI can be seen in LLMs such as ChatGPT and Gemini. In their analysis, Ms Morris and her colleagues stated that while superhuman AI could be seen in narrow systems such as AlphaFold, which predicted the structure of protein molecules using machine learning, it had not yet been achieved in general AI.^[4]

2. Superintelligent artificial intelligence (ASI)

The concept of artificial superintelligence (ASI) is explained by Tim Mucci and Cole Stryker of US technology company IBM as follows:

ASI is a hypothetical software-based artificial intelligence (AI) system with an intellectual scope beyond human intelligence. At the most fundamental level, this superintelligent AI has cutting-edge cognitive functions and highly developed thinking skills more advanced than any human. A big step toward developing an ASI would be to realise an AGI or strong AI. An AGI is a next-generation AI system that can understand the world and learn and apply problem-solving intelligence as broadly and flexibly as a human can. AGI would be capable of cross-domain learning and reasoning with the ability to make connections across different fields. Just like ASI, true AGI has yet to be developed.^[5]

They argued that before superintelligence could become a reality key technologies and processes must develop, such as access to massive datasets and the development of complex, powerful and advanced neural networks modelled on how neurons operate in the human brain.^[6]

2.1 How long could ASI take to develop?

Views about the world’s progress in developing ASI are mixed, and there is also significant debate about how intelligent AI is currently.^[7]

In September 2024, Sam Altman, chief executive officer of the artificial intelligence company OpenAI, stated “it is possible that we will have [ASI] in a few thousand days […] it may take longer, but I’m confident we’ll get there”.^[8]

However, Flora Salim, from the School of Computer Science and Engineering at the University of New South Wales, argues that superintelligence is “not as imminent as many have suggested”. She cites recent research showing the limitations in solving mathematical reasoning problems for many language models, which suggests “sophisticated pattern-matching rather than true advanced reasoning”.^[9] Similarly, Brent Smolinski, IBM vice-president and global head of technology and data strategy, argued “it’s totally exaggerated […] I don’t think we’re even in the right zip code for getting to superintelligence”.^[10] Writing for Brookings, Mark MacCarthy is also sceptical, stating “AI firms are not very close to developing an AI system with capabilities that could threaten us”.^[11] However, he did note that “this assertion runs against a consensus in the AI industry”.

Research published in 2024 examining aggregate forecasts from a survey of over 2,700 AI researchers suggested that the chance of unaided machines outperforming humans in every possible task was 10% by 2027 and 50% by 2047.^[12]

2.2 Potential benefits and risks of ASI

As with other forms of AI, the development of ASI poses risks as well as benefits. Risks and benefits of AI more generally are explored in greater depth in the House of Lords Library briefing ‘Artificial intelligence: Development, risks and regulation’ (18 July 2023).

Writing for IBM, Tim Mucci and Cole Stryker summarised the potential impact of developing ASI:

The technology that goes into developing an ASI would transform the way the world works at a fundamental level, and some say that ASI will be the last invention humanity will ever invent. The benefits of such a technology are science fiction-like in their implications. In essence, an ASI would be an inexhaustible, hyper-intelligent super-being. A nearly perfect supercomputer available 24/7, with the ability to process and analyse any amount of data with speed and precision that we can’t yet comprehend.

With such capabilities, human agents could use ASI to make the best possible decisions and solve the most complex problems facing healthcare, finance, scientific research, politics and every industry. Such advanced thinking could be enough to solve the most persistent medical puzzles to develop life-saving medicines and treatments and unlock the mysteries of physics to aid humanity’s goal of exploring the stars. With its ability to significantly reduce human errors, particularly in programming and risk management, ASI could write and debug programs and deploy robots to perform dangerous physical tasks like bomb defusing or deep-sea exploration.

Because ASI can operate continuously, it would be ideal for tasks like safely navigating networks of self-driving cars and assisting in space exploration. Furthermore, ASI’s superior creativity and ability to analyse vast amounts of data might lead to solutions humans can’t even imagine, resulting in, hopefully, better quality of life and perhaps even a prolonged life.^[13]

However, concerns about ASI have been raised for a number of years. In 2014, AI researchers Max Tegmark and Stuart Russell, alongside Stephen Hawking, noted:

[…] we cannot predict what we might achieve when this intelligence is magnified by the tools AI may provide, but the eradication of war, disease, and poverty would be high on anyone’s list. Success in creating AI would be the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks.

In the near term, for example, world militaries are considering autonomous weapon systems that can choose and eliminate their own targets […] Looking further ahead, there are no fundamental limits to what can be achieved: there is no physical law precluding particles from being organized in ways that perform even more advanced computations than the arrangements of particles in human brains. […] One can imagine such technology outsmarting financial markets, out-inventing human researchers, out-manipulating human leaders, and developing weapons we cannot even understand. Whereas the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all.^[14]

In his 2014 book, ‘Superintelligence: Paths, Dangers and Strategies’, Nick Bostrom discussed the idea that AI would advance to a point where it might turn against and delete humanity. More recently, Eliezer Yudkowsky and Nate Soares’s book drew a distinction between the AI that already exists today, which they argue has “limitations, such as an inability to form new long-term memories”, and ASI.^[15] The authors describe the “headlong charge toward superhuman AI” by AI companies as “a race to the bottom” and “careening toward disaster”, warning:

If any company or group, anywhere on the planet, builds artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.^[16]

However, they note:

The default outcome is lethal, but the situation is not hopeless; machine superintelligence doesn’t exist yet, and its creation can still be prevented.^[17]

3. Regulation and calls for a moratorium

The potential benefits and harms of AI have led to calls for governments to adapt quickly to the changes AI is already delivering and the potentially transformative changes to come. These include calls to pause AI development and for countries, including the UK, to deliver a step-change in regulation, potentially before the technology passes a point when such regulation can be effective.

One example was the open letter signed by more than 1,000 artificial intelligence experts, researchers and backers in March 2023.^[18] The letter called for an immediate pause on the creation of “giant” AIs for at least six months so that the capabilities and dangers of such systems could be properly studied and mitigated. The Future of Life Institute, which coordinated the production of the letter, also published a policy paper entitled ‘Policymaking in the pause’, which offered a number of recommendations to govern the future of AI development.

In October 2025, the Future of Life Institute organised a statement calling for a global ban on superintelligence.^[19] As at 21 January 2026, there were over 133,000 signatories, including prominent figures in the AI sphere such as Geoffrey Hinton and Yoshua Bengio, AI safety researchers including Stuart Russell from the University of California, Berkeley, and figures from business and politics. The statement said:

We call for a prohibition on the development of superintelligence, not lifted before there is:

• broad scientific consensus that it will be done safely and controllably, and
• strong public buy-in.^[20]

In the UK, the non-profit organisation Control AI, which states that it “works to reduce the risks to humanity from artificial intelligence”, campaigned for greater regulation of the development of AI. In a campaign statement it argued:

Nobel Prize winners, AI scientists, and CEOs of leading AI companies have stated that mitigating the risk of extinction from AI should be a global priority. Specialised AIs—such as those advancing science and medicine—boost growth, innovation, and public services. Superintelligent AI systems would compromise national and global security. The UK can secure the benefits and mitigate the risks of AI by delivering on its promise to introduce binding regulation on the most powerful AI systems.^[21]

The campaign received the support of over 100 cross-party parliamentarians.

In contrast, Mark MacCarthy notes that while “it is crucial for policymakers to understand the existential threat”, ASI is not close to being developed and therefore:

[…] there is still plenty of time to address the problem of aligning superintelligence with values that make it safe for humans. It is not today’s most urgent AI research priority. As AI researcher Andrew Ng is reputed to have said back in 2015, worrying about existential risk might appear to be like worrying about the problem of human overpopulation of Mars.^[22]

4. Government policy on AI and ASI

The UK does not have any AI-specific regulation or legislation. AI is instead regulated in the context in which it is used through existing legal frameworks and non-statutory principles. For example, the Conservative government’s 2023 white paper ‘A pro-innovation approach to AI regulation’ established principles for regulating the use of AI but not the technology itself.^[23] This framework was intended “to respond to the level of risk in a proportionate manner and avoid stifling innovation or missing opportunities”.

However, the UK has launched an AI Security Institute (AISI), which is tasked with testing AI systems to identify safety concerns, such as behaviours that could lead to loss of control.^[24] The AISI also develops and tests AI risk mitigation methods. The AISI was initially created as the AI Safety Institute by the then Conservative government in 2023, but was renamed in 2025 to “reflect its focus on serious AI risks with security implications”. The institute had been initially launched by then prime minister Rishi Sunak during the 2023 ‘AI safety summit’, a meeting of governments and AI companies in the UK to discuss the risks of AI and to arrange joint efforts to mitigate them.^[25]

The 2024 Labour Party manifesto did include a commitment to “ensure the safe development and use of AI models by introducing binding regulation on the handful of companies developing the most powerful AI models”.^[26] This was reaffirmed in the 2024 King’s Speech.^[27] In March 2025, Feryal Clark, then minister for AI and online safety, stated that the government was “continuing to refine its proposals and will launch a public consultation in due course”.^[28]

As of the time of writing, no such legislation has been published. The Guardian reported in June 2025 that the government had changed its plans from introducing a short “AI bill” in the current (2024–26) parliamentary session, to introducing a more comprehensive bill in the next session.^[29]

While giving evidence to the House of Commons Science, Innovation and Technology Committee on 3 December 2025, Secretary of State for Science, Innovation and Technology Liz Kendall spoke about the need for AI safety legislation but did not commit to an AI bill.^[30] Ms Kendall stated: “I am thinking about it more in terms of specific areas where we may need to act rather than a big all-encompassing bill”.^[31] In a recent Westminster Hall debate on AI safety, the committee’s chair, Dame Chi Onwurah, said the committee would hold Ms Kendall to account on her commitment to introduce legislation.^[32]

However, the government did publish an ‘AI opportunities action plan’ in January 2025. This set out how the government seeks to embrace and drive the use and development of AI, but also referenced the importance of ensuring safety and security. For example, it highlighted the importance of the AISI in this context, and also set out its position on regulation:

The UK’s current pro-innovation approach to regulation is a source of strength relative to other more regulated jurisdictions and we should be careful to preserve this.

Well-designed and implemented regulation, alongside effective assurance tools, can fuel fast, wide and safe development and adoption of AI. Regulators themselves have an important role in supporting innovation as part of their growth duty. Government must protect UK citizens from the most significant risks presented by AI and foster public trust in the technology, particularly considering the interests of marginalised groups. That said, we must do this without blocking the path towards AI’s transformative potential.^[33]

The government recently stressed its general position in response to a parliamentary question about potential AI legislation on 19 January 2026, stating:

The government does not speculate on legislation ahead of future parliamentary sessions. A range of existing rules already apply to AI systems, such as data protection, competition, equality legislation and other forms of sectoral regulation. AI is a general-purpose technology with a wide range of applications, which is why the UK believes that the vast majority of AI systems should be regulated at the point of use. In response to the AI action plan, the government committed to work with regulators to boost their capabilities. This is complemented by the work of the AI Security Institute which has deepened our understanding of the critical security risks posed by frontier.^[34] However, the government will not hesitate to act where evidence suggests that further action is necessary.^[35]

Asked in December 2025 about the development of ASI in particular, the government said:

There is considerable debate and uncertainty around AGI and ASI, but the possibility of their development must be taken seriously. The increasing capabilities of AI may exacerbate existing risks and present new risks, for which the UK need to be prepared.

The role of the AISI is to build an evidence base on these risks, so the government is equipped to understand them. AISI focuses on emerging AI risks with serious security implications, including the potential for AI to help users develop chemical and biological weapons, and the potential for loss of control presented by autonomous systems.^[36]

5. Read more

• House of Lords Library, ‘Potential future risks from autonomous AI systems’, 5 January 2026
• ‘Mark MacCarthy, ‘Are AI existential risks real—and what should we do about them?’, Brookings, 11 July 2025
• Flora Salim, ‘What is AI superintelligence? Could it destroy humanity? And is it really almost here?’, The Conversation, 28 October 2024
• Parliamentary Office of Science and Technology, ‘Artificial intelligence: An explainer’, 14 December 2023
• House of Lords Library, ‘Artificial intelligence: Development, risks and regulation’, 18 July 2023

---

Image by Freepik